A staggered wave launching multi-radar data fusion method, device, equipment and medium

Abstract

The application discloses a multi-radar data fusion method, device and equipment based on staggered wave emission, and a medium. The method comprises the following steps: determining a current wave emission period according to the earliest wave emission time and the latest wave emission time in each wave emission time, and determining a current candidate key frame from a vehicle data frame in the current wave emission period; determining a target key frame according to the current candidate key frame and a historical candidate key frame, and determining a vehicle motion trajectory in a time period from a first target key frame to the latest wave emission time according to vehicle body data corresponding to the target key frame and a vehicle motion model; determining wave emission trajectory points of each millimeter wave radar on the vehicle motion trajectory according to the wave emission time of each millimeter wave radar; converting point cloud data of each millimeter wave radar to a vehicle coordinate system at the wave emission time of the millimeter wave radar with the latest wave emission according to the wave emission trajectory points of each millimeter wave radar and a conversion relationship, and performing data fusion. The application can improve the accuracy of original point cloud data.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing technology, and in particular to a method, apparatus, device and medium for multi-radar data fusion with staggered wave transmission. Background Technology

[0002] To meet emerging demands such as assisted driving and automatic obstacle avoidance, automobiles typically incorporate multiple millimeter-wave radars to identify the vehicle's surroundings. A typical arrangement is one front-corner radar and four corner radars. In this configuration, the front radars usually have a longer detection range, while the corner radars have a wider detection angle, balancing forward long-range detection with omnidirectional surround-view detection. However, the field of view (FOV) of these millimeter-wave radars overlaps.

[0003] In some cases, to avoid interference between multiple millimeter-wave radars working together, a staggered emission strategy is adopted. That is, radars with non-overlapping fields of view (FOV) emit emission signals simultaneously for detection, while radars with overlapping FOVs emit emission signals at different times to minimize mutual interference between multiple radars. Taking the "one front, four corners" arrangement as an example, emission signals are emitted in the order of front radar, left front radar and right rear radar, and left rear radar and right front radar. Staggered emission effectively reduces mutual interference between multiple radars.

[0004] However, due to the inconsistent emission times, there will be some deviation in the position of the radar emission when performing data fusion of multiple radars. At low speeds, the impact of such deviations caused by inconsistent emission times is very limited, but at higher vehicle speeds, it will have a significant impact on data fusion and final target detection. Therefore, it is necessary to process such situations to improve the accuracy of detection results. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for fusing multi-radar data with staggered wave transmission to improve the accuracy of raw point cloud data.

[0006] According to one aspect of the present invention, a multi-radar data fusion method with staggered time emission is provided, comprising:

[0007] Acquire the emission times and point cloud data of at least two millimeter-wave radars that emit waves at staggered times;

[0008] The current wave cycle is determined based on the earliest and latest wave times among the wave times, and the current candidate key frame is determined from the vehicle data frames within the current wave cycle.

[0009] The target key frame is determined based on the current candidate key frame and the historical candidate key frame, and the vehicle motion trajectory from the first target key frame to the latest wave emission time period is determined based on the vehicle body data and vehicle motion model corresponding to the target key frame.

[0010] The emission trajectory points of each millimeter-wave radar on the vehicle's motion trajectory are determined based on the emission time of each millimeter-wave radar.

[0011] Based on the emission trajectory points and transformation relationships of each millimeter-wave radar, the point cloud data of each millimeter-wave radar is transformed into the vehicle coordinate system at the time of emission of the latest millimeter-wave radar, and then the data is fused.

[0012] According to another aspect of the present invention, a multi-radar data fusion device with staggered time emission is provided, comprising:

[0013] The radar data acquisition module is used to acquire the emission time and point cloud data of at least two millimeter-wave radars that emit waves at different times.

[0014] The candidate keyframe determination module is used to determine the current wave cycle based on the earliest wave cycle and the latest wave cycle among the wave cycles, and to determine the current candidate keyframe from the vehicle data frames within the current wave cycle.

[0015] The motion trajectory determination module is used to determine the target key frame based on the current candidate key frame and the historical candidate key frame, and to determine the vehicle motion trajectory from the first target key frame to the latest wave time period based on the vehicle body data and vehicle motion model corresponding to the target key frame.

[0016] The emission trajectory point determination module is used to determine the emission trajectory point of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar.

[0017] The radar data fusion module is used to convert the point cloud data of each millimeter-wave radar to the vehicle coordinate system at the time of the latest millimeter-wave radar's emission, based on the emission trajectory points and conversion relationships of each millimeter-wave radar, and then perform data fusion.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multi-radar data fusion method with staggered wave transmission as described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multi-radar data fusion method with staggered wave transmission as described in any embodiment of the present invention.

[0023] This invention calculates the vehicle's trajectory using vehicle body data and, based on the emission time of the millimeter-wave radar, obtains the conversion relationship between the corresponding emission point and trajectory point and the vehicle's current position. Based on this conversion relationship, all millimeter-wave radar point cloud data are converted to the vehicle body coordinate system at the same time, effectively reducing the deviation of the original point cloud data caused by vehicle movement when the millimeter-wave radar emission time is different, and improving the accuracy of the original point cloud data.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a multi-radar data fusion method with staggered wave emission according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of a multi-radar data fusion method with staggered wave emission according to another embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a multi-radar data fusion device with staggered wave transmission according to another embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device that implements an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Figure 1 This is a flowchart illustrating a multi-radar data fusion method with staggered emission times, according to an embodiment of the present invention. This embodiment is applicable to situations where multiple millimeter-wave radars on a vehicle emit signals at staggered times during vehicle movement, and the point cloud data obtained from these staggered emission times is fused and transformed. This method can be executed by a multi-radar data fusion device with staggered emission times. This device can be implemented in hardware and / or software and can be configured in an electronic device with corresponding data processing capabilities, such as an in-vehicle device. Figure 1 As shown, the method includes:

[0033] S110: Acquire the emission times and point cloud data of at least two millimeter-wave radars that emit waves at staggered times.

[0034] S120. Determine the current wave generation cycle based on the earliest and latest wave generation times among the wave generation times, and determine the current candidate key frame from the vehicle data frames within the current wave generation cycle.

[0035] The vehicle is equipped with at least two millimeter-wave radars, preferably arranged in a front-four-corner configuration: a front left radar and a rear right radar, or a rear left radar and a front right radar. Each radar emits waves at staggered intervals, with a specific order in the emission. Vehicle data frames store the vehicle's motion status and corresponding timestamps.

[0036] Specifically, the emission times and point cloud data of all millimeter-wave radars on the vehicle are acquired sequentially. Based on the radar's calibration relationship with the vehicle, the original point cloud data of the radar is transformed into the vehicle's coordinate system. The earliest and latest emission times are used as the cycle start times to determine the current emission cycle. During radar operation, the vehicle may move violently, causing instability in the vehicle data frames of certain parts. Therefore, unstable vehicle data frames in the current emission cycle are removed, and the remaining stable vehicle data frames are used as the current candidate keyframes.

[0037] S130. Determine the target key frame based on the current candidate key frame and the historical candidate key frame, and determine the vehicle motion trajectory from the first target key frame to the latest wave emission time period based on the vehicle body data and vehicle motion model corresponding to the target key frame.

[0038] Among them, historical candidate keyframes are stable vehicle keyframes with timestamps prior to the earliest wave generation time.

[0039] Specifically, relying solely on the current candidate keyframe is insufficient to obtain an effective and stable vehicle trajectory. This invention references historical candidate keyframes and fuses and filters them to obtain the target keyframe. The vehicle body data corresponding to the target keyframe is input into the vehicle's CTRV (Constant Turn Rate and Velocity) motion model to obtain the vehicle's motion points at the corresponding timestamps. These motion points are then transformed to the vehicle coordinate system corresponding to the radar with the latest emission time and fitted to obtain the vehicle trajectory. It should be noted that the timestamp of the first target keyframe corresponds to a time earlier than the earliest emission time. This ensures a larger number of keyframes are obtained, preventing the current emission period from being too short to collect enough keyframes and thus affecting subsequent trajectory fitting. The specific process for establishing the CTRV motion model is as follows:

[0040] If a CTRV (Constant Turn Rate and Velocity) motion model of the vehicle is established, then the vehicle's state variables are:

[0041]

[0042] Where, p x p represents the x-coordinate. y Let v represent the vertical axis, θ represent the velocity, θ represent the heading angle, and ω represent the yaw rate. Then we have:

[0043]

[0044]

[0045] Where k+1 represents the current time, k represents the previous time, and X k+1 and X k They represent X respectively k+1 and X k+1 Let T represent the time difference between time k+1 and time k, and let the state variable of the vehicle at time k be [the state variable]. Then the motion of the vehicle between two adjacent time points is [the motion variable].

[0046]

[0047]

[0048] Therefore, after obtaining vehicle body data such as speed and yaw rate, the vehicle motion points at the corresponding historical time can be converted to the vehicle coordinate system at the latest wave emission time through the model.

[0049] S140. Determine the emission trajectory point of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar.

[0050] S150. Based on the emission trajectory points and transformation relationships of each millimeter-wave radar, the point cloud data of each millimeter-wave radar is transformed to the vehicle coordinate system at the time of emission of the latest millimeter-wave radar, and data fusion is performed.

[0051] In this cycle, the last radar to emit a signal at a staggered time is usually the latest radar to emit a signal. The emission trajectory point is the location on the vehicle's trajectory when the radar emits a signal; it is actually the vehicle's position at the time the radar emits a signal.

[0052] Specifically, based on the emission time of each millimeter-wave radar, the corresponding trajectory points on the fitted vehicle motion trajectory at the time of each radar emission are determined as emission trajectory points. Based on the emission trajectory points on the vehicle motion trajectory, a transformation matrix is ​​obtained to represent the change in vehicle position caused by the different radar emission times. Using this transformation matrix, the original point clouds of all radars are transformed to the vehicle coordinate system at the time of the last radar emission. The transformed point cloud data are then fused and used as input for subsequent processing steps, such as using the transformed point cloud data for assisted driving.

[0053] This invention calculates the vehicle's trajectory using vehicle body data and, based on the emission time of the millimeter-wave radar, obtains the conversion relationship between the corresponding emission point and trajectory point and the vehicle's current position. Based on this conversion relationship, all millimeter-wave radar point cloud data are converted to the vehicle body coordinate system at the same time, effectively reducing the deviation of the original point cloud data caused by vehicle movement when the millimeter-wave radar emission time is different, and improving the accuracy of the original point cloud data.

[0054] Optionally, after determining the emission trajectory points of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar, the method further includes:

[0055] Based on the emission trajectory points of each millimeter-wave radar and the vehicle's motion trajectory, the vehicle's motion speed and direction corresponding to the emission time of each millimeter-wave radar are determined; based on the vehicle's motion speed and direction, static and dynamic point clouds are selected from each original point cloud.

[0056] Among them, points that do not change the location cloud information after being detected by imaging millimeter-wave radar can be classified as static point clouds, including walls, railings, and parked vehicles, which are obstacles that need to be avoided when planning the route later. Continuous point clouds that change the location cloud information after being detected by imaging millimeter-wave radar can be classified as dynamic point clouds. When planning the route later, dynamic point clouds may also be used to make specific decisions on actions. For example, if it is determined that the target corresponding to the dynamic point cloud will pass through the planned route of the vehicle, it may be necessary to stop and wait for it to pass or detour.

[0057] Specifically, the tangent of the vehicle's trajectory passing through the radar emission point is obtained, and the direction of the tangent is taken as the direction of the vehicle's movement. The vehicle's speed corresponding to the radar emission time is obtained. Based on the vehicle's speed and direction of movement, the speed and direction of movement are compared with the speed magnitude and direction of each original point cloud detected by the millimeter-wave radar, and the static and dynamic point clouds in each original point cloud are selected.

[0058] Figure 2 This is a flowchart illustrating a multi-radar data fusion method with staggered wave transmission, provided as another embodiment of the present invention. This embodiment is an optimization and improvement upon the above embodiments. Figure 2 As shown, the method includes:

[0059] S210: Acquire the emission times and point cloud data of at least two millimeter-wave radars that emit waves at staggered times.

[0060] S220. Determine the current wave transmission cycle based on the earliest and latest wave transmission times among all wave transmission times.

[0061] S230. Obtain vehicle data frames within the current wave generation cycle. If the vehicle body data corresponding to any vehicle data frame meets the stability requirements, then determine the vehicle data frame as the current candidate key frame.

[0062] Specifically, vehicle body data and timestamps are acquired, and the vehicle body data is smoothed and filtered to reduce the impact of sudden changes in the vehicle body data caused by various factors. Vehicle data frames are generated based on the smoothed and filtered vehicle body data and the corresponding timestamps. The stability of the vehicle body data in each vehicle data frame is assessed, and vehicle data frames whose vehicle body data meets the stability requirements are selected as the current candidate keyframes.

[0063] Optionally, the stability requirement includes that the vehicle body data is stable and within a set range, and the vehicle body data includes vehicle speed and vehicle yaw rate.

[0064] Specifically, if vehicle speed, yaw rate, and other vehicle body data are relatively stable (the rate of change is below the threshold) and within the set threshold range, the corresponding vehicle body data frame will be marked as a candidate keyframe.

[0065] S240. Obtain the first historical candidate keyframe within a reference time before the earliest transmission time; if the number of the first historical candidate keyframes is greater than a set number, merge the first historical candidate keyframes and the current candidate keyframes, and filter the merged candidate keyframes through mean deviation to obtain the target keyframe.

[0066] The reference time is dynamically adjusted based on the vehicle's current speed; the faster the vehicle is, the shorter the reference time, and the slower the vehicle is, the longer the reference time.

[0067] Specifically, the number of potential keyframes within a reference time period prior to the earliest launch time is obtained. If the number of potential keyframes exceeds a set threshold, historical candidate keyframes within the reference time period and newly generated candidate keyframes in the current launch cycle are merged. After merging, all candidate keyframes from the first to the last are obtained, and the mean of the vehicle body data in the candidate keyframes is calculated. If the deviation of the vehicle body data in a candidate keyframe from the mean exceeds a certain threshold, that candidate keyframe is filtered, and the remaining unfiltered vehicle data frames are the target keyframes. Based on the filtered target keyframes, the vehicle's motion trajectory within the current launch cycle is fitted.

[0068] Optionally, after obtaining the first historical candidate keyframe within a reference time period prior to the earliest transmission time, the method further includes:

[0069] If the number of the first historical candidate keyframes is less than the set number, then the second historical candidate keyframe within the reference smooth trajectory before the earliest wave transmission time is obtained; if the number of the second historical candidate keyframes is greater than the set number, then the second historical candidate keyframe and the current candidate keyframe are merged, and the merged candidate keyframe is filtered by mean deviation to obtain the target keyframe.

[0070] The reference smooth trajectory is also determined based on the vehicle's current speed; the faster the vehicle is, the shorter the reference smooth trajectory is, and the slower the vehicle is, the longer the reference smooth trajectory is.

[0071] Specifically, if the number of historical candidate frames within the reference time does not meet the requirements, the number of potential keyframes of the vehicle in a previous smooth trajectory (the coefficient equation of the trajectory satisfies the threshold and the distance from the vehicle's current position is less than the threshold) is obtained. If the set number threshold is met, the trajectory is fitted according to the above process.

[0072] Optionally, after determining the current candidate keyframe from the vehicle data frames within the current emission period, the method further includes:

[0073] If the number of the first historical candidate keyframe and the second historical candidate keyframe are both not greater than the set number, then the historical motion trajectory within the most recent time before the earliest wave transmission time is obtained; if the time distance of the historical motion trajectory meets the reference time threshold requirement, then the vehicle trajectory equation obtained by filtering and predicting the historical motion trajectory is determined as the vehicle motion trajectory from the first target keyframe to the latest wave transmission time period.

[0074] Optionally, if the number of historical candidate keyframes within the reference time and reference smooth trajectory still cannot meet the set number requirement, the historical trajectory closest to the current time is obtained. If the time distance of the fitted historical trajectory meets the threshold requirement, Kalman filtering is applied to the historical trajectory, and the vehicle trajectory equation predicted by Kalman filtering is used as the vehicle motion trajectory.

[0075] S250. Determine the vehicle motion trajectory from the first target key frame to the latest wave emission time period based on the vehicle body data and vehicle motion model corresponding to the target key frame.

[0076] S260. Determine the emission trajectory points of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar.

[0077] S270. Based on the emission trajectory points and transformation relationships of each millimeter-wave radar, the point cloud data of each millimeter-wave radar is transformed to the vehicle coordinate system at the time of emission of the latest millimeter-wave radar, and data fusion is performed.

[0078] This invention improves the accuracy of calculating the vehicle's trajectory within the current wave cycle by referencing historical vehicle motion data.

[0079] Figure 3 This is a schematic diagram of a multi-radar data fusion device with staggered wave transmission, provided as another embodiment of the present invention. Figure 3 As shown, the device includes:

[0080] Radar data acquisition module 310 is used to acquire the emission time and point cloud data of at least two millimeter-wave radars that emit waves at different times;

[0081] The candidate keyframe determination module 320 is used to determine the current wave cycle based on the earliest wave cycle and the latest wave cycle among the wave cycles, and to determine the current candidate keyframe from the vehicle data frames within the current wave cycle.

[0082] The motion trajectory determination module 330 is used to determine the target key frame based on the current candidate key frame and the historical candidate key frame, and to determine the vehicle motion trajectory from the first target key frame to the latest wave time period based on the vehicle body data and vehicle motion model corresponding to the target key frame.

[0083] The emission trajectory point determination module 340 is used to determine the emission trajectory point of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar.

[0084] The radar data fusion module 350 is used to convert the point cloud data of each millimeter-wave radar to the vehicle coordinate system at the time of the latest millimeter-wave radar's emission, based on the emission trajectory points and transformation relationships of each millimeter-wave radar, and then perform data fusion.

[0085] The multi-radar data fusion device with staggered emission provided in this embodiment of the invention can execute the multi-radar data fusion method with staggered emission provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0086] Optionally, the candidate keyframe determination module 320 includes:

[0087] The vehicle data frame acquisition unit is used to acquire vehicle data frames within the current transmission cycle.

[0088] The candidate keyframe determination unit is used to determine the vehicle data frame as the current candidate keyframe if the vehicle body data corresponding to any vehicle data frame meets the stability requirements.

[0089] Optionally, the stability requirement includes that the vehicle body data is stable and within a set range, and the vehicle body data includes vehicle speed and vehicle yaw rate.

[0090] Optionally, the motion trajectory determination module 330 includes:

[0091] The first historical frame acquisition unit is used to acquire the first historical candidate key frame within a reference time before the earliest broadcast time.

[0092] The first target frame determination unit is used to merge the first historical candidate keyframe and the current candidate keyframe if the number of the first historical candidate keyframe is greater than a set number, and to obtain the target keyframe by filtering the merged candidate keyframe through mean deviation.

[0093] Optionally, the motion trajectory determination module 330 also includes:

[0094] The first historical frame acquisition unit is used to acquire the second historical candidate key frame within the reference smooth trajectory before the earliest emission time if the number of the first historical candidate key frames is less than a set number.

[0095] The first target frame determination unit is used to merge the second historical candidate keyframe and the current candidate keyframe if the number of the second historical candidate keyframe is greater than a set number, and to obtain the target keyframe by filtering the merged candidate keyframe through mean deviation.

[0096] Optionally, the device further includes a historical trajectory reference module, used to acquire the historical motion trajectory within the most recent time period before the earliest broadcast time if the number of both the first historical candidate keyframe and the second historical candidate keyframe is not greater than a set number; and to determine the vehicle motion trajectory from the first target keyframe to the latest broadcast time period based on the vehicle trajectory equation obtained by filtering and predicting the historical motion trajectory, according to the time distance of the historical motion trajectory meeting the reference time threshold requirement.

[0097] Optionally, the device further includes:

[0098] The motion information determination module is used to determine the vehicle's speed and direction of motion corresponding to the emission time of each millimeter-wave radar based on the emission trajectory points of each millimeter-wave radar and the vehicle's motion trajectory.

[0099] The original point cloud filtering module is used to filter out static and dynamic point clouds from each original point cloud based on the vehicle's speed and direction of movement.

[0100] The multi-radar data fusion device with staggered wave emission, as further explained, can also execute the multi-radar data fusion method with staggered wave emission provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0101] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0102] like Figure 4As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0103] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as the multi-radar data fusion method with staggered wave transmission.

[0105] In some embodiments, the staggered-emission multi-radar data fusion method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the staggered-emission multi-radar data fusion method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the staggered-emission multi-radar data fusion method by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0111] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0112] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-radar data fusion method with staggered time-transmission, characterized in that, The method includes: Acquire the emission times and point cloud data of at least two millimeter-wave radars that emit waves at staggered times; The current wave cycle is determined based on the earliest and latest wave times among the wave times, and the current candidate key frame is determined from the vehicle data frames within the current wave cycle. The target key frame is determined based on the current candidate key frame and the historical candidate key frame, and the vehicle motion trajectory from the first target key frame to the latest wave emission time period is determined based on the vehicle body data and vehicle motion model corresponding to the target key frame. The emission trajectory point of each millimeter-wave radar on the vehicle's motion trajectory is determined based on the emission time of each millimeter-wave radar; the emission trajectory point is the point on the vehicle's motion trajectory when the radar emits a wave, indicating the vehicle's position when the radar emits a wave; Based on the emission trajectory points and transformation relationships of each millimeter-wave radar, the point cloud data of each millimeter-wave radar is transformed into the vehicle coordinate system at the time of emission of the latest millimeter-wave radar, and data fusion is performed. The step of determining the current candidate keyframe from the vehicle data frames within the current wave generation cycle includes: Obtain vehicle data frames within the current broadcast cycle. If the vehicle body data corresponding to any vehicle data frame meets the stability requirements, then that vehicle data frame is determined as the current candidate keyframe.

2. The method according to claim 1, characterized in that, The stability requirement includes that the vehicle body data is stable and within a set range, and the vehicle body data includes vehicle speed and vehicle yaw rate.

3. The method according to claim 1, characterized in that, The step of determining the target keyframe based on the current candidate keyframe and the historical candidate keyframe includes: Obtain the first historical candidate keyframe within the reference time before the earliest broadcast time; If the number of the first historical candidate keyframes is greater than the set number, the first historical candidate keyframes and the current candidate keyframes are merged, and the merged candidate keyframes are filtered by mean deviation to obtain the target keyframe.

4. The method according to claim 3, characterized in that, After obtaining the first historical candidate keyframe within the reference time before the earliest transmission time, the method further includes: If the number of the first historical candidate keyframes is less than the set number, then the second historical candidate keyframe within the reference smooth trajectory before the earliest wave transmission time is obtained. If the number of the second historical candidate keyframes is greater than the set number, the second historical candidate keyframes and the current candidate keyframes are merged, and the merged candidate keyframes are filtered by mean deviation to obtain the target keyframe.

5. The method according to claim 4, characterized in that, After determining the current candidate keyframe from the vehicle data frames within the current wave generation cycle, the process further includes: If the number of the first historical candidate keyframe and the second historical candidate keyframe are both no greater than the set number, then the historical motion trajectory within the most recent time before the earliest wave transmission time is obtained. If the time distance of the historical motion trajectory meets the reference time threshold requirement, then the vehicle trajectory equation obtained by filtering and predicting the historical motion trajectory will be determined as the vehicle motion trajectory from the first target key frame to the latest wave emission time period.

6. The method according to claim 1, characterized in that, After determining the emission trajectory points of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar, the method further includes: Based on the emission trajectory points of each millimeter-wave radar and the vehicle's motion trajectory, determine the vehicle's speed and direction of motion corresponding to the emission time of each millimeter-wave radar. Based on the vehicle's speed and direction of motion, static and dynamic point clouds are selected from each original point cloud.

7. A multi-radar data fusion device with staggered time-transmission, characterized in that, The device includes: The radar data acquisition module is used to acquire the emission time and point cloud data of at least two millimeter-wave radars that emit waves at different times. The candidate keyframe determination module is used to determine the current wave cycle based on the earliest wave cycle and the latest wave cycle among the wave cycles, and to determine the current candidate keyframe from the vehicle data frames within the current wave cycle. The motion trajectory determination module is used to determine the target key frame based on the current candidate key frame and the historical candidate key frame, and to determine the vehicle motion trajectory from the first target key frame to the latest wave time period based on the vehicle body data and vehicle motion model corresponding to the target key frame. The emission trajectory point determination module is used to determine the emission trajectory point of each millimeter-wave radar on the vehicle's motion trajectory based on the emission time of each millimeter-wave radar; the emission trajectory point is the point on the vehicle's motion trajectory when the radar emits a wave, indicating the vehicle's position when the radar emits a wave; The radar data fusion module is used to convert the point cloud data of each millimeter-wave radar to the vehicle coordinate system at the time of the latest millimeter-wave radar's emission, based on the emission trajectory points and conversion relationships of each millimeter-wave radar, and then perform data fusion. The candidate keyframe determination module includes: The vehicle data frame acquisition unit is used to acquire vehicle data frames within the current transmission cycle. The candidate keyframe determination unit is used to determine the vehicle data frame as the current candidate keyframe if the vehicle body data corresponding to any vehicle data frame meets the stability requirements.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-radar data fusion method of staggered wave transmission as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the multi-radar data fusion method with staggered wave transmission as described in any one of claims 1-6.

Images