Method and apparatus for multi-sensor data fusion

Abstract

Disclosed are a method and device for multi-sensor data fusion. The method comprises: obtaining a plurality of groups of sensor coordinate data detected by a plurality of sensors on a target scene in a target time period; determining a plurality of coordinate data sets according to a plurality of preset angle ranges determined based on a target origin and a preset unit angle, wherein each coordinate data set in the plurality of coordinate data sets is a set of coordinate data in which each group of sensor coordinate data in the plurality of groups of sensor coordinate data is located in a corresponding preset angle range in the plurality of preset angle ranges; marking the coordinate data closest to the target origin in each coordinate data set as selected coordinate data; and integrating all corresponding selected coordinate data in the plurality of coordinate data sets to obtain a multi-sensor data set.

Description

Technical Field

[0001] This invention relates to the field of sensor data processing technology, and in particular to a method and apparatus for multi-sensor data fusion. Background Technology

[0002] Combining multiple sensors offers advantages such as complementarity and redundancy, overcoming the limitations of a single sensor and achieving more comprehensive and reliable sensing. However, as the number of sensor types increases, the processing difficulty of sensor data also rises. Summary of the Invention

[0003] A method for multi-sensor data fusion according to an embodiment of the present invention includes: acquiring multiple sets of sensor coordinate data obtained by multiple sensors detecting a target scene during a target time period; determining multiple coordinate datasets according to multiple preset angle ranges determined based on the target origin and a preset unit angle, wherein each coordinate dataset is a set of coordinate data of each set of sensor coordinate data located within a corresponding preset angle range of the multiple preset angle ranges; recording the sensor coordinate data in each coordinate dataset that is closest to the target origin as selected coordinate data; and integrating all selected coordinate data in the multiple coordinate datasets to obtain a multi-sensor data set.

[0004] An apparatus for multi-sensor data fusion according to an embodiment of the present invention includes: a processor; and a memory storing computer-executable instructions thereon, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform the above-described method for multi-sensor data fusion.

[0005] According to an embodiment of the present invention, a computer-readable storage medium stores computer-executable instructions thereon, wherein, when executed by a processor, these computer-executable instructions cause the processor to perform the above-described method for multi-sensor data fusion.

[0006] A computer program product according to an embodiment of the present invention includes computer-executable instructions, wherein, when executed by a processor, these computer-executable instructions cause the processor to perform the above-described method for multi-sensor data fusion. Attached Figure Description

[0007] The invention can be better understood from the following description of specific embodiments of the invention in conjunction with the accompanying drawings, wherein:

[0008] Figure 1 A flowchart illustrating a method for multi-sensor data fusion according to an embodiment of the present invention is shown.

[0009] Figure 2A schematic diagram illustrating an example application scenario of a method for multi-sensor data fusion according to an embodiment of the present invention is shown.

[0010] Figure 3 It shows Figure 2 A schematic diagram of the data information flow.

[0011] Figure 4 A schematic diagram of a computer system is shown that can implement the method and apparatus for constructing a large multi-turn dialogue model system according to embodiments of the present invention. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configurations and algorithms presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention.

[0013] Data from multiple sensors can be complementary and redundant, but due to the different sampling frequencies and data formats of different sensors, and the rapid increase in data volume as the number of sensor types increases, fusing large amounts of data from different types requires significant processing power. Prolonged processing time can negatively impact the real-time performance and accuracy of decision-making.

[0014] In view of the above problems, a method and apparatus for multi-sensor data fusion according to embodiments of the present invention are proposed. By dividing each set of sensor coordinate data into coordinate datasets corresponding to multiple preset angular ranges, the coordinate data closest to the target origin in each coordinate dataset is determined as the selected coordinate data, and all selected coordinate data are integrated to obtain a multi-sensor data set. This preserves the data characteristics of multiple sets of sensor coordinate data, while significantly reducing the amount of sensor coordinate data, thereby lowering the data processing threshold and improving the real-time performance and accuracy of decision-making.

[0015] Figure 1 A flowchart illustrating a method for multi-sensor data fusion according to an embodiment of the present invention is shown. Figure 1As shown, the method 100 for multi-sensor data fusion according to an embodiment of the present invention includes: S101: acquiring multiple sets of sensor coordinate data obtained by multiple sensors detecting a target scene during a target time period; S102: determining multiple coordinate datasets according to multiple preset angle ranges determined based on the target origin and a preset unit angle, wherein each coordinate dataset is a set of coordinate data within a corresponding preset angle range of each set of sensor coordinate data in the multiple sets of sensor coordinate data; S103: recording the coordinate data in each coordinate dataset that is closest to the target origin as selected coordinate data; and S104: integrating all selected coordinate data in the multiple coordinate datasets to obtain a multi-sensor data set.

[0016] The method according to embodiments of the present invention can be applied to a mobile robot, wherein the target origin is located on the mobile robot and moves with it. The mobile robot can use a multi-sensor data set to make action decisions regarding path planning, obstacle avoidance, and other operations. Here, multiple sensors can be installed on the mobile robot and move with it, or they can be installed at other locations that do not move with the mobile robot.

[0017] Figure 2 A schematic diagram illustrating an example application scenario of a method for multi-sensor data fusion according to an embodiment of the present invention is shown. For example... Figure 2 As shown, the multiple sensors may include, but are not limited to, a stereo vision sensor, at least one LiDAR sensor, and at least one ultrasonic sensor. The stereo vision sensor, which can be implemented using a binocular camera, structured light, or a 3D ToF sensor, is used to acquire stereo information of the target scene (which can be represented as a depth map or point cloud information). The LiDAR sensor can be distributed at different locations to acquire distance information of the target scene from multiple angles (which can be represented as point cloud information). The ultrasonic sensor can be dispersed around the mobile robot for close-range distance detection.

[0018] like Figure 2 As shown, each sensor detects the target scene during the target time period to obtain a set of sensor coordinate data (including the coordinate data of each location point in the target scene). The main controller executes the sensor data fusion method according to the embodiment of the present invention to obtain a multi-sensor data set based on the multiple sets of sensor coordinate data obtained from multiple sensors.

[0019] In the method according to embodiments of the present invention, multiple sets of sensor coordinate data are the results of detection of the same environment under test by sensors of different types or locations. In the time dimension, the same environment under test refers to the timestamps of multiple sets of sensor coordinate data falling within the same target time period; in the spatial dimension, the same environment under test refers to the detection objects of multiple sets of sensor coordinate data being the same target scene. Therefore, within the target time period, the positional relationship between each element in the target scene and the sensor remains relatively static. In some embodiments, the target time period can be a relatively short period, such as 100ms. If the positional relationship between the target scene and the sensor changes slowly, the length of the target time period can also be appropriately increased, for example, 1s or 5s.

[0020] In the method according to an embodiment of the present invention, multiple preset angle ranges are determined based on a target origin and a preset unit angle. The target origin serves as the common vertex of the multiple preset angle ranges, and the preset unit angle can be regarded as the angular resolution for a multi-sensor data set. Therefore, a coordinate data can be selected from the set of coordinate data in the same group within a preset angle range (i.e., the coordinate dataset) to represent the preset angle range.

[0021] In the method according to an embodiment of the present invention, the selection of coordinate data is based on the distance between the target origin and the coordinate data of each sensor. The target origin can be the origin of a polar coordinate system, a cylindrical coordinate system, or a spherical coordinate system. For example, when the target origin is the origin of a polar coordinate system, multiple preset angle ranges are obtained by dividing the target plane containing the target origin into angles with the target origin as the vertex and by a preset unit angle. The coordinate data in each group of sensor coordinate data are coordinate data projected onto the target plane. Similarly, when the target origin is the origin of a spherical coordinate system, multiple preset angle ranges are obtained by dividing the three-dimensional space containing the target origin into angles with the target origin as the vertex and by a preset unit angle.

[0022] In some embodiments, the target origin can be used as the origin of the polar coordinate system, and the entire target plane can be divided into multiple preset angle ranges by a preset unit angle, that is, all preset angle ranges correspond to [0, 360] in the polar coordinate system. Alternatively, in some embodiments, the target origin can be used as the origin of the polar coordinate system, and a specific part of the target plane (e.g., the specific part may only be half of the target plane) can be divided into multiple preset angle ranges by a preset unit angle, in which case all preset angle ranges correspond to [0, 180] in the polar coordinate system.

[0023] In some embodiments, by integrating the selected coordinate data to obtain a multi-sensor data set, the multi-sensor data set can give full play to the advantages of different sensors and make up for the shortcomings of a single sensor, thereby improving the accuracy and comprehensiveness of the perceived environmental information. It can be used for subsequent data registration and fusion. Therefore, the selected coordinate data from multiple sensors can be unified into the same target coordinate system. The target coordinate system can be a coordinate system with the target origin as the origin, which facilitates the comparison of the distance between the coordinate data in each coordinate dataset and the target origin.

[0024] In some embodiments, for each of the multiple sensors, acquiring a set of sensor coordinate data obtained by the sensor detecting the target scene within a target time period includes: acquiring a set of initial coordinate data in the sensor coordinate system used by the sensor, which describes the positional relationship between the target scene and the sensor; and converting the set of initial coordinate data into a set of sensor coordinate data in the target coordinate system based on the transformation relationship between the target coordinate system and the sensor coordinate system.

[0025] In some embodiments, both the target coordinate system and the sensor coordinate system are Cartesian coordinate systems, and the transformation relationship is determined by a rotation matrix and a translation matrix. Assume that an initial coordinate data point in the sensor coordinate system S is p. s (x s ,y s ,z s This initial coordinate data is then converted into sensor coordinate data p in the target coordinate system B. b (x b ,y b ,z b The specific conversion process is as follows:

[0026]

[0027] The transformation relationship between the sensor coordinate system S and the target coordinate system B is given by the rotation matrix R. s Translation matrix T s Sure.

[0028] In some embodiments, the multiple sensors may include an ultrasonic sensor. Acquiring a set of initial coordinate data obtained by the ultrasonic sensor detecting the target scene within a target time period may include: acquiring one-dimensional distance data obtained by the ultrasonic sensor detecting the target scene within the target time period; and converting the one-dimensional distance data into a set of initial coordinate data in the sensor coordinate system used by the ultrasonic sensor based on the one-dimensional distance data and the detection angle range corresponding to the ultrasonic sensor. For example, a polar coordinate system is established with the location of the ultrasonic sensor as the origin, and the detection angle range of the ultrasonic sensor can be represented in this polar coordinate system as follows: Assuming the unit distance data detected by the ultrasonic sensor is R, the detected coordinate data is represented in the polar coordinate system as an arc with radius R within the detection angular range. The detection angular range is divided into multiple angular intervals with a detection angular resolution θ. For each angular interval, an initial coordinate data is taken for the corresponding arc. For example, the coordinate data of the arc corresponding to the midpoint angle of the angular interval can be taken as the initial coordinate data. Therefore, for the midpoint angle... The corresponding initial coordinate data (x) j ,y j It can be calculated using the following formula:

[0029]

[0030] Here, each angle interval corresponds to an initial coordinate data point, and multiple initial coordinate data points corresponding to all angle intervals form a set of initial coordinate data. When determining the initial coordinate data for each angle interval, in addition to the midpoint angle, the angle of one of the endpoints of the angle interval or other angles can also be selected; there are no restrictions here. Here, the single-dimensional distance data and the initial coordinate data are located on the same detection plane of the sensor coordinate system (which can be denoted as z = z0). Therefore, during the transformation, the normal vector dimension of the detection plane (i.e., the z-axis) can be temporarily ignored. The value of the normal vector dimension remains unchanged and can be directly added to the initial coordinate data. After addition, the initial coordinate data is (x... j ,y j ,z0).

[0031] In some embodiments, multiple preset angle ranges are obtained by dividing the target plane containing the target origin into angles with the target origin as the vertex and according to preset unit angles. The coordinate data in the multiple sets of sensor coordinate data are coordinate data projected onto the target plane. The ultrasonic sensor uses a two-dimensional Cartesian coordinate system, and the plane containing the two-dimensional Cartesian coordinate system (i.e., the detection plane) is parallel to the target plane. At this time, the initial two-dimensional coordinate data does not need to be supplemented with the normal dimension of the detection plane. j ,y j Transform the sensor coordinate data (x) into the target coordinate systemb1 ,y b1 When ), the transformation relation can be derived from the rotation matrix R. s Translation matrix T s In addition to determining the exact coordinates, the rotation angle ψ and translation vector (t) between the origin and the target origin of the two-dimensional Cartesian coordinate system can also be used. x ,t y ) Confirmed. The specific conversion process is as follows:

[0032]

[0033] In some embodiments, the data detected by the sensors may include, in addition to sensor coordinate data, attribute point cloud data, such as intensity data, reflectivity, color, and normal vectors. Therefore, the method according to embodiments of the present invention may further include: acquiring attribute point cloud data obtained by multiple sensors detecting a target scene within a target time period; and integrating the attribute point cloud data corresponding to all selected coordinate data within multiple coordinate datasets into a multi-sensor dataset.

[0034] Figure 3 It shows Figure 2 A schematic diagram of the data information flow. (For example...) Figure 3 As shown, for each set of initial coordinate data obtained from the sensor, preprocessing can be performed, mainly including at least one of noise removal and downsampling. Specifically, for point cloud data acquired by the stereo vision sensor, outlier noise removal is first performed (specifically, noise points that significantly deviate from the normal range are removed through statistical analysis of the point cloud data distribution). Then, a voxel grid downsampling method is used to downsample the point cloud data, reducing the data volume and improving the efficiency of subsequent processing. Outlier noise removal is performed on the point cloud data acquired by the lidar sensor. For distance data acquired by the ultrasonic sensor, a moving average filtering method is used to filter the data to remove abnormal noise caused by signal interference and other factors.

[0035] like Figure 3 As shown, the initial coordinate data, after preprocessing, can be stored in a Last-In-First-Out (LIFO) queue. After being stored in the LIFO queue, the initial coordinate data is converted into sensor coordinate data using a transformation relationship. Multi-sensor data fusion is then performed on the sensor coordinate data, and finally, a multi-sensor data set is output. Specifically, determining the selected coordinate data in each coordinate dataset can be achieved by creating an array corresponding to each coordinate dataset, iterating through the sensor coordinate data, and updating the array accordingly.

[0036] Figure 4 A schematic diagram of a computer system is shown that can implement the method and apparatus for multi-sensor data fusion according to embodiments of the present invention. It should be understood that... Figure 4The computer system 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the method and apparatus for multi-sensor data fusion according to embodiments of the present invention.

[0037] like Figure 4 As shown, the computer system 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the computer system 400. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0038] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, cameras, accelerometers, gyroscopes, sensors, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, motors, electronic speed controllers, etc.; storage devices 408 including, for example, flash cards; and communication devices 409. Communication device 409 allows computer system 400 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 4 A computer system 400 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0039] In particular, according to some embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, a computer-readable medium is provided having a computer program stored thereon, the computer program comprising methods for executing... Figure 1 The program code shown is for a method of multi-sensor data fusion. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it implements the functional units defined in the apparatus for multi-sensor data fusion according to an embodiment of the present invention.

[0040] It should be noted that the computer-readable medium according to embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium according to embodiments of the present invention may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, the computer-readable signal medium according to embodiments of the present invention may include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0041] Computer program code for performing operations according to embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0042] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0043] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A method for multi-sensor data fusion, comprising: Acquire multiple sets of sensor coordinate data obtained from multiple sensors detecting the target scene within a target time period; Multiple coordinate datasets are determined according to multiple preset angle ranges based on the target origin and a preset unit angle. Each of the multiple coordinate datasets is a set of coordinate data of each set of sensor coordinate data located within the corresponding preset angle range of the multiple preset angle ranges. The coordinate data in each of the multiple coordinate datasets that is closest to the target origin is recorded as the selected coordinate data. as well as By integrating all selected coordinate data from the multiple coordinate datasets, a multi-sensor data set is obtained.

2. The method according to claim 1, wherein, The multiple preset angle ranges are obtained by dividing the target plane where the target origin is located into angles with the target origin as the vertex and according to the preset unit angle. The coordinate data in the multiple sets of sensor coordinate data are the coordinate data projected onto the target plane.

3. The method according to claim 1, wherein, For each of the plurality of sensors, acquiring a set of sensor coordinate data obtained by the sensor detecting the target scene within the target time period includes: Acquire a set of initial coordinate data in the sensor coordinate system used by the sensor, obtained by the sensor detecting the target scene within the target time period, wherein the sensor coordinate system can describe the positional relationship between the target scene and the sensor; and Based on the transformation relationship between the target coordinate system and the sensor coordinate system, the initial set of coordinate data is converted into a set of sensor coordinate data in the target coordinate system.

4. The method according to claim 3, wherein, Both the target coordinate system and the sensor coordinate system are Cartesian coordinate systems, and the transformation relationship is determined by a rotation matrix and a translation matrix.

5. The method according to claim 4, wherein, The plurality of sensors includes an ultrasonic sensor, and the initial coordinate data obtained by the ultrasonic sensor detecting the target scene within the target time period includes: Acquire one-dimensional distance data obtained by the ultrasonic sensor detecting the target scene within the target time period; and Based on the single-dimensional distance data and the detection angle range corresponding to the ultrasonic sensor, the single-dimensional distance data is converted into a set of initial coordinate data in the sensor coordinate system used by the ultrasonic sensor.

6. The method according to claim 5, wherein, The multiple preset angle ranges are obtained by dividing the target plane containing the target origin into angles based on the preset unit angle, with the target origin as the vertex. The coordinate data in the multiple sets of sensor coordinate data are coordinate data projected onto the target plane. The ultrasonic sensor uses a two-dimensional Cartesian coordinate system, and the plane containing the two-dimensional Cartesian coordinate system is parallel to the target plane.

7. The method according to claim 1, further comprising: Acquire attribute point cloud data obtained by the multiple sensors detecting the target scene during the target time period; as well as The attribute point cloud data corresponding to all selected coordinate data in the multiple coordinate datasets are integrated into the multi-sensor data set.

8. An apparatus for multi-sensor data fusion, comprising: processor; as well as A memory having stored computer-executable instructions thereon, wherein, when executed by the processor, the computer-executable instructions cause the processor to perform the method for multi-sensor data fusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon computer-executable instructions, wherein, When executed by a processor, the computer-executable instructions cause the processor to perform the method for multi-sensor data fusion as described in any one of claims 1 to 7.

10. A computer program product comprising computer-executable instructions, wherein, When executed by a processor, the computer-executable instructions cause the processor to perform the method for multi-sensor data fusion as described in any one of claims 1 to 7.

Images