System and method for underground spatial perception

Abstract

A sensor module, spatial perception system and an object detection method for a vehicle operating underground. The spatial perception system comprises at least one sensor module. The sensor module comprising at least two RADAR sensors. Each RADAR sensor is configured to generate a point cloud of a surrounding space of the vehicle. The point clouds generated by the at least two RADAR sensors are combined into a combined point cloud. The combined point cloud is transformed onto a frame of reference relative to the vehicle based on a position and a pose of the sensor module relative to the vehicle.

Description

SYSTEM AND METHOD FOR UNDERGROUND SPATIAE PERCEPTIONField

[0001] The present invention relates generally to spatial perception technologies for vehicles such as vehicles operating underground. In particular, the present invention relates to systems and methods for detecting objects around an underground vehicle and vehicle-object collision prediction and prevention.

[0002] While the present invention is described with reference to underground vehicles, some aspects of the present invention may also find applications in above-ground mining and construction vehicles.Background

[0003] Safety issues for vehicles operating underground, such as in an underground tunnel, include how to effectively pre-warn, or prevent, collisions of vehicles between each other, between vehicles and fixed infrastructure or machinery, and between vehicles and human personnel.

[0004] The safety and productivity of vehicles operating in underground tunnels has been at a disadvantage compared to surface-based applications due to the lack of practical technologies and methodologies available for sensing and control. One problem faced by vehicles operating underground is that vehicle localisation using the global positioning system (GPS) is typically not available underground due to the inability of GPS radio signals to penetrate below the earth’s surface.

[0005] Further compounding the problem are those underground environments, such as certain mining applications, where the presence of explosive gas or dust may pose a safety hazard due to the risk of uncontrolled or undesired explosion or fire. Uncontrolled or undesired explosion or fire may further result in damage of electrical equipment and / or financial loss. Simply migrating sensing and navigation technologies used in non-explosive environments to underground tunnels with explosive atmospheres is typically impractical and not cost effective.Summary

[0006] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more of the above disadvantages.

[0007] One aspect of the present disclosure provides a sensor module for a vehicle operating underground, the sensor module comprising: at least two RADAR sensors, each RADAR sensor configured to generate a point cloud of a surrounding space of the vehicle; and a processing unit and a memory storing instructions, which when executed by the processing unit, cause the processing unit to combine the point cloud generated by each of the RADAR sensors into a combined point cloud and transform the combined point cloud onto a frame of reference relative to the vehicle based on a position and a pose of the sensor module relative to the vehicle.

[0008] In one or more examples, the two RADAR sensors are identical.

[0009] In one or more examples, the sensor module further comprises at least one of a LiDAR sensor, a time of flight (ToF) sensor, and an imaging sensor.

[0010] In one or more examples, the imaging sensor comprises at least one of a visual imaging sensor and an Infra-Red (IR) thermal imaging sensor.

[0011] In one or more examples, the frame of reference is a frame of the vehicle.

[0012] In one or more examples, the instructions comprise instructions to apply Cell Averaging Constant False Alarm Rate (CACFAR) algorithm to each of the generated point clouds.

[0013] In one or more examples, the point clouds generated by the two RADAR sensors are combined by overlaying the point cloud generated by one of the two RADAR sensors with the point cloud generated by the other one of the two RADAR sensors.

[0014] In one or more examples, the point cloud generated by one of the two RADAR sensors is compared with the point cloud generated by the other one of the two RADAR sensors.

[0015] In one or more examples, in response to a comparison result that the point cloud generated by one of the two RADAR sensors differs from the point cloud generated by the other one of the two RADAR sensors, point cloud data associated with identified differences is removed from one or both of the point clouds.

[0016] Another aspect of the present disclosure provides a spatial perception system for a vehicle operating underground, the system comprising: at least one sensor module configured for installation on the vehicle, the at least one sensor module comprising: at least two RADAR sensors, each RADAR sensor configured to generate a point cloud of a surrounding space of the vehicle; and a processing unit and a memory storing instructions, which when executed by the processing unit, cause the processing unit to combine the point cloud generated by each of the RADAR sensors into a combined point cloud and transform the combined point cloud onto a frame of reference relative to the vehicle based on a position and a pose of the at least one sensor module relative to the vehicle; a central processing unit connected to the at least one sensor module via a network; and a central memory connected to the central processing unit and storing instructions, which when executed by the central processing unit, cause the central processing unit to: receive the transformed point cloud from the at least one sensor module via the network; identify an object in the received point cloud; and determine a predicted future trajectory of the identified object.

[0017] In one or more examples, the two RADAR sensors are identical.

[0018] In one or more examples, the at least one sensor module comprises at least one of a LiDAR sensor, a time of flight (ToF) sensor, and an imaging sensor.

[0019] In one or more examples, the imaging sensor comprises at least one of a visual imaging sensor and an Infra-Red (IR) thermal imaging sensor.

[0020] In one or more examples, the frame of reference is a frame of the vehicle.

[0021] In one or more examples, the instructions comprise instructions to apply Cell Averaging Constant False Alarm Rate (CACFAR) algorithm to each of the generated point clouds.

[0022] In one or more examples, the point clouds generated by the two RADAR sensors are combined by overlaying the point cloud generated by one of the two RADAR sensors with the point cloud generated by the other one of the two RADAR sensors.

[0023] In one or more examples, the point cloud generated by one of the two RADAR sensors is compared with the point cloud generated by the other one of the two RADAR sensors.

[0024] In one or more examples, in response to a comparison result that the point cloud generated by one of the two RADAR sensors differs from the point cloud generated by the other one of the two RADAR sensors, point cloud data associated with identified differences is removed from one or both of the point clouds.

[0025] In one or more examples, the instructions stored on the central memory, when executed by the central processing unit, further cause the central processing unit to: determine whether the vehicle will collide with the identified object; and generate a recommendation for a collision avoidance action in response to predicting that the vehicle will collide with the identified object.

[0026] Another aspect of the present disclosure provides a method of detecting objects surrounding a vehicle operating underground, the method comprising: generating a point cloud, by each of at least two RADAR sensors, of a surrounding space of the vehicle; combining the point cloud generated by each of the at least two RADAR sensors into a combined point cloud; transforming the combined point cloud onto a global frame of reference relative to the vehicle based on position and pose of the spatial perception sensor module relative to the vehicle; identifying an object in the transformed point cloud; and determining a predicted future trajectory of the identified object.

[0027] In one or more examples, the method further comprises determining whether the vehicle will collide with the identified object.

[0028] In one or more examples, the method further comprises generating a recommendation for a collision avoidance action in response to predicting that the vehicle will collide with the identified object.

[0029] Other aspects (of the invention) are also disclosed.Brief Description of Drawings

[0030] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0031] Figure 1 is a block diagram of a spatial perception sensor module according to the present invention.

[0032] Figure 2 is a perspective view of an example of an enclosure for housing the spatial perception sensor module of Figure 1.

[0033] Figure 3 is a diagram illustrating a Time of Flight (ToF) distance measurement concept.

[0034] Figures 4A to 4C are a perspective view, top view, and side view, respectively, of an object located in the field of view of the spatial perception sensor module of Figure 1.

[0035] Figure 5 is an example of 3-dimensional (3D) point cloud representation generated by the spatial perception sensor module of Figure 1.

[0036] Figures 6A and 6B collectively form a schematic block diagram representation of an electronic device upon which described arrangements can be practised.

[0037] Figure 7 is a flow chart of a method performed by the spatial perception sensor module of Figure 1 according to the present invention.

[0038] Figure 8 is a block diagram of a spatial perception system according to the present invention.

[0039] Figure 9 is a schematic diagram showing an example spatial arrangement of a plurality of sensor modules of the spatial perception system of Figure 8 with respect to a vehicle.

[0040] Figure 10 is a schematic diagram showing an example spatial arrangement of a plurality of sensor modules of the spatial perception system of Figure 8 with respect to another vehicle.

[0041] Figure 11 is a perspective view of an example of an enclosure for housing a central processing unit of the spatial perception system of Figure 8.

[0042] Figure 12A is a flow chart of a method performed by the spatial perception system of Figure 8 according to the present invention.

[0043] Figure 12B is a data flow diagram showing the data flow of the method of Figure 12A.

[0044] Figures 13 A and 13B are diagrams illustrating an example scenario where a single object is detected and identified by the spatial perception system of Figure 8.

[0045] Figures 14A and 14B are diagrams illustrating an example scenario where two objects are detected and identified by the spatial perception system of Figure 8.

[0046] Figures 15A-15D are diagrams illustrating an example process of tracking an identified object and predicting a future trajectory of the object using the method of Figure 12.

[0047] Figures 16A-16D are diagrams illustrating example scenarios where the vehicle of Figure 9 collides with an object that is moving relative to the vehicle.

[0048] Figure 17 is a schematic diagram showing an example configuration of safety zones configured for the vehicle of Figure 9.

[0049] Figure 18 is a block diagram of a collision avoidance system in which the application of collision avoidance can be implemented according to the present invention.

[0050] Figure 19 is a flow chart of a method performed by the collision avoidance system of Figure 18 according to the present invention.

[0051] Figure 20 is a diagram showing an example of an object being tracked by the collision avoidance system of Figure 18 that is predicted to be travelling into a safety zone of Figure 17.Detailed Description

[0052] One of the important safety and productivity goals for many underground vehicle applications is to remove human personnel from underground work sites so that vehicles can be monitored and controlled from a remote location, for example, a remote control room.Removing human personnel from any exposure to moving vehicles operating in the underground work sites also reduces human safety hazards. Additionally, removing human personnel from underground work sites provides scope for semi-autonomous and fully- autonomous vehicle operation to improve vehicle productivity.

[0053] Some embodiments of the present disclosure provide spatial perception for underground vehicle applications such that the underground vehicle applications do not require human personnel to be present at underground work sites.

[0054] According to some embodiments, spatial perception is implemented using a sensor module that comprises at least two Radio Detection and Ranging (RADAR) sensors and optionally additional sensors of the same type or different type. Each RADAR sensor is configured to provide point cloud data of a surrounding space of a vehicle on which the sensor module is installed. In some implementations, one or more sensor modules are installed at various locations on the vehicle for improved reliability of spatial perception.

[0055] Some embodiments provide a system that comprises the sensor module, a central processing unit and memory storing instructions configured to perform spatial perception analysis based on the point cloud data from the sensor module. The spatial perception analysis can comprise, for example, object detection, collision prediction, and collision prevention. In the present application, objects can include, but not limited to people, other vehicles, and / or stationary infrastructure.

[0056] Embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it will be appreciated that additions, adaptions, and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure.

[0057] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.Spatial Perception Sensor Module

[0058] Figure 1 shows a spatial perception sensor module 100. The sensor module 100 is configured for installation on a vehicle operating underground, such as in an underground work site like a tunnel. The sensor module 100 comprises sensor elements for perceiving objects in the field of view (FoV) of the sensor module 100, a processor 104 connected to the sensor elements, and a memory 106 connected to the processor 104. The memory 106 stores instructions, when executed by the processor 104, cause the sensor module 100 to perform various functions or effect the steps of the methods discussed below. The instructions may be formed as one or more software modules, each for performing one or more particular tasks. Theprocessor 104 and / or memory 106 can be built into the sensor module 100 or separate units coupled to the sensor module 100.

[0059] The sensor elements can be of any type of radiation emitter / detector device that allows points (point cloud) of an external physical surface to be generated and located in a 3- dimensional (3D) coordinate system. The generated point cloud comprises data points representing physical surfaces of nearby objects such humans, other vehicles, stationary infrastructure, or the like and 3D location information of the points in space. The FoV of each sensor element is aligned with the FoVs of other sensor elements in the sensor module 100 and the FoV of the sensor module 100 itself based on measured offsets between the FoV of the sensor element and the FoV of the sensor module 100.

[0060] The sensor elements include at least two RADAR sensors. Dust particles do not typically present a reflective surface to RADAR signals due to the size and low volumetric distribution in air of the particles. Therefore, RADAR sensors are typically less prone to failure or malfunction in applications where dust may be present. However, other sensors like Light Detection and Ranging (LiDAR) sensors or Time-of-Flight (ToF) sensors, in addition to or instead of, may also be used. Each RADAR sensor comprises a transmitter radio circuit and a receiver radio circuit (not shown). The transmitter radio circuit and the receiver radio circuit are each coupled with a matched antenna (not shown). The transmitter radio circuit and receiver radio circuit allow detection of radio pulse reflections from nearby surfaces to build a point cloud of a surrounding space of the vehicle in which the vehicle is travelling in a 3 -dimensional coordinate system. The location and relative velocity vectors of points in the point cloud can be determined from the point cloud. As discussed above, location information is included in the point cloud data. Velocity information is included in the point cloud data in some implementations or can be calculated based on the location information included in the point cloud data in some other implementations.

[0061] In the example of Figure 1, the spatial perception sensor module 100 includes two substantially identical RADAR sensors 102. The two RADAR sensors 102 being substantially identical is not to be narrowly understood as the sensors being limited to have identical electronic and mechanical design, but rather the RADAR sensors 102 provide substantially the same sensing behaviour. For example, in some implementations, both RADAR sensors 102effectively provide substantially identical three-dimensional point cloud mapping for the same environment in which the vehicle is travelling.

[0062] For example, two sensors 102 are substantially identical if the sensors 102 capture the same object or the same portion of an object at substantially same time. The generated point cloud mappings output from the two sensors 102 in that case may have discrepancies caused by noise, dust and / or other artefacts, however, the spatial perception of the captured object or the captured portion of the object as defined by the generated point cloud mappings would be substantially identical.

[0063] However, it would be appreciated that the RADAR sensors 102 can have identical electronic and mechanical design in some implementations.

[0064] The provision of point cloud data from two identical RADAR sensors 102 offers redundant sensing of objects in the FoVs of the respective RADAR sensors. For example, in some implementations, the two identical RADAR sensors 102 provide two point cloud data sets associated with the same object which can then be compared with each other. If there are discrepancies between the data sets, the corresponding point cloud data can be disregarded or further analysed to verify the validity thereof. For example, if one of the point cloud data sets includes outliers next to a cluster while the other one of the point cloud data sets does not, the outliers can be determined to be related to noise and invalid. Invalid data can be removed from one or both of the point cloud data sets from the RADAR sensors 102 so as to be removed from processes such as clustering identification discussed below, thereby improving confidence of object detection.

[0065] It would be appreciated that the spatial perception sensor module 100 could include more than two identical sensors. A person skilled in the art would also appreciate that the identical sensors do not need to be RADAR sensors and can be any radiation emitter / detector devices that allow points (point cloud) of an external physical surface to be generated and located in a 3- dimensional (3D) coordinate system. For example, two substantially identical LiDAR sensors that capture the same object or the same portion of an object at substantially same time can be used instead.

[0066] In some implementations, the sensor module 100 comprises a communications interface 108 connected to the processor 104 for communicating with a central processing unit 802 discussed below with reference to Figure 8 or other sensor modules on the vehicle.

[0067] To supplement the sensing provided by the RADAR sensors 102, the spatial perception sensor module 100 in some implementations can include a number of other sensor elements in addition to the RADAR sensors 102. The additional sensor elements can be of the same type as the RADAR sensors 102 or of different technology types. For example, the additional sensor elements can include one or more sensors that likewise provide a point-cloud data representation of physical surfaces in the FoV of the sensor module 100, such as LiDAR sensors or ToF sensors. In some implementations, the additional sensor elements additionally or alternatively include one or more imaging sensors such as visual imaging sensor or Infra-Red (IR) thermal imaging sensor. The increase in richness of point cloud data due to the addition of one or more LiDAR, ToF, or RADAR sensors and the corresponding improvement due to sensor diversity can provide additional confidence of object detection and tracking discussed below. Further, the additional sensor elements can provide additional object detection redundancy as well as a more comprehensive perception mapping of the environment in which the vehicle is travelling.

[0068] In the example of Figure 1, the spatial perception sensor module 100 includes a LiDAR 110, a Time of Flight (ToF) sensor 112, a visual imaging sensor 114, and an Infra-Red thermal imaging sensor 116 in addition to the two RADAR sensors 102.

[0069] The LiDAR sensor 110 and the ToF sensor 112 are each configured to provide either a 2-dimensionnal or 3-dimensional point cloud data set to the processor 104. The ToF sensor 112 can be any type of technology where propagation or travel time of a reflected energy signal is measured to determine the distance to a nearby object. Examples of existing sensing technologies that achieve this outcome include, but not limited to, radio-wave RADAR, sound wave RADAR, and LiDAR.

[0070] The visual imaging sensor 114 can be any device that captures human-visible light, such as cameras, and digitizes the data in a 2-dimensional image captured by the imaging sensor 114 which is then stored into the memory 106.

[0071] The infrared (IR) thermal imaging sensor 116 can be any device that captures infrared radiation and digitizes each captured frame as a 2-dimensional map which is then stored into the memory 106.

[0072] The sensor elements in the spatial perception module 100 and the processor 106 can be in bi-directional communication to allow configuration of the sensor elements and transmissionof sensing data from the sensor elements to the processor 104 by execution of instructions stored on the memory 106 by the processor 104 which has access to the memory 106. The memory 106 can be formed from non-volatile semiconductor read only memory (ROM) and semiconductor random access memory (RAM). Software is stored in the ROM memory to operate the processor.

[0073] In some implementations, the sensor module 100 is configured to allow signal transmission between the processor 104 and each sensor to be carried out independently of each other in order to minimise interference between sensors of similar radiation type. Additionally or alternatively, each sensor element can use signals that occupy a different radiation spectrum frequency band for transmitting / receiving signals to / from the processor 104 to prevent interference.

[0074] Figure 2 shows an example of an enclosure 200 for housing the sensor module 100. Certain underground work sites are so-called “hazardous environments” due to the presence of explosive gasses or dust in such sites. To provide protection against undesirable explosion or fire for sensor module 100 used in hazardous environments, the enclosure 200 is of an explosion-protected design and construction. The explosion protection can be based on any technique or design standard, for example, those that are accepted and recognized by either local or international standards in relation to explosion protection. Examples of such standards include the international IEC standard IEC 60079 and the European Standard EN 60079. The standard used may vary depending on the country in which the systems and methods of the present disclosure are practiced. In some implementations, the enclosure 200 is an 'Ex d' (flameproof) enclosure. In some other implementations the enclosure (and the internal electronic circuits) can be protected against explosion by so-called “Intrinsic Safety” or “Ex i" type of protection. The protection types ‘Ex d’ and ‘Ex I’ are design protection techniques codified in standards such as IEC 60079.

[0075] In the example of Figure 2, a sensing window 202 is provided in a sidewall 204 of the enclosure 200 to allow transmission of radiant energy for sensing objects outside of the enclosure 200. The sensing window 202 is transparent to the type of radiation being used by the sensor elements of the sensor module 100. If no visual radiation sensor (e.g., imaging camera or certain types of visual-based ToF sensor) is used, the window may not be transparent for human-vision, but will be transparent to the type of radiation that is used (e.g., RADAR).

[0076] The enclosure 200 is provided with one or more attachment mechanisms, such as fasteners (not shown), for attachment to the vehicle, and an opening for insertion of an electrical cable 208 for data communications and power supply. The enclosure 200 can also include unused openings for additional cable entry. The unused openings can be plugged with a protective plug 206 that complies with the requirements of the explosion-protection design technique used.

[0077] In the example of Figure 2, the enclosure 200 has a substantially square or cubic shape. However, it will be appreciated that the enclosure 200 is not limited to the example shape shown in Figure 2.

[0078] In the example of Figure 2, all of the components of the sensor module 100 are housed in a single enclosure 200. However, it will be appreciated that the internal components of the sensor module 100 may also be arranged in separate enclosures, or any combination of enclosures, depending on the practical requirements of each application. Figure 2 is representative of an explosion-protected enclosure for use in hazardous areas where explosive gas or dust may be present. However, it will be appreciated that for applications where no such explosion risk exists, the enclosure can also be of a non-explosion-protected type.

[0079] Turning to Figure 3, the principle used by a sensor element including RADAR, LiDAR or ToF sensor in the spatial perception sensor module 100 to detect a point in any nearby object is illustrated. The sensor element periodically transmits a signal 5. The transmitted signal 5 can be light, radio wave radiation or sound wave radiation, depending on the type of sensor used. The transmitted signal 5 is reflected by a nearby object, for example, a human 302 as shown in the example of Figure 3. A reflected signal r then arrives back at the sensor module 100. The sensor element measures, using a timer 304 built into the sensor element, the total propagation time t taken for the transmitted signal .s to travel from the sensor module 100 to the human 302 and for the reflected signal r to travel back to the sensor module 100. The distance S between the sensor module 100 and the object is then calculated using Equation (1) below. vt S = — 2wherein v denotes the velocity of the signal 5 and signal r.

[0080] In Equation (1), the velocity v of the transmitted signal 5 and reflected signal r is known, being the constant velocity of the chosen radiation used by the sensor element, thereby allowing the distance to be calculated by measuring the propagation time only.

[0081] In the case of RADAR, LiDAR and ToF sensors that measure distance to points as illustrated in Figure 3, the sensor element is configured to provide the processor 104 with 3- dimensional distance ranging of points in any external surfaces in the FoV of the sensing module 100.

[0082] Figures 4A, 4B, and 4C illustrate an example scenario where an object 408 is within the FoV of the sensor module 100. Figure 4A is a perspective view 400 showing that a point 406 on an external physical surface of the object 408 is 2274mm away from the sensor module 100. Figure 4B is a top view 402 showing that the point 406 has a rotation (polar) angle of 27.1° relative to the sensor module 100. Figure 4C is a side view 404 showing that the point 406 has an elevation angle (azimuthal angle) of 14.4° relative to the sensor module 100. The spherical coordinate position P of the point 406 can be expressed by Equation (2) below:P = (r, 0, (p) (2) where r denotes the radial distance between the sensor module 100 and the point 406, Q denotes the rotation (polar) angle of the point 406 relative to the sensor module 100, and (p denotes the elevation angle (azimuthal angle) of the point 406 relative to the sensor module 100. In the example of Figures 4A to 4C, the spherical coordinate position of the point 406 can be expressed as P = (2274mm, 27.1°, 14.4°). The coordinates specify the position of the object 408 respective to the sensor module 100.

[0083] Figure 5 shows an example of raw point cloud data collected by a point cloud sensor element (e.g., RADAR, LiDAR or ToF sensor) of the spatial perception sensor module 100. The data is presented visually in Figure 5 for illustration purposes only. In other words, visual presentation of point cloud data is not mandatory. The point cloud data from each of such sensor elements can be stored as 3 -dimensional coordinate data in the memory 106.

[0084] Figures 6A and 6B collectively form a schematic block diagram of a general purpose electronic device 601 including embedded components, upon which the methods to be described are desirably practiced, for example methods implemented on the module 100 or a system 800,to be described. The electronic device 601 may be, for example, a mobile phone, a portable media player or a digital camera, in which processing resources are limited. Nevertheless, the methods to be described may also be performed on higher-level devices such as desktop computers, server computers, and other such devices with significantly larger processing resources.

[0085] As seen in Figure 6A, the electronic device 601 comprises an embedded controller 602. Accordingly, the electronic device 601 may be referred to as an “embedded device.” In the present example, the controller 602 has a processing unit (or processor) 605 which is bidirectionally coupled to an internal storage module 609. An example implementation of the processing unit 605 is the processor 104 or the central processing unit described below. An example implementation of the internal storage module 609 is the memory 106 or the central memory 806 described below. The storage module 609 may be formed from non-volatile semiconductor read only memory (ROM) 660 and semiconductor random access memory (RAM) 670, as seen in Figure 6B. The RAM 670 may be volatile, non-volatile or a combination of volatile and non-volatile memory.

[0086] The electronic device 601 can include a display controller 607, which is connected to a video display 614, such as a liquid crystal display (LCD) panel or the like. The display controller 607 is configured for displaying graphical images on the video display 614 in accordance with instructions received from the embedded controller 602, to which the display controller 607 is connected.

[0087] The electronic device 601 can also include user input devices 613 which are typically formed by keys, a keypad or like controls. In some implementations, the user input devices 613 may include a touch sensitive panel physically associated with the display 614 to collectively form a touch-screen. Such a touch-screen may thus operate as one form of graphical user interface (GUI) as opposed to a prompt or menu driven GUI typically used with keypad-display combinations. Other forms of user input devices may also be used, such as a microphone (not illustrated) for voice commands or a joystick / thumb wheel (not illustrated) for ease of navigation about menus.

[0088] As seen in Figure 6A, the electronic device 601 can also comprise a portable memory interface 606, which is coupled to the processor 605 via a connection 619. The portablememory interface 606 allows a complementary portable memory device 625 to be coupled to the electronic device 601 to act as a source or destination of data or to supplement the internal storage module 609. Examples of such interfaces permit coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCMIA) cards, optical disks and magnetic disks.

[0089] The electronic device 601 also has a communications interface 608 to permit coupling of the device 601 to a computer or communications network 620 via a connection 621. An example implementation of the communications interface 608 is the communications interface 108. The connection 621 may be wired or wireless. For example, the connection 621 may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes Bluetooth™ type local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family), Infrared Data Association (IrDa) and the like.

[0090] Typically, the electronic device 601 is configured to perform some special function. The embedded controller 602, possibly in conjunction with further special function components 610, is provided to perform that special function. In the context of the present disclosure, where the device 601 is the sensor module 100, the components 610 may represent the sensor elements of the sensor module 100. The special function components 610 is connected to the embedded controller 602. As another example, where the device 601 is a digital camera, the components 610 may represent a lens, focus control and image sensor of the camera. As another example, the device 601 may be a mobile telephone handset. In this instance, the components 610 may represent those components required for communications in a cellular telephone environment. Where the device 601 is a portable device, the special function components 610 may represent a number of encoders and decoders of a type including Joint Photographic Experts Group (JPEG), (Moving Picture Experts Group) MPEG, MPEG-1 Audio Layer 3 (MP3), and the like.

[0091] The methods described hereinafter may be implemented using the embedded controller 602, where the processes of Figures 7, 12, and 19 may be implemented as one or more software application programs 633 executable within the embedded controller 602. The electronic device 601 of Figure 6A implements the described methods. In particular, with reference to Figure 6B, the steps of the described methods are effected by instructions in thesoftware 633 that are carried out within the controller 602. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the described methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0092] The software 633 of the embedded controller 602 is typically stored in the non-volatile ROM 660 of the internal storage module 609. The software 633 stored in the ROM 660 can be updated when required from a computer readable medium. The software 633 can be loaded into and executed by the processor 605. In some instances, the processor 605 may execute software instructions that are located in RAM 670. Software instructions may be loaded into the RAM 670 by the processor 605 initiating a copy of one or more code modules from ROM 660 into RAM 670. Alternatively, the software instructions of one or more code modules may be pre-installed in a non-volatile region of RAM 670 by a manufacturer. After one or more code modules have been located in RAM 670, the processor 605 may execute software instructions of the one or more code modules.

[0093] The application program 633 is typically pre-installed and stored in the ROM 660 by a manufacturer, prior to distribution of the electronic device 601. However, in some instances, the application programs 633 may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the portable memory interface 606 of Figure 6 A prior to storage in the internal storage module 609 or in the portable memory 625. In another alternative, the software application program 633 may be read by the processor 605 from the network 620, or loaded into the controller 602 or the portable storage medium 625 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and / or data to the controller 602 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the device 601. Examples of transitory or non -tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the device 601 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded onWebsites and the like. A computer readable medium having such software or computer program recorded on it is a computer program product.

[0094] The second part of the application programs 633 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 614 of Figure 6A. Through manipulation of the user input device 613 (e.g., the keypad), a user of the device 601 and the application programs 633 may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via the microphone (not illustrated).

[0095] Figure 6B illustrates in detail the embedded controller 602 having the processor 605 for executing the application programs 633 and the internal storage 609. The internal storage 609 comprises read only memory (ROM) 660 and random access memory (RAM) 670. The processor 605 is able to execute the application programs 633 stored in one or both of the connected memories 660 and 670. When the electronic device 601 is initially powered up, a system program resident in the ROM 660 is executed. The application program 633 permanently stored in the ROM 660 is sometimes referred to as “firmware”. Execution of the firmware by the processor 605 may fulfil various functions, including processor management, memory management, device management, storage management and user interface.

[0096] The processor 605 typically includes a number of functional modules including a control unit (CU) 651, an arithmetic logic unit (ALU) 652, a digital signal processor (DSP) 653 and a local or internal memory comprising a set of registers 654 which typically contain atomic data elements 656, 657, along with internal buffer or cache memory 655. One or more internal buses 659 interconnect these functional modules. The processor 605 typically also has one or more interfaces 658 for communicating with external devices via system bus 681, using a connection 661.

[0097] The application program 633 includes a sequence of instructions 662 though 663 that may include conditional branch and loop instructions. The program 633 may also include data,which is used in execution of the program 633. This data may be stored as part of the instruction or in a separate location 664 within the ROM 660 or RAM 670.

[0098] In general, the processor 605 is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the electronic device 601. Typically, the application program 633 waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices 613 of Figure 6A, as detected by the processor 605. Events may also be triggered in response to other sensors and interfaces in the electronic device 601.

[0099] The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM 670. The disclosed method uses input variables 671 that are stored in known locations 672, 673 in the memory 670. The input variables 671 are processed to produce output variables 677 that are stored in known locations 678, 679 in the memory 670. Intermediate variables 674 may be stored in additional memory locations in locations 675, 676 of the memory 670. Alternatively, some intermediate variables may only exist in the registers 654 of the processor 605.

[0100] The execution of a sequence of instructions is achieved in the processor 605 by repeated application of a fetch-execute cycle. The control unit 651 of the processor 605 maintains a register called the program counter, which contains the address in ROM 660 or RAM 670 of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit 651. The instruction thus loaded controls the subsequent operation of the processor 605, causing for example, data to be loaded from ROM memory 660 into processor registers 654, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

[0101] Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program 633, and is performed by repeated execution of a fetch-execute cycle in the processor 605 or similar programmatic operation of other independent processor blocks in the electronic device 601.

[0102] Figure 7 shows a method 700 that can be performed by the spatial perception sensor module 100. The method 700 may be implemented by a sensor module 100 and execution of software modules stored as instructions in memory 106 and controlled by execution of the software modules by processor 104.

[0100] The method starts from step 702 in which each of the RADAR sensors 102 generates a point cloud of a surrounding space of the vehicle. Additionally or optionally, the LiDAR sensor 110 and / or ToF sensor 112, if included in the sensor module 100, each generates a point cloud of the surrounding space of the vehicle. Additionally or optionally, sensing data is also collected from each additional sensor element such as visual imaging sensor 114 or IR thermal imaging sensor 116, if included in the spatial perception sensor module 100, at step 702.

[0101] Data from each sensor element of the spatial perception sensor module 100 is collected at a periodic discrete sampling rate. The interval between samples can be defined by a configuration parameter known as the sampling interval. The sampling interval typically has a value in the order of milliseconds and can be configured between 10 milliseconds and 100 milliseconds. In some implementations, the sampling interval is 50 milliseconds.

[0102] In some implementations, the step 702 additionally comprises a filtering step of filtering out invalid data such as noise from raw point cloud data collected from the RADAR sensors 102, and optionally from the LiDAR sensor 110 or ToF sensor 112, using, for example, Cell Averaging Constant False Alarm Rate (CACFAR) algorithm. The sensitivity of the CACFAR algorithm is at least partially dependent on the type of the sensor used and / or whether the vehicle the system 800 is attached to is in motion. When the vehicle is in motion, the quantity of noise tends to increase, but the filtering of such noise can be as effective. Accordingly, by dynamically reducing the sensitivity of the CACFAR algorithm when the vehicle is in motion, the filtering step is optimised to identify points that are representative of a physical object. On the other hand, when the vehicle is stationary, the sensitivity of the CACFAR algorithm can be increased to improve reliability of close object detection.

[0103] The method 700 proceeds from the step 702 to a combining step 704 in which the point cloud data collected from each RADAR sensor 102, and optionally from the LiDAR sensor 110 or ToF sensor 112, is combined into a single data set. The single data set provides a combined 3- dimentional point cloud (e.g., a 3-dimentional map) comprising data from each sensor.

[0104] The combining step 704 comprises aligning data from each sensor generated with respect to the sensor to a common point of reference, such as the centre of the sensor module 100, and overlaying (e.g., superimposing) data from each sensor to generate the combined 3- dimensional point cloud. For example, the overlaying comprises overlaying the point cloud data from one of the RADAR sensors 102 with the point cloud data from the other one of the RADAR sensors 102. Known offsets between relative physical position of each sensor in the sensor module 100 can be used to align, and overlay, the point cloud data of each sensor on the same local reference relative to the sensor module itself. The alignment of data can be achieved by transforming the data onto a common coordinate system, using matrix multiplication, for example. In matrix multiplication, transformation between coordinates can be represented as a 4x4 matrix that defines the rotation (from quaternions) and translation to be applied. When a point cloud is transformed between coordinate systems, each point in the point cloud is multiplied by the matrix to convert the point to the new coordinate system, effectively shifting (translating) and rotating the point to a corresponding point in the new coordinate system.

[0105] In some implementations, the combining step 704 additionally or optionally comprises aligning the image data from the visual imaging sensor 114 and / or the thermal image data from the IR imaging sensor 116 to the common point of reference and overlaying (e.g., superimposing) the image data and / or thermal image data with the combined point cloud.

[0106] The combining step 704 can also include comparing the point cloud data set from one of the RADAR sensors 102 (hereinafter “the first RADAR sensor”) with the point cloud data set from the other one of the RADAR sensors 102 (hereinafter “the second RADAR sensor”) to determine whether the point cloud data sets provide statistically identical, or sufficiently similar, point cloud data. In one example, the comparison enlarges each point in the point cloud data set from the first RADAR sensor to be a cube with the point being at the centre of the cube, thereby generating a first set of cubes. The dimension of such cubes is configurable but typically has an edge length between 50mm and 200 mm. Similarly, each point in the point cloud data set from the second RADAR sensor is also enlarged to be a cube with the point being at the centre of thecube, thereby generating a second set of cubes. Each cube in the first set of cubes is then checked against the cubes in the second set of cubes to calculate the number of cubes overlaps between the first set of cubes and second set of cubes. If the percentage of cube overlap exceeds a pre-configured threshold, then data from both RADAR sensors 102 is considered to be statistically identical, or sufficiently similar, and valid. The pre-configured threshold is, for example, between 50% and 75%. In other implementations, other statistical algorithms typically used for point cloud analysis may be used.

[0107] In some implementations, point cloud data sets that do not provide statistically identical, or sufficiently similar, point clouds are considered to be invalid or non-coherent data. Identification of invalid data can trigger the entire data set from all sensors to be discarded.

[0108] Additionally or alternatively, identification of invalid data can trigger the entire data set from all sensors to be temporarily discarded util it can be further analysed to further verify the validity thereof. In some implementations, for example, implementations where the sensor module 100 additionally comprises the LiDAR sensor 110 and / or ToF sensor 112, the identification of invalid or non-coherent data can invoke a further sub-step of comparing the point cloud data from one or both RADAR sensors 102 with the point cloud data collected from the LiDAR sensor 110 and / or ToF sensor 112 to further verify the validity of the collected data. For example, if the point cloud data from one of the RADAR sensors 102 includes outliers next to a cluster while the point cloud data from the other one of the RADAR sensors 102 does not, the point cloud data from the LiDAR sensor 110 and / or a ToF sensor 112 is checked to determine whether there are outliers next to the corresponding cluster. In response to determining that the point cloud data from the LiDAR sensor 110 and / or a ToF sensor 112 does not include such outliers, then the outliers can be determined to relate to noise and removed from the combined point cloud. Similarly, in implementations where the sensor module 100 additionally comprises one or more imaging sensors such as visual imaging sensor 114 or IR thermal imaging sensor 116, the point cloud data collected by one or both RADAR sensors 102 can be compared with the image data captured by the one or more imaging sensors to further verify the validity of the point cloud data. For example, if the image data does not include any data that corresponds with the discrepancies in the point cloud data between the RADAR sensors 102, the point cloud data can be determined to be invalid. In another example, if the image data does include data that corresponds with the discrepancies in the point cloud databetween the RADAR sensors 102, for example, data indicative of an object, the point cloud data can be determined to be valid and retained in the combined point cloud.

[0109] Additionally or optionally, the combining step 704 comprises a buffering sub-step of placing the combined point cloud into a buffer. An example of such a buffer is shown in Figure 12B as buffer 1218, described below. The buffer can be implemented within the memory 106 or a central memory 806 of Figure 8 described below. The buffer depth of the buffer is a configuration parameter defining the number of data samples that can be stored in the data buffer at any point in time. In some implementations, the buffer is configured to have a buffer depth between one to six. In an example implementation, the accumulation buffer is configured to have a buffer depth of six samples. The buffer can be a first-in-first-out (FIFO) buffer so that when a new sample is obtained, the oldest sample in the buffer is disregarded (i.e., removed from the buffer) and the new data sample is added to the buffer.

[0110] The method 700 proceeds from the step 704 to a transforming step 706 in which the combined point cloud is transformed onto a global frame of reference, such as the frame of the vehicle. For example, the combined point cloud can be transformed from the frame of the sensor module 100 defined relative to the centre of the module 100 onto the frame of the vehicle defined relative to the centre of the vehicle where the vehicle is assumed to be stationary based on the position and pose of the sensor module 100 relative to the vehicle. The relative position and pose of the sensor module 100 can be obtained by a calibration process in which the coordinate reference offsets of the sensor module 100 are measured relative to the vehicle and stored into a non-volatile calibration memory implemented within the memory 106. Alternatively, the coordinate reference offset can be stored into a calibration memory of a storage space additionally included in or accessible to the spatial perception sensor module 100. An example of the calibration memory is shown in Figure 12B as calibration memory 1216, discussed below.

[0111] In some implementations, the same matrix multiplication as discussed above can be used to transform the combined point cloud onto the global frame of reference. The transformation onto the global point cloud allows data from the sensor modules 100 and other sensor module(s) installed on the vehicle to be positioned relative to the vehicle and to each other.

[0112] The method 700 ends at step 706.Spatial Perception System

[0113] Figure 8 shows a spatial perception system 800 of a vehicle operating underground. The spatial perception system 800 comprises at least one spatial perception sensor module 100 of Figure 1, a central processing unit 802 connected to the at least one spatial perception sensor module 100 via a network 804, and a central memory 806 connected to the central processing unit 802.

[0114] The spatial perception system 800 is adapted for installation on a vehicle, for example, vehicle 900 shown in Figure 9 and vehicle 1000 shown in Figure 10, described below, configured to travel in an underground environment, such as an underground tunnel network of a mine.

[0115] The number of sensor modules 100 in the system 800 is dependent upon the type, size, dimensions, and / or complexity of the vehicle and / or the practical requirements of the application. The number of sensor modules 100 in the system 800 typically increases in larger vehicles or where the vehicles travel at higher speeds. Point clouds are generated independently at each spatial perception sensor module 100.

[0116] Each of the at least one sensor module 100 has a corresponding FoV. If the spatial perception system 800 comprises a plurality of sensor modules 100, the FoVs of the plurality of sensor modules 100 can be set up to fully or at least in part differ from each other, for example, by installing the sensor modules 100 at various locations on the vehicle and / or with various poses. The combination of the FOVs of the plurality of the sensor modules 100 provides an increased object visibility coverage which can be beneficial in some applications. The point cloud data sets collected from the plurality of sensor modules 100 can provide a combined map of point cloud data with reference to the vehicle, thereby building a more comprehensive, holistic spatial awareness mapping of the space in which the vehicle is travelling.

[0117] Some or all of the components of the spatial perception system 800 can communicate via network 804. The network 804 can be an on-board local data network. In some implementations, components such as networking switching equipment of the network 804 can be integrated into the same enclosure, for example, enclosure 1100 of Figure 11 discussed below, as the central processing unit 802. In some other implementations, components of the network 804 and the central processing unit 802 can be housed in separate enclosures.

[0118] In some implementations, the central processing unit 802 is in data communication with a vehicle motion control system 810 of the vehicle, via a vehicle-wide network 812, allowing object tracking and collision prediction data to be shared with the vehicle motion control system 810.

[0119] In some implementations, the central processing unit 802 is in data communication with a wide area network, WAN, (not shown) via the vehicle-wide network 812 and a gateway 808 to enable sharing of data for applications such as remote monitoring and data collection. The external network connection via the vehicle-wide network 812 allows the system to support a number of possible applications including, but not limited to, remote monitoring of the system, remote configuration of system parameters, data logging of system events, and tele-remote control of the vehicle, etc.

[0120] Figure 9 shows an example sensor module arrangement for a vehicle 900 with the spatial perception system 800 of Figure 8 installed. In the example of Figure 9, the spatial perception system 800 installed on vehicle 900 comprises six (6) sensor modules 100a through lOOf. Sensor modules 100a to lOOf correspond to the sensor module 100 of Figure 1. Sensor modules 100a and lOOf are located towards the front portion of the vehicle 900, 100b and lOOe at around the middle portion of the vehicle 900, and 100c and lOOd towards the rear portion of the vehicle 900. The central processing unit 802 of the spatial perception system 800 can be arranged in an area between sensor module lOOd and sensor module lOOe for example.

[0121] Figure 10 shows an example sensor module arrangement for another vehicle 1000 with the spatial perception system 800 of Figure 8 installed. The vehicle 1000 has a shorter length compared to vehicle 900 of Figure 9. In the example of Figure 10, the spatial perception system 800 installed on vehicle 1000 comprises four (4) sensor modules lOOf through lOOi. Sensor modules lOOf to lOOi are the same as the sensor module 100 of Figure 1. Sensor modules lOOf and 100g are located on the sides and sensor modules lOOh and lOOi are located towards the front and rear of the vehicle 1000, respectively. The central processing unit 802 can be arranged in a region between sensor module 100g and sensor module lOOf. The number and placement of sensor modules can vary in each implementation based on factors such as vehicle size, environment, and the like.

[0122] Figure 11 shows an example of an explosion-protected enclosure 1100 of the central processing unit 802. The explosion protected enclosure 1100 provides protection for the central processing unit 802 against explosion in hazardous environments where explosive gas and / or dust may be present. The enclosure 1100 provides a plurality of cable entries 1102 to allow for connection such as data network connection to a number of spatial perception sensor modules 100 and to external devices and / or systems. The cable entries 1102 can be of cable-gland type or quick-release connector type, depending on the application. Similar to the enclosure 200, the enclosure 1100 can be of an explosion -protected design and construction. The explosion protection can be based on any technique or design standard, for example, those that are accepted and recognized by either local or international standards such as the standards discussed above. In some implementations, the enclosure 1100 is an 'Ex d' (flameproof) enclosure. In some other implementations the enclosure 1100 (and the internal electronic circuits) can be protected against explosion by so-called “Intrinsic Safety” or “Ex i" type of protection. The protection types ‘Ex d’ and ‘Ex I’ are design protection techniques codified in standards such as IEC 60079.Detection and Tracking of Objects

[0123] One application of the spatial perception system 800 is object detection.

[0124] Referring now to Figures 12A and 12B, an object detection method 1200 of performed by the spatial perception system 800 for a vehicle and a data flow diagram 1212 of method 1200 are illustrated. The method 1200 may be implemented by one or more sensor modules 100 and execution of software modules stored as instructions in memory 106 and central memory 806 and controlled by execution of the software modules by processor 104 and central processing unit 802.

[0125] In the example of Figs 12A and 12B, the spatial perception system 800 comprises a plurality of spatial perception sensor modules 100. It will however be appreciated that the spatial perception system 800 in some implementations can comprise a single spatial perception sensor module 100 only. Each spatial perception module 100 is configured to perform a method 1221a to generate a transformed point cloud 1221 discussed in more detail below.

[0126] As shown in Figure 12 A, the method 1200 starts from step 1202 in which the RADAR sensors 102 of each spatial perception sensor modules 100 each generate a point cloud of a surrounding space of the vehicle.

[0127] In some implementations, sensing data is also collected from each additional sensor element, if included in the spatial perception sensor module 100, at step 1202. For example, additional point cloud data can be collected from sensors such as LiDAR sensor 110 or the ToF sensor 112. Image data or thermal image data can be collected from sensors such as visual imaging sensor 114 or IR thermal imaging sensor 116.

[0128] As shown in Figure 12B, the step 1202 in some implementations comprises a filtering step 1214 of filtering out invalid data such as noise from raw point cloud data collected from the RADAR sensors 102, and optionally from the LiDAR sensor 110 and / or ToF sensor 112, using, for example, Cell Averaging Constant False Alarm Rate (CACFAR) algorithm. The sensitivity of the CACFAR algorithm is at least partially dependent on the type of the sensor used and / or whether the vehicle on which the system 800 is installed is in motion. When the vehicle is in motion, the quantity of noise tends to increase, but the filtering of such noise can be as effective. Accordingly, by dynamically reducing the sensitivity of the CACFAR algorithm when the vehicle is in motion, the filtering step 1214 is optimised to identify points that are representative of a physical object. On the other hand, when the vehicle is stationary, the sensitivity of the CACFAR algorithm can be increased to improve reliability of close object detection.

[0129] Returning to Figure 12 A, the method 1200 proceeds from step 1202 to a combining step 1204 in which the point cloud data collected from each RADAR sensor 102, and optionally from the LiDAR sensor 110 and / or ToF sensor 112, is combined into a single data set. The single data set provides a combined 3-dimentional point cloud (e.g., a 3-dimentional map) comprising data from each sensor.

[0130] The combining step 1204 comprises aligning data from each sensor to a common point of reference, such as the centre of the sensor module 100, and overlaying (e.g., superimposing) the aligned data to generate the combined 3-dimentional point cloud. For example, the overlaying comprises overlaying the point cloud data from one of the RADAR sensors 102 with the point cloud data from the other one of the RADAR sensors 102. Known offsets between relative physical position of each sensor in the sensor module 100 can be used to align, andoverlay, the point cloud data of each sensor on the same local reference relative to the sensor module itself. The alignment of data can be achieved by transforming the data onto a common coordinate system using matrix multiplication. In matrix multiplication, transformation between coordinates can be represented as a 4x4 matrix that defines the rotation (in terms of quaternions) and translation to be applied. Each point in the point clouds from the RADAR sensors 102 is multiplied by the respective matrices to convert, i.e., shift (translate) and rotate, the point to the common coordinate system.

[0131] In some implementations, the combining step 1204 additionally or optionally comprises aligning the image data from the visual imaging sensor 114 and / or the thermal image data from the IR imaging sensor 116 to the common point of reference and overlaying (e.g., superimposing) the image data and / or thermal image data with the combined point cloud.

[0132] The combining step 1204 can also include comparing the point cloud data set from one of the RADAR sensors 102 (“the first RADAR sensor”) with the point cloud data set from the other one of the RADAR sensors 102 (“the second RADAR sensor”) to determine whether the point cloud data sets provide statistically identical, or sufficiently similar, point cloud representations. In one example, the comparison enlarges each point in the point cloud data set from the first RADAR sensor to be a cube with the point being at the centre of the cube, thereby generating a first set of cubes. The dimension of such cubes is configurable but typically has an edge length between 50mm and 200 mm. Similarly, each point in the point cloud data set from the second RADAR sensor is also enlarged to be a cube with the point being at the centre of the cube, thereby generating a second set of cubes. Each cube in the first set of cubes is then checked against the cubes in the second set of cubes to calculate the number of cube overlaps between the first set of cubes and second set of cubes. If the percentage of cube overlap exceeds a pre-configured threshold, then data from both RADAR sensors 102 is considered to be statistically identical, or sufficiently similar, and valid. The pre-configured threshold is, for example, between 50% and 75%.

[0133] In some implementations, point cloud data sets that do not provide statistically identical, or sufficiently similar, point clouds are considered to be invalid or non-coherent data.

[0134] Identification of invalid data can trigger the entire data set from all sensors to be discarded. If data collected from a sensor module 100 continue to be non-coherent, for example,if non-coherent for a continuous period that exceeds a maximum allowable non-coherent period (threshold ), or if the number of successive non-coherent samples exceeds a maximum allowable number (threshold) of successive non-coherent samples, step 1204 can default the spatial perception system 800 to a safe state. The maximum allowable non-coherent period (threshold) can be configured, for example, between 2 to 20 data fetch cycles. For instance, if data is fetched every 50ms, the maximum allowable non-coherent period (threshold) is 100ms to 1000ms. The maximum allowable number (threshold) of successive non-coherent samples can be configured, for example, between 2 to 20 samples. For instance, if two successive samples are determined to be non-coherent, the spatial perception system 800 is defaulted to the safe state. In the safe state, functions such as object detection and collision prediction, described below, can be halted and warnings provided to the operator of the vehicle that such functions are unavailable.

[0135] Additionally or alternatively, identification of invalid data can trigger the data sets from all sensors to be analysed to further verify the validity thereof. In some implementations, for example, implementations where the sensor module 100 additionally comprises the LiDAR sensor 110 and / or ToF sensor 112, the identification of invalid or non-coherent data can invoke a further sub-step of comparing the point cloud data from one or both RADAR sensors 102 with the point cloud data collected from the LiDAR sensor 110 and / or ToF sensor 112 to further verify the validity of the collected data. For example, if the point cloud data from one of the RADAR sensors 102 includes outliers next to a cluster while the point cloud data from the other one of the RADAR sensors 102 does not, the point cloud data from the LiDAR sensor 110 and / or a ToF sensor 112 is checked to determine whether there are outliers next to the corresponding cluster. In response to determining that the point cloud data from the LiDAR sensor 110 and / or a ToF sensor 112 does not include such outliers, then the outliers can be determined to relate to noise and removed from the combined point cloud. Similarly, in implementations where the sensor module 100 additionally comprises one or more imaging sensors such as visual imaging sensor 114 or IR thermal imaging sensor 116, the point cloud data collected by one or both RADAR sensors 102 can be compared with the image data captured by the one or more imaging sensors to further verify the validity of the point cloud data. For example, if the image data does not include any data that corresponds with the discrepancies in the point cloud data between the RADAR sensors 102, the point cloud data can be determined to be invalid. In another example, if the image data does include data that corresponds with the discrepancies in the point cloud data between the RADAR sensors 102, forexample, data indicative of an object, the point cloud data can be determined to be valid and retained in the combined point cloud.

[0136] As shown in Figure 12B, the method 1200 additionally or optionally comprises a buffering step 1216 of placing the combined point cloud into the buffer 1218. The buffer 1218 can be implemented within the memory 106 or the central memory 806. The buffer depth of the buffer is a configuration parameter defining the number of data samples that can be stored in the data buffer at any point in time. In some implementations, the buffer is configured to have a buffer depth between one to six. In an example implementation, the accumulation buffer is configured to have a buffer depth of six samples. The buffer can be a first-in-first-out (FIFO) buffer so that when a new sample is obtained, the oldest sample in the buffer is disregarded (i.e., removed from the buffer) and the new data sample is added to the buffer.

[0137] Returning to Figure 12 A, the method 1200 proceeds from step 1204 to a transforming step 1206 that transforms the combined point cloud onto a global frame of reference, such as the frame of the vehicle. The same matrix multiplication as discussed above can be used to transform the combined point cloud onto the global frame of reference. For example, the combined point cloud can be transformed from the frame of the sensor module 100 defined relative to the centre of the module 100 onto the frame of the vehicle defined relative to the centre of the vehicle where the vehicle is assumed to be stationary, based on the position and pose of the sensor module 100 relative to the vehicle. The relative position and pose of the sensor module 100 can be obtained by the same calibration process as discussed above with respect to method 700 and stored into the non-volatile calibration memory 1216 implemented within the memory 106 or a storage space additionally included in or accessible to the spatial perception sensor module 100. The transforming step 1206 thereby remaps the data points from the combined 3 -dimensional point cloud in the global frame of reference. The transformed point cloud 1221 is then transmitted to the central processing unit 802 via network 804 as shown in Figure 12B. Where the method 1200 comprises the buffering step 1216, the entire set of the combined point clouds buffed in the buffer 1218 is transformed onto the global frame of reference in each transformation. The transformation onto the global point cloud allows data from each of the at least one sensor module 100 to be positioned relative to the vehicle and to each other.

[0138] The method 1200 proceeds from step 1206 to an identifying step 1208 which identifies objects in the received point cloud 1221 using a clustering algorithm. In implementations where the spatial perception system 800 comprises a single sensor module 100, the point cloud received from the single sensor module 100 provides a global point cloud of the space in which the vehicle is travelling relative to the vehicle. In implementations where spatial perception system 800 comprises a plurality of sensor modules 100 as in the example of Figures 12A and 12B, the identifying step 1208 additionally or optionally comprises a data concatenation substep 1222 of concatenating the point clouds received from the plurality of sensor modules 100 into a concatenated point cloud. The concatenated point clouds form a global point cloud, providing a map with increased object visibility coverage.

[0139] Referring to Figure 12B, the identification of objects at step 1208 comprises a cluster identification process 1224 of identifying clusters of points in the global point cloud as objects and an object identification process 1226 of determining whether clusters that are identified as objects have been identified as objects previously. By increasing the richness and sensordiversity of point cloud data, for example, by increasing the number of RADAR, LiDAR, or ToF sensors and including more sensor types in each spatial perception sensor module 100, clusters can be more readily identified.

[0140] In some implementations, the cluster identification process 1224 uses a noise reducing clustering algorithm to further reduce noise in the point cloud data and to identify clusters of data as objects. Such noise reducing clustering algorithms include, but not limited to, Density- Based Spatial Clustering of Applications with Noise (DBSCAN), Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN), Ordering Points To Identify the Clustering Structure (OPTICS), or the like.

[0141] When a cluster has been identified as an object, the size, corresponding point cloud data, and the position of the cluster are stored in an object detection database 1228. The object detection data base 1228 can be implemented within the central memory 806 as a table where data, including size, point cloud data, and position, related to the objects identified in the present sample of the global point cloud is stored.

[0142] In some implementations, the cluster identification process 1224 comprises defining a prism in the central memory 806 for representing each identified object in the global point cloudto facilitate further processing of identified object, such as object tracking and collision prediction described below. The prism is of dimensions that enclose all of the points in a cluster detected as an object. The prism can be of a rectangular shape. Data, such as position and dimensions, associated with the prism defined for the identified object can be stored in the object detection data base 1228. In some implementations, the prism can be optionally displayed in a visual representation of the global point cloud.

[0143] In the object identification process 1226, objects identified in the present sample of global point cloud are compared with objects identified in one or more previous samples of the global point cloud to determine whether clusters identified as the objects have been identified as objects previously. Data related to objects identified in previous samples is stored in an object tracking database 1230. The object tracking database 1230 can be implemented within the central memory 806 or in a storage space additionally included in or accessible to the spatial perception system 800.

[0144] In some implementations, the number of previous samples for comparison is configurable between 2 to 10. In some implementations, the number of previous samples for comparison is 4. The number of previous samples for comparison can be increased to provide a more reliable tracking of objects. In some implementations, the comparison between the present sample and previous samples of the global point cloud is carried out independently for each identified object.

[0145] The comparison in the object identification process 1226 comprises determining whether an object identified in the present sample of the global point cloud overlaps with any object identified previously or a potion thereof. If the object of the present sample does not overlap any object previously identified or a portion thereof, the object is determined to be a new object and assigned a unique object identification (ID) number. The object identification number can be an integer. The prism representing the object can be tagged (for example, labelled) with the assigned object identification number to facilitate continuous monitoring and tracking of the object over time. The object identification number, and tracking data of the object including the size (as a prism) and position of the object, are then stored in the object tracking database 1230.

[0146] On the other hand, if the object of the present sample overlaps a previously identified object or a portion thereof, it is determined that the object is part of the previously identifiedobject to which an object identification number has been assigned. The tracking data with that object identification number is then updated to include additional tracking data of the data points of, and the prism representing, the object of the present sample. It will be appreciated that after the update, the size data of the prism representing the object includes size data from more than one sample. Accordingly, the prism can appear to be larger than the actual object if the position of the object relative to the vehicle changes between samplings.

[0147] The object tracking database 1230 can store tracking data from a plurality of present and previous samplings with respect to the same object so that objects can be identified and tracked over time. For example, the tracking data of an object stored in the object tracking database 1230 comprises tracking data from a maximum of 5 samplings. As another example, the tracking data of an object stored in the object tracking database 1230 comprises tracking data from a maximum of 10 samplings. The tracking data of an object stored in the object tracking database 1030 therefore includes both the historical positions of the object and the position of the object from the most recent sampling.

[0148] Returning to Figure 12A, the method 1200 proceeds from step 1208 to a step 1210 in which a predicted future trajectory of each identified object is determined. In some implementations, a recursive algorithm for estimating the state of a system such as Kalman filter or extended Kalman filter is used to estimate object velocity (speed and direction) and predict a future position of the object based on historical positions of the object identified in step 1208 stored in object tracking database 1230. The prediction is based on, for example, received positions of the centre-most point(s) of the points representing the object. The predicted future position can then be used to calculate a velocity vector, including speed and direction, of the object. Based on the determined velocity and the last detected position of the object, the predicted future trajectory of the identified object can be determined assuming that the object maintains the calculated velocity.

[0149] In some implementations, the step 1210 additionally or optionally comprises defining a new prism to represent the identified object at the predicted future location.

[0150] In some implementations, the method 1200 additionally or optionally comprises an object classification step (not shown) of classifying the identified object based on the image data and / or thermal image data overlaid with the combined point cloud, using any suitable objectclassification technique. Examples of object classification algorithms that can be used on 2- dimensional data sets from sensors such as the vision imaging sensor 114 or the thermal imaging sensor 116 include, but are not limited to, You Only Look Once (YOLO) and Convolutional Neural Network (CNN). Such algorithms can be adapted to recognise (e.g., classify) common objects of interest that are expected to be found around the vehicle. When aligned and combined with object detection based on 3 -dimensional point-cloud data from sensors such as the RADAR sensors 102, LiDAR sensor 110, and / or ToF sensor 112, the object classification step provides additional confidence of object tracking. The object classification step is also useful in recording the types of objects that were detected.

[0151] As seen from Figure 12B, the step 1210 is effectively a two-stage process to detect and track objects that are present in the field of view of any spatial perception sensor module 100 installed on the vehicle.

[0152] The first stage primarily takes place at each spatial perception sensor module 100 and includes steps 1202, 1204, and 1206 and optionally includes one or more of processes or steps 1214 and 1216. In the first stage, sensing data is collected independently and locally at each spatial perception sensor module 100 and transformed onto a global frame of reference.

[0153] The second stage primarily takes place at the central processing unit 802 and includes steps 1208 and 1210, and optionally includes step 1222. In the second stage, data collected from each spatial perception sensor module 100 is received at the central processing unit 802 and processed to build a map of objects and the relative movement thereof around the vehicle.

[0154] In some implementations, the second stage begins when the central processing unit 802 receives the next set of combined point clouds from each spatial perception sensor modules 100.

[0155] The method 1200 ends at step 1210.Example of Detecting a Single Object

[0156] Figures 13 A and 13B show an example scenario where a single object is detected by the spatial perception system 800. While Figures 13 A and 13B are discussed with reference to the vehicle 900 of Figure 9, it will however be appreciated that the spatial perception system 800 can be installed on other vehicles travelling underground and perform similar functionalities.

[0157] As shown in Figure 13A, a person 1301 is standing in front of the vehicle 900.

[0158] Turning to Figure 13B, operation of the spatial perception system 800 has identified the person 1301 as an object and defined a rectangular prism 1302 representing the person 1301, using the method 1000. Specifically, at step 1208 of the method 1200, the collection (cluster) of points from the front facing surface of the person 1301 is identified by the clustering algorithm. The central processing unit 802 then uses the rectangular prism 1302 encompassing the collection of points to represent the identified person 1301. The object is assigned an object identification number (ID) “1” and the prism 1302 is labelled with the assigned ID. The object ID, and associated data for the person 1301 including the size, corresponding point cloud data, and the position of the collection of points, is stored and maintained in the object tracking database 1230.Example of Detecting Multiple Objects

[0159] Figures 14A and 14B show another example where two objects are detected by the spatial perception system 800. While Figures 14A and 14B are discussed with reference to the vehicle 900 of Figure 9, it will be appreciated that the spatial perception system 800 can be installed on other vehicles travelling underground and perform similar functionalities.

[0160] As shown in Figure 14 A, two people are standing, at different locations, in front of the vehicle 900. The first person 1401 is closer to the vehicle 900 and the second person 1402 is further away from the vehicle 900 compared to the first person 1401.

[0161] Turning to Figure 14B, operation of the spatial perception system 800 has identified the first person 1401 and the second person 1402 as objects at step 1208 of the method 1200. The first person 1401 is represented as an object by a rectangular prism 1403 and assigned a unique object identification number “1”. The second person 1402 is represented as an object by a rectangular prism 1405 and assigned a unique object identification number “2”. The object ID, and associated data for the person 1401 including the size, corresponding point cloud data, and the position of the collection of points) of each of the identified object 1401 and 1402 is stored and maintained in the object tracking database 1230.Tracking Object Trajectory and Predicting Future Trajectory

[0162] A further application of the spatial perception system 800 is object tracking in which the spatial perception system 800 continuously locating objects in the vicinity of the vehicle on which the spatial perception system 800 installed whilst tracking object movement and predicting future object trajectory.

[0163] Figures. 15A-15D show an example of tracking an identified object walking in front of a stationary vehicle over a timeline and estimating a future trajectory of the object using the method 1200.

[0164] Referring to Figure 15 A, a person 1501 is at a first location in front and to the far left of the vehicle 900 at time instance to+0. Operation of the spatial perception system 800 identifies the person 1501 at the first location as an object at step 1208. Additionally or optionally, operation of the spatial perception system 800 represents the person 1501 by a prism 1503 at a corresponding first location in the global point cloud and labelled with object ID “1” at step 1208.

[0165] Turning to Figure 15B, the person 1501 moves to a second location after a time interval ti. Operation of the spatial perception system 800 identifies the person 1501 who has moved to the second location at time instance to+ti and determines that the person 1501 has been identified previously at step 1208. Additionally or optionally, operation of the spatial perception system 800 represents the person 1501 by a prism 1505 at a corresponding second location in the global point cloud and labelled with object ID “1” at step 1208.

[0166] Turning to Figure 15C, the person 1501 moves to a third location after another time interval t2. Operation of the spatial perception system 800 identifies the person 1501 who has moved to the third location at time instance to+ti +t2 and determines that the person 1501 has been identified previously, at step 1208. Additionally or optionally, operation of the spatial perception system 800 represents the person 1501 by a prism 1507 at a corresponding third location in the global point cloud and labelled with object ID “1” at step 1208.

[0167] Turning to Figure 15D, operation of the spatial perception system 800 predicts that the person 1501 will move to a fourth, future location after yet another time interval tj based on the historical locations of the person 1501, including the first, second, and third locations, using an extended Kalman filter algorithm at step 1210. The change in location over time instances can be used to determine velocity vectors of the object(s) detected. Based on the velocity vectors, aset of trajectory lines 1511, 1513, 1515, 1517 that predict where the object will arrive in the next interval of time are determined. The trajectory lines represent the predicted path the object will take to arrive at the predicted future location. The projected trajectory lines extending from each of the vertices of the prism 1507 provide a predicted future trajectory of the object 1501. The location to which the projected trajectory lines extend provides a predicted future location (i.e., the fourth location in the example of Figures 15A-15D) that the vehicle will arrive at a time instance to + ti+ t2 + C Based on the predicted future location, operation of the spatial perception system 800 represents the person 1501 by a prism 1509 (displayed in dotted lines in Figure 15D) at a corresponding location in the global point cloud and labelled with object ID “1” at step 1210.

[0168] It will be appreciated that a vehicle in use will often be in motion relative to the ground over which the vehicle travels. According to the present invention, processes such as object detection and tracking and future trajectory prediction are based on a global frame of reference where the vehicle is assumed to be stationary. In particular, the spatial perception system 800 allows objects to be detected relative to the global frame of reference as if the vehicle were stationary and only external objects were moving, thereby simplifying object tracking and anticollision management which will be discussed below.Scenarios of Collision

[0169] Figures. 16A to 16D illustrate example scenarios where the vehicle 900 of Figure 9 collides with an object 1601. Two scenarios are presented. In the example scenario of Figures 16A and 16B, both the vehicle 900 and the object 1610 are in motion with respect to the ground. In the example scenario of Figure 16C and Figure 16D, only the object 1610 is moving in relation to the ground while the vehicle 900 is stationary.

[0170] In both example scenarios, the vehicle 900 and object 1610 start at the same positions at time instance to. As can be seen from Figure 16A, the vehicle 900 is moving due East at a speed of 1 m / s and the object is moving due North at a speed of 1 m / s. As can be seen from Figure 16C, the vehicle is stationary and the object is moving North-West at a speed of 1.41 m / s.

[0171] Figure 16C and Figure 16D show the locations of vehicle 900 and object 1610 at time instance to+ti. In both example scenarios, the object 1601 arrives at a location where the object1601 collides with the vehicle 900, despite the vehicle not moving at all in the example scenario of Figures 16C and 16D.Collision Prevention

[0172] A further application of the spatial perception system 800 is to predict collisions between the vehicle and other objects in the environment.Configuration of Safety Zones

[0173] To predict collisions between a vehicle and objects identified using the method 1200 described above, a plurality of safety zones (or collision prediction zones) can be defined and the spatial perception system 800 is configured to continuously monitor whether the predicted future trajectory of any identified object intersects with any safety zone. The shape of each collision prediction zone can be a polygon such as a rectangle, or a trapezoid where the longer of the parallel edges being further away from the front of the vehicle compared to the shorter of the parallel edges. The plurality of safety zones can comprise at least one danger zone and at least one safe zone. The at least one danger zone can be further divided into a number of subzones each associated with a different severity of danger if an object is predicted to travel along a trajectory that intersects that sub-zone. The spatial perception system 800 is configured to allow customization of safety zones to suit different types of vehicles, cruising speeds and tunnel applications.

[0174] Figure 17 shows an example configuration of safety zones for vehicle 900 of Figure 9. Two main safety zones have been defined. Bounded by line 1700 is a danger zone 1701. The zone outside the danger zone 1701 is a SAFE zone 1703 where the object is at a safe distance from the vehicle. The danger zone 1701 comprises three sub-zones, namely ALARM zone 1705, SLOW zone 1507, and STOP zone 1709, bounded by line 1711, 1713, and 1715, respectively. The STOP zone 1709 is immediately around vehicle 900 and of the highest severity of danger where the object is extremely close to the vehicle. The SLOW zone 1707 surrounds the STOP zone 1709 and is of moderate severity of danger where the object is very close to the vehicle. The ALARM zone 1705 surrounds subzone 1707 and is of the least severity of danger where the object is at a close distance from the vehicle. Each of the safety zones in the example of Figure 17 has the shape of a rounded rectangular.

[0175] The size and shape of the collision prediction zones can be dynamically configured based on both the velocity of the vehicle and / or the predicted future trajectory. For example, when the vehicle is travelling in a straight line the zones are rectangles of increasing size, similar to the configuration shown in Figure 17. When the vehicle is making a turn to the left, the shape of the zones will change to a trapezoid where the left edge is angled to the left. When the vehicle is then making a turn to the right, the shape of the trapezoid will change so that the right edge angles to the right. By dynamically changing the shape of the zones in this way, collision prediction, described below, is cognisant of the expected future location of the vehicle. The length of the zones in the direction of travelling can be increased or decreased with respect to the vehicle speed.Collision Prediction

[0176] Using the method 1200 as discussed above, the spatial perception system 800 installed on a vehicle can identify an object nearby and predict a future trajectory of the identified object at regular time intervals, for example, at each sampling point. To predict collisions with the object, the spatial perception system 800 is configured to continuously check whether the predicted future trajectory of the object in the next time interval intersects with any of the safety zones. Based on the result of the checking, the spatial perception system 800 can predict whether the vehicle will collide with the object. For example, the spatial perception system 800 predicts that collision with the object will occur if at least one of the trajectory lines of the predicted future trajectory intersects the danger zone 1701. As another example, the spatial perception system 800 predicts that collision with the object will not occur if none of the trajectory lines of the predicted future trajectory intersects the danger zone 1701. Table 1 below provides an example of the relationship between each safety zone and the corresponding prediction.Table 1Collision Avoidance

[0177] A further application of the spatial perception system 800 is to avoid collisions between the vehicle and other objects in the environment. If the result of the checking indicates that the vehicle will collide with the object, the spatial perception system 800 determines a recommended action to be taken to prevent collision. The determination of the recommended action can be based on the safety zone with which the predicted trajectory of the object intersects. Table 2 below provides an example of the relationship between each safety zone and the corresponding prediction and action that can be taken to avoid collisions.Table 2

[0178] Table 1 and table 2 can be stored in the central memory 806 of the spatial perception system 800.

[0179] With the configuration of recommended actions shown in Table 1, an audible and / or visual warning can be provided. For example, when collision is predicted to occur in one of the safety zones 1705, 1707, and 1709, an audible and / or visual indicator on the vehicle can be activated. A human operator of the vehicle can then respond to the warning signal so as to prevent a collision.

[0180] The spatial perception system 800 can additionally or alternatively provide recommendations of a suitable traveling speed in response to the prediction. For example, when collision is predicted to occur in the SLOW zone 1707, the recommended action is to slow down the vehicle, for example, by recommending a reduced speed that is between 0 and a maximum permitted vehicle speed.

[0181] The spatial perception system 800 can additionally or alternatively provide a recommendation to stop the vehicle. For example, when collision is predicted to occur in the STOP zone 1509, the recommended action is to stop the vehicle, for example, by recommending that the traveling speed be reduced to zero and braking be applied.

[0182] In a tunnel environment, the traveling speed of the vehicle is typically not to exceed the maximum permitted vehicle speed. The maximum permitted vehicle speed is independent of the travelling speed recommended by spatial perception system 800, but rather a limit that is typically configured by an operator or an autonomous algorithm of the vehicle to limit the speed to a level that is safe and practical. The operator or the autonomous algorithm can drive the vehicle up to the configured limit, but the vehicle motion control system 810 can reduce the speed, or stop the vehicle altogether, based on the recommendations received from the spatial perception system 800 in response to a collision being predicted. In some implementations, if the detected object is in a safe zone such as the SAFE zone 1703 or a zone that will not present immediate danger such as the ALARM zone 1705, the vehicle is allowed to attain the maximum permitted vehicle speed.

[0183] Collision avoidance can additionally or alternatively be based on detection of the object within one of the safety zones / sub-zones. Depending on the safety zone / sub-zone in which the object is detected a different response can be made. For example, warnings can be given if the detected object is within the ALARM zone 1705 which is a sub-zone of the danger zone 1701 but will not present immediate danger. For example, an audio and / or visual warning signal can be generated and emitted to alert the operator or the remote control of the vehicle to take precautionary actions such as decelerate or stop the vehicle. As another example, if the detected object is within the SLOW zone 1707 or STOP zone 1709 of the danger zone 1701, warnings can be given. The spatial perception system 800 can additionally determine an action that is recommended to be applied by the vehicle 900 to decelerate or stop the vehicle and provide the determined action to the vehicle motion control system 810. Based on the recommended action received from the spatial perception system 800, the vehicle motion control system 810 can control the speed and trajectory of the vehicle to avoid collisions.Collision Avoidance System

[0184] Figure 18 shows a collision avoidance system 1800 in which collision avoidance can be realised. The collision avoidance system 1800 comprises the spatial perception system 800 of Figure 8, the vehicle motion control system 810 of Figure 8, and an example vehicle drivetrain 1802. The spatial perception system 800 maintains data communication with the motion control system 810 of the vehicle to exchange data. The control system 810 is coupled to the vehicle drivetrain 1804 to implement motion controls via manipulation of any combination of multi-speed transmission, vehicle traction, brakes, and steering. The control system 810 can be a programmable driving system by which the ability to modulate the vehicle’s movement can be configured.

[0185] The drivetrain 1804 comprises actuators and motors coupled to the multi-speed transmission system 1806, traction system 1808, brake system 1810, and steering system 1812. The actuators and motors can be electric, hydraulic, or pneumatic.

[0186] The spatial perception system 800 is configured to transmit data indicative of a recommended action to the control system 810 so that the control system 810 can implement control based on the recommended action to modulate the vehicle’s movement, thereby preventing the predicted collision from arising. Such a configuration allows the ability to modulate the vehicle’s movement to be configured to achieve safe outcomes where potential collisions are detected and avoided.

[0187] The architecture shown in Figure 18 is based on the spatial perception system 800 predicting possible collisions and communicating recommendations such as vehicle speed and trajectory changes to the vehicle motion control system 810. The recommendations can then be implemented by the vehicle motion control system 810.

[0188] When detection of an object is associated with a safety decision such as recommending an action in response to detection of collision, the safety decision is configured to default to a safe outcome upon failure of one or both of the RADAR sensors 102, for example, when data collected therefrom is not aligned sufficiently with the other one of the RADAR sensors.

[0189] Referring now to Figure 19, a collision avoidance method 1900 implemented by the collision avoidance system 1800 is illustrated. The method 1700 may be implemented by one or more sensor modules 100 and execution of software modules stored as instructions in memory 106 and central memory 806 and controlled by execution of the software modules by processor 104 and central processing unit 802. The method 1800 includes steps 1802 to 1810 that are identical to steps 1202 to 1210 of the method 1200 of Figures 12 A and 12B.

[0190] The method 1900 proceeds from step 1910 to a step 1912 that determines whether the vehicle will collide with the identified object based on whether the predicted future trajectory ofthe identified object intersects with any of predefined safety zones, for example as shown in relation to Figure 17 and Figure 20 described below.

[0191] The method proceeds from step 1912 to step 1914 that generates a recommendation for a collision avoidance action in response to determining that the vehicle will collide with the identified object. The recommendation can be generated in manners described above. The recommendation is then transmitted to the vehicle motion control system 810. The method ends at step 1914.

[0192] One or more controls can then be applied by the vehicle motion control system 810 to implement the received recommendation.

[0193] Figure 20 provides an example of an object being tracked by the collision avoidance system 1800 that is predicted to be travelling into the ALARM zone 1705 as defined in Figure 17 for vehicle 900 of Figure 9 in the next time interval.

[0194] As shown in Figure 20, an object 2001 being tracked by the spatial perception system 800 is located in the SAFE zone 1503, meaning there is no collision predicted and no action such as recommending changes to vehicle motion will be taken.

[0195] However, operation of the spatial perception system 800 of the collision avoidance system 1800 predicts, at step 1912 of method 1900, that the vehicle 900 will collide with the object 2001 travelling into the ALARM zone 1505 in the next time interval. The prediction is based on determining, at step 1912 of method 1900, that projected trajectory lines shown as 2003, 2005, 2007, and 2009 all intersect the ALARM zone 1505 extending to a prism 2011 at a location within the ALARM zone 1505. According to the action defined for the ALARM zone 1505 in Table 1, the spatial perception system 800 generates a warning signal causing an audio and / or visual alarm to be emitted through, for example, an audio warning system or a visual indicator of the vehicle. The alarm can alert the operator of the vehicle or the object, if the object is a person, to take precautions to avoid the predicted collision.Industrial Applicability

[0196] The arrangements described are applicable to the mining industries and / or civil construction industries and particularly to underground tunnel network where collision avoidance is salient to worker safety and asset protection.

[0197] The arrangements described utilize a combination of different types of sensing technologies to overcome the lack of availability of GPS signals in an underground setting. Use of different types of sensing to detecting objects via radio signals and visually increases the likelihood that a collision can be avoided in an underground environment.

[0198] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0199] In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of’. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings.

Claims

CLAIMS1. A sensor module for a vehicle operating underground, the sensor module comprising: at least two RADAR sensors, each RADAR sensor configured to generate a point cloud of a surrounding space of the vehicle; and a processing unit and a memory storing instructions, which when executed by the processing unit, cause the processing unit to combine the point cloud generated by each of the RADAR sensors into a combined point cloud and transform the combined point cloud onto a frame of reference relative to the vehicle based on a position and a pose of the sensor module relative to the vehicle.

2. The sensor module of claim 1, the two RADAR sensors are identical.

3. The sensor module of claim 1 or 2, further comprising at least one of a LiDAR sensor, a time of flight (ToF) sensor, and an imaging sensor.

4. The sensor module of claim 3, wherein the imaging sensor comprises at least one of a visual imaging sensor and an Infra-Red (IR) thermal imaging sensor.

5. The sensor module of any one of claims 1-4, wherein the frame of reference is a frame of the vehicle.

6. The sensor module of any one of claims 1-5, wherein the instructions comprise instructions to apply Cell Averaging Constant False Alarm Rate (CACFAR) algorithm to each of the generated point clouds.

7. The sensor module of any one of claims 1-6, wherein the point clouds generated by the two RADAR sensors are combined by overlaying the point cloud generated by one of the two RADAR sensors with the point cloud generated by the other one of the two RADAR sensors.

8. The sensor module of any one of claims 1-7, wherein the point cloud generated by one of the two RADAR sensors is compared with the point cloud generated by the other one of the two RADAR sensors.

9. The sensor module of claim 8, wherein in response to a comparison result that the point cloud generated by one of the two RADAR sensors differs from the point cloud generated by the other one of the two RADAR sensors, point cloud data associated with identified differences is removed from one or both of the point clouds.

10. A spatial perception system for a vehicle operating underground, the system comprising: at least one sensor module configured for installation on the vehicle, the at least one sensor module comprising: at least two RADAR sensors, each RADAR sensor configured to generate a point cloud of a surrounding space of the vehicle; and a processing unit and a memory storing instructions, which when executed by the processing unit, cause the processing unit to combine the point cloud generated by each of the RADAR sensors into a combined point cloud and transform the combined point cloud onto a frame of reference relative to the vehicle based on a position and a pose of the at least one sensor module relative to the vehicle; a central processing unit connected to the at least one sensor module via a network; and a central memory connected to the central processing unit and storing instructions, which when executed by the central processing unit, cause the central processing unit to: receive the transformed point cloud from the at least one sensor module via the network; identify an object in the received point cloud; and determine a predicted future trajectory of the identified object.

11. The spatial perception system of claim 10, the two RADAR sensors are identical.

12. The spatial perception system of claim 10 or 11, wherein the at least one sensor module comprises at least one of a LiDAR sensor, a time of flight (ToF) sensor, and an imaging sensor.

13. The spatial perception system of claim 12, wherein the imaging sensor comprises at least one of a visual imaging sensor and an Infra-Red (IR) thermal imaging sensor.

14. The spatial perception system of any one of claims 10-13, wherein the frame of reference is a frame of the vehicle.

15. The spatial perception system of any one of claims 10-14, wherein the instructions comprise instructions to apply Cell Averaging Constant False Alarm Rate (CACFAR) algorithm to each of the generated point clouds.

16. The spatial perception system of any one of claims 10-15, wherein the point clouds generated by the two RADAR sensors are combined by overlaying the point cloud generated by one of the two RADAR sensors with the point cloud generated by the other one of the two RADAR sensors.

17. The spatial perception system of any one of claims 10-16, wherein the point cloud generated by one of the two RADAR sensors is compared with the point cloud generated by the other one of the two RADAR sensors.

18. The spatial perception system of claim 17, wherein in response to a comparison result that the point cloud generated by one of the two RADAR sensors differs from the point cloud generated by the other one of the two RADAR sensors, point cloud data associated with identified differences is removed from one or both of the point clouds.

19. The spatial perception system of claim 18, wherein the instructions stored on the central memory, when executed by the central processing unit, further cause the central processing unit to: determine whether the vehicle will collide with the identified object; and generate a recommendation for a collision avoidance action in response to predicting that the vehicle will collide with the identified object.

20. A method of detecting objects surrounding a vehicle operating underground, the method comprising: generating a point cloud, by each of at least two RADAR sensors, of a surrounding space of the vehicle;combining the point cloud generated by each of the at least two RADAR sensors into a combined point cloud; transforming the combined point cloud onto a global frame of reference relative to the vehicle based on position and pose of the spatial perception sensor module relative to the vehicle; identifying an object in the transformed point cloud; and determining a predicted future trajectory of the identified object.

21. The method of claim 20, further comprising determining whether the vehicle will collide with the identified object.

22. The method of claim 21, further comprising generating a recommendation for a collision avoidance action in response to predicting that the vehicle will collide with the identified object.

Images