Multi-modal sensor positioning system

Abstract

Methods and systems are provided that include a first sensor of a platform, a second sensor, and a processor. The first sensor has a first modality and is configured to obtain first sensor data regarding a target relative to the platform. The second sensor has a second modality different from the first modality, is configured to obtain second sensor data regarding the target relative to the platform, and is used to detect the target from a short distance with a relatively higher accuracy than the first sensor of the first modality. The processor is configured to locate the target using the first sensor data and the second sensor data, where the second sensor data is used to enhance the first sensor data based on a validity of the first sensor data, and the use of the first sensor data and the second sensor data is based on a distance of the target from the platform.

Description

TECHNICAL FIELD

[0001] The technical field is generally related to platforms such as vehicles, and more specifically to methods and systems for localizing a target using multiple different types of modalities of sensors. BACKGROUND

[0002] Many vehicles and other platforms utilize sensors, such as ultra-wideband sensors, to localize a target. However, in certain situations, such technology can not always be optimal.

[0003] Accordingly, it is desirable to provide improved methods and systems for localizing a target proximate to a platform, such as a vehicle, using different modalities of sensors. Additionally, other desirable features and characteristics of the present disclosure will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background. SUMMARY

[0004] According to embodiments, a method is provided that includes obtaining, via one or more first sensors of a platform, first sensor data regarding a target relative to the platform, the one or more first sensors having a first modality; obtaining, via one or more second sensors of the platform, second sensor data regarding the target, the one or more second sensors having a second modality different from the first modality; and localizing, via a processor of the platform, the target using the first sensor data and the second sensor data, wherein the second sensor data is used to augment the first sensor data based on a validity of the first sensor data, and the using of the first sensor data and the second sensor data is based on a distance of the target from the platform; wherein the one or more first sensors of the first modality are configured for detecting the target at a relatively greater distance from the platform than the one or more second sensors of the second modality; and the one or more second sensors of the second modality are configured for detecting the target from a short distance with a relatively higher accuracy than the one or more first sensors of the first modality.

[0005] Further in embodiments, the platform comprises a vehicle, and the processor is further configured to at least facilitate performing a control action regarding movement of the vehicle based on the localization of the target relative to the vehicle.

[0006] Further in embodiments, the one or more second sensors of the second modality comprise ultra-wideband (UWB) sensors.

[0007] Further in embodiments, the one or more first sensors of the first modality comprise RSSI sensors.

[0008] Further in embodiments, the second sensor data is used via the processor to identify a plurality of candidate locations for the target; and the first sensor data is used via the processor to select a preferred candidate of the plurality of candidate locations based on a pattern match for the second sensor data using a path loss model.

[0009] Further in embodiments, the positioning is performed via the processor by using only the first sensor data without using the second sensor data when the target is greater than a first predetermined distance from the platform; and using both the first sensor data and the second sensor data when the target is less than the first predetermined distance from the platform.

[0010] Further in embodiments, the positioning is performed via the processor by using first sensor data from a single first sensor of the first modality when the target is greater than a first predetermined distance from the platform and also greater than a second predetermined distance from the platform, where the second predetermined distance is greater than the first predetermined distance; and using first sensor data from a plurality of sensors of the first modality when the target is greater than the first predetermined distance from the platform and less than the second predetermined distance from the platform.

[0011] Further in embodiments, the positioning is performed via the processor by using first sensor data from a plurality of first sensors of the first modality and second sensor data from a single second sensor of the second modality when the target is less than the first predetermined distance from the platform but greater than a third predetermined distance from the platform, where the third predetermined distance is less than the first predetermined distance; and using first sensor data from a plurality of first sensors of the first modality and second sensor data from a plurality of second sensors of the second modality when the target is less than the first predetermined distance from the platform and less than the third predetermined distance from the platform.

[0012] Further in embodiments, the positioning is performed via the processor by using first sensor data from a plurality of first sensors of the first modality and second sensor data from two second sensors of the second modality when the target is less than a third predetermined distance from the platform but greater than a fourth predetermined distance from the platform, where the fourth predetermined distance is less than the third predetermined distance; and using first sensor data from a plurality of first sensors of the first modality and second sensor data from at least three second sensors of the second modality when the target is less than the fourth predetermined distance from the platform by applying triangulation with respect to the second sensor data from the at least three second sensors.

[0013] In another embodiment, a system is provided that includes: one or more first sensors of a platform, the one or more first sensors having a first modality and configured to obtain first sensor data about a target relative to the platform; one or more second sensors of the platform, the one or more second sensors having a second modality different from the first modality and configured to obtain second sensor data about the target relative to the platform, and wherein the second sensors of the second modality are configured for detecting the target from a short distance with a relatively higher accuracy compared to the first sensors of the first modality; and a processor of the platform coupled to the one or more first sensors and the one or more second sensors and configured to facilitate at least positioning of the target using the first sensor data and the second sensor data, wherein the second sensor data is used to augment the first sensor data based on a validity of the first sensor data, and the use of the first sensor data and the second sensor data is based on a distance of the target from the platform.

[0014] Further in an embodiment, the platform comprises a vehicle, and the processor is further configured to facilitate at least performing a control action regarding movement of the vehicle based on the positioning of the target relative to the vehicle.

[0015] Further in an embodiment, the one or more second sensors of the second modality comprise ultra-wideband (UWB) sensors.

[0016] Further in an embodiment, the one or more first sensors of the first modality comprise RSSI sensors.

[0017] Further in an embodiment, the second sensor data is used via the processor to identify a plurality of candidate locations for the target; and the first sensor data is used via the processor to select a preferred candidate among the plurality of candidate locations based on a pattern matching for the second sensor data using a path loss model.

[0018] Further in an embodiment, the positioning is performed via the processor by: using only the first sensor data without the second sensor data when the target is greater than a first predetermined distance from the platform; and using both the first sensor data and the second sensor data when the target is less than the first predetermined distance from the platform.

[0019] Further in an embodiment, the positioning is performed via the processor by: using first sensor data from a single first sensor of the first modality when the target is greater than a first predetermined distance from the platform and also greater than a second predetermined distance from the platform, wherein the second predetermined distance is greater than the first predetermined distance; and using first sensor data from a plurality of first sensors of the first modality when the target is greater than the first predetermined distance from the platform and less than the second predetermined distance from the platform.

[0020] Further in embodiments, the locating is performed via the processor by: using first sensor data from a plurality of first sensors of the first modality and second sensor data from a single second sensor of the second modality when the target is less than a first predetermined distance from the platform but greater than a third predetermined distance from the platform, where the third predetermined distance is less than the first predetermined distance; and using first sensor data from a plurality of first sensors of the first modality and second sensor data from a plurality of second sensors of the second modality when the target is less than the first predetermined distance from the platform and less than the third predetermined distance from the platform.

[0021] Further in embodiments, the locating is performed via the processor by: using first sensor data from a plurality of first sensors of the first modality and second sensor data from two second sensors of the second modality when the target is less than a third predetermined distance from the platform but greater than a fourth predetermined distance from the platform, where the fourth predetermined distance is less than the third predetermined distance; and using first sensor data from a plurality of first sensors of the first modality and second sensor data from at least three second sensors of the second modality when the target is less than the fourth predetermined distance from the platform, by applying triangulation with respect to the second sensor data from the at least three second sensors.

[0022] In another embodiment, a system is provided that includes: a body, one or more first sensors, one or more second sensors, and a processor. The one or more first sensors are disposed within the body, have a first modality, and are configured to obtain first sensor data with respect to a target relative to the platform. The one or more second sensors are disposed within the body, have a second modality different from the first modality, and are configured to obtain second sensor data with respect to the target relative to the platform. The second sensors of the second modality are configured for detecting the target from a short distance with a relatively higher accuracy compared to the first sensors of the first modality. The processor is disposed within the body, coupled to the one or more first sensors and the one or more second sensors, and is configured to at least facilitate locating the target using the first sensor data and the second sensor data, where the second sensor data is used to augment the first sensor data based on a validity of the first sensor data, and the using the first sensor data and the second sensor data is based on a distance of the target from the platform.

[0023] Further in embodiments, the platform includes a vehicle, and the processor is further configured to at least facilitate performing a control action with respect to movement of the vehicle based on the locating of the target relative to the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0024] The present disclosure will be described below in conjunction with the following drawings, in which like numbers denote like elements, and in which:

[0025] Figure 1 is a functional block diagram of a platform, such as a vehicle, according to example embodiments, including a control system including systems of different modalities, and a processor configured to localize a target proximate to the platform using sensors of different modalities;

[0026] Figure 2 is a process flow diagram of a process for localizing a target proximate to a platform, such as a vehicle, using sensor data from sensors of different modalities, and can be implemented in conjunction with Figure 1 is a functional block diagram of a platform, such as a vehicle, according to example embodiments, including a control system including systems of different modalities, and a processor configured to localize a target proximate to the platform using sensors of different modalities;

[0027] Figure 3 is an illustration of the various stages utilized in an implementation of the process of Figure 2 is a flow diagram of an example sub-process of the process of

[0028] Figure 4 is a flow diagram of an example sub-process of the process of Figure 2 Figure 3 is a flow diagram of an example sub-process of the process of

[0029] Figure 5 is a flow diagram of another example sub-process of the process of Figure 2 Figure 3 is a flow diagram of another example sub-process of the process of

[0030] Figure 6 , Figure 7 , Figure 8A , Figure 8B , Figure 9A , Figure 9B and Figure 9C depict example implementations of pattern matching associated with steps of the process of Figures 2-5 including the pattern matching steps described in Figure 5 is a flow diagram of an example sub-process of the process of DETAILED DESCRIPTION

[0031] The following detailed description is merely exemplary in nature and is not intended to limit the disclosure or the application and uses of it. Furthermore, there is no intention to be bound by any theory of operation presented in the preceding background or the following detailed description.

[0032] Figure 1 is a functional block diagram of a platform 100 according to example embodiments. As described in further detail below, the vehicle 100 includes a plurality of sensors 116 of different modalities and a controller 102 that uses the sensors 116 of different modalities to provide localization of a target proximate to the platform 100, as well as other components. As described below in conjunction with Figure 1 ​​and Figure 2 The process 200 and Figures 3-9C Further described in more detail with respect to the sub-processes and implementations of

[0033] In various embodiments, the platform 100 comprises a vehicle, and is also referred to herein as a vehicle 100. In various embodiments, the vehicle 100 comprises an automobile, such as any of several different types of automobiles, such as for example a sedan, a van, a truck, a sport utility vehicle (SUV), etc. In certain embodiments, the vehicle 100 can also comprise a motorcycle or other vehicle, such as an aircraft, a spacecraft, a watercraft, etc., and / or one or more other types of mobile platforms (e.g., a robot and / or another mobile platform). Although the term "vehicle" 100 is used throughout this application, it will be understood that in various embodiments the platform 100 can comprise any number of mobile platforms (such as those mentioned above) or non-mobile platforms (such as for example, one or more mobile telephones or other electronic devices, one or more buildings, other structures, other devices or systems, etc.).

[0034] In the depicted embodiment, the sensors 116 comprise a first sensor type 118 and a second sensor type 119. In various embodiments, the first sensor type 118 can detect targets at relatively greater distances from the vehicle 100 than the second sensor type 119. Conversely, also in various embodiments, the second sensor type 119 has a higher target detection accuracy than the first sensor type 118, at least when the target is very close to the vehicle 100.

[0035] In certain embodiments, the first sensor type 118 comprises a radio signal strength indication (RSSI) sensor, which has a range of approximately thirty to forty meters (30-40 m). Also in certain embodiments, the second sensor type 119 comprises an ultra-wideband (UWB) sensor, which has a range of approximately fifteen meters (15 m). However, this can vary in other embodiments.

[0036] As Figure 1As depicted in the exemplary embodiment, the vehicle 100 also includes a body 104 disposed on a chassis 106. The body 104 substantially surrounds the other components of the vehicle 100. The body 104 and the chassis 106 may together form a frame. As described above, the vehicle 100 also includes a plurality of wheels 112, each wheel being rotatably coupled to the chassis 106 near a corresponding corner of the body 104 to facilitate movement of the vehicle 100. In various embodiments, the vehicle 100 also includes a plurality of doors 110.

[0037] The drive system 114 is mounted on the chassis 106 and drives the wheels 112, for example, via axle 108. In some embodiments, the drive system 114 includes a motor having one or more motors. Figure 1 The propulsion system (not depicted herein, and includes, for example, one or more internal combustion engines, electric motors, etc. in various embodiments) may also include or be coupled to a braking system, a steering system, and / or one or more other systems of the vehicle 100.

[0038] In various embodiments, the vehicle 100 also includes one or more display systems 120. In some embodiments, the one or more display systems 120 provide displays and information to the driver and / or user of the vehicle 100, including displays and information regarding the location of targets adjacent to the vehicle 100 and / or in conjunction with one or more other vehicle control actions taken therein. In various embodiments, the display system 120 can provide audio, visual, tactile, and / or other types of notifications.

[0039] like Figure 1 As depicted, in some embodiments, sensor 116 and controller 102 may be collectively considered, or referred to as control system 101, which controls the positioning of targets adjacent to vehicle 100, and in various other embodiments also controls other aspects of vehicle functionality (e.g., control actions such as propulsion, braking, steering, etc.).

[0040] In addition, such as Figure 1 As depicted, in various embodiments, controller 102 is coupled to sensor 116, as well as drive system 114, display system 120, and other vehicle systems, and performs... Figure 2 The process involves 200 steps and Figures 3-9C The sub-procedures and their implementation are described in further detail below.

[0041] like Figure 1As depicted, in various embodiments, the controller 102 includes a computer system (also referred to herein as the computer system 102) and includes a processor 122, a memory 124, an interface 126, a storage device 128, and a computer bus 130.

[0042] The processor 122 performs the computational and control functions of the controller 102 and can comprise any type of processor or multiple processors, a single integrated circuit such as a microprocessor, or any suitable number of integrated circuit devices and / or circuit boards that cooperate to cause the processing unit to function as described. During operation, the processor 122 executes one or more programs 132 contained in the memory 124 and thus controls the general operation of the controller 102 and the general operation of the computer system of the controller 102 (typically in response to commands from a user and / or instructions from another computer system or device). Figure 2 the process 200 of Figures 3-9C and implementations thereof, and as further described in connection therewith below).

[0043] The memory 124 can be any suitable type of memory, including various types of non-transitory computer-readable storage media. In certain examples, the memory 124 is located and / or co-located on the same computer chip as the processor 122. In the depicted embodiment, the memory 124 stores the programs 132 described above as well as stored values 134 (e.g., lookup tables, threshold values, and / or other values related to the process 200).

[0044] The interface 126 allows communication, for example, from system drives and / or another computer system to the computer system of the controller 102, and can be implemented using any suitable method and apparatus. In one embodiment, the interface 126 obtains various data from the sensors 116 and other possible data sources. The interface 126 can include one or more network interfaces that communicate with other systems or components. The interface 126 can also include one or more network interfaces that communicate with a technician and / or one or more storage interfaces that connect to storage devices such as the storage device 128.

[0045] The storage device 128 can be any suitable type of storage device, including various different types of direct access storage and / or other memory devices. In one example embodiment, the storage device 128 includes a program product from which the memory 124 can receive the programs 132 that perform one or more embodiments of one or more processes of the present disclosure, such as the steps of the process 200 of Figure 2 and implementations thereof, and as further described in connection therewith below). Figures 3-9Cimplementation, and as further described below in connection with FIG. 1. In another example embodiment, the program product can be directly stored in and / or accessed from memory 124 and / or a disk (e.g., disk 136), such as is referenced hereinafter.

[0046] Bus 130 serves to transmit programming, data, status and other information or signals between the various components of computer system of controller 102. Bus 130 can be any suitable physical or logical means for communicating information or signals between computer system components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared and wireless bus techniques, and so forth. During operation, programming 132 is stored in memory 124 and executed by processor 122.

[0047] It will be appreciated that, although this example embodiment is described in the context of a fully functional computer system, those skilled in the art will recognize the mechanism of the present disclosure is capable of being distributed as a program product in one or more types of non-transitory computer-readable signal bearing media having stored thereon the program and instructions and executed by a computer processor, such as processor 122, to perform and run the program.

[0048] Figure 2 is a flowchart of a process 200 in example embodiments for localizing a target proximate to a platform, such as a vehicle, using sensor data from sensors of different modalities. In various embodiments, process 200 can be implemented in conjunction with Figure 1 vehicle 100 (including sensors 116, controller 102 and other components thereof).

[0049] As Figure 2 depicted, in various embodiments, process 200 begins when vehicle 100 is stationary and a target approaches vehicle 100. However, in other embodiments, this can vary.

[0050] As Figure 2 depicted, process 200 utilizes three models, namely: (1) a sensing model 202; (2) a localization model 204; and (3) a prediction model 206, as described in greater detail below.

[0051] In various embodiments, as part of sensing model 202, a first measurement is obtained (step 208). In various embodiments, the first measurement of step 208 includes sensor data from sensors 116 of a first type 118. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on vehicle 100. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100, that is stationary. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100, that is stationary and that is proximate to a target. Figure 1 In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on vehicle 100. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100, that is stationary. In various embodiments, these include sensor data from sensors 116 of a first type 118 that are mounted on a platform, such as vehicle 100, that is stationary and that is proximate to a target. Figure 1The sensor data of the second type 119 sensor 116 is compared with the measurement which has a relatively larger distance but relatively lower accuracy in the short range. In some embodiments, the first measurement in step 208 includes meter-level measurements from multiple RSSI sensors of the vehicle 100. Furthermore, in some embodiments, an initial determination of the location of the target is obtained or determined with respect to the first measurement (step 210) (e.g., initial RSSI location determination).

[0052] Furthermore, in various embodiments, and also as part of sensing model 202, a second measurement is obtained (step 212). In various embodiments, the second measurement in step 212 includes measurements from... Figure 1 The sensor data of the second type 119 sensor 116. In various embodiments, these include data from... Figure 1 The first sensor data of the first type 118 sensor 116 is compared with measurements that have a relatively smaller distance but relatively higher accuracy in the short range. In some embodiments, the second measurement in step 212 includes centimeter-level measurements from multiple ultra-wideband (UWB) sensors from the vehicle 100. Furthermore, in some embodiments, an initial determination of the target's position (step 214) is performed with respect to the second measurement via two-way ranging (e.g., regarding initial UWB position determination).

[0053] In various embodiments, feature extraction is performed (step 216). In various embodiments, Figure 1 The processor 122 performs feature extraction from the first measurement of step 208 and the initial determination of step 210 (e.g., from an RSSI sensor in an exemplary embodiment). In various embodiments, feature extraction involves the detection and initial localization of one or more targets (e.g., one or more people, other vehicles, and / or other objects) adjacent to vehicle 100.

[0054] Furthermore, in various embodiments, real-time training is provided (step 218). In various embodiments, during step 218, real-time training concerns the localization of the target using various sensors 116, feature extraction based on step 216 and the initial determination in step 214, via... Figure 1 It is executed by processor 122.

[0055] Furthermore, in various embodiments, the path loss model is updated (step 220). In various embodiments, during step 220, Figure 1 The processor 122 uses the real-time training of step 218 to update the path loss matters for the sensor 116 (e.g., RSSI sensor) of the first type 118.

[0056] Furthermore, in various embodiments, region detection is performed (step 222). In various embodiments, during step 222, if the target is at least a predetermined distance from the vehicle 100, the processor 122 performs region detection for the target detected only by the sensor 116 of the first type 118. (See below for further details.) Figure 3 In a further, more detailed description, in various embodiments, this corresponds to where Figure 3 The target 306 is far enough from the vehicle 100 that the current situation falls within either Phase 1 (301) or Phase 2 (302) scenarios, such as... Figure 3 As described below and in further, more detailed description thereof. Furthermore, in various embodiments, one or more vehicle control actions are subsequently provided to the vehicle based on the location of the target, such as (A) via instructions provided by processor 122. Figure 1 The display system 120 provides one or more notifications (e.g., one or more audio, visual and / or tactile notifications) to the driver or other passengers of the vehicle 100; and / or (B) executes one or more vehicle movement commands for adjusting motor / propulsion torque, braking torque and / or steering angle via the drive system 114 (and / or one or more motors, braking systems, steering systems and / or other systems belonging to or coupled to it) according to instructions provided by the processor 122.

[0057] In various embodiments, signal aggregation is performed as part of positioning model 204 (step 224). In various embodiments, during step 224, if the target is close enough to be detected by sensor 116 of type 219 (i.e., in some embodiments, when the target is less than a predetermined distance from vehicle 100), then Figure 1 The processor 122 collects and aggregates data from... Figure 1 The signal from the second type 119 sensor 116 (e.g., a UWB sensor in an exemplary embodiment). In various embodiments, this corresponds to Figure 3 Phases 303, 404, and 505 are described in further detail below.

[0058] In various embodiments, a determination is made regarding whether the number of sensors 116 currently detecting the second type 119 of the target is greater than or equal to a predetermined threshold (step 226). In various embodiments, this is determined by... Figure 1the processor 122 determines. For example, in various embodiments, certain particular sensors 116 of the second type 119 can not be able to detect the target if the target is too far away from the particular sensor 116, and / or if the particular sensor 116 (e.g., a UWB sensor) is prevented from detecting the target due to one or more individuals or objects between the target and the particular sensor 116, etc. Further in one example embodiment, the threshold of step 226 is three, such that the determination of step 226 is whether there are at least three sensors 116 of the second type 119 that detect the target.

[0059] In various embodiments, if the number of sensors 116 of the second type 119 that detect the target is determined to be greater than or equal to the predetermined threshold of step 226 (e.g., at least three UWB sensors in the example embodiment) in step 226, the process proceeds to step 228. In various embodiments, during step 228, a triangulation of the location of the target is performed using the sensors 116 of the second type 119 (e.g., three UWB sensors in the example embodiment). Further in various embodiments, the triangulation of step 228 is performed via Figure 1 the processor 122 using the signal aggregation of step 224 and the initial determination of step 214. Additionally, in the example embodiment, the triangulation of step 228 is performed in response to the determination of step 226. Further in various embodiments, the triangulation of step 228 is performed in response to the determination of step 226. Figure 3 under the conditions of phase five 305 of the process, as described in further greater detail below in connection therewith. Further in various embodiments, subsequent to the localization of the target, one or more vehicle control actions are provided for the vehicle, such as (A) via the drive system 114 (and / or one or more motors, braking systems, steering systems, and / or other systems belonging to or coupled with the vehicle 100) in accordance with instructions provided by the processor 122 to adjust the motor / propulsion torque, braking torque, and / or steering angle; and / or (B) one or more notifications (e.g., one or more audio, visual, and / or haptic notifications) are provided for a driver or other passenger of the vehicle 100 via the display system 120 in accordance with instructions provided by the processor 122. Figure 1 the display system 120 of the vehicle 100; and / or (B) one or more vehicle movement commands for adjusting the motor / propulsion torque, braking torque, and / or steering angle are executed via the drive system 114 (and / or one or more motors, braking systems, steering systems, and / or other systems belonging to or coupled with the vehicle 100) in accordance with instructions provided by the processor 122.

[0060] In contrast, in various embodiments, if instead the number of sensors 116 of the second type 119 that detect the target is determined to be less than the predetermined threshold of step 226 (e.g., two or fewer UWB sensors in the example embodiment) in step 226, the process instead proceeds to step 230. In various embodiments, during step 230, a potential location (also referred to herein as a candidate location) for the target is determined by Figure 1The processor 122 is identified using the available sensors 116 of the second type 119 (e.g., one or two UWB sensors in an exemplary embodiment).

[0061] Furthermore, in various embodiments, after step 230, a prediction is performed regarding the candidate locations (step 232). In various embodiments, the prediction is based on the path loss model from step 220. Figure 1 The data from the first type 118 sensor 116 (e.g., an RSSI sensor) is used to perform step 230 with regard to different candidate locations.

[0062] Furthermore, in various embodiments, style matching is performed (step 234). Specifically, in various embodiments, in addition to the initial determination in step 210, Figure 1 The processor 122 also uses the prediction from step 232 to perform pattern matching for each of the candidate locations from step 230. In various embodiments, pattern matching utilizes data from sensor 116 of the first type 118 (e.g., an RSSI sensor) to efficiently evaluate candidate locations identified by sensor 116 of the second type 118 (e.g., a UWB sensor) in order to select the candidate location considered to have the highest level of certainty as the most accurate location.

[0063] In various embodiments, position selection is performed (step 236). In various embodiments, the position selection in step 236 is performed in accordance with... Figure 3 It is executed under the conditions of Phase 3 303 and Phase 4 304, as follows: Figure 3 As described in further, more detailed description. Furthermore, in various embodiments, one or more vehicle control actions are subsequently provided to the vehicle based on the location of the target, such as (A) via instructions provided by processor 122. Figure 1 The display system 120 provides one or more notifications (e.g., one or more audio, visual and / or tactile notifications) to the driver or other passengers of the vehicle 100; and / or (B) executes one or more vehicle movement commands for adjusting motor / propulsion torque, braking torque and / or steering angle via the drive system 114 (and / or one or more motors, braking systems, steering systems and / or other systems belonging to or coupled to it) according to instructions provided by the processor 122.

[0064] As mentioned above, Figure 3 According to exemplary embodiments Figure 2 The diagram 300 illustrates the various stages utilized in the implementation of process 200.

[0065] like Figure 3 The first phase 301 (as described) Figure 2occurs when the target is at least a first predetermined distance from the vehicle 100 (or, in various embodiments, from a particular respective sensor 116). In various embodiments, the passive scan 308 is performed by a single sensor of the first type 118 (e.g., an RSSI sensor) and not by any of the sensors 116 of the second type 119. Moreover in various embodiments, the distance between the target 306 and the vehicle 100 (or from a particular respective sensor 116) is too great for the sensors 116 of the second type 119 (e.g., UWB sensors) to detect the target 306 and also too great for additional sensors 116 of the first type 118 (in addition to the single sensor 116 of the first type 118 that is able to detect the target 306). Moreover in various embodiments as Figure 3 depicted, the target 306 is just outside of a first (outer) region 310 around the vehicle 100 and entirely outside of a second (inner) region 312 around the vehicle 100. Moreover as Figure 3 depicted, there is a resulting error 314 in the accuracy of the location.

[0066] Moreover as Figure 3 depicted, the second stage 302 (again as Figure 2 referenced in the process 200) occurs when the target is less than the first predetermined distance (of the first stage 301) from the vehicle 100 but greater than a second (relatively smaller) predetermined distance from the vehicle 100 (or, in various embodiments, from a particular respective sensor 116) such that the target 306 is now located within the first region 310 around the vehicle 100. In various embodiments, the passive scan 320 is performed by multiple sensors 116 of the first type 118 (e.g., multiple RSSI sensors, each of which is now able to detect the target 306) and not by any of the sensors 116 of the second type 119. Moreover in various embodiments, the distance between the target 306 and the vehicle 100 (or from a particular respective sensor 116) is still too great for the sensors 116 of the second type 119 (e.g., UWB sensors) to detect the target 306, although now the multiple sensors 116 of the first type 118 (e.g., RSSI sensors) are able to detect the target 306. Moreover as Figure 3 depicted, each of the multiple sensors 116 of the first type 118 provides scanning and localization of the target 306 in a different respective corresponding region 321, 322, 323, and 324, each region corresponding to a different respective one of the sensors 116 of the first type 118.

[0067] Moreover as Figure 3The described third stage 303 (as well as...) Figure 2 The process referred to in process 200 occurs when the target is less than both the first predetermined distance (of the first stage 301) and the second predetermined distance (of the second stage 302) from the vehicle 100 (or, in various embodiments, from a specific corresponding sensor 116), such that the target 306 is now within a portion of the region 312 surrounding the vehicle 100. However, in the third stage 303, the target 306 is still at a sufficient distance from the vehicle 100 (or sensor 116) (i.e., greater than a third predetermined threshold) (or, in some embodiments, has a blocked path with some sensors 116 of the second type 119), such that only a single sensor (i.e., a single UWB sensor) of the sensors 116 of the second type 119 is able to detect the target 306. In various embodiments, activation 330 is performed by the single sensor (e.g., a single UWB sensor) of the sensors 116 of the second type 119 that provides detection and localization of the target 306. Additionally, as Figure 3 As depicted, within the third stage 303, positioning is further assisted by continuous scanning and localization of the target 306 by multiple sensors 116 of the first type 118 (e.g., multiple RSSI sensors), for example, corresponding to... Figure 3 The different regions shown are 331, 332, 333, and 334.

[0068] In addition, such as Figure 3 The described fourth stage 304 (as well as...) Figure 2 The process referred to in process 200 occurs when the target is closer to the vehicle 100 (or, in various embodiments, closer to a specific corresponding sensor 116) (i.e., less than the third predetermined threshold distance of the third stage 303), such that the detection and localization of the target 306 can now be performed by two (and only two) of the sensors 116 of the second type 119. However, in the fourth stage 304, the target 306 is still at a sufficient distance from the vehicle 100 (or sensor 116) (i.e., the fourth predetermined distance) (or, in some embodiments, has a blocked path with some of the sensors 116 of the second type 119), such that only two of the sensors 116 of the second type 119 (i.e., two UWB sensors) can detect the target 306. In various embodiments, activation 330 is performed by the two sensors (e.g., two UWB sensors) of the sensors 116 of the second type 119 that provide the detection and localization of the target 306. Additionally, as Figure 3 As depicted, within the fourth stage 304, positioning is further assisted by continuous scanning and localization of the target 306 by multiple sensors 116 of the first type 118 (e.g., multiple RSSI sensors), for example, corresponding to... Figure 3 The area shown is 344.

[0069] Further as Figure 3 depicted, a fifth stage 305 (also as referenced in the process 200 of Figure 2 occurs when the target is closer (i.e., less than the fourth predetermined threshold distance of the fourth stage 304) to the vehicle 100 (or, in various embodiments, to a particular respective sensor 116) such that detection and localization of the target 306 can now be performed by at least three sensors 116 of the second type 119. In various embodiments, the localization of the target 306 is performed via triangulation via three or more sensors 116 of the second type 119. In various embodiments, this is performed via the processor 122 of the Figure 1 In addition, in various embodiments, the localization (e.g., triangulation) is also aided by continuous scanning and localization of the target 306 by multiple sensors 116 of the first type 118 (e.g., multiple RSSI sensors), such as corresponding to the area 354 as shown in Figure 3

[0070] Figure 4 is a flowchart of an exemplary sub-process of the process 200 of Figure 2 corresponding to the localization in the particular first and second stages 301 and 302 of Figure 2 step 222 of Figure 3

[0071] As depicted in Figure 4 an initial determination is performed (step 402) with respect to the sensors 116 of the first type 118. In various embodiments, this step corresponds to step 210 of Figure 2 where an initial determination of the location of the target is obtained or determined (e.g., with respect to an initial RSSI location determination) with respect to the first measurements of step 208.

[0072] In addition, in various embodiments, feature extraction is performed (step 404). In various embodiments, this step corresponds to step 216 of Figure 2 where the processor 122 of the Figure 1 performs feature extraction from the first measurements of step 208 and the initial determination of step 210 (e.g., step 402) described above. In various embodiments, the feature extraction involves detection and initial localization of one or more targets (e.g., one or more people, other vehicles, and / or other objects) proximate to the vehicle 100.

[0073] In addition, in various embodiments, a path loss model is implemented and / or updated (step 406). In various embodiments, the path loss model comprises a neural network. In addition, in various embodiments, this step corresponds to step 218 of Figure 2 ​​of step 220, wherein Figure 1 The processor 122 of the system 100 uses the real-time training of step 218 to update path loss matters for the first type 118 of sensors 116 (e.g., RSSI sensors).

[0074] In various embodiments, a determination is made as to whether the target is within the core area (step 410). In various embodiments, during this step, Figure 1 The processor 122 of the system 100 determines Figure 3 whether the target 306 is within Figure 3 the second area 312 of the system 100 (i.e., within a predetermined distance of the vehicle 100 and / or applicable sensors 116 such that one or more sensors 116 of the second type 119 can detect the target 306).

[0075] If it is determined in step 410 that the target is within the core area, then in example embodiments, an initialization is triggered and provided for one or more of the second type 119 of sensors 116. Specifically, in various embodiments, the processor 122 initiates utilization of one or more sensors of the second type 119 (e.g., one or more UWB sensors). Further in various embodiments, a positioning is subsequently performed in step 414 via the processor 222 using sensors 116 of both the first type 118 (e.g., RSSI sensors) and the second type 119 (e.g., UWB sensors), resulting in a two-level accurate positioning of the target 306.

[0076] Conversely, if it is instead determined in step 410 that the target is not within the core area, then in example embodiments, a positioning is provided by the processor 122 in step 416 using only the first type 118 of sensors 116 (e.g., RSSI sensors), resulting in a one-level positioning (e.g., providing a coarse location of the target 306).

[0077] In example embodiments, as part of (or subsequent to) the positioning of step 414 or 416, based on the positioning of the target, one or more vehicle control actions are provided for the vehicle, such as (A) providing one or more notifications (e.g., one or more audio, visual, and / or haptic notifications) for a driver or other passenger of the vehicle 100 via the display system 120 in accordance with instructions provided by the processor 122; and / or (B) executing one or more vehicle movement commands for adjustments to motor / propulsion torque, braking torque, and / or steering angle via the drive system 114 (and / or one or more motors, braking systems, steering systems, and / or other systems belonging to or coupled with the same) in accordance with instructions provided by the processor 122. Figure 1

[0078] ​Figure 5 This is according to an exemplary embodiment. Figure 2 Another exemplary subprocess of process 200 (corresponding to) Figure 2 Step 226, namely Figure 3 The flowcharts for the specific positioning in phases 303 and 304 of the third and fourth stages. (See reference) Figure 5 In an exemplary embodiment, Figure 5 The steps described in the text are about Figure 1 Each of the sensors 116 in the vehicle 100 performs this action.

[0079] like Figure 5 As depicted, in an exemplary embodiment, an initial determination (step 502) is performed with respect to sensor 116 of type 118. In various embodiments, this step corresponds to Figure 2 Step 402 and Figure 2 Step 210, wherein the initial determination of the target's location is obtained or determined with respect to the first measurement in step 208 (e.g., initial RSSI location determination).

[0080] Furthermore, in various embodiments, the observed patterns are identified (step 504). In various embodiments, the processor 122 uses the initial determination of step 502 to observe patterns in sensor data obtained from sensor 116 of the first type 118 (e.g., RSSI sensor).

[0081] Furthermore, in an exemplary embodiment, feature extraction is performed (step 506). In various embodiments, this step corresponds to Figure 4 Step 404 and Figure 2 Step 216. Furthermore, in various embodiments, during step 506, Figure 1 The processor 122 performs feature extraction based on the initial determination in step 502 and the pattern observation determination in step 504.

[0082] Furthermore, in various embodiments, the initial determination of the target's location is performed via a sensor 116 of the second type 119 (e.g., one or more UWB sensors) (step 508). In some embodiments, step 508 corresponds to... Figure 2 Step 214, and includes about Figure 2 The second measurement in step 212 is used to make an initial determination (e.g., regarding the initial UWB position determination) via two-way ranging.

[0083] Furthermore, in various embodiments, one or more distances initially determined from step 508 are used to update the path loss model (step 510). In various embodiments, during step 510, processor 122 uses the distances to update... Figure 2 Step 220 and Figure 4Step 406 involves the path loss model. As described above, in various embodiments, the path loss model includes a neural network model.

[0084] In addition, such as Figure 5 As depicted, in various embodiments, a potential location is provided for the target (step 512). In various embodiments, this corresponds to the above. Figure 2 Step 230. Furthermore, in various embodiments, step 512 is via... Figure 1 The processor 122 performs this action. Specifically, in various embodiments, the processor 122 uses the initial determination of the sensor 116 (e.g., a UWB sensor) according to the second type 119 in step 508 to determine a plurality of potential locations (also referred to as candidate locations) for the positioning or location of the target 306 relative to the vehicle 100.

[0085] In various embodiments, an estimation pattern is determined (step 514). Specifically, in various embodiments, during step 514, processor 122 uses sensor data from sensor 116 of the second type 119 (and specifically includes the updated path loss model from step 510 and the potential locations from step 512) to predict an estimated pattern for the sensor data from sensor 116 of the first type 118. In various embodiments, the corresponding estimation pattern is predicted for each of the potential locations.

[0086] Furthermore, in various embodiments, pattern matching is performed (step 516). Specifically, in various embodiments, the observed pattern of the data from sensor 116 of the first type 118 (from step 504) is compared with the predicted pattern of the data from sensor 116 of the first type 118 (from step 514). In various embodiments, pattern matching is performed by processor 122 with respect to each of the potential locations in step 512.

[0087] In various embodiments, location selection is performed (step 518). Specifically, in various embodiments, processor 122 selects one of the potential locations from step 512 as the most probable location of the target based on the pattern matching from step 516. In an exemplary embodiment, the potential location with the closest match between the corresponding observed pattern and the estimated pattern is determined as the most probable location of the target.

[0088] In various embodiments, positioning is performed (step 520). In various embodiments, the processor 122 utilizes sensors 116 of both the first type 118 and the second type 119 to perform further positioning of the target (thereby providing a two-level accurate position of the target) with respect to the selected position of step 518. Further in various embodiments, based on the positioning of the target, one or more vehicle control actions are subsequently provided for the vehicle, such as (A) providing one or more notifications (e.g., one or more audio, visual, and / or haptic notifications) for a driver or other passenger of the vehicle 100 via the display system 120 in accordance with instructions provided by the processor 122; and / or (B) executing one or more vehicle movement commands for adjustments to motor / propulsion torque, braking torque, and / or steering angle via the drive system 114 (and / or one or more motors, braking systems, steering systems, and / or other systems belonging to or coupled with the same) in accordance with instructions provided by the processor 122. Figure 1

[0089] Figure 6 Figure 7 Figure 8A Figure 8B Figure 9A Figure 9B Figure 9C depicts an exemplary implementation of pattern matching associated with steps of the process 200 of FIG. 6 in accordance with exemplary embodiments (including pattern matching of step 516 of FIG. 5). Figures 2-5 Figure 5

[0090] Referring first to FIG. 6, Figure 6 the exemplary illustration 600 relates to an implementation in which there are four sensor locations Al (601), A2 (602), A3 (603), and A4 (604). The sensors of the second type 119 propose two candidate locations, namely: LI (605) and L2 (606). In various embodiments, the sensor data of the sensors of the first type 118 is used to determine which of the two candidate locations (i.e., LI (605) or L2 (606)) is correct. In various embodiments, this is performed using the following equations:

[0091] If the target is at LI, the theoretically predicted RSSI based on the RSSI path loss model trained on each sensor by NN or LS) (Equation 1);

[0092] If the target is at L2, the theoretically predicted RSSI based on the RSSI path loss model trained on each sensor by NN or LS) (Equation 2); ​​​​​​​​​

[0093] RSSI sequence observed by the sensor (Equation 3); and

[0094] For any i == j, Where m = 1, 2 (Equation 4).

[0095] Referring to Figure 7 , there is shown a graph with various sensors A1 (701), A2 (702), A3 (703), and A4 (704), and any number of possible candidate values L1 (705), L2 (706), L3 (707)... L m (720) and the like.

[0096] Figure 8A and Figure 8B Exemplary implementations of pattern matching associated with steps of process 200 of Figures 2-5 are also depicted. Specifically, in exemplary embodiments: (i) Figure 5 Exemplary implementations of pattern matching associated with steps of process 200 of Figure 8A are also depicted. Specifically, in exemplary embodiments: (i) Figure 8B A second graph 800(B) is depicted with a second RSSI pattern for a second candidate location L2 based on UWB data from one or more second UWB sensors. In both graphs: the X-axis is represented by 801; the Y-axis is represented by 802, sensed RSSI location values are represented by 810, true RSSI location values are represented by 820, and false RSSI location RSSI values are represented by 830. In the depicted example, the first candidate location L1 provides a better fit than the second candidate location L2, as the sum of errors for the first candidate location L1 is less than the sum of errors for the second candidate location L2.

[0097] Figure 9A , Figure 9B and Figure 9C Further graphs of pattern matching are provided with respect to illustrative examples of Figure 8A and Figure 8B . Specifically: (i) Figure 9A A first representation 900(A) is provided with an RSSI ratio chart for a first candidate location L1; (ii) Figure 9B A second representation 900(B) is provided with an RSSI ratio chart for a second candidate location L2; and (iii) Figure 9CA third representation 900(C) is provided, which has an RSSI ratio chart of the observed RSSI data. In each of these three charts: (i) the first quadrant (901(A), 901(B), or 901(C), respectively) represents the respective ratios with respect to the first sensor; (ii) the second quadrant (902(A), 902(B), or 902(C), respectively) represents the respective ratios with respect to the second sensor; (iii) the third quadrant (903(A), 903(B), or 903(C), respectively) represents the respective ratios with respect to the third sensor; and (iv) the fourth quadrant (904(A), 904(B), or 904(C), respectively) represents the respective ratios with respect to the fourth sensor. In the illustrative example, the first candidate location LI provides a better fit than the second candidate location L2, as the RSSI component ratios / distributions for the first candidate location LI are closer to the actual / observed values.

[0098] Accordingly, methods, systems, and vehicles for detecting and localizing targets proximate to a vehicle or other platform are provided. As shown and described above in connection therewith, in various embodiments, the disclosed methods and systems utilize sensors of different modalities in combination with one another to detect targets proximate to a platform. In certain embodiments, the platform comprises a vehicle, and the methods and systems use a combination of different types of sensors (e.g., RSSI sensors and UWB sensors) to detect and localize targets at different distances from the vehicle.

[0099] It will be appreciated that the systems, vehicles, and methods can differ from those depicted in the drawings and described herein. For example, in embodiments different from those depicted in Figure 1 and / or described above in connection therewith, Figure 1 The vehicle 100 (including the control system 101, the controller 102, and / or other components thereof) can vary. Similarly, it will be appreciated that the steps of the process 200 and their implementation can differ from those depicted in Figures 2-9C and / or described above in connection therewith, and / or various steps of the process 200 can occur simultaneously and / or in a different order than depicted in Figures 2-9C and / or described above in connection therewith.

[0100] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary embodiment or exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as set forth in the appended claims and the legal equivalents thereof, which can be practiced or carried out in various fashions.

Claims

1. A method comprising: obtaining, via one or more first sensors of a platform, first sensor data regarding a target relative to the platform, the one or more first sensors having a first modality; obtaining, via one or more second sensors of the platform, second sensor data regarding the target, the one or more second sensors having a second modality different from the first modality; and using, via a processor of the platform, the first sensor data and the second sensor data to locate the target, wherein the second sensor data is used to augment the first sensor data based on a validity of the first sensor data, and using the first sensor data and the second sensor data is based on a distance of the target from the platform; wherein the one or more first sensors of the first modality are configured for detecting the target at a relatively greater distance from the platform than the one or more second sensors of the second modality; and the one or more second sensors of the second modality are configured for detecting the target from a short distance with a relatively higher accuracy than the one or more first sensors of the first modality.

2. The method of claim 1, wherein the platform comprises a vehicle, and the processor is further configured to at least facilitate performing a control action regarding movement of the vehicle based on the locating of the target relative to the vehicle.

3. The method of claim 2, wherein the one or more second sensors of the second modality comprise ultra-wideband (UWB) sensors.

4. The method of claim 3, wherein the one or more first sensors of the first modality comprise RSSI sensors.

5. The method of claim 1, wherein: the second sensor data is used, via the processor, to identify a plurality of candidate locations for the target; and the first sensor data is used, via the processor, to select a preferred candidate among the plurality of candidate locations based on a pattern matching for the second sensor data using a path loss model.

6. The method of claim 5, wherein the locating is performed, via the processor, by: when the target is greater than a first predetermined distance from the platform, using only the first sensor data without using the second sensor data; and when the target is less than the first predetermined distance from the platform, using both the first sensor data and the second sensor data.

7. The method of claim 6, wherein the locating is performed, via the processor, by: when the target is greater than the first predetermined distance from the platform and also greater than a second predetermined distance from the platform, using the first sensor data from a single first sensor of the first modality, wherein the second predetermined distance is greater than the first predetermined distance; and when the target is less than the first predetermined distance from the platform and also less than the second predetermined distance from the platform, using both the first sensor data from the single first sensor of the first modality and the second sensor data from the one or more second sensors of the second modality. ​ when the target is greater than the first predetermined distance from the platform and less than the second predetermined distance from the platform, using the first sensor data from a plurality of sensors of the first modality.

8. The method of claim 7, wherein the locating is performed via the processor by: when the target is less than the first predetermined distance from the platform but greater than a third predetermined distance from the platform, using the first sensor data from a plurality of first sensors of the first modality and the second sensor data from a single second sensor of the second modality, wherein the third predetermined distance is less than the first predetermined distance; and when the target is less than the first predetermined distance from the platform and less than the third predetermined distance from the platform, using the first sensor data from a plurality of first sensors of the first modality and the second sensor data from a plurality of second sensors of the second modality.

9. The method of claim 8, wherein the locating is performed via the processor by: when the target is less than the third predetermined distance from the platform but greater than a fourth predetermined distance from the platform, using the first sensor data from a plurality of first sensors of the first modality and the second sensor data from two second sensors of the second modality, wherein the fourth predetermined distance is less than the third predetermined distance; and when the target is less than the fourth predetermined distance from the platform, using the first sensor data from a plurality of first sensors of the first modality and the second sensor data from at least three second sensors of the second modality, by applying triangulation with respect to the second sensor data from the at least three second sensors.

10. A system comprising: one or more first sensors of a platform, the one or more first sensors having a first modality and configured to obtain first sensor data with respect to a target relative to the platform; one or more second sensors of the platform, the one or more second sensors having a second modality different from the first modality and configured to obtain second sensor data with respect to the target relative to the platform, and wherein the second sensors of the second modality are configured for detecting the target from a short distance with a relatively higher accuracy compared to the first sensors of the first modality; and a processor of the platform, the processor coupled to the one or more first sensors and the one or more second sensors and configured to facilitate at least locating the target using the first sensor data and the second sensor data, wherein the second sensor data is used to augment the first sensor data based on a validity of the first sensor data, and using the first sensor data and the second sensor data is based on a distance of the target from the platform.

Images