Tracking Aggregation and Alignment
A context tracking system efficiently aggregates and communicates data between sensors to enhance tracking in open environments, addressing the challenges of crowd density and obstruction by leveraging contextual information for improved identification and filtering.
Patent Information
- Application Number
- JP2021558812
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-03-26
- Filing Date
- 2020-03-30
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2040-03-30
AI Technical Summary
Accurately tracking unique individuals in large/open environments is difficult due to crowd density and obstruction issues, especially when precise location and activity tracking is desired.
A context tracking system that aggregates and communicates tracking data between multiple sensor systems, utilizing contextual information from one sensor to enhance the tracking capabilities of others, improving identification and filtering of objects.
Enhances tracking efficiency and accuracy in open environments by leveraging contextual data from one sensor to refine the tracking process, reducing resource usage and improving processing speed.
Smart Images

Figure 0007596295000001 
Figure 0007596295000002 
Figure 0007596295000003
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 828,198, entitled “TRACKING AGGREGATION AND ALIGNMENT,” filed April 2, 2019, which is incorporated herein by reference in its entirety for all purposes.
[0002] The present disclosure relates generally to tracking systems and, more specifically, certain embodiments of the present disclosure relate to the aggregation and handoff of tracking system data between tracking systems to facilitate more efficient and effective tracking of objects within an environment. [Background technology]
[0003] In the digital age, with the proliferation of digital sensors, object tracking has become increasingly desirable. Unfortunately, in large / open environments, tracking users is an extremely challenging prospect, especially when precise location and activity tracking is desired. As used herein, an open environment refers to an area in which the object being tracked can move in multiple directions with relatively little containment. For example, such environments can include amusement parks, airports, shopping malls, or other relatively large environments that may have multiple tracking coverage zones. Accurately tracking unique individuals is difficult, especially in open environments, as well as situations where crowd density presents obstacle problems and one individual may block another.
[0004] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present technology, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art. Summary of the Invention
[0005] Certain embodiments falling within the scope of the initially claimed invention are summarized below. These embodiments are not intended to limit the scope of the disclosure, but rather, these embodiments are intended only to provide a brief summary of certain disclosed embodiments. Indeed, the disclosure may encompass a variety of forms that may be similar to or different from the embodiments set forth below.
[0006] The embodiments described herein relate to a tracking system that efficiently aggregates and / or communicates tracking data between tracking systems, allowing the context of one tracking sensor to enhance tracking of other tracking sensors. More specifically, context information determined by one tracking sensor (e.g., location, time, identity of tracked object) can be used to facilitate more efficient and / or effective tracking of other sensors. For example, such context information can increase the confidence of tracked object identity and provide efficient filtering of possible identities that may be attributed to tracked objects. This can improve processing efficiency and also enable more fine-grained tracking of objects in open environments.
[0007] As an example, in a first embodiment, a tangible, non-transitory machine-readable medium includes machine-readable instructions that, when executed by one or more processors of the machine, cause the machine to receive tracking target context of a first tracked object from a first tracking sensor system, provide tracking target context from the first tracking sensor system to a second tracking sensor system different from the first tracking sensor system, and provide an identification of a newly observed tracking target by the second tracking sensor system based on the tracking target context from the first tracking sensor system.
[0008] In a second embodiment, a computer-implemented method includes receiving a tracking target context of a first tracked object from a first tracking sensor system, providing the tracking target context from the first tracking sensor system to a second tracking sensor system different from the first tracking sensor system, and providing an identification of a newly observed tracking target by the second tracking sensor system based on the tracking target context from the first tracking sensor system.
[0009] In a third embodiment, a system includes a first tracking sensor system, a second tracking sensor system, and a context tracking sensor system. The first tracking sensor system tracks a first tracking object in a first coverage area. The second tracking sensor system tracks a second tracking object in a second coverage area. The context tracking system receives a tracking target context of the first tracking object from the first tracking sensor system, provides the tracking target context from the first tracking sensor system to a second tracking sensor system different from the first tracking sensor system, and provides an identification of the second tracking object based on the tracking target context from the first tracking sensor system. [Brief description of the drawings]
[0010] These and other features, aspects and advantages of the present disclosure will become better understood from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout.
[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating a multi-sensor tracking component with a context tracking system according to one embodiment of the present disclosure.
[0012] [Diagram 2] FIG. 2 is a schematic diagram illustrating an open environment using the system of FIG. 1 according to one embodiment of the present disclosure.
[0013] [Diagram 3] 1 is a flowchart illustrating a process for identifying a tracking context according to one embodiment.
[0014] [Figure 4] 10 is a flowchart illustrating a process for identifying a context in a subsequent tracking sensor using acquired context, according to one embodiment.
[0015] [Diagram 5] 1 is a flowchart illustrating a process for using confidence intervals to determine sufficient context for target identification, according to one embodiment.
[0016] [Figure 6] FIG. 2 is a schematic diagram illustrating an exemplary diversion for increasing tracking input, according to one embodiment.
[0017] [Figure 7] 1 is a flowchart illustrating a process for filtering possible discrimination predictions based on a provided sensor context, according to one embodiment.
[0018] [Figure 8] FIG. 1 is a schematic diagram illustrating an example control action based on tracking identity, according to one embodiment.
[0019] [Figure 9] 1 is a flowchart for grouping individuals into groups using machine learning techniques, according to one embodiment.
[0020] [Figure 10] FIG. 2 is a schematic diagram of grouped and ungrouped individuals according to one embodiment.
[0021] [Figure 11]FIG. 1 is a schematic diagram of a use case for modifying an interactive environment based on compliance rules, according to one embodiment.
[0022] [Figure 12] FIG. 1 is a schematic diagram illustrating an example control action based on tracking identity, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0023] One or more specific embodiments of the present disclosure are described below. These described embodiments are merely examples of the technology of the present disclosure. In addition, in order to provide a concise description of these embodiments, not all features of an actual implementation may be described herein. As with any engineering or design project, it will be appreciated that the development of any such actual implementation will require numerous implementation-specific decisions to be made to achieve the developer's particular goals, which may vary from implementation to implementation, such as compliance with system-related and business-related constraints. Moreover, it will be appreciated that such development efforts may be complex and time-consuming, but will nevertheless be routine undertakings of design, fabrication and manufacture for those skilled in the art having the benefit of this disclosure.
[0024] When introducing elements of various embodiments of the disclosure, the articles "a," "an," and "the" are intended to mean that there are one or more of the element. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Further, it is to be understood that references to "one embodiment" or "embodiments" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the listed features.
[0025] The present disclosure generally relates to tracking systems that accumulate and / or hand off context information for efficient and effective tracking processing. By using previously determined context of other tracking sensors, an independent tracking sensor can more efficiently determine the identity of a tracked object. With this in mind, FIG. 1 is a schematic diagram illustrating a multi-sensor tracking system 100 with a context tracking system 102, according to one embodiment of the present disclosure. As shown, the multi-sensor tracking system 100 includes multiple tracking sensors, such as one or more Light Detection and Ranging (LIDAR) systems 104, one or more Radio Frequency Identification (RFID) reader systems 106, one or more Time of Flight (ToF) systems 107, one or more computer vision systems 108, and / or one or more millimeter wave (mmWave) systems 109.
[0026] The LIDAR system 104 can track individuals, objects, and / or groups of individuals or objects by illuminating the target with pulsed light and measuring the reflected pulse. The difference in wavelength and time between the pulsed light and the reflected pulse can be used to generate a spatial representation of the location of the target individuals, groups, and / or objects. The LIDAR system 104 can cover a large spatial area while detecting objects with relative ease and efficiency. However, the LIDAR system 104 may not be effective at actually providing identification of the tracked object, and instead may best be used to identify the presence and location of the object independent of the object's identification.
[0027] The RFID reader system 106 can read digital data encoded in an RFID tag (e.g., worn by an individual or placed on a particular object to track the individual or object). When an RFID tag comes into the vicinity of an RFID reader, the RFID reader can provide an energizing signal to the RFID tag, causing the RFID tag to emit radiant energy that can be interpreted by the RFID reader to identify the particular RFID tag. Because each RFID tag has a unique identification, the RFID reader system 106 can effectively and efficiently identify tracking targets. However, the RFID reader system 106 requires that the RFID tags be placed in relatively close proximity to the RFID reader, which reduces coverage and / or significantly increases the hardware costs to implement a large number of RFID readers.
[0028] Similar to the LIDAR system 104, the ToF system 107 (e.g., a three-dimensional time-of-flight sensor system) can track individuals, groups, and / or objects by illuminating a target with a pulse of light and measuring the characteristics of the reflected pulse. Specifically, the ToF system 107 can emit a pulse of infrared light and measure the time corresponding to the return pulse. The ToF system 107 can also map texture (e.g., skin texture) to identify individuals, groups, and / or objects. Thus, the ToF system 107 can obtain three-dimensional location and texture attributes of the tracked identity. Another advantage is that the ToF system 107 may not be dependent on visible lighting conditions, and thus the ToF system 107 is not subject to lighting condition limitations that may characterize some camera-based vision acquisition systems. However, the ToF system 107 may be less effective and accurate in certain environmental conditions, such as on a rainy day, so a redundant system may still be useful.
[0029] The computer vision system 108 can receive camera data (e.g., still images and / or video) for context analysis. For example, the camera data can be analyzed to perform object recognition and / or tracking. Using face recognition, the computer vision system 108 can identify tracking targets. Unfortunately, however, computer vision systems 108 can be quite costly while often only covering a limited tracking area. Additionally, the computer vision system 108 can take a significant amount of time to analyze the computer images.
[0030] A millimeter wave (mmWave) system 190 (e.g., a millimeter wave radar sensor system) can provide high bandwidth and authenticate the presence of a tracked identity. In particular, the mmWave system 190 can enable a high rate of data transfer with low latency. For example, the mmWave system 190 can include a device that emits and / or receives millimeter waves to communicate with one or more computing devices (e.g., wearable devices) associated with an individual to quickly identify the individual (or authenticate an identity proposed by the individual). Furthermore, the mmWave system 190 can maintain track of the tracked identity even through surfaces (e.g., radio wave-permeable physical barriers such as windows and walls). However, the mmWave system 109 can utilize components that are relatively close to the tracked identity, resulting in reduced coverage.
[0031] As can be appreciated, each of the tracking systems has its tradeoffs. Thus, it may be desirable to use a combination of tracking systems, allowing each of the advantages of the various tracking systems to be used in combination with one another. To do this, the context tracking system 102 is responsible for maintaining and / or trading tracking data from each of the various tracking systems. For example, the context tracking system 102 may receive location information from the LIDAR system 104, object identifiers obtained from RFID tags in proximity to one or more of the RFID reader systems 106, and / or object identities and / or locations from the computer vision system 108.
[0032] Using this information, object tracking in the open environment can be obtained more effectively and efficiently. Figure 2 is a schematic diagram illustrating an open environment 200 using the system of Figure 1, according to one embodiment of the present disclosure. Although Figure 2 illustrates a theme park environment, the discussion is not intended to limit the application of the context tracking system to such an embodiment. Indeed, the present technology can be used in a variety of environmental applications.
[0033] As shown, the open environment 200 can have many separate coverage zones 202A, 202B, 202C, and 202D, each tracked by one or more sensor systems. For example, coverage zone 202A is a parking lot tracked by a computer vision system 204. Coverage zone 202B is the area between an entrance gate 206 and an attraction 208. Coverage zone 202B has a LIDAR system 210, an RFID reader system 212, and an mmWave system 213. Coverage zone 202C is tracked by a second LIDAR system 214 and a ToF system 215. Coverage zone 202D is tracked using a second mmWave system 217.
[0034] The context tracking system 216 is communicatively coupled to various tracking systems in the open environment 200. Similar to the context tracking system 102 of FIG. 1, the context tracking system 216 can maintain and / or exchange tracking data between the tracking systems. For example, the computer vision system 204 can detect a particular car 219 within the coverage zone 202A. The computer vision system 204 can analyze a visual image of the car 219 to identify the car 219. For example, the computer vision system 204 can identify alphanumeric characters on a license plate of the car 219 to identify the car 219.
[0035] The identified object can be used to identify other objects in the context tracking system 216. For example, an identification of the car 219 can be provided to the context tracking system 216. The car 219 can be identified, for example, as corresponding to one or more people (e.g., a group of people) that the computer vision system 204 detected exiting from the car 219 at a particular time. Based on this information, the context tracking system 216 can determine that one or more people are likely to be in the coverage zone 202A. Furthermore, the identity of the person or group based on the computer vision system 204 can be used in determining the identity of the person or group in another coverage zone. In particular, the context tracking system 216 can record characteristics unique to one or more people identified in the coverage zone 202A and can use the recorded characteristics in determining the identity of the tracked object in other coverage zones, such as the coverage zone 202B and the coverage zone 202D.
[0036] By providing such context to the context tracking system 216 and / or other tracking sensors in the coverage zone, the tracking analysis can become more aware of possible approaching candidate objects. For example, if the computer vision system 204 provides a notification to tracking sensors in nearby coverage zones that Car A is in a parking lot (or that Person A or Group A associated with Car A is likely to be in the parking lot [e.g., based on the presence of Car A in the parking lot]), the tracking sensors in these nearby coverage zones can be "preheated" with data identifying likely object identifiers. Thus, in some embodiments, a larger coverage system with fewer object identification capabilities, such as the LIDAR system 210, can be used to track the location of a user, relying at least in part on the identification provided by the computer vision system 204. Furthermore, as briefly mentioned above, the identified objects can be used to identify other objects, such as groups of people. Indeed, it can be beneficial for the context tracking system 216 to track groups. For example, a car 219 can be identified as corresponding to a group of people (e.g., Group A). In this case, the context tracking system 216 may determine that all people exiting the car 219 are related to one another and therefore may comprise group A. As will be explained later, other methods of identifying and determining groups are possible.
[0037] If only one user is associated with car A, a single identified object exiting the car may have a high probability of being the associated user. However, in some cases, the context provided from the preceding tracking system may not meet the threshold level of probability. In such cases, additional tracking may be selectively enabled to identify a particular object / user. For example, another computer vision system 218 at the entrance gate 206 for identifying the object / user. Since the entrance gate 206 is a funnel-shaped location with a desired direct line of sight access of the object / user, this location may be the primary location of the computer vision system 218. In such an embodiment where individual objects / users are detected by the computer vision system 218, the object identification analysis performed by the computer vision system 218 may be heavily influenced by the context data provided by the computer vision system 204 via the context tracking system 216. For example, viable candidates for identification may be filtered based on the data provided by the computer vision system 204, resulting in more efficient and faster processing of object / user identification. Furthermore, the mmWave system 213 may also be used in the coverage zone 202B to identify or authenticate the presence of individual objects / users. Indeed, the mmWave system 213 can work in conjunction with the computer vision system 218 in identifying objects. For example, the mmWave system 213 can use contextual data provided by the computer vision system 204 via the context tracking system 216 to act as a second mechanism for filtering possible candidates to improve the accuracy and speed of object identification.
[0038] The context tracking system 216 can infer / predict the identity of an object / user using location information in some embodiments. For example, if the RFID reader system 212 informs that person A enters the attraction 208 at 12:00 noon and the attraction 208 has an exit point that is typically reached in 10 minutes, the second LIDAR system 214 can infer / predict that an object that arrives at the exit point at 12:10 is likely to be person A. Thus, at the exit of the attraction 208, a tracking system that provides detailed identification of the object / user may not be needed, resulting in a more efficient use of resources. Furthermore, the tracking system does not necessarily need to be within the line of sight of the object / user. Indeed, as shown in the coverage zone 202D, the second mmWave system 217 is positioned to track objects within the restaurant 221, even though it is not within the line of sight of the objects located in the restaurant 221. The second mmWave system 217 may be able to enable this type of communication because certain materials (e.g., fabrics, fiberglass reinforced plastic, etc.) associated with the restaurant 221 and / or tracked objects may be transparent to radiation (e.g., transparent to radio frequencies) emanating from the second mmWave system 217.
[0039] As discussed in more detail below, the tracking information (e.g., object / user identification and location) can be used for many different purposes. For example, in one embodiment, kiosk 220 can provide specific information useful to an identified object / user in tracking the particular object / user to kiosk 220. Additionally, in some embodiments, this tracking can help theme park personnel understand the interests of an object / user based on where the object / user is tracked (e.g., an attraction, a restaurant, etc.).
[0040] Having discussed the basic utility of the context tracking system, FIG. 3 is a flow chart illustrating a process 300 for identifying and maintaining a tracking context, according to one embodiment. The process 300 begins with selecting a tracking target (block 302). The tracking target may be selected based on one or more criteria of objects observed in the environment, and may differ for each coverage zone. For example, for the open environment 200, the coverage zone 202A may be of particular interest in tracking vehicles in a parking lot. Thus, the tracking target may be selected based on the extent of the moving speed of the object attributed to the vehicle, the size of the object, the shape of the object, etc. In contrast, in the coverage area 202B, it may be assumed that there is no vehicle, but instead a person associated with the vehicle. Thus, the criteria for selecting the tracking object in the coverage zone 202B may be the extent of the moving speed of the object attributed to a person or a group of people, the size of the object, the shape of the object, etc.
[0041] The process 300 proceeds to identify the tracked target identity (block 304). For example, as discussed above, the identity may be determined based on other identified objects, based on computer vision analysis of the object (e.g., at the entrance gate 206), where the relationship between the other identified objects and the tracked target is recorded (e.g., in tangible storage, such as the context tracking system 216).
[0042] Once the identity is determined, the position and / or location of the tracked target is identified (block 306). This information, in combination with the identity, can be useful to other tracking sensor systems, which can filter in or out certain identities (e.g., a subset of identities) as possible identities of the objects they track. For example, as previously described, with reference to FIG. 2, the identity and location showing Person A entering the attraction, along with a duration estimate of the length of the attraction from start to finish, can enable a tracking system covering that exit of the attraction to filter in Person A as a possible identity of the object detected at the exit point of the attraction at or near the end of the duration estimate. Furthermore, if Person A is in the attraction, then Person A is obviously not in the parking lot. Thus, the tracking system sensor in the parking lot can filter out Person A as a candidate identity, allowing for faster response time of the discrimination analysis.
[0043] As can be appreciated, the tracked target context is maintained and / or traded by the context tracking system (block 308). As discussed herein, the tracked target context can include the observed activity of a particular tracked sensor system, such as tracking time, tracking location, tracking identity, tracking identity associated with a tracked object, etc. In some embodiments, the context tracking system can maintain the context data and filter in or out candidates based on the context data to provide the filtering results to the tracking system for efficient discrimination analysis. In other embodiments, the context tracking system can provide the context to the tracking system and enable the tracking system to perform filtering and discrimination based on filtering.
[0044] 4 is a flow chart illustrating a process 400 for identifying a context at a subsequent tracking sensor using the acquired context, according to one embodiment. The process 400 begins with receiving a tracked target context (or filtering information, if the tracked target context is maintained in a context tracking system) (block 402). Once received, the tracking sensor can determine the identity of the object using the filtering-in and / or filtering-out techniques described above (block 404).
[0045] Targeted control actions are performed based on the subsequent tracking identifier (block 406). For example, as discussed above, a kiosk can be controlled to display specific information useful to the identified person. In other embodiments, access to a specific restricted portion of the environment can be granted based on a determination that the identity is authorized for access to the restricted portion of the environment. Additionally, in some embodiments, metric tracking such as visits to specific areas, logged activity of the user, etc. can be maintained for subsequent business analysis, research, and / or reporting.
[0046] As mentioned above, sometimes a certain threshold level of confidence may be required to make a positive match of an identity to a tracked object. Figure 5 is a flow chart illustrating a process 500 for using a confidence interval to determine sufficient context for target identification, according to one embodiment. Process 500 begins with collecting tracking sensor input (e.g., context) (block 502). As mentioned above, a context tracking system may receive tracking sensor input that provides identity, location, and / or other information that provides actionable context for observed objects of other tracked sensors.
[0047] Based on these tracking inputs, one or more tracking identities of the observed object can be predicted by other tracking sensors (block 504). For example, as described above, context information indicating that a particular person A has entered the attraction can be used to assume that person A will eventually exit the attraction at a predicted time, possibly based on the person's attributes and / or the attraction's attributes. For example, if the tracking system observes a person to have a below-average or above-average movement speed, the average duration of the attraction can be adjusted upwards or downwards accordingly. This adjusted duration is used to determine the time that person A is likely to exit the attraction, which allows the tracking system at the exit of the attraction to predict that the person leaving the attraction at the adjusted duration is person A.
[0048] In some embodiments, a predicted confidence score can be calculated. The predicted confidence score can indicate the likelihood that the tracked object has a particular identity. For example, if the target is highly likely to be Person A based on known information, the predicted confidence score can be larger than if the target is somewhat likely or unlikely based on known information. The predicted confidence score can vary depending on several factors. In some embodiments, redundant information (e.g., similar identifications using two or more data sets) can increase the predicted confidence score. Additionally, observable characteristics of the tracked object can be used to affect the predicted confidence score. For example, a known size associated with the identity can be compared to the size of the tracked target. The predicted confidence score can be increased based on the proximity of the known size and the size of the target observed by the tracking sensor.
[0049] After the predicted confidence score is obtained, a determination is made as to whether the predicted confidence score meets a score threshold (decision block 508). The score threshold may indicate a minimum score that can result in an identity associated with the tracked target.
[0050] If the predicted confidence score does not meet the threshold, additional tracking inputs can be acquired (block 510). To acquire additional tracking inputs, the tracking sensors in the open environment can continue to accumulate tracking information and / or acquire contextual data of the tracked target. In some embodiments, the tracked target can be prompted to move toward a particular sensor to acquire new tracking inputs regarding the tracked target. For example, FIG. 6 is a schematic diagram illustrating an example scenario 600 of increasing tracking inputs, according to one embodiment. In the scenario 600, the predicted confidence score of the tracked target 602 is less than the threshold, as indicated by the callout 604. Thus, the electronic display 606 can be controlled to display a message directing the target to a location where additional tracking inputs can be acquired, here, the pizzeria 608. Here, the target is prompted to move toward the pizzeria 608 by providing an electronic notification via an electronic sign that directs the target to the pizzeria 608. At the entrance of the pizzeria 608, there is an additional tracking sensor (such as an RFID reader 610). The tracking sensor obtains additional tracking input of the target 602 as the target 602 moves into the vicinity of the pizzeria 608 .
[0051] Returning to Figure 5, as additional tracking input is collected, additional predictions of the target's identity are made (block 504), a new predicted confidence score is determined using the newly collected additional tracking input (block 506), and an additional decision is made as to whether the predicted confidence threshold has been met (decision block 508). This process can continue until the predicted confidence threshold is met.
[0052] Once the predicted confidence threshold is met, the identifier may be attributed to the tracked object (block 512). For example, returning to Figure 6, additional tracking information received via RFID reader 610 results in an increased predicted confidence score to a level equal to or greater than the threshold, as indicated by bubble 612. Thus, tracked target 602 (e.g., tracked object) is attributed to identifier 614 (e.g., Person A).
[0053] As previously mentioned, the context provided by one tracking sensor system can be useful for analysis by other tracking sensor systems. For example, candidate predicted identities can be "filtered in" (e.g., whitelisted) or "filtered out" (e.g., blacklisted) based on the context provided from another tracking sensor system. Figure 7 is a flow chart illustrating a process 700 for filtering possible identity predictions based on the provided sensor context, according to one embodiment.
[0054] The process 700 begins with receiving context data from another sensor system (block 702). For example, the context data may include a tracked object identity, a location of the tracked object, and a timestamp indicating that the tracked object was at a particular location.
[0055] The contextual data can be used to supplement the tracking capabilities of the tracking sensors. For example, viable candidate identities of a newly observed tracking object can be filtered (e.g., filtered in or out) based on contextual information indicative of the location of the identified tracking object provided by other tracking sensors (block 704). For example, if a target object with an identity of person A was tracked in a parking lot five minutes ago and the parking lot tracking sensor indicates that it will take 20 minutes to move from the parking lot to the coverage area associated with the tracking sensor that observed the new target object, then the identity of person A can be filtered out as a viable candidate identity of the new target object since it is not possible for person A to reach the coverage area from the parking lot in five minutes. Additionally and / or alternatively, filtering of candidate identities can occur by obtaining contextual data indicative of which identities could have reached the coverage area at the time the new target object was observed. These identities can be whitelisted as viable candidate identities of a new target object observed in the coverage area. This can save processing time and improve operation of computer systems employed in accordance with the disclosed embodiments.
[0056] Once the candidate identities are filtered, the tracking system can predict the identity of a new tracked object from the filtered candidate identities (block 706). For example, an identity can be chosen from a list of identities that are not filtered out (e.g., not on a blacklist) and / or can be selected from a list of identities that are filtered in (e.g., on a whitelist). As can be appreciated, the context information can reduce the number of candidate identities considered to identify a tracked object, thereby enabling more efficient object tracking. Furthermore, in some cases, identity tracking may not be possible without such context. For example, certain LIDAR systems may not be able to perform target object identification without such context. Thus, the present technology provides more efficient and improved tracking capabilities than conventional tracking systems.
[0057] Having discussed enhanced context tracking techniques, FIG. 8 is a schematic diagram illustrating an example control action 800 that can be accomplished based on an identity tracked via the context tracking techniques provided herein, according to one embodiment. In one embodiment, a kiosk 802 or other device can include a display 804. The display 804 can present a graphical user interface (GUI) 806 that provides information about the open environment. As shown, the GUI 806 can be personalized based on the tracked identity. For example, as indicated by the status bubble 808, the context tracking system can identify that the tracked object is "James" based on a trade-off of context data between various target sensors. In response to a tracked object approaching the display 804, a control action command can be presented on the display 804 (e.g., the underlying hardware that controls the display 804) instructing the GUI 806 to display information related to the identified person. Here, for example, James is provided with a personalized wait time associated with his identity. As can be appreciated, each identity can be associated with a unique identifier and other information (e.g., wait time, demographic information, and / or other personalized data) to perform personalized control actions. Additionally, as discussed with respect to FIG. 3, control actions can be accomplished based on identities corresponding to groups, as shown in FIG.
[0058] In another embodiment, the wearable device 810 can be placed on the tracked object 812. Based on the interaction with the open environment feature 814, here a virtual game mystery box, personalized data associated with the tracked object's identity can be updated. For example, here, the game database 816 can be updated via control action commands to reflect the updated game status based on the interaction with the open environment feature 814. Thus, as shown in database record 818, the identity 123 associated with James is provided with five additional points in the database. Additionally, the display 820 of the wearable device 810 can be controlled via control action commands to display a notification 822 confirming the status change based on the interaction of the tracked object 812 with the open environment feature 814.
[0059] As discussed above, the context tracking system can identify and track groups. Particular patterns of data indicative of group identification and / or tracking are discussed below, but these are merely examples. As can be appreciated, with machine learning, additional data patterns indicative of grouping can be observed by the machine learning system described herein. FIG. 9 is a flow chart of a method 900 for grouping individuals using machine learning techniques. One or more steps of the method 900 can be performed by one or more components of the context tracking system 102 of FIG. 1 or the context tracking system 216 of FIG. 2. Indeed, the grouping system (e.g., a machine learning system) can be integrated into the context tracking system 102 of FIG. 1 or the context tracking system 216 of FIG. 2. In particular, the grouping system can receive training data of patterns indicative of an active group (block 902). For example, the patterns indicative of an active group can include a proximity level of one or more persons at a particular time, a notice of an association of a vehicle with one or more persons, etc. In some embodiments, the patterns can include a notice of association of one or more persons (e.g., adults) and / or presumed children based on the height of the tracked individuals. Based on the received training data, the grouping system can become adept at detecting the presence of active groups, block 904. In particular, the grouping system can identify patterns in the raw data that are indicative of active groups in an environment (e.g., open environment 200).
[0060] In some embodiments, the time series data can be mined for patterns to identify and track groups. The time series data can provide data for a period of time that can help machine learning logic identify patterns of people over time. These patterns can be used to identify and / or track groups based on activities / patterns observed over time. For example, in an example of identification, a group of users can be identified as part of a group when they participate in a particular common activity or activities at a threshold time, which can be observed as a pattern from the time series data.
[0061] In another embodiment, the machine learning system can also be equipped to recognize that not all members of a group can be within a certain proximity to each other. For example, one or more persons of the active group can go to the bathroom, while one or more persons of the same active group do not go to the bathroom. In this case, the grouping system can retain the identity of all members of the active group and / or determine subgroups of the active group, where the subgroups can correspond to one or more persons in the active group. Indeed, following the above example, a first subgroup of the active group can be the members of the active group that go to the bathroom, and a second subgroup can be the members of the active group that do not go to the bathroom, while the first subgroup is in the bathroom. Furthermore, an active group can correspond to a group that remains in the database of detected groups for a certain time period. For example, an active group can remain active as long as one or more members of the group are present in the open environment 200 of FIG. 2 within a certain time period.
[0062] In block 906, the machine learning system may determine whether an individual or group (e.g., a subgroup) is at least part of an active group based on at least the training data and the identified patterns indicative of an active group. For example, the machine learning system may determine that two individuals who spent the past hour together are a group. As another example, the grouping system may group individuals who show affection for each other, such as holding each other's hands. Indeed, the grouping system may use weighted probabilities in determining whether one or more individuals should be grouped. Specifically, one or more behavioral patterns, characteristics, and / or attributes observed for a particular tracked identity may be weighted with a probability and utilized to determine a group. For example, the grouping system may attach a weighted probability (e.g., a value between 0 and 1) of belonging to a group to an individual of short stature, since infants and young children may not typically be without adult supervision. Nevertheless, the grouping system may use weighted probabilities for other attributes and / or behaviors of the individuals (e.g., common locations, proximity, association with the same car, etc.). The grouping system can sum the weighted probability values of being in a group and can determine that each individual with which the weighted probability value is associated is part of or forms a group. In an embodiment, one of the weighted probability values may be sufficient to exceed a threshold to determine that each individual with which the weighted probability value is associated is part of or forms a group.
[0063] At block 908, the grouping system can update the database containing active groups. As newly identified groups are determined, the database containing active groups can be updated and tracked. Thus, the machine learning system can keep track of who has been grouped. In some embodiments, grouping can affect downstream processing. For example, as discussed below, group tracking can be utilized in gaming experiences.
[0064] As described herein, the grouping system can identify groups of individuals based on activities / patterns observed in the time series data (e.g., as captured by sensors of a sensor system). Any number of patterns can be identified by a machine learning algorithm, which can implement supervised machine learning to train the grouping system how to identify groups. For discussion purposes, FIG. 10 is a schematic diagram 1000 that provides an example of a set of individuals deemed likely to be a group and / or unlikely to be a group by a grouping system using the techniques described herein. Specifically, as shown in diagram 1002, individuals who enter and exit the same vehicle can be deemed likely to be grouped by the grouping system. In contrast, as shown in diagram 1004, individuals who enter and exit different vehicles with a relatively large distance between the vehicles can be deemed unlikely to be grouped. Furthermore, as shown in diagram 1006, individuals who are in close proximity to each other for a significant amount of time are likely to be grouped. However, widely spaced individuals who can pass a certain threshold distance in the queue of the attraction can be deemed unlikely to be grouped. For example, in diagram 1008, each person 1011 represents 20 individuals in a line. As shown in diagram 1008, individual 1010 is separated from another individual 1012 in a line of 100 people for an attraction, and therefore the grouping system may determine that individual 1010 and individual 1012 should not be grouped together. However, in some cases, the grouping system may determine that individuals are likely to be grouped together even if they are spaced apart. As shown in diagram 1014, person 1016, person 1018, and person 1020 may be likely to be grouped together even if person 1016 is separated from persons 1018, 1020. In this case, persons 1018, 1020 are heading to the restroom 1022 while person 1016 is waiting outside.The grouping system can determine that the persons 1016, 1018, 1020 are a group even though the person 1016 may be alone for a period of time. Additionally, the grouping system can also group individuals who are separated but have spent time together past a certain threshold period of time. For example, a person in a group may choose to sit in a rest area rather than wait in a queue for an attraction with other persons from the same group. Thus, the grouping system can keep a group active for a period of time even if one or more members of the group are separated.
[0065] As can be seen, once a group is identified, additional functionality can be implemented. For example, FIG. 11 is a diagram 1100 illustrating how a group recognized by a grouping system can modify the interactive environment of an interactive game. In diagram 1100, compliance with rule 1102 on display 1103 is required to unlock door 1104 (e.g., a portal in a virtual and / or augmented reality experience). To enter door 1104, rule 1102 must be satisfied: "all team members must enter together." Here, the grouping system has identified persons 1114 and 1112 as part of person 1110's group / team. However, these persons 1114 and 1112 are not present with person 1110. Since not all team members are with team member 110 at one time, the door is not unlocked. Later, at time 1113, all team members (e.g., persons 1110, 1112, 1114) are present at door 1104. Effects occur from the interactive environment (e.g., door 1104 is unlocked) in response to the presence of the entire group. Thus, by tracking the group, one or more components of the interactive environment can be controlled based on aspects of the group (e.g., location of team members, accumulated activity of team members, etc.) to provide an enjoyable interactive experience for guests.
[0066] FIG. 12 is a schematic diagram illustrating an example control action 1200 that can be accomplished based on a group identity tracked via the context tracking techniques provided herein, according to one embodiment. Similar to FIG. 8, FIG. 12 includes a kiosk 1202, or another device can include a display 1204. The display 1204 can present a graphical user interface (GUI) 1206 that provides information about the open environment. As shown, the GUI 1206 can be personalized based on the tracked identity. For example, as indicated by the status bubble 1208, the context tracking system can identify that a tracked object is part of a tracked group "Group 1" based on a tradeoff of context data between various target sensors. In response to a tracked object approaching the display 1204, a control action command can be presented on the display 1204 (e.g., the underlying hardware that controls the display 1204) instructing the GUI 1206 to display information related to the identified group. Here, for example, Group 1 is provided with instructions associated with the identity. Specifically, the GUI 1209 reads "Hello Group 1, complete the task in under 10 minutes." As can be appreciated, each group identity can be associated with a unique identifier and other information (e.g., wait times, instructions, demographic information, and / or other personalized data), allowing personalized control actions to be performed.
[0067] In another embodiment, each member of the tracked group 1212 (i.e., group 1) can have a wearable device 1210 communicatively coupled to one or more components of the context tracking system. Based on interactions with the open environment feature 1214, here a virtual game mystery box, personalized data associated with the tracked group's identities can be updated. For example, here, the game database 1216 can be updated via control action commands to reflect an updated game status based on interactions with the open environment feature 1214. Thus, as shown in database record 1218, the identity 456 associated with group 1 has 15 points added in the database. Additionally, the display 1220 of each of the group members' wearable devices 1210 can be controlled via control action commands to display a notification 1222 confirming the change in status based on interactions with the open environment feature 1214 and each member of the tracked group 1212.
[0068] The technology presented and claimed herein refers to and applies to specific examples of a tangible and practical nature that clearly improve the technical field of the present invention, and is therefore not abstract, intangible, or purely theoretical. Moreover, when any claim appended hereto includes one or more elements designated as "means for performing a function" or "step for performing a function," such elements are intended to be construed under 35 U.S.C. 112, paragraph 6. However, for any claim that includes an element designated in any other manner, such elements shall not be construed under 35 U.S.C. 112(f).
Claims
1. A tangible, non-transitory, machine-readable medium, comprising: When executed by one or more processors of a machine, receiving a tracking target context of a first tracking object from a first tracking sensor system that tracks a first coverage area; providing the tracking target context from the first tracking sensor system to a second tracking sensor tracking a second coverage area different from the first coverage area, the second tracking sensor system being different from the first tracking sensor system; identifying a newly observed tracked target by the second tracking sensor system based on the tracked target context from the first tracking sensor system; filtering out candidates from a set of candidates that the newly observed tracked target can identify based on the tracked target context from the first tracking sensor system; and determining that the candidate was observed at a previous location outside a range of identified locations, the range of identified locations being a range within which the candidate is predicted to be reachable during a time difference between a time the candidate was observed at the previous location and a time the newly observed tracked target was observed by the second tracking sensor system; providing the set of candidates not including the candidate to the second tracking sensor system; and A machine-readable medium comprising machine-readable instructions that cause the machine to
2. When executed by the one or more processors of the machine, filtering a first subset of the set of candidates from the set of candidates in which the newly observed tracked target is not identifiable based on the tracked target context from the first tracking sensor system; identifying the newly observed tracked target from the set of candidates not including the first subset via the second tracking sensor system; 10. The machine-readable medium of claim 1 comprising machine-readable instructions that cause the machine to:
3. When executed by the one or more processors of the machine, generating a blacklist based on the first subset; providing the blacklist to the second tracking sensor system; 3. The machine-readable medium of claim 2 comprising machine-readable instructions that cause the machine to:
4. When executed by the one or more processors of the machine, generating a blacklist based on the first subset; providing the set of candidates not including the blacklist to the second tracking sensor system; 3. The machine-readable medium of claim 2 comprising machine-readable instructions that cause the machine to:
5. When executed by the one or more processors of the machine, filtering a second subset of the set of candidates from which the newly observed tracked target can be identified based on the tracked target context from the first tracking sensor system; identifying the newly observed tracked target from the second subset via the second tracking sensor system; 10. The machine-readable medium of claim 1 comprising machine-readable instructions that cause the machine to:
6. When executed by the one or more processors of the machine, determining the second subset as a portion of the set of candidates observed at a previous location within a range of an identified location, the range of the identified location being a range within which the candidates are predicted to be reachable during a time difference between a time the set of candidates was tracked at the previous location and a time the newly observed tracked target was observed by the second tracking sensor system; 6. The machine-readable medium of claim 5, comprising machine-readable instructions that cause the machine to:
7. When executed by the one or more processors of the machine, generating a whitelist based on the second subset; providing the whitelist to the second tracking sensor system; 6. The machine-readable medium of claim 5, comprising machine-readable instructions that cause the machine to:
8. the first tracking sensor system, the second tracking sensor system, or both, include a Light Detection and Ranging (LIDAR) system, a Radio Frequency Identification (RFID) system, a computer vision system, a Time of Flight (ToF) system, a millimeter wave (mmWave) system, or any combination thereof; The machine-readable medium of claim 1 .
9. the second tracking sensor system includes the LIDAR system; 9. The machine-readable medium of claim 8.
10. When executed by the one or more processors of the machine, determining a predicted confidence score indicating a confidence level of an identification of the newly observed tracked target; responsive to the predicted confidence score not meeting a confidence threshold, collecting additional tracking sensor system inputs for another identification of the newly observed tracked target; 10. The machine-readable medium of claim 1 comprising machine-readable instructions that cause the machine to:
11. When executed by the one or more processors of the machine, collecting additional tracking sensor system input by providing a direction to the newly observed tracked target to proceed into a coverage area of a tracking sensor system; 11. The machine-readable medium of claim 10 comprising machine-readable instructions that cause the machine to:
12. When executed by the one or more processors of the machine, Providing assistance by motivating the newly observed tracked target to proceed into a coverage area of the tracking sensor system.
12. The machine-readable medium of claim 11, comprising machine-readable instructions that cause the machine to:
13. the first tracked object comprises a different object type than the newly observed tracked target; The machine-readable medium of claim 1 .
14. the first tracking object includes a vehicle; the newly observed tracking targets include one or more people; 14. The machine-readable medium of claim 13.
15. When executed by the one or more processors of the machine, Receiving training data indicative of groups of persons and associated attributes; identifying patterns of the associated attributes to identify grouping attributes indicative of whether the persons should be grouped; determining whether the first tracked object is associated with the pattern; in response to determining that the first tracked object is associated with the pattern, identifying the first tracked object as part of an active group; 10. The machine-readable medium of claim 1 comprising machine-readable instructions that cause the machine to:
16. When executed by the one or more processors of the machine, determining a time that the first tracked object and the second tracked object spend in a threshold proximity to each other; associating the second tracked object with the active group when the time exceeds a threshold; determining whether the second tracked object is a member of the active group based on the pattern; 20. The machine-readable medium of claim 15, comprising machine-readable instructions that cause the machine to:
17. When executed by the one or more processors of the machine, associating with the second tracked object a first weighted probability representing a likelihood that the second tracked object is a member of the active group; associating with the second tracked object a second weighted probability representing a second likelihood that the second tracked object is a member of the active group; associating the second tracked object with the active group if the sum of the first weighted probability and the second weighted probability exceeds a threshold; determining whether the second tracked object is a member of the active group based on the pattern; 20. The machine-readable medium of claim 15, comprising machine-readable instructions that cause the machine to:
18. the first weighted probability being a value representing the time that the second tracked object spent in the vicinity of the first tracked object; 20. The machine-readable medium of claim 17.
19. receiving a tracking sensor system input from a first tracking sensor system tracking a first coverage area, the tracking sensor system input including a tracking target context of a first tracked object; providing the tracking target context from the first tracking sensor system to a second tracking sensor that tracks a second coverage area different from the first coverage area, the second tracking sensor system being different from the first tracking sensor system; identifying a newly observed tracked target by the second tracking sensor system based on the tracked target context from the first tracking sensor system; filtering out candidates from a set of candidates that the newly observed tracked target can identify based on the tracked target context from the first tracking sensor system; and determining that the candidate was observed at a previous location outside a range of identified locations, the range of identified locations being a range within which the candidate is predicted to be reachable during a time difference between a time the candidate was observed at the previous location and a time the newly observed tracked target was observed by the second tracking sensor system; providing the set of candidates not including the candidate to the second tracking sensor system; and 4. A computer-implemented method comprising:
20. determining a predicted confidence score indicating a confidence level of an identification of the newly observed tracked target; responsive to the predicted confidence score not meeting a threshold, collecting additional tracking sensor system inputs for another identification of the newly observed tracked target; 20. The computer-implemented method of claim 19, comprising:
21. the first tracked object and the newly observed tracked target include a first group of individuals; 20. The computer-implemented method of claim 19.
22. a first tracking sensor system configured to track a first tracking object in a first coverage area; a second tracking sensor system configured to track a second tracked object in a second coverage area different from the first coverage area; A context tracking system; Equipped with The context tracking system comprises: receiving a tracking target context for the first tracked object from the first tracking sensor system; providing the tracking target context from the first tracking sensor system to the second tracking sensor system different from the first tracking sensor system; identifying the second tracked object based on the tracked target context from the first tracked sensor system; filtering out candidates from a set of candidates that the second tracked object can identify based on the tracked target context from the first tracking sensor system; and determining that the candidate was observed at a previous location outside a range of identified locations, the range of identified locations being a range within which the candidate is predicted to be reachable during a time difference between a time the candidate was observed at the previous location and a time the second tracked object was observed by the second tracking sensor system; providing the set of candidates not including the candidate to the second tracking sensor system; It is structured to bring about the second tracking sensor system is different from the first tracking sensor system; system.
23. the first tracking sensor system includes a different sensor type than the second tracking sensor system; 23. The system of claim 22.
24. determining a range of identified locations within which the candidate is predicted to be reachable during the time difference includes including an estimated time the candidate will remain at the previous location; The machine-readable medium of claim 1 .
25. determining a range of identified locations within which the candidate is predicted to be reachable during the time difference includes including an estimated time the candidate will remain at the previous location; 20. The computer-implemented method of claim 19.
Citation Information
Patent Citations
Plural radar linkage system and its operation method and plural radar managing control device
JP2001174547A
Network radar system, radar, and central control device
JP2008139166A
Target detection apparatus and target detection method
JP2011112457A
Sensor integration device and sensor integration method
JP2014169872A
Position estimation system, position estimation method and program
JP2016219990A
