System and method for controlling interactive hybrid environment expressing motor sports event in track

Abstract

To control an interactive hybrid environment expressing a motor sports event in a track.SOLUTION: In a detection system for providing position data of one or multiple moving bodies operating in a match area in a central server, the detection system includes multiple sensor groups, the respective sensor groups are configured so as to monitor one portion of the match area, the sensor groups have multiple position detection devices disposed around the match area, the position detection devices are infrared sensors having a field of view (FOV), and sensor output of the infrared sensors or information originating from it are transmitted to another position detection device operating as a communication node of a sensor group. The detection system also includes communication facilities configured so as to transmit sensor output of the respective infrared sensors of a sensor group or information originating from it to a central collation server.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system and method for controlling an interactive hybrid environment representing a motorsport event on a track. More particularly, but not exclusively, the present invention relates to a method and system for enabling remotely located gaming computers and other imaginable entertainment devices to compete against real-life motorsport events. The present invention also extends to capturing and live broadcasting high-precision, real-time vehicle tracking and vehicle control data from live motorsport events, and using this data to provide new forms of gaming or viewing experiences that allow remotely located gamers to compete against actual participants in the motorsport event, or remotely located viewers to interactively compete with the event. Capturing accurate kinematic data allows the present system and method to be applied to competition in a range of other types of indoor and outdoor sporting events, such as soccer, basketball, cycling, and skiing. [Background technology]

[0002] Various approaches have been used to provide lifelike systems and methods for player-controlled virtual representations of motorsport events. Most of these are entirely virtual and use complex models that attempt to provide an appearance of realism by determining the kinematic behavior of a virtual vehicle in response to user actuator inputs. This realism is often provided by building models that use real kinematic data obtained by recording the vehicle's movement during a race. However, such models are not only complex and difficult to build, but are also often based on inaccurate kinematic data. Furthermore, these prior art approaches are often aimed at a single or small number of players, which means that the scope of interaction with other players is very limited. All of this results in simulation and gaming experiences that are not scalable and are unrealistic.

[0003] None of the known prior art attempts to define systems and methods for player-controlled simulations to interact with real, live racing in a practical way that would allow large numbers of players and followers worldwide to simultaneously compete against real drivers, cars, race teams, tracks, off-track racetracks, and event environments in a way that would prove satisfying and challenging for computer game players, support collective competition in e-sports tournaments, or offer this feature to non-gaming followers who desire a more immersive and enjoyable experience of live motorsports events. Achieving any of these objectives would broaden and enhance the appeal of, for example, auto racing to a wider fanbase that includes traditional television viewers or fans of famous drivers, through hardcore auto racing computer game fans, and to consumers of "over-the-top" racing and technical data services.

[0004] The present inventor is the author of prior art UK Patent No. 2518602B which describes a system, method and technique for accurately tracking cars in real-life live motor racing (e.g. Formula 1) whilst remaining passive and in all but the worst weather conditions, whilst providing computer gamers with the ability to virtually compete in a live race by replacing one of their cars with a virtual car. The data provided by UK Patent No. 2518602B includes information on the position of cars on the track. This concerns position information that is sensed non-invasively by a single IR tracking sensor placed at a considerable altitude (1-2 km) above the racing circuit, for example on a helicopter, drone, or light aircraft. This is based on the field of view (FOV) of the single sensor covering the entire competition area. Taking Formula 1 as an example, this approach is not feasible when the FOV is obstructed by vegetation (growing trees) or the ground's spectator seats, or in the case of urban racing circuits, by all buildings and other structures. Therefore, this prior art approach is somewhat limited in its practical applications.

[0005] Video games that interact with live events are known (see, e.g., U.S. Patent Application Publication No. 2010 / 0271367), which describes an in-game method in which a player-controlled virtual car interacts with a representation of a real car in a limited and somewhat artificial way via data streamed from the live event to a conventional computer racing game. The performance of the virtual car is determined by a combination of player input, a software model of the car, and its environment. This physics-based model, which is part of the video game software, is subject to many limitations in the fidelity it can achieve in attempting to simulate the highly complex and dynamic scenarios typical of auto racing events. This results in poor quality interaction between the virtual car and the representation of the real car.

[0006] Other prior art documents (e.g., GB 2365360A) attempt to solve this major deficiency by suggesting that a physics model simulation of the dynamics of a virtual car and its environment in a computer game can be pre-defined by data collected from actual driving, or can be defined in real time from performance data transmitted live from the real car and its environment, generating an "optimal physics model" of the car and its environment that can then be controlled by the game player. This approach suffers from the same fundamental limitation in that there is no realistic means described for achieving very high performance tracking. Furthermore, software models of real-life scenarios involving very complex cars operating in very complex environments involve so many variables and such complex relationships between variables that even when some of the more obvious variables are continuously measured and used, the model a) Deviating from reality very quickly due to many approximations and lack of models, or b) They require enormous amounts of computing power and in some cases cannot operate in real time even on the largest computers.

[0007] Still other prior art documents (e.g., U.S. Pat. No. 6,155,927) describe in abstract form a system that enables computer game players to compete in live and recorded real-world races, but make no mention of the specific methods and systems described in GB Patent No. 2,518,602 B that would make the abstract concept practically realizable.

[0008] GB 2585165 A, co-authored by the present inventor, describes an approach to infrared (IR) tracking of passenger cars, heavy trucks, and the like on highway, roadway, and street traffic networks. Here, tracking devices are placed on low-altitude infrastructure such as lampposts, and IR emitters are placed on the vehicle to enable vehicle detection, or alternatively, IR reflectors are placed on the vehicle and optionally, the tracking device is equipped with an IR lamp. The tracking devices then communicate highly accurate, real-time tracking data for the vehicle and its vicinity to the vehicle, aiding in its navigation and autonomous or semi-autonomous driving. The tracking devices are connected to each other to form a reliable linear network, allowing vehicles to be tracked along a continuous, uniform stretch of roadway. The tracking device placement described in GB 2585165 A has features unrelated to the present invention (e.g., reliable data collection and distribution from the tracking devices to the vehicles themselves). Furthermore, motorsport events are typically used for transporting large amounts of data. Generally, this requires tracking high performance vehicles such as cars, motorcycles etc. along closed, non-homogeneous circuits with challenging geometries (sharp curves, obstacles, inclines, bumps, pit stops etc.) on which motor vehicles travel at extremely high accelerations, speeds, cornering speeds etc., and so the systems and methods described in GB 2585165 A, which are generally designed to monitor slower moving vehicles, would not be able to cope well.

[0009] Therefore, it is an object of the present invention to overcome the limitations of the above-mentioned prior art documents. In another embodiment, it is an object of the present invention to overcome the limitations regarding accurate tracking capabilities set forth in GB Patent No. 2518602B and described above, and to provide, for the first time, a practically feasible and improved system and method for interactive real-to-virtual car racing mass competitions for high-performance motorsport events in complex environments, with a large number of players and followers from around the world simultaneously. In another embodiment, it is an object of the present invention to identify how virtual drivers can interact with real races and real drivers to enhance the player experience, enabling local, regional, national, or global e-sports or car racing events to be conducted fairly while being highly integrated with real motorsport events and competitions. The present invention is also applicable to the generation and use of recorded data for real motorsport events, in which the kinematic data and vehicle control data (and, in some cases, driver input) of competing vehicles are recorded or reconstructed to adhere to the accuracy and latency requirements for computer gaming set forth in GB Patent No. 2518602B. Summary of the Invention

[0010] According to one aspect of the present invention, there is provided a computer-implemented method for controlling an interactive hybrid environment representing a motorsport event on a track, the interactive hybrid environment including representations of real and virtual vehicles on the track, the method comprising the steps of receiving a stream of real data including real kinematic data of the real vehicle on the track captured by infrared sensors at the track and real control data relating to a driver's control of the real vehicle captured by vehicle sensors and obtained from the real vehicle via a telemetry system, determining a position and kinematic behavior of the representation of the real vehicle within the interactive hybrid environment using the real kinematic data, using the real control data and the real kinematic data to generate a black-box determination of the position of the real vehicle on the track based on the real control data, receiving a stream of computer-generated control data resulting from user interaction with a computer that presents the interactive hybrid environment to the user and captures user inputs to control the kinematic behavior of the virtual vehicle representation, and determining the position and kinematic behavior of the virtual vehicle representation within the interactive hybrid environment by using the black-box determination and the computer-generated control data.

[0011] The use of real data to include both kinematic and control data in an interactive hybrid environment allows accurate black-box decisions to be made. This, in turn, provides a reference for the received computer-generated control data, enabling lifelike and accurate kinematic control of the virtual vehicle in relation to a representation of the real vehicle on the track. The degree to which the user manipulates the actuators is accurately reflected in the effect it has on, for example, the virtual vehicle, in the same way that the driver's manipulation of the controls in the real vehicle affects the kinematic data (e.g., speed, heading, acceleration, and position) of the real vehicle. This provides a level of realism not possible with prior art approaches, allowing movements in both the virtual and real domains to be accurately reflected. Furthermore, this combination of features allows representations of vehicles from different domains to be presented in the same hybrid environment in an equally realistic and accurate manner. The present invention overcomes interoperability challenges when real data is live data, i.e., data is streamed from events occurring simultaneously with the control of an interactive hybrid environment, for example, the present invention allows a virtual driver in a virtual domain to compete in real time against a real driver in a real domain, which has not previously been possible.

[0012] Preferably, the real sensor data includes real kinematic data of multiple real vehicles on the track and real control data relating to the control of each of the multiple real vehicles by their drivers. Embodiments of the present invention are configured to enable the competitive behavior of multiple vehicles to be captured holistically, for example, recognized in a Formula 1 race. Having streams of real data representing the behavior of multiple real vehicles makes it possible to generate such a competitive environment. Each real data stream can be processed separately and may include a unique vehicle identifier.

[0013] Similarly, the stream of computer-generated control data may include multiple streams of computer-generated data, each stream generated by a different user interaction with the computer and capture of each user input. This allows multiple users to be associated with a sporting event, advantageously allowing a team game to be realized. Each computer-generated data stream may be separately processable and may include a unique computer device identifier.

[0014] Preferably, the multiple real vehicles are fewer than the multiple streams of computer-generated data, and the method further includes associating a subset of the multiple representations of the virtual vehicles with a representation of a real vehicle to generate the associated representation. Obviously, in a swarm gaming environment, the number of users participating via gaming computers / devices significantly exceeds the number of vehicles in the race (because there are typically physical limitations on the number of vehicles that can participate in a race for safety reasons). Therefore, by associating representations of two or more virtual vehicles with a single representation of a real vehicle, it is possible to accommodate any number of users. This makes it possible to extend this method to swarm gaming scenarios where hundreds of thousands of users in the virtual domain can simultaneously compete against drivers in the real domain.

[0015] In some embodiments, the method further includes using the associated representations to represent a subset of the plurality of representations of the virtual vehicle within the interactive hybrid environment while the positions of the subset of virtual vehicles are within tolerance limits of the real vehicle.

[0016] The multiple streams of computer-generated data are typically multiple times greater than the multiple real vehicles in many embodiments, and the associating step may include associating each of the multiple streams of computer-generated data with multiple representations of the real vehicles with a mean distribution. In other embodiments, where the multiple streams of computer-generated data are multiple times greater than the multiple real vehicles, the associating step may include associating each of the multiple computer-generated data with multiple representations of the real vehicles with a logarithmic distribution. In either case, it is possible to accommodate a large number of players / users in an interactive hybrid environment where only a small number of real vehicle representations are present.

[0017] In one embodiment, it is possible to have a central gaming server that generates the interactive hybrid environment and then provides it to all gaming devices over a communications network, in which case the method includes the steps of updating the interactive hybrid environment with new positions of the representations of the real and virtual vehicles as determined by the received real sensor data and computer-generated data, and displaying the updated interactive hybrid environment. The method may further include generating the hybrid environment and broadcasting the updated interactive hybrid environment from a central server to multiple remotely located computers. The primary solution may require more processing power but is relatively easy to update and control.

[0018] In an alternative embodiment, each user / player's gaming device locally generates the interactive hybrid environment, and there are multiple such generated local environments. In this case, the method may further include broadcasting the black-box determinations and real sensor data from a central server to multiple remotely located computers, generating the interactive hybrid environment at each remotely located computer, updating the interactive hybrid environment with new positions of the representations of the real and virtual vehicles as determined by the received real sensor data and computer-generated data, and transmitting the new positions of the representations of the virtual vehicles to the central server. Such a distributed solution may require more management, but does not suffer from potential failures and is generally not subject to time delays or lags in the generation of the interactive hybrid environment.

[0019] In some embodiments, the method further includes using an artificial intelligence engine that references black-box adjudication to vary the association between the computer-generated control data and the resulting position of the virtual vehicle. The use of such an artificial intelligence engine allows assistance to be provided to each player in controlling their virtual vehicle. Such assistance can provide a handicap factor that allows weaker virtual gamers to compete fairly with professional drivers in the real world.

[0020] In most embodiments, the received real kinematic data includes longitudinal position data relative to the track, lateral position data relative to the track, and vehicle heading data relative to the track. These types of data allow the vehicle's kinematic behavior to be accurately mapped into the virtual domain.

[0021] In some embodiments, the real-world control data includes one or more of the steering wheel position, accelerator position, brake pedal position, and gear selection of the real vehicle. These are exemplary control data, the information of which is provided by a telematics system, and the control data assists in determining driver control inputs that determine the track position of the real vehicle. Advantageously, these data can also be easily associated with corresponding actuators that can be controlled by a player in the virtual world.

[0022] Different embodiments generate the interactive hybrid environment from real data from different sources. In one embodiment, the source is a store of previously recorded real data. Thus, the method may further include retrieving the real sensor data from a data store that stored a copy of the real sensor data at the time the real sensor data was generated. In an alternative embodiment, the source is the sporting event itself, in which case the receiving step includes receiving the real sensor data in near real time as the sporting event is taking place.

[0023] In one embodiment, the stream of real sensor data has a sampling rate of at least 25 Hz, and the position of the real vehicle is captured at a point in time and provided to the interactive hybrid environment within 40 milliseconds of being captured. This allows for real-time realization of the real domain within a virtual domain operating at a minimum refresh rate of 25 Hz. More specifically, in some embodiments, the stream of real sensor data has a sampling rate of at least 60 Hz, and the position of the real vehicle is captured at a point in time and provided to the interactive hybrid environment within 40 milliseconds of being captured. Positions are captured and provided to the interactive hybrid environment within 16.7 milliseconds of being captured, a refresh rate that is typically found on most computer monitors and therefore supports high-quality representation of real-time events within the virtual domain.

[0024] The method may further include using the stored data model to assist in generating the interactive virtual environment, which may make the virtual environment more realistic for the player.

[0025] Additionally, video and audio data streams may be received from a real vehicle to enhance the interactive hybrid environment, in which case the method further comprises receiving a stream of video data or audio data from the real vehicle and including the stream of video data or audio data in the interactive hybrid environment.

[0026] To facilitate collective gaming, in some embodiments, the method further includes associating a representation of one of the plurality of virtual vehicles with a representation of one of the plurality of real vehicles when the position of the representation of the virtual vehicle is within a predetermined threshold of the position of the representation of the real vehicle, and using the representation of the real vehicle as the representation of the virtual vehicle in the interactive hybrid environment. This advantageously allows a vast number of virtual vehicles to be included in the interactive hybrid environment without cluttering the screen with a vast number of representations of virtual vehicles. Indeed, this form of representation solves the technical problem of how to present a motorsports event that potentially includes millions of players within a limited screen size. Furthermore, some of the embodiments solve the problem of connecting millions of computer game players and other fans worldwide live with a motorsports event. This can provide a challenging, satisfying, and entertaining interactive experience for gamers and viewers, allowing all players to be fairly managed and ranked as an integrated part of the real-to-virtual auto racing event, which is important for e-sports events that are increasingly attracting professional gamers.

[0027] In some embodiments where the audio and / or video data is provided from a real vehicle, the associating (also referred to herein as "snapping") step may operate by providing the stream of audio or video data received from the real vehicle to a computer that presents the interactive hybrid environment to the user. This allows the sights and sounds experienced by a particular real vehicle to be provided within the virtual domain, making the interactive hybrid environment more realistic.

[0028] In some embodiments, the method further includes disassociating the representation of one of the plurality of virtual vehicles from the representation of one of the plurality of real vehicles when the position of the representation of the virtual vehicle is outside a predetermined threshold of the position of the representation of the real vehicle, and presenting the representation of the virtual vehicle separately from the representation of the real vehicle within the interactive hybrid environment, which allows the representation of the virtual vehicle to be displayed when it does not match the representation of the real vehicle, thereby making the actual position of the virtual vehicle relative to the representation of the real vehicle visible to the player and enabling transitions between representations of real vehicles within the interactive hybrid environment.

[0029] In some embodiments, aspects of the interactive hybrid environment may be relayed to a team associated with a real vehicle, as described below, in which case the method may further include providing details of any virtual vehicle associated with the representation of the real vehicle to a remotely located third party computer.

[0030] In some embodiments, the performance of the virtual vehicle can be matched with the performance of a representation of a nearby real vehicle. A fairer game is possible if each real vehicle has a different set of performance characteristics. In this embodiment, the method further includes determining a representation of a real vehicle from among the multiple real vehicles that is located closest to the representation of the virtual vehicle, and introducing the set of performance characteristics of the closest real vehicle representation as performance characteristics of the virtual vehicle.

[0031] In some embodiments, the method further includes capturing position data of a real vehicle on the track using an infrared sensor, converting the position data into a stream of real kinematic data over time, and transmitting the stream of real kinematic data in real time to a central server.

[0032] Preferably, the location data is captured using a group of sensors monitoring different locations on the track, with each sensor in each group detecting infrared radiation reflected or transmitted from one or more vehicles operating on the track within the sensor's field of view (FOV). This configuration is particularly advantageous for providing accurate real-time information for the interactive hybrid environment described below.

[0033] In some embodiments, the method further includes processing the infrared emissions detected by the infrared sensors to determine kinematic data of one or more real vehicles operating on the track. Preferably, this processing is performed at each sensor, allowing a small amount of data to be transmitted for use in the interactive hybrid environment.

[0034] According to another aspect of the present invention, a computer system for controlling an interactive hybrid environment representing a motorsport event on a track is provided. the interactive hybrid environment includes a representation of a real vehicle on a track and a virtual vehicle, the system including a receiver for receiving a stream of real sensor data, the real sensor data including real kinematic data of the real vehicle on the track and real control data relating to control of the real vehicle by a driver, the real kinematic data being captured by infrared sensors at the track and the real control data being captured by vehicle sensors and obtained from the real vehicle via a telemetry system; and a virtual race command processor configured to receive a stream of computer-generated control data resulting from user interaction with a computer that presents the interactive hybrid environment to the user and captures user inputs to control the kinematic behavior of the representation of the virtual vehicle; a virtual race simulation engine, the virtual race simulation engine including: a race simulation output engine for determining the position and kinematic behavior of a representation of a real vehicle within an interactive hybrid environment using real kinematic data; a reference black-box model generator configured to use the real control data and the real kinematic data to generate a black-box determination of the position of the real vehicle on the track based on the real control data; and a gaming black-box implementation engine configured to determine the position and kinematic behavior of a representation of a virtual vehicle within the interactive hybrid environment by using the black-box determination and the computer-generated control data.

[0035] The computer system may further include an artificial intelligence engine configured to vary the correlation between the received computer-generated control data and the final position of the virtual vehicle. The artificial intelligence engine may be configured to increase a threshold required from the received computer-generated control data to generate a predetermined position of the virtual vehicle.

[0036] As described in more detail below, some embodiments of the present invention relate to systems and methods that allow millions of players of motorsports computer games worldwide to simultaneously interact with real, live motorsports events for entertainment purposes, including, but not limited to, computer gaming, e-sports tournaments, streaming, spectating, gambling, and generally enhancing fan engagement with enhanced motorsports. One embodiment allows a computer game player to initiate a race by digitally pairing with one of the real cars, transitioning from car to car or acting as an additional car, depending on certain parameters. It will be understood that the terms "car" and "vehicle" are used interchangeably herein and have a broader meaning of any vehicle. When pairing with a car, dynamic black-box simulation methods can be used to closely and reliably match the performance of the virtual car in its environment, thus providing realistic, challenging, fun, and fair content between the computer game player and the real driver. The system and method for one player then forms the basis of an expanded system and method for millions of players to simultaneously and competitively interact with the real event. The systems and methods may be used to enable participants who simply watch motor racing, rather than playing computer games, to become more involved and interact with the motor sport event.The systems and methods may be used to enhance the engagement of viewers and computer gamers with a wide range of sports.

[0037] Broadly, embodiments of the invention relate to improvements over known prior art for capturing and live broadcasting highly accurate real-time vehicle tracking data in motor car racing (e.g., Formula 1), thereby enabling truly fair, competitive, and entertaining races between computer gamers and professional drivers, as well as enabling a wide range of other spectator and entertainment enhancement features. The improvements include capturing and live broadcasting tracking data from any motorsport venue of a significant number of high-performance vehicles competing in a live event, such that the broadcasted data is real-time, sufficiently accurate, and in a format suitable to enable computer games and other entertainment media to integrate with and utilize the data, whether live or previously recorded, thereby enhancing the computer gaming experience and / or providing additional benefits to motorsport organizers and fans, such as user-selectable viewpoints, tailored streaming, user-focused advertising, live gambling, etc. In certain embodiments, while competing in live, connected races, gamers may change from the real car selected at the start of the race to a different real car during the race, or to an additional car, operating as a predetermined set of specific parameters, thereby allowing the gamer to match their capabilities with the drivers of other cars at any location the gamer finds themselves in during the race. Upon selecting a new real car, the gamer's car's capabilities are matched to the new real car, again establishing fair and competitive racing with the new real drivers. The systems and methods described herein thus extend to allowing an unlimited number of gamers to participate in an engaging, fair, and competitive manner. While in certain circumstances, gamers may also be able to view and interact with other virtual cars, in all circumstances, the systems and methods enable fair and competitive racing between an unlimited number of gamers in a virtual environment and real drivers in a real environment. Together, these are referred to as an interactive hybrid environment.The systems and methods described herein are applicable to other sporting events, a number of representative examples of which are described.

[0038] Thus, embodiments of the present invention advantageously provide an interactive reality-virtual car racing A system and method of operation for an event is provided, whereby, when using the system and method, any number of participants using a virtual environment (interactive hybrid environment): a) By competing as individual virtual drivers of all experience and skill levels, and between real drivers and possibly a limited number of other virtual drivers (any number up to the total number of real drivers), in a realistic, satisfying, challenging and fair manner in compatible live motorsport events; or b) By competing in a massively multiplayer e-sports computer gaming event or any other (often self-managed among friends or associates) computer gaming event or activity that is highly integrated with a live competitive motorsport event in a realistic, satisfying, challenging and fair manner; or c) By enhancing their live viewing or live streaming experience by being included as an observer using the systems and methods of the present embodiments; It is possible to interact with real-life live events and real-life drivers.

[0039] According to another aspect of the present invention, there is provided a detection system for providing position data of one or more mobile objects operating in a playing area to a central server, the detection system including a plurality of sensor groups, each sensor group configured to monitor a portion of the playing area, each sensor group comprising a plurality of position detection devices arranged around the playing area, each position detection device configured to monitor a portion of the playing area different from a position above the playing area, each position detection device including an infrared sensor having a field of view (FOV) for detecting infrared radiation emitted, reflected, or transmitted from one or more mobile objects operating over the playing area within the FOV and generating a sensor output, a transmitter configured to transmit the sensor output of the infrared sensor or information derived therefrom to another one of the position detection devices in the sensor group that operates as a communication node for the sensor group, and communication equipment communicatively coupled to the position detection device operating as a communication node in the sensor group, the communication equipment configured to transmit the sensor output of each infrared sensor of the sensor group or information derived therefrom to a central verification server.

[0040] In some embodiments, each sensor group includes 10 or fewer location sensing devices. Minimizing the number of location sensing devices advantageously ensures an optimal balance between data transmission latency and system complexity due to the need for additional communication paths.

[0041] In some embodiments of the group, at least one of the position sensing devices of the sensor group includes a processor configured to determine current kinematic data in at least two dimensions of one or more moving bodies operating over the competition area within the FOV based on the sensor output or information derived therefrom. This feature can significantly improve the reduction in the amount of data to be transmitted around the system, and can thereby increase the information transmission rate because the sensor output is processed before being transmitted to the communication node.

[0042] In some embodiments, a first sensor group of the plurality of sensor groups is configured to relay sensor output or information derived therefrom determined by the first sensor group to a second sensor group of the plurality of sensor groups.

[0043] In various embodiments, one or more of the plurality of position sensing devices may include a long wave infrared (LWIR) microbolometer or a mid wave infrared (MWIR) photon detection camera configured to detect thermal IR emitted by one or more real moving objects on the playing area. One or more may include a short wave infrared (SWIR) or near infrared (NIR) photon detection camera for detecting broadband or narrowband light emitted, reflected, or transmitted from the moving object.

[0044] Preferably, to accurately detect very fast moving vehicles, one or more of the position sensing devices may have a frame rate of at least 60 Hz, more preferably at least 100 Hz, which compares favorably with minimum gaming refresh rates, which are typically between 25 Hz and 60 Hz.

[0045] Preferably, in some embodiments, one or more of the plurality of location sensing devices are configured to detect a unique identifier of a mobile object based on an infrared signature. This is extremely useful for tracking purposes where there are multiple mobile objects being tracked within the same FOV of the location sensing device. The unique identifier may be a modulated IR signal from an emitter on the mobile object, with each different mobile object having a different modulated signal.

[0046] Another optional feature of one or more of the plurality of position sensing devices includes an LED floodlight directed toward a portion of the playing area and configured to detect reflected light from the LED floodlight. Such lighting of the real vehicle provides more resilience in low-light and inclement weather conditions. In some embodiments, one or more of the plurality of position sensing devices is configured to detect an infrared signature of the vehicle, which comprises modulated infrared light. Such modulation provides more resilience to variable environmental factors and advantageously allows for more accurate distance measurements.

[0047] To assist in determining relative position, one or more of the multiple position sensing devices may be configured to detect infrared radiation reflected or emitted at the edges of the playing area, and the system can use the detected information as a frame of reference to determine the lateral position of the moving object.

[0048] In some embodiments, the position sensing devices may be oriented to face an approaching real vehicle. They may be positioned at angles relative to the horizontal and vertical directions to capture the infrared signature of the real vehicle. More specifically, in these embodiments, at least some of the position sensing devices are positioned to have a field of view (FOV) boresight (centerline) at an acute angle relative to the horizontal and vertical planes and, in use, to face an approaching vehicle as the vehicle advances through the competition area. Different configurations are possible, and thus, in some embodiments, at least some of the plurality of position sensing devices include a 20-30 degree FOV and a detection range of up to 50 meters. In other embodiments, at least some of the plurality of position sensing devices include a 70 degree FOV and a detection range of up to 15 meters.

[0049] In some embodiments, the system further comprises a GPS receiver that provides a time stamp for the sensor output or information derived therefrom, and the system is configured to use the time stamp to establish a common time reference for the sensor output or data derived therefrom from at least some of the location sensing devices.

[0050] As mentioned above, in some embodiments, the communications facility is configured to operate at a minimum refresh rate of 25 Hz to provide sensor output or information derived therefrom of one or more mobile objects operating on the competition area to a central server. Using this refresh rate provides sufficient resolution data to enable tracking of high speed vehicles (e.g., operating at up to 220 mph) and providing these kinematic data to an interactive hybrid environment. However, in a more preferred embodiment, the communications facility is configured to operate at a minimum refresh rate of 25 Hz to provide sensor output or information derived therefrom of one or more mobile objects operating on the competition area to a central server. Using this refresh rate provides sufficient resolution data to enable tracking of high speed vehicles (e.g., operating at up to 220 mph) and providing these kinematic data to an interactive hybrid environment. The monitor is configured to operate at a 60 Hz refresh rate and provide sensor output or information derived therefrom of one or more moving objects operating on a playing area to a central server. The use of this refresh rate is consistent with that of most computer gaming monitors, and thus helps to provide a lifelike, realistic representation of the movement of moving objects within, for example, an interactive hybrid environment.

[0051] The processor of each position sensing device, in some embodiments, is configured to determine a longitudinal position along the playing area, a lateral position across the playing area, and a rotational orientation of the moving object.

[0052] In some embodiments, multiple location sensing devices in a sensor cluster are arranged in a series, with a location sensing device located at the midpoint of the series acting as a communication node for the sensor cluster. This configuration reduces communication hops between location sensing devices to communication nodes. Thus, for example, in a cluster of nine sensors, the fifth sensor is the communication node, and the maximum number of hops for the sensor output or information derived therefrom of any location sensing device to reach the communication node is four hops.

[0053] In a motorsport event embodiment, the moving object comprises a vehicle and the competition area comprises a track.

[0054] According to another aspect of the present invention, there is provided a system for generating and controlling an interactive hybrid environment representing a motorsport event on a track, the interactive hybrid environment including representations of real and virtual vehicles on the track, the system including a combination of the computer system described above and the sensing system described above.

[0055] According to another aspect of the invention, there is provided a computer-implemented method for determining updated positions of one or more user-operated virtual vehicles on a virtual representation of a track using data from one or more physical vehicles on the track, the method including the steps of: associating, in a processor, each of the one or more user-operated virtual vehicles with one of the one or more physical vehicles; receiving, in the processor, initial position data indicating positions of the one or more physical vehicles on the track at a first time; determining, in the processor, the initial position data for each of the one or more user-operated virtual vehicles on the virtual representation of the track based on the initial position data of the physical vehicles with which it is associated; continuously receiving, in the processor, position data indicating positions of the one or more physical vehicles on the track at a second time, driver input data for each of the one or more physical vehicles, and user inputs for controlling operation of the one or more user-operated virtual vehicles; and determining, in the processor, positions of the one or more user-operated virtual vehicles at a third time based on the continuously received position data, driver input data, and user inputs.

[0056] Some embodiments of the present invention provide tracking capabilities with data latency and coordination of data from multiple vehicles required by vehicles moving at extremely high speeds around heterogeneous racing circuits, particularly at racing events that may be held worldwide in any type of environment, whether city, urban, or metropolitan. Specifically, these embodiments identify how tracking of real-world drivers can be achieved simultaneously around complex, cluttered city-based tracks, where tunnels, adjacent buildings, and other obstacles can hinder tracking with single-line-of-sight, high-altitude IR sensors. Systems embodying the present invention feature a specific architecture of computer, communications, and sensor hardware, including computer software, all of which are independently configured. Alternatively, the system may be configurable and advantageously deployable across a wide range of motorsport venues, allowing all competing vehicles to simultaneously track around complex racing circuits in a variety of configurations and environments tailored to push the vehicles to their performance limits. Each architecture of equipment and software tailored to a particular racing circuit can deliver a single, real-time, dynamic data stream indicating the precise positions of all real competing vehicles within a highly representative computer-based model that is, at least in shape, a real racing circuit. This data stream is suitable for accuracy, latency, and potentially other aspects of representational fidelity when simultaneously distributed to millions of computer gaming devices via the Internet or other communications technologies. Computer gaming devices implement some or all of the methods described below and typically include any range of computer hardware traditionally used by participants to play motorsport-related computer games or to experience motorsport events or recordings via digital data streams provided by commercial or other suppliers.

[0057] The present embodiment provides an enhancement to the technology described in GB Patent No. 2,585,165A below, which addresses the difficult challenge of tracking high performance motorsport vehicles in complex environments, and is configured in an architecture that creates a single integrated real-time data stream containing high accuracy, real-time tracking data for all competitors simultaneously, suitable for broadcast and use in gaming and other entertainment environments. While the IR detection capabilities of the tracking device described herein include that described in GB Patent No. 2,585,165A, the inclusion of the thermal IR tracking technology described in GB Patent No. 2,585,165A in this application goes beyond the technology described in GB Patent No. 2,518,602B, enabling the ability to track motorsport vehicles based solely on their thermal IR signature.

[0058] The features of the above-described embodiments can be combined in various ways and, unless otherwise stated, can be added to the specific description of the embodiments of the invention presented below. [Brief explanation of the drawings]

[0059] In order that the invention may be more readily understood, reference will now be made by way of example to the accompanying drawings, in which: [Figure 1] 1 is a schematic diagram illustrating a three-part gaming environment including multiple gaming devices, a gaming server, and a live event data capture system, according to one embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing the gaming server of FIG. 1 in more detail. [Figure 3a] FIG. 2 is a schematic block diagram illustrating the format of real data received from the live event data capture system of FIG. 1. [Figure 3b] FIG. 2 is a schematic block diagram illustrating the format of virtual race commands received from the gaming computer / device of FIG. 1. [Figure 4] 2 is a flow diagram illustrating a method of operation of the gaming server of FIG. 1. [Figure 5] 2 is a schematic diagram illustrating a gaming and entertainment server according to another embodiment of the present invention, which replaces the gaming server of FIG. 1. [Figure 6] FIG. 1 is a plan view of the Monaco Grand Prix route map, showing the locations of different types of IR sensors. [Figure 7a] 1 is a schematic diagram illustrating two possible strategies used in the present embodiment for generating a highly representative, computer-based data model of a racing circuit. FIG. [Figure 7b] 1 is a schematic diagram illustrating two possible strategies used in the present embodiment for generating a highly representative, computer-based data model of a racing circuit. FIG. [Figure 8]1A and 1B collectively show a three-dimensional diagram illustrating an array of sensors around a racing track and their respective fields of view, according to one embodiment of the present invention. [Figure 9] FIG. 2 is a three-dimensional view showing the communication arrangement between the sensors of FIG. 1 and their respective fields of view around the racing track in more detail. [Figure 10a] 1 is a schematic plan view showing the initial configuration of a Formula 1 motor race associated with a single player video game, including 24 real cars and 24 real drivers, each of which has been assigned to its starting position or selected by the player as its starting position. [Figure 10b] FIG. 1 is a schematic plan view showing a representation of a real vehicle on a track and a player's simulated vehicle, the player's simulated vehicle deviating from the real vehicle representation by a known amount in instantaneous position and possibly in other attributes such as speed, velocity and acceleration. [Figure 11a] FIG. 1 is a schematic plan view of a racing situation in which a simulated car is positioned between the three nearest representations of a real car. [Figure 11b] 11b is a schematic plan view of a continuation of the racing situation of FIG. 11a, showing a situation a little later when the simulated car has moved forward and is approaching the leading real car representation. [Figure 11c] 11b is a schematic plan view of the racing situation of FIG. 11b after a longer time interval, in another part of the circuit, with the simulated car approaching the leading real car representation. [Figure 12a] 1 is a schematic plan view of a starting location showing the starting configuration for a race linked and integrated live with an eSports or online gaming race with 10 million video game players, including 24 real cars evenly distributed at the starting location. [Figure 12b]12a is a schematic plan view of the starting position showing an alternative starting configuration to that shown in FIG. 12b, in which 100 million video game players are logarithmically distributed and only the highest ranked virtual car is associated with a leading real car. [Figure 13] FIG. 1 shows a racing situation up to the middle of a race between 24 real cars and a number of computer game players, each associated with one real car. [Figure 14] 1 is a graph showing various parameters transmitted by a live event data capture server over time to illustrate the principles of the black-box dynamic simulation method; [Figure 15] FIG. 3 is a schematic diagram showing the virtual racing simulation engine of FIG. 2 in more detail. [Figure 16] 2 is a schematic block diagram of the gaming device of FIG. 1 in a distributed processing system according to a further alternative embodiment. [Figure 17] 17 is a schematic block diagram of the gaming server of FIG. 1 of a distributed processing system for use with the gaming device of FIG. 16. [Figure 18a] 17 is a schematic block diagram illustrating the format of real data received from a live event data capture system for use in the gaming device of FIG. 16. [Figure 18b] 17 is a schematic block diagram illustrating the format of virtual car position data received from the gaming device of FIG. 16. [Figure 19] FIG. 18 is a schematic block diagram illustrating the virtual race management engine of FIG. 17 in more detail. [Figure 20] 17 is a flow chart showing a method of operation of the gaming device of FIG. 16. DETAILED DESCRIPTION OF THE INVENTION

[0060] All of the computer gaming methods described below involve each computer gaming element of the system (including the game player's computer / tablet / phone, etc. and / or remote server hardware and all associated software) controlling a motor. Assume that a game system can provide traditional features typical of a sports-related game, allowing a player to provide input to a computer system to control the progress of a simulated car around a detailed data model of a real racing track. The player's simulated vehicle can interact with the track model and other simulated vehicles. The other simulated vehicles are controlled by the game's physics model and artificial intelligence (AI) functions, by other types of simulations, or by other players. These other players can be present and provide input to the same computer system, or they can use remote computer systems at other locations. The remote computer systems communicate with the original player or players' computer systems via the Internet or other suitable network (which may include a game server) to provide realistic interaction between all vehicles and between all vehicles and the track. All of the methods of the present embodiment described below add to, modify, or replace these traditional features.

[0061] Taken as a whole, this new environment in which systems embodying the present invention reside can be thought of as a multi-component infrastructure that supports computer game players and other interactive viewers in collective competition with live motorsports and other types of sporting events. The new environment in which this embodiment resides includes three elements: a. A multitude of devices used by players and viewers, such as gaming consoles, personal computers, simulators, arcade gaming equipment, satellite receiving and processing equipment, smartphones, tablets, etc., in some cases including camera and audio capture devices for broadcasting from participants anywhere in the world. b. The Internet and associated computing facilities such as servers, real-time data centers, real-time server clouds (and in some cases dedicated game and entertainment servers) to facilitate the receipt, manipulation, mixing, storage and dissemination of data from players and viewers in real time, including, in some cases, without limitation, audio, video game data and control inputs when driving a virtual car. c. A real-life, live motorsport or other sporting event, including, in particular, the enhancements and developments described herein to the systems of GB Patent No. 2518602B and GB Patent Application Publication No. 2365360A and other systems that collect and provide other data about the real-life, live event as it occurs and to systems that store data related to the live event for future use.

[0062] Specific embodiments will now be described with reference to the accompanying drawings. Referring first to FIG. 1, the three basic elements described above that comprise a non-limiting embodiment of the present invention are illustrated in schematic form. More specifically, FIG. 1 illustrates several different gaming devices 2 connected to a central gaming server 4 via a communications network 6. The gaming server also includes a local data store 5 for collating different types of kinematic control data. FIG. 1 also illustrates a track 8 of a motorsport event 10, which includes real vehicles 12 positioned at different locations around the track 8. Each vehicle is equipped with a vehicle telemetry system. The vehicle telemetry system wirelessly provides a stream of data via a communications infrastructure around the circuit to a corresponding team's telemetry data capture computer 14, allowing analysis of control data (the driver's control of that vehicle) and other data related to the operation of the vehicle 12. This telemetry data stream can include live video data from an on-board camera on the vehicle 12 and acoustic data from a microphone mounted on the vehicle 12. This telemetry system For the entire team, the data for each team is a. It is not open to anyone other than the team, b. A predetermined portion is available to the managing and governing body; c. It is possible to keep certain portions available for broadcast on TV or other media streams.

[0063] The track 8 is also equipped with a number of sensor clusters 16 (though nine clusters are shown schematically, this is not limiting and other embodiments may provide a different number of clusters). Each sensor cluster 16 consists of a number of trackside infrared sensors 18. These sensors are configured to detect infrared radiation emitted by the vehicles 12 as they race around the track 8 and convert the sensor data into real-world kinematic data for each vehicle 12 in the sensor's field of view (FOV). Each sensor cluster 16 is capable of communicating this real-world kinematic data to a position data capture system 20. The position data capture system 20 collates the individual kinematic data streams and provides a stream of kinematic data 22 for all vehicles 12 on the track 8 to a live event data capture server 24. A stream of live telemetry data 26 from the team telemetry data capture systems 14 is also provided to the live event data capture server 24.

[0064] A live event data capture server 24 collates information received from a position data capture system 20, which includes multiple tracking sensors 16, with the team telemetry system 14. All of this information (real-world sensor data streams 28) is provided to a gaming server 4, which in this embodiment provides data organization and generates a virtual race simulation. The virtual race simulation is provided to all gaming devices in this embodiment. However, in another further embodiment (described below with reference to Figures 16-20), the virtual race simulation is generated on each gaming device / computer 2, and the gaming server 4 simply orchestrates the entire gaming process.

[0065] In a further embodiment, described below with reference to FIG. 5, the live event data capture system also collates other information received from the teams, including all data generated by the teams, and comprises a general event data capture system that covers all other data related to the motorsport event that may be relevant to entertainment and gaming functions. In this further embodiment, all of the information is provided to an entertainment and gaming server that provides an entertainment feed for non-gaming fans via authorized distributors, all while ensuring that the data privacy needs of all stakeholders are met, as exemplified for telemetry data in paragraph

[0062] above, as described below.

[0066] The elements forming the gaming aspects of the gaming server of the system shown in FIG. 6 are shown diagrammatically in FIG. 2. The gaming server includes a real data processor 30 for processing real data (sensor data streams) 28 received from a live event data capture server 24, which, as described in more detail below, ensures synchronization of all contributing data sources (streams of kinematic data 22 and streams of live telemetry data 26). This received live event data 28 is stored in a data store 5 as currently recorded real data from the live event 32 and is also transmitted to a virtual racing simulation engine 36, where it is used to generate a reference black-box model generator (described below) that maps the real driver inputs (control data 26) of each vehicle 12 to the kinematic position data 22 of that vehicle 12. The term "black-box" is used in its conventional sense, i.e., in the art of a system or engine characterized by its response to signals applied to its input ports. This reference black-box model generator is then used to generate a reference black-box model generator. The output of the generator is used within the virtual race engine 36 to access player virtual race commands (driving inputs) 38 received from the gaming devices 2 over the communications network 6, and consequently to determine where each virtual vehicle should be located. In this embodiment, a virtual race command processing engine 40 is provided to receive these player virtual race command data streams 38, convert them as necessary into a common format, and provide them to the virtual race simulation engine 36.

[0067] As an alternative to real-time live race data 28, previously recorded real data 42 from a live event stored in data storage device 5 may be used to generate the virtual black-box functions. The virtual race simulation engine 36 may also use data models 44 stored in data storage device 5 to generate the virtual race simulation 46. Finally, an AI engine 48 is optionally provided to assist in adjusting the virtual car's response to the driver's inputs (virtual race commands 38) with reference to the black-box model generated by the reference black-box model generator (described below), and in some cases to assist the virtual car's driver (player) in adjusting the handicap earned by the virtual driver, for example, to fair racing between real and virtual drivers of different skill and experience levels. For example, this assistance may allow for expanding the thresholds around each player's inputs (virtual race commands 38) to optimally position the virtual vehicle. Thus, while using the reference black-box model generator to determine the optimal set of control data inputs that will result in a specified vehicle position, the player control inputs 38 can be within the thresholds of those particular control data inputs 26 that will result in a specified car position. The degree to which these thresholds are adjusted can determine the handicap applied by the AI ​​engine 48 and its corresponding control of the virtual car for a given player. It will be appreciated that the AI ​​engine 48 is used because it can be trained to produce specific vehicle position outputs upon a complex array of combinations of control data inputs.

[0068] The real vehicle data 28 received from the live event data capture server 24 in this embodiment is shown schematically in Figure 3a. Here, it can be seen that the specific vehicle data 28 is associated with a vehicle ID 50 and includes kinematic real vehicle data 22 (e.g., longitudinal position 52b, lateral position 52c, and heading data 52a) and driver input data (control data) 26 such as steering position 54a, brake position 54b, accelerator position 54c, gear selection 54d, and possibly other actuator control inputs (not shown). The specific vehicle data 28 is provided for each real vehicle 12 competing around the track 8.

[0069] The virtual vehicle control data (virtual race commands) 38 received in this embodiment at the gaming server 4 from the gaming device / simulator / computer 2 is shown schematically in Figure 3b. This virtual control data 38 includes a gaming computer ID 56 and player actuator inputs 58 for controlling the virtual vehicle and possibly other data. The vehicle control player inputs 58 include steering position data 58a, brake position data 58b, accelerator position data 58c, gear selection data 58d, and possibly other actuator control inputs (not shown), which typically match those of a real driver or differ by measured amounts so that, using a reference black box, the virtual vehicle position can be determined as a result of these inputs.

[0070] The general method of operation of the system on the gaming server 4 is shown in the overview flow diagram of Figure 4. The method 60 begins with a setup phase at step 62, in which, among other things, all virtual participants are assigned to one of the real vehicles 12 participating in the race. Each real vehicle 12 may have multiple players assigned to it, as will be described in more detail below. Thereafter, in step 64, a virtual racing environment model 46 is generated, including a race circuit data model and initial positions of the virtual representations of the real vehicles. As previously mentioned, this may be based on live data 28 from the live event 10 or pre-recorded data 42 from a previously recorded live event. The circuit data model and possibly other data models 44 are stored in the local data storage device 5 to generate the virtual racing environment 46. Once the race begins in step 66, a stream of data 28, 42 from the live or pre-recorded event, including precise kinematic data 22 for all real vehicles 12 as measured and synchronized at a series of precise points in time over the duration of the race, is received in step 68 and can be used to reposition the representations of the real racing vehicles 12. These new positions are used to generate new positions for the representations of the real vehicles in the virtual racing environment 46 in step 68. This new position is then transmitted to each gaming device 2 registered for the race (not shown) and displayed to the associated player. In response, the gaming server 4 receives gaming control data 38 indicative of the player's user input to control their virtual vehicle, at step 70. The player's user input 38 is then compared to the input of a reference black-box model generator, which is used to generate, at step 72, the next position of the virtual vehicle as a result of the player's user input 38. In some embodiments (see detailed description of another embodiment below), these new positions of the virtual vehicle may then be transmitted to each gaming device 2 (not shown). This process of steps 68-74 continues until the end of the race, which is determined at step 76.

[0071] The method of operation of the system of this embodiment is characterized by the following advantageous features: a) Providing accurate and timely live data from real-world motorsport events in any type of environment to millions of players and followers worldwide simultaneously; b) Providing computer game players of all skill and experience levels with a realistic, challenging, satisfying and fair simulation of participating in a real race; i. as one player racing against a real-life driver and a limited number of other players (up to one minus the real-life driver), all of whom are represented in the game; or ii. as a player and competitor in an organized e-sports event or other computer gaming event involving any number (potentially millions) of other players and possibly real-life drivers, some or all of whom are depicted in the game; c) optionally, during a racing event, providing data from any number of players and followers worldwide to real racing teams and real drivers for any purpose, including but not limited to presenting to such teams (or drivers) real-time information about players participating in the event; and d) Optionally, provide motorsport followers with any data from the real-life event and / or all players and / or esports competitors associated with the live event to enable them to interact with and compete against said event to enhance their enjoyment of it.

[0072] In a further embodiment, the gaming server 4 described above with reference to Figure 2 may be replaced by a gaming and entertainment server 4a as shown in Figure 5. This gaming and entertainment server 4a operates in the same way as the gaming server but provides additional functionality. Therefore, for the sake of simplicity, the gaming server 4a of Figure 2 will be referred to herein as a gaming and entertainment server 4a. Only the differences between the gaming server 4 and the gaming and entertainment server 4a of FIG. 5 will be described.

[0073] Referring to Figure 5, the components of the gaming and entertainment server 4a are the same as those of the gaming server of Figure 2 and are therefore designated by the same reference numerals. Where differences exist, new reference numerals have been introduced. It can thus be seen that in this embodiment, the gaming and entertainment server 4a further includes a race simulation and entertainment engine 36a that receives virtual race commands, viewer commands, and other data 38a from gaming devices 4 and other entertainment devices (not shown). This data is collected and transmitted to the race simulation and entertainment engine 36a via an entertainment / race command and data processing engine 40a. Additionally, a gambling engine 78 and an advertising engine are also provided that provide data to the race simulation and entertainment engine 36a.

[0074] There are many entertainment-enhancing features that can be provided based on the availability of accurate tracking data from a positional data capture system, of which only three are described below as examples. The first example is a roving viewpoint, which allows a live video stream, for example from the driver's viewpoint in one of the real cars, to seamlessly switch or pan, based on viewer commands from an entertainment device, to a computer-generated viewpoint, for example above the car, via software functionality in an entertainment engine common to or very similar to the virtual racing simulation engine of FIG. 2. A second example is augmenting the live video stream with computer-generated displays or overlays convincingly integrated into the live video, for example, with temperature-dependent tire tint or "under-the-hood" images of the car's mechanics based on live technical data from sensors installed in the car or other sources. This could be extended, for example, to augmenting a live video stream from the driver's viewpoint in one of the real cars with trackside billboards or bridges, or with advertising imagery convincingly and seamlessly integrated on the car's side. Computer-generated advertisements from the advertising engine 80 dynamically move with the trackside scenery and cars, making them indistinguishable from advertisements appearing on the live video stream. Because the advertisements are tailored for streaming to specific, known users, they can be tailored to the user based on data received in the entertainment command and the data processing capabilities and algorithms contained in the advertising engine 80. A third example is the ability for entertainment viewers to bet live on events as they unfold in a real race, such as "car 1 will overtake car 2 on the inside line in 20 seconds." The gaming engine enables such functionality and provides it to the racing simulation and entertainment engine 36a.

[0075] The specific operation of the position data capture system 20 including the tracking sensor group 16 as implemented in the above-described non-limiting embodiment will now be described as an example of the system and method provided by the present invention. To more clearly understand the position data capture system 20 of the above-described embodiment, reference will first be made to a challenging racing track.

[0076] Referring to Figure 6, an example of a particularly challenging racing circuit 8 is shown, which cannot be monitored from above by a single high performance IR sensor and requires a novel system of sensors 18 and communications equipment to provide the data required for real-time, interactive gaming. An illustrative example of a live motorsport event 10 is an international Formula One race, involving 24 real cars from 12 teams, each with their own pits, garages, and other facilities. Formula 1 racing. The race location is a particularly complex circuit, as shown in Figure 6. It is an assembled site that includes tall buildings, bridges, tunnels, and has a variety of permanent street and traffic infrastructure, as well as temporary infrastructure such as spectator seating. This is subject to the prevailing range of atmospheric, weather, and lighting conditions depending on the time and location of the event.

[0077] 7a and 7b show two possible strategies used in this embodiment for generating a highly representative computer-based data model 44 of the racing circuit 8. These strategies are suitable for ensuring real-time correlation between the measured kinematic data of the real vehicle 12 and the virtual kinematic data of the game player's virtual car, overcoming the challenge of verifying that the fidelity of the circuit representation is sufficiently accurate and correct. The model 44 is a living link between the real environment and real event 10 and the computer gaming environment and computer gaming event 46, since each tracking sensor 18 in the network of tracking sensors will have a real field of view (FOV) of a portion of the circuit 8 and a relationship between the real FOV and a portion of the data model 44 of the racing circuit 8 in its memory. In this way, an image of the car's IR reflector or IR radiation, or an image of the car's natural thermal infrared radiation, is converted within the tracking sensor 18 into kinematic data expressed in terms of the racing circuit data model 44. Note that in another embodiment, raw data from the tracking sensors 18 can be first communicated to the components of the overall system. The relationship of the raw data to the circuit data model is then calculated before being transferred to the gaming server 4 for processing.

[0078] 7a, an example of the basis for the data model 44 is shown, which includes an orthogonal grid 82 of closely spaced horizontal and vertical lines. The spacing of the grid lines is adapted to the tracking accuracy desired by the computer gaming software, which can be achieved by the tracking sensors 18. The kinematic data for this model typically includes at least longitudinal data 52b, lateral data 52c, and rotational orientation data 52a for each vehicle 12. The circuit model may also include precise altitude data for each cell in the horizontal-vertical grid 82, which allows the virtual interactive hybrid environment to simulate normal G-forces, accelerations, etc.

[0079] Figure 7b shows an alternative model that facilitates computational accuracy. The circuit is modeled as a series of horizontal strips, each having a geographic location (horizontal, vertical and elevation) of the horizontal centre point of the horizontal strip, a heading, a lateral width and possibly (in some embodiments) a lateral tilt angle or elevation profile. The kinematic data for each car then includes at least a longitudinal position 52b along the track 8, a lateral position 52c across the width of the track 8, and a rotational heading 52a, as reflected in Figure 3a.

[0080] Previously, establishing a highly accurate relationship between a computer game data model of a real-world race track and the circuit's topography has been surprisingly difficult. This can be overcome with the present overall system by calibrating each sensor 18 during system setup, for example by placing stationary IR-reflecting / emitting markers (not shown) within each camera's FOV at precise measurement locations along the perimeter of the race track 8 to create highly accurate object to data model mapping. In some embodiments, dynamic calibration is achieved, for example by driving or stepping a special-purpose calibration vehicle along each edge of the race track 8 to adjust tracking performance and achieve continuity between the sensors 18.

[0081] The inventors have concluded that the geometry of such a complex environment, as shown in FIG. 6, can be arranged in a "star" configuration. This excludes connecting the 100 or more tracking sensors required in a city-wide data capture system so that they all communicate in parallel. For wireless communication, finding 100 independent, reliable signal paths would likely be impossible, and a wired connection would involve 100 kilometers or so of cable stretching radially across the cityscape, a major drawback. The only reliable path for either technology would be to connect the tracking sensors 18 in a closed loop around the circuit 8. In this closed loop, at least one point on or near the circuit must accumulate data in real time. With only one point, there would be at least 50 individual data hops from several sensors to the required device, and the accumulated data transfer delays would be unacceptable.

[0082] Real-time interactive gaming between two geographically separated players / computers A and B requires that player input to gaming device A be received and processed by gaming device B within 20-40 ms, as measured by a typical computer game update rate of 25-50 Hz, and vice versa. It should be understood that the term "computer game" is synonymous with simulated environment and includes interactive hybrid environments. In this embodiment, the entire positional data capture system, including tracking sensors 18, can be considered to provide the equivalent of player input for all 24 real cars on the race track, and this data must be provided to remote gaming device 2 within a similar latency delay. While it is possible to introduce an overall time lag, e.g., one second, into the data stream by using data time tagging to ensure synchronization (as one option, described in more detail below), this would undesirably exclude the entire live gaming experience from consideration of other information streams, such as live video. Therefore, with an infrared camera frame rate of 100 Hz (maximum 10 ms delay), allowing an additional 5 ms for raw image signal processing, only approximately 20-30 ms may be allowed for aggregation of data from sensors 18 around circuit 8 and its onward communication. For wired or wireless technologies, the inventors have taken this into consideration and determined that in this embodiment, 50 hops around the tracking sensor circuit are unacceptable, and that only 10 hops around the tracking sensor circuit are essential, even taking into account expected developments in communication technology.

[0083] Thus, the exemplary sensor constellation 16 shown in Figure 1 is transformed into an IR sensor network architecture with a small group of sensors 16 arranged in a high performance local network, but adapted to the real circuit of Figure 6, taking into account a range of other factors such as viewing angle geometry, line of sight, obscuration (between vehicles 12 and between vehicles and infrastructure), available mounting points, IR sensor capabilities such as image resolution (typically 640 x 512 or 1280 x 1024) and frame rate (typically 100 Hz), image processing parameters such as the number of "pixels-on-target" (typically greater than four), required tracking accuracy (10 cm or better laterally and less precisely vertically for high but much lower relative vehicle speeds), maximum vehicle speed in each sensor's FOV, and worst-case processing run times, all with the objective of having an output from the constellation data acquisition system 20 of Figure 1. This output includes the kinematic data of Figure 3a (for example) for each of the 24 cars 12 synchronized to a common precise time reference with an update frequency of at least 25 Hz, and an overall real-time delay of only 1 s if a real-time lag proves necessary (note that the real driver input data of Figure 3a, which is sensitive to the team and confidential outside the team, is protected by fixed or dynamic encryption when sent to the gaming devices, or when used at a central gaming server and not sent to the gaming devices at all).

[0084] To illustrate how to define an architecture for a particular circuit 8, exemplary approaches to possible multiple sensor groups 16 are described in more detail below. It should be noted that these examples are based in part on the capabilities of currently available technology, e.g., IR camera image resolution, frame rate, LED intensity, etc., and that these technologies are rapidly evolving and improving.

[0085] FIG. 8 shows a representative configuration of a specific sensor array 18 suitable for monitoring a particular type of racing circuit section for a particular class of racing car 12. These particular (F1) cars 12 have a low, aerodynamic, forward-leaning profile. This means that these cars are most likely to be reliably detected at high speeds if the tracking sensors 18, mounted higher than the track, are aimed at oncoming cars—that is, if the FOV has a boresight that is at an acute angle to the horizontal when looking toward the oncoming car. The boresight of the FOV can be considered the central axis of the FOV. This configuration also means that the sensors 18 can be conveniently placed closer to the ground and have a larger FOV compared to the simply downward-facing sensors of GB 2585165A. In some situations, a very tall pole or other structural mounting point 84 may be preferred. In this case, fewer sensors 18 are required to monitor the track, and the boresight angle to the horizontal will be increased. Alternative configurations can be defined for other types of locations, as well as for other types of auto racing events, such as powerboat racing or NASCAR auto racing.

[0086] With particular reference to the example circuit shown in FIG. 6, the following describes how sensors 18 may be arranged to provide a position data capture system 20.

[0087] First, for the full 2 ​​km section of the circuit of Figure 6 equipped with standard lampposts, Figure 8 shows one of many possible arrangements of a group 16 of IR sensors 18 mounted on lampposts 84 in this section of the track 8. Each sensor (detector) has an exemplary 640x512 pixel resolution, a field of view of approximately 20-30 degrees (see views A and B), and a detection range of up to 50 m. With this geometry, reliable detection down to a worst-case resolution of 10 cm is believed to be achievable with any of the following mounting sensor device 18 arrangements: a) A long wave infrared (LWIR) microbolometer or mid wave infrared (MWIR) photon detection (or other suitable technology) camera that detects thermal IR emitted by the car (its tires, engine, exhaust, etc.) or that detects thermal IR "cold spots" created by painting or attaching non-IR emitting markers or materials to the car. b) A shortwave infrared (SWIR) or near infrared (NIR) photon detection (or other suitable technology) camera that detects broadband (or narrowband) light emitted by at least two small LEDs or LED clusters mounted on the car, or detects natural (sunlight-derived) IR radiation from the car's reflectors, or detects cold spots from IR non-radiators on the car. The light emitted by the LEDs can be constant or dimmed over time. c) A SWIR or NIR camera that detects broadband (or narrowband) SWIR or NIR light provided by an LED floodlight placed near the IR camera and reflected by at least two small reflectors mounted on the car, or detects cold spots in the reflected image from an IR non-emitter on the car. This would allow for night racing, and the LED floodlights would be pulsed and synchronized with the imaging on the camera to improve detection performance in extreme weather conditions.

[0088] As mentioned above, the position of every individual target (LED emitter, reflector, non-emitter or car / tire) in the raw image on the camera focal plane array can be converted to its exact location within the circuit model by signal processing and geometric calculations tailored to the specific location, configuration or orientation of the tracking sensor. The specific location, configuration, or orientation of the tracking sensors may include the relative position of each of the two emitters (or reflectors or non-emitters) of the vehicle 12, measured relative to its outer physical limits. Such techniques are well within the capabilities of those skilled in the art and therefore need not be described below.

[0089] Continuous identification and tracking of individual cars 12 can be achieved in GB Patent Application Publication No. 2585165A by each tracking sensor communicating the car's position to the next sensor ahead as it leaves its FOV. Note that in conjunction with the SWIR or NIR camera placement as described in b) above, the LED light may be constant or may be dimmed to emit a pattern (or IR signature) characteristic of each car 12. This allows each tracking sensor to track and identify simultaneously, eliminating the need for a tracking sensor to "hand off" the vehicle identifier to the next tracking sensor.

[0090] Second, around the circuit 8, there may be positions (e.g., black box 81 in FIG. 6 ) that lend themselves to a single IR camera 18 placed at higher elevations, mounted on top of a building or a mobile boom lift (up to 45 m) or a work platform (up to 100 m), or carried by an aircraft such as a drone or tethered drone (which can provide power for extended flight time and data broadcast to a ground station). At a height of 100 m, a single IR sensor 18 is fitted with a wide-angle lens that extends the FOV to approximately 140 m x 140 m (black box 81 in FIG. 6 is representative in scale). For some circuits 8, it may be preferable to arrange several drone-mounted sensors 18 around the circuit to form a complete closed-circuit network. Each drone stays stationed using its own onboard GPS, and fixed IR beacons placed on the ground provide an accurate reference image, from which absolute tracking and detection accuracy can be achieved. This is an efficient arrangement. This is because fewer sensors are needed to cover the entire circuit and the ease of deployment of this arrangement makes it extremely convenient for circuits where sensors are not fixedly attached. It is possible to achieve safe conditions for drones flying over high-performance motor racing because the drone does not need to be directly above the circuit or directly above the spectator stands. Tethered drones may have a particular advantage because if the drone breaks down, it will immediately rotate and fall into a defined crash zone around the base station.

[0091] Third, the short lampposts 84a in Figure 8 may be fitted with wide FOV (70 degree) IR sensors. Each IR sensor covers a short roadway section approximately 15 m long. The same applies to tunnel 83, where sensors 18 are mounted on the tunnel roof. In another alternative embodiment, it may be more cost-effective to not use short lampposts 84a but to deploy mobile poles up to 30 m high, each covering roadway sections up to 50 m long. In summary, the complex 3.3 km long circuit in this example would require 50 standard sensors, 50 wide-angle sensors, and two long-range high-platform or drone-mounted sensors.

[0092] It should be noted that on many racing circuits, lampposts may not be present or may be more widely spaced for safety reasons, or may need to be protected so as not to present a safety risk to cars. In different embodiments, it may be necessary or preferable to temporarily mount tracking sensors on buildings or suitably placed movable poles.

[0093] Finally, to achieve the real-time data streaming performance criteria described above, the ground tracking sensors and / or drone-mounted tracking sensors of this example are organized into groups 16, as generally shown in FIG. 1 and more specifically in FIG. 9. 9 is an illustrative example of how sensors 18 can be arranged in groups 16 to achieve the overall real-time performance of position data capture system 20, and how sensors 18 and groups 16 can communicate data to provide a complete, continuously synchronized single data stream 28 with a suitable latency for transmission to a real-time interactive gaming environment 46. While FIG. 19 shows one group 16 of seven sensors 18 mounted on a lamppost, other configurations are possible in different embodiments. Multiple groups 16 can be created around track 8 to provide the required complete coverage. In this way, the complete set of tracking sensors 18 positioned around racing circuit 8 can be operated simultaneously as a single group to generate a single data stream for all real cars, while also achieving the low latency and concurrency required for interactive gaming.

[0094] Referring more particularly to FIG. 9 , the sensors in each group are connected to a high-performance, low-latency communication link 86. Such communication link 86 uses a suitable communication technology (either a wired technology such as Gigabit Ethernet with a data rate of greater than 900 Mbit / s and a latency of 100-500 μs for ground-based sensors, or a wireless technology such as dual-band Wi-Fi with a data rate of greater than 90 Mbit / s and a latency of 1-2 ms for ground-based or drone-mounted sensors). Each group of sensors 16 is equipped with a forward communication node 88, typically used in fiber optic ("wired") or wireless networks, often used for streaming telemetry data in auto racing sports. This forward communication node 88 is configured to communicate wired or wirelessly to a live event data capture server, where position data for all 24 cars is collected at a refresh rate of at least 25 Hz for forward transmission. Additionally, and depending on communication time delays from circuit to circuit, and effectively around the network, every set of sensors or each sensor group 16, or in some embodiments each sensor 18, may be equipped with a GPS receiver so that a common, accurate time reference can be established across the sensor network. The precise position measurements of the cars 12 are then time-tagged, allowing the live event data capture server 24 to assemble the car position data into the required contemporaneous set, even if this introduces a small time lag (<1 second) in the onward transmission to the gaming server 4.

[0095] Next, the specific operation of the above-described non-limiting embodiment of the system and method for the interactive hybrid (real-virtual) gaming environment 46 provided by the present invention will be described in further detail with reference to FIGS. 10a-14.

[0096] In this example, Figure 10a shows a Formula 1 motor racing starting configuration including 24 real cars and drivers. These real car 12 representations (89a, 89b, 89c, 89d, ..., 89x) are associated with a single player computer game, where a real car representation 89f is assigned to the player or selected by the player as its starting position so that its simulated car 90 representation is exactly overlaid on the selected real car representation 89f.

[0097] 10a, the first case described above involves one computer game player controlling a simulated car 90 that, at the start of the race, is in the same position in the computer game as the real car 12f in the real event. After the race begins, the progress of the simulated car 90 is determined by the inputs the player makes to his computer device (gaming device / computer 2) and by a simulation of the car and its environment by the gaming server 4. The progress of the real car representation 89 is determined by the inputs the driver makes to the real car 12f and by the physical behavior of the highly complex car 12f in its highly complex environment. The forward movement of the simulated car 90 may differ from the forward movement of the real car 12f due to i) a lack of fidelity in the simulation of the car and its environment, or ii) differences in inputs between the player and driver. This embodiment minimizes the former reason, thereby making the competition between real drivers and virtual players as fair, realistic, and enjoyable as possible.

[0098] One of the main features of the method of the described embodiment is that the computer game is particularly dynamic and increases interaction with the real vehicle 12, while maintaining the simulated car 90 sufficiently close to the real vehicle representation 89f (the following description refers to the simulated car 90 being "snapped" to the real vehicle representation 89f) to a tolerance that can be defined in a number of ways, including but not limited to: a) The real car representation 89 is not displayed and does not interact with the player's simulated car 90 as part of the snapped player's game. b) The snapped player's car 90 is "digitally paired" with a real car representation 89, and the simulated car 90 replicates the performance of the real car representation 89 so that the competition between the driver and the player is sufficiently close to being a fair contest. c) The inputs (control data 26) of the real driver (steering, acceleration, gear selection, etc.) can be streamed and made available to the interactive hybrid environment generated in real time by the gaming server. d) Audio, video and other data feeds from the real car 12, its driver and the wider racing team may be streamed into a computer game (an interactive hybrid environment) generated in real time by a gaming server. e) Audio, video and other data feeds from the snapped player may be available backwards to the real car 12 and / or team and / or other computer gamers.

[0099] 10b illustrates a racing situation in which a player's simulated car 90 deviates from a real car representation 89f by a known amount in instantaneous position, and possibly other attributes such as speed, velocity, and acceleration. Here, we describe a non-limiting, simple method for defining when a player's car 90 is snapped to a real car representation 89f. Within computer game software running on the gaming server 4, the location on the racetrack of a reference point 94f on the real car representation 89f is used to define a region of proximity points 96f, in this case a rectangular box. If the location of an equivalent reference point 91 on the simulated car 90 falls within the region of proximity points 96f, the player's car is "snapped" to the real car representation 89f.

[0100] Many other snapping methods are possible, including those that may depend on other parameters, such as how far ahead or behind the player's car representation 90 is in time relative to the real car representation 89 (such as the method used in Formula 1's Drag Reduction System (DRS), which allows a chasing car to "boost" if it is within a certain range of the car in front). Other snapping methods may include speed, velocity vector, angular momentum, etc., but the principle is clear. Additionally, the computer game AI engine 48 may use algorithmic or other strategies to implement a degree of "stiction" that provides more flexibility and tolerance so that the snap is not subject to jitter. Here, a handicapping system may be used, with higher-ranked players being given less "AI assistance" than lower-ranked players. Also, game software running on the gaming server may take into account the proximity of other players' virtual cars when "un-snapping" from the real car representation 89.

[0101] If the player's simulated car 90 is not snapped to one of the 24 real car representations in the race, all of the car representations will be in the same position as conventional computer games. When a player's simulated car 90 is snapped to a real car representation 89, only the other 23 real car representations 89 are visible to the player, and the player is likely to feel as if they are driving a real car and interacting very closely with the real driver and his or her race team, especially if a live audio feed is provided to the player by the team.

[0102] This snapping method allows the player to advance during the race by avoiding real car representations 89 or by moving from real car representation 89 to real car representation 89 and snapping them (or not, as the player chooses). Figure 11a illustrates the racing situation shortly after the start of the race. Here, simulated car 90 is somewhat between the three closest real car representations 89c, 89b, and 89e. The player's simulated car 90 has unsnapped from its original pair (race car representation 89f) by improving its performance and is now ahead of it. At this point, the closest real car representations to the player's virtual car are real car representations 89b, 89c, and 89e. Depending on the skill and actions of the player and the three real drivers, the positional relationships 98b, 98c, and 98e will change rapidly. At the time of Figure 11a, the computer game (virtual racing simulation) displays real car representation 89c in front of the player's car 90 and real car representation 89e in its rearview mirror.

[0103] Figure 11b continues the racing situation of Figure 11a, but shows a later point in time when simulated car 90 moves forward and approaches real car representation 89c. Simulated car 90 is shown entering proximity point zone 96c of real car representation 89c, where it is snapped to. At this point, the gaming server prevents real car representation 89c from being displayed on the player's screen, and the features described above become active.

[0104] FIG. 11c is a further continuation of the racing situation of FIG. 11b after a longer time interval, illustrating another portion of the circuit 8 in which a simulated car 90 is designated and approaching the leading real car representation 89a. Here, the computer game player's simulated car 90 can be seen moving clearly ahead of the real car representation 89c, more than halfway between the real car representation 89c and the race leader's real car representation 89a. However, the player's virtual car 90 has not yet reached the snap zone 96a of the real car representation 89a and is pursuing it. This illustration introduces two additional zones around each real car representation 89a. A first zone 98a is located in front of the real car representation 89a and is called the "post-overtake zone," while a second zone 100a is located behind the real car representation 89a and is called the "chasing zone." Together, these three zones 96a, 98a, and 100a may form a contiguous set, as illustrated in FIG. 11c. Thus, when the simulated car 100 enters any of these three zones associated with the real vehicle representation 89, it is digitally paired with the simulated car 90, with its performance closely matching that of the real vehicle representation. This feature equalizes the chances that the virtual car 90 will chase the real vehicle representation 89, or vice versa, allowing for natural integration with real-world features such as DRS. Thus, the simulated car 90 is now in the chasing zone 100a of the real vehicle representation 89a. As soon as the simulated car 90 moves from the post-overtake zone 98c of the real vehicle representation 96c to the chasing zone 100a of the real vehicle 89a, its performance characteristics are switched between those of the real vehicle representation 89c and those of the real vehicle representation 89a. This means that the performance of the simulated car 90 always matches the real vehicle representation 89 with which it is most closely interacting and competing, ensuring fair competition based solely on driving skill between computer game players and real drivers. This zoning method is one example of the zoning principle, and there are several possible variations of this method.

[0105] As noted in paragraph a) above, there may be more than one computer game player involved in a race simulation associated with a live or recorded race. In prior art motorsports computer games, it is common for multiple players (ranging from one to as many as the actual regular participants in the actual race) to play together online wherever they are located around the world. Thus, in the Formula 1 example described above, it would be common for up to 24 players to play, i.e., each player takes a starting position, with any empty positions occupied by simulated cars controlled by the game software's algorithms or AI. This type of limited player gaming mode would also be possible in this embodiment. However, in other embodiments described below, other gaming modes not limited by the number of players may be implemented. According to the above-described embodiment of the present invention, it is possible to implement the limited player gaming mode described above, i.e., a player gaming mode in which empty player positions are assigned to representations 89 of real cars in the actual race, and their behavior in the game is controlled by data streamed live from the event. In this case, there will be 24 drivers and / or players displayed in the computer game at any one time, with a number, i.e., "n," of computer players competing against "24-n" real cars. All aspects of the "chase and snap" method described above are applicable, with the performance of each car's simulation 90 being paired with the performance of the real car representation 89 that replaces it in its starting position, except that each car is in the "chase and snap" zone of the real car representation 89 displayed in the race.

[0106] According to an alternative embodiment implementing a limited player gaming mode, there are 24 game players and 24 real drivers, and whenever a player's virtual car 90 is not snapped to a real car representation 89, their virtual car 90 will appear in the computer game environment and be visible to and interactable with all other players' virtual cars 90. Thus, there will always be 24 real car representations and anywhere from 0 to 24 unsnapped simulated cars 90 visible in the computer racing simulation. These two method variations of this gaming mode are exemplary, and many other variations are conceivable, but the principles of the method are clear.

[0107] In many types of motorsports (Formula 1 being just one example of a very common application of the present invention), real cars can take time out of the race for maintenance or modification operations, such as repairing a malfunction, refueling, or putting on new tires. Thus, in this embodiment, a computer game player can choose to take a simulated version of the race for the same maintenance or modification operations as the real car. In this case, the game player's simulated car 90 will inherit the altered performance characteristics of the real car representation 89. Alternatively, if the computer game player's car 90 is not snapped with the real car representation 89, the computer game player can choose to take a race time out for the simulated maintenance or modification operations. This will entitle the computer game player, upon returning to the actual race, to associate the computer game player's car 90 with any real car representation 89 that has undergone similar or compatible maintenance or modification operations. This is one of many possible examples of how to ensure competition is fair when real cars can undergo maintenance or modifications during a race.

[0108] As mentioned above, other gaming modes can be implemented that are not limited by the number of players. One such embodiment corresponds to the situation described in paragraph b) above, where a large number of computer game players participate in a game and play simultaneously. Embodiments are described below in which a game is linked to a live or recorded real-world event. A mass esports race is just one example of such a situation, where players are required to interact with the real-world race in a realistic, engaging, and challenging way, and where a fair way of ranking a large number of players based on their performance and final placement in the esports event is required.

[0109] 12a and 12b show two examples of possible starting configurations for a race involving 24 real cars interacting live in an e-sports game, or an online gaming race involving tens of millions of game players whose starting positions are distributed among the starting positions of the real cars. More specifically, FIG. 12a shows a starting configuration for a race involving 24 real cars associated with 10 million video game players evenly distributed among the starting positions. FIG. 12b shows an alternative starting configuration to that shown in FIG. 12a, a starting configuration with 100 million video game players logarithmically distributed across the 24 real car representations, but associated with real car representations 89 in which only the highest-ranking virtual car 90 leads. These gaming modes are referred to as unlimited player gaming modes. These examples are merely two of many similar options, and for purposes of the method of the present embodiment, each participating player in a computer game need only be assigned one of these cars. Thus, each player begins the race snapped to one of the real car representations. Each player snapped to the same real vehicle representation 89 is digitally paired with that real vehicle and has the same performance model as the real vehicle representation 89 in its environment.

[0110] FIG. 13 illustrates a racing situation midway through a race between 24 real cars and multiple computer game players, each with a virtual car 90 associated with a single real car representation 89. The actual racing situation in FIG. 13 resembles the situation in FIG. 11c, with real car representation 89a in the lead, followed by real car representation 89c, which in turn is followed by real car representation 89b. Each computer game player experiences the race in much the same way as a single player interacting with the 24 real car representations described above. The situation shown in FIG. 13 is one in which the simulated cars 90 of three players 102a(3) are in zone 98a ahead of the leader (real car representation 89a). This is indicated by a single dashed car on the track and, specifically, in a close-up view, by three individual simulated cars 90c, 90f, and 90z in precisely and closely separated positions. In this lead zone, each player may be displayed with information and / or representations of some or all of the other players in that zone, facilitating an interactive competition for lead position. Four cars 104a(4) are located within the snap zone 96a of the real car representation 89a and have been snapped to the real car representation 89a. These are shown in FIG. 13 as a single dashed car on the track 8 and in a close-up view as four individual dashed car representations 90b, 90m, 90w, and 90s, shown in their exact location in the game relative to the real car representation 89a. Nine players 106a(9) are located in the chase zone 100a of the real car representation 89a. All simulated cars within zones 96a, 98a, and 100a are digitally paired with the real car representation 89a. The real car representation 89c has a snap zone 96c with 25 cars 104c(25) currently snapped to it. The post-pass zone 98c includes 18 players 102c (18), and the chase zone 100c includes 57 players 106c (57). All players are digitally paired with a real vehicle representation 89c.In the overtake and chase zones, each player's virtual car 90 may be displayed along with information and / or representations of some or all of the other players who have their virtual cars 90 in the same zone, promoting interactive competition between virtual players in the overtake and chase zones 98, 100. This representation method advantageously provides a large amount of player information within a small screen size. This allows the gaming device to maximize the amount of information displayed. If a real car crash occurs in the actual race, the snapped virtual car 90 will follow unless the player's input commands are such as to mitigate and avoid the crash as determined by reference to the black-box model. Otherwise, it will continue as the unsnapped car 90 from the point where it successfully left the road.

[0111] Thus, in this unlimited-player mode, the above-described method conveniently allows any number of computer game players to interact with a live race, each player experiencing the event 10 and competing against a full range of real cars in an engaging and exciting way. This is made possible by each player's gaming device 2 receiving a live data stream of the real cars' progress around the circuit and, where appropriate, driver input (into the virtual race simulation) via other data streams such as team audio and driver video. In addition, when players are participating in a race associated with a live mass competition, each player's progress around the circuit is typically communicated and compiled at a central gaming server 4. This allows for a unified view of the entire circuit 8, shown in FIG. 13, and is analyzed in real time so that the relative positions of multiple players' virtual cars 90 in the race can be continuously and reasonably accurately monitored (e.g., a computer game software simulation can present the instantaneous position of any simulated car at any time with any desired accuracy). This then allows for the following: a) The final positions of all players at the end of the race are determined unambiguously and fairly; and b) In environments where competition between computer game players may be enhanced, individual players may be presented with images of the cars of other relevant players in their vicinity, as described above.

[0112] Additionally, each player's computer game displays their behavior and interactions with the real-life event 10, enhanced by the collective competition of many players. For example, not only is a live audio feed of the real drivers and teams streamed to all snapped players, but the number of snapped, chasing, and overtaken players may also be presented to the real teams or a number of spectators or real drivers (which may include individual player names). Real drivers may comment on some or all of the snapped drivers, and selected snapped drivers may reply to the comments (subject, of course, to evaluation for acceptability from a safety perspective). There are potentially many other such immersive entertainment features enabled by this embodiment.

[0113] In events 10 associated with live mass competition of computer game players, depending on how they are organized, there may be a wide range of skill levels in the game player population. This embodiment can be easily extended to allow for "all who wish to participate" or "open" events by including a handicap parameter for each player, as described above (based on an unbiased assessment of prior experience, as with all handicap systems). For example, each player could achieve a handicap between 0 and 100, where 0 represents professional gamer status and 100 represents novice gamer status. A handicap of 0 would mean that the AI ​​engine 48 of the gaming server 4 would not provide any assistance to the game in accepting the player's inputs and translating them into advancement of their virtual car around the circuit 8; conversely, a handicap of 5 could mean that a player's data inputs 38 would be considered suitable if they were within 5% of the inputs of a real driver 26.

[0114] All of the above methods advantageously exhibit the following characteristic: a computer game simulation of real-car performance in its environment can present fair and enjoyable competition not only between game players, but also between game players and real drivers. In other words, if a game player applies equivalent simulated inputs 38 to the simulated car as a real driver would apply realistic control inputs 26 to the real car, the forward movement of the simulated car 90 will match with acceptable accuracy the forward movement of the real-car representation 89. Achieving a high level of fidelity has long been a goal of motor car racing computer game development and professional simulators, and considerable capabilities already exist. However, for events associated with live, group battles, there will be many factors in the play of any event. These factors will need to be measured and communicated to the computer game and realistically accounted for in the computer game algorithms and AI. As noted above, prior art suggests sending many such measurements live to a computer game and augmenting traditional physics models based on the simulation. An alternative and preferred approach is realised in the final embodiment described below, and is effectively summarised as a method of "real-time dynamic black-box simulation for real-virtual interaction".

[0115] This embodiment allows for the implementation of a dynamic black-box simulation model within a computer game associated with a live event. In these embodiments, in addition to providing the computer game with accurate position data 22 of the real vehicle in real time via the position data capture system 20 shown in Figure 1, the control inputs 26 made by the real driver to the real vehicle 12 are also measured and transmitted in real time to the gaming server 4. Figure 14 is a graph showing various parameters 28 transmitted by the live event data capture server 24 over time to illustrate the principles of the black-box dynamic simulation method.

[0116] Referring to FIG. 14, this embodiment uses a real-time dynamic black-box simulation model for an exemplary and simplified set of driver inputs (control inputs) for a real vehicle: steering wheel position 54a, accelerator position 54c, and brake pedal position 54b. In reality, there may be more driver inputs, such as a gear change (not shown in FIG. 14). These are the dynamic data inputs to the black-box model. The dynamic outputs from the black-box model are the advancement of the real vehicle 89 in terms of position over time; in this example model, these are shown as longitudinal distance traveled 52b and lateral position 52c on the track 8. FIG. 5 shows that variations in the driver's inputs can be matched with variations in the output of the track position / location of the real vehicle 89 over time. Recording and correlating the time history of inputs and outputs over one lap provides the simplest version of the black-box dynamic model, which is then used to determine the possible track positions of the virtual vehicle 90 based on the user's computer-generated inputs 38.

[0117] This embodiment is based on a racing simulation occurring on the central server 4. An example of how the virtual racing simulation engine 36 implements a dynamic black-box simulation in this case is described below and illustrated in FIG. 15. Kinematic race position data 22 is matched to real driver inputs 26 from a live or recorded event 10 by a reference black-box model generator 110 (described above). The output of this data is provided to a gaming black-box implementation generator 112, which references it to determine whether player input data (driving commands) 38 are affecting the position of the virtual vehicle 90 that it is controlling in the race. In effect, the reference black-box generator defines a function that transfers inputs to outputs, which is the gaming The stored data model 44 may be used by the gaming black box implementation engine 112 to assist the player in competing against professional drivers of real vehicles. As mentioned above, an AI engine 48 is also provided to assist the player in competing against professional drivers of real vehicles. Once the position of the virtual vehicle 90 is determined by the gaming black box implementation engine 112, it is sent to the racing simulation output engine 114, which is used to generate a representation 90 of that vehicle in the race. An interactive hybrid racing environment for all racing vehicles 89, 90 is generated by the racing simulation output engine 114, which uses the stored data model 44 to output the virtual racing environment 46 to all participating gaming devices 2.

[0118] The black-box dynamic model minimizes the need to generate a highly accurate "white-box" model (i.e., a model that combines more detailed models based on complex physics, which in turn combine more detailed models, etc.) of an extremely complex vehicle and its extremely complex and dynamically changing environment. Thus, complex factors that cannot be dynamically measured and modeled—such as wear and tear on the airframe, the effect of tire compound buildup on the track surface on wheel adhesion, and wind gusts on the vehicle's forward aerodynamics (to name just three of many)—are all integrated into the realism of the measured black-box model. The measured black-box model generated by the reference black-box generator 110 then serves as a reference model for the AI ​​algorithms of the AI ​​engine 48 in the game, changing its outputs consistently, proportionally, and realistically when inputs 38 provided by the computer game player differ from the reference model inputs.

[0119] A black-box model of a single car will vary from lap to lap due to large variations in driver inputs from lap to lap, and variations (usually continuous and gradual, and sometimes discrete) in other physical factors. However, there are many detailed strategies and techniques that can be used in the present embodiment to ensure that the method provides optimal dynamic and accurate simulation during a race. These include, but are not limited to: i. Average or otherwise combine black-box model data from multiple laps throughout practice sessions and real-world races to generate a reference black-box lap model that best represents the driver, car, and track on that day. Do the same as in ii.a) but on a shorter section of the track, for example a particular corner or a particular straight. iii. Take a lap with the best lap time as a reference black box model for the final performance of the driver, car and truck on the day. iv. Do the same as c) but on a shorter section of the track, for example on a particular corner or on a particular straight line. v. Building a black box model second by second, comparing game player inputs 38 with real driver inputs 26, and adjusting outputs in direct and immediate response (this method would be particularly useful for the first lap when no history is available, or for the first lap after a "pit stop" and therefore modifications to the real car). vi. Supplement the measured black-box model with modeling of known physical laws that are known to apply continuity and relative simplicity. vii. Model known limitations on vehicle performance, such as maximum straight line acceleration, top straight line speed, etc.

[0120] There are many other techniques, but the gist of this method is that rather than generating white-box models of highly complex physical systems in real time, it is much more practical to model and simulate changes relative to one or more dynamic black-box reference models that are known to be valid.

[0121] As well as being linked live to a real motorsport event 10, other embodiments of the present invention are also applicable to situations where the computer game player's interaction is with a recorded version 42 of a past live race. This recording 42 consists of the highly accurately measured dynamic performance 22 of a real car 89 in the environment of the past race and the real driver's inputs 26 to the real car during that race. In these embodiments, all of the above discussion is applicable to pitting a computer game player against a recorded race.

[0122] Alternatively, in another embodiment where the actual race recording 42 consists solely of the dynamic performance 22 of the real car 89 measured with high precision in this environment, without any recording of the real driver's inputs 26, the digital pairing may be performed based on any other method that ensures that the simulated performance of the game player's car 90 in the environment is close enough to represent the performance of the real car 89 to create fair competition between the game player and a real driver or autonomous vehicle. This may include, for example, reverse engineering the driver inputs 26 using a high-fidelity racing car simulator, where the driver rehearses and then records an input history that allows the driver to recreate the performance of each real car 89 in the recorded race. This is an illustrative example of a method for reconstructing inputs 26 into a dynamic black-box model when data from the original event is not available.

[0123] With reference to Figures 16-20, a further embodiment of the present invention is described, which differs from the above-described embodiment primarily in that it is implemented in a more distributed manner. This further embodiment has many components that function similarly to those of the embodiment of Figures 1-4, and only the differences will be described below to avoid unnecessary repetition. With reference to Figure 16, the configuration of a gaming device 2 is shown. Here, the gaming device 2 is equipped with a virtual racing simulation engine 36 rather than a central server 4, and player input commands 38 derived from player input can be entered directly into the virtual racing simulation engine 36 and do not need to be transmitted to the central server 4. This significantly improves efficiency and bandwidth consumption. The virtual racing simulation engine 36 is provided with at least the position data 22 and driver input data 26 of the real car 89 from the live event 10, recorded reference black box data 120, and supplemental data 122 from the gaming server 4 via a communications engine 124. This received data is stored in a local data storage device 5 and used by the virtual racing simulation engine 36 in the same manner as described above for the embodiment of Figure 2. Similarly, a data model 44 provides further support for the generation of the simulated racing environment 46. As previously discussed, an AI engine 48 is provided to provide support to help players compete against real drivers. As a result of having a more distributed approach to virtual racing simulation, the central server 4 needs to be updated with the position of each player's virtual vehicle 90. Accordingly, a communications engine 124 is arranged to transmit virtual vehicle position data 126 as determined by the virtual racing simulation engine 36 to the central gaming server 4 during the race, as well as race car selection 126 for the start of the race and changes 126 thereto during the race.

[0124] 17, a gaming server 4a of this distributed embodiment is shown. The gaming server 4a has a virtual vehicle position processor 128 that collates each virtual vehicle position 90 and provides it to a virtual race management engine 130. The virtual race management engine 130 then provides each gaming device 2 with information regarding their position and relative performance during the race, and potentially (if it is determined that the race between virtual players is intensifying or becoming more competitive) the positions of other virtual vehicles 90 during the race, The virtual race management engine 130 can also generate a reference black box for previously recorded data 42, which can be provided to all gaming devices 2. Because this is pre-recorded data 42, the information is not provided in a time-critical manner that would be tied to real-time racing. The gaming server 4a then simply handles monitoring of the race, knowing the locations of all competitors, both virtual and real, possibly providing telemetry and other data to the "snapped" cars (thus requiring a link from the live event data capture server 24), possibly providing selected race images to each virtual competitor, and possibly providing statistics about the virtual race to the real teams.

[0125] Also, in this embodiment, the real live event data 28 is sent from the live event data capture server 24 to the gaming server 4a, and distributed from the gaming server 4a to the gaming devices 2. This is convenient for synchronization. However, in another embodiment, the real live event data can be sent directly to each gaming device 2, which has the advantage of reducing latency and the processing power required at the central server.

[0126] Figure 18a illustrates data from the live event data capture server 24 and virtual car position data received at the gaming server 4a from the gaming devices 2. While Figures 8a and 13a are very similar, in this embodiment other data 132 is also provided to the central server 4a. This other data 132 may be live telemetry data, wireless communication data between teams and drivers, etc. These additional data streams 132 may make gaming more realistic or engaging, as discussed above.

[0127] 18b shows virtual car position data 134 sent from gaming device 2 to gaming server 4a. This data further includes gaming computer identifier 56. This position data 134 is used to update the position of virtual vehicle 90, as determined by gaming server 4a, so that the position can be communicated to other participants in the same race using other gaming devices 2. Other data fields 136 relate to the selection of the real vehicle to be associated or other data, such as handicap data.

[0128] Referring now to FIG. 19, a virtual race management engine 130 within the gaming server 4a is shown. The virtual race management engine 130 determines which gaming devices 2 will receive which updated position data 38. At its heart is a race engine management processor 140, which aggregates race photos provided to the central server 4a from different data streams. The virtual race management engine 130 also includes a reference black-box model generator 142, which is the same as that described in the embodiment above with reference to FIG. 15, except that it operates only on historically recorded event data 42. Creation of the live data reference black-box model is delegated to each gaming device 2. A communications engine 144 provides position data 38 and reference black-box model data 146 to each appropriate gaming device 2. The communications engine 144 can also provide other race data 132 (as described above) to each gaming device 2 and can transmit live event data if not provided directly to each gaming device 2.

[0129] Referring to Figure 20, an operational method 150 for the gaming device 2 of the system of this embodiment is shown. The method 150 begins with the gaming device in step 152: The gaming server 4a then begins sending a request to the gaming server 4a to participate in the event. This request includes certain configuration details that allow the gaming server 4a to interact with the gaming device 2 and identify the real vehicle 89 with which the player of that device 2 will be associated. For example, if the gaming device 2 requires a direct feed of live data from the live event data capture server 24 and / or the live telematics feed 26, these gaming options can be identified. In the next step, the gaming server 4a provides the requested information, in step 154, such as links to the live data feed from the live event data capture server 24 and the locations of other virtual players 90 in the race as determined by the virtual race management engine 130. While not shown, evaluated with this information, the gaming server 4a can generate a global picture of the active gaming devices 2 assigned to a particular event 10. As with the previously described embodiment, a single real vehicle 89 can have multiple players associated with it. Thereafter, in step 156, a virtual racing environment is generated locally on the gaming device, including the initial locations of the virtual representations of the real vehicles 89. As explained above, this is based on live data 28 from a live event or pre-recorded data 42 from a previously recorded live event. A data model 44 stored on the local data storage device 5 is used to generate a virtual racing environment. When a race begins in step 158, the data streams 28, 42 from the live or pre-recorded event are received in step 160 and used to change the positions of the representations of the real racing vehicles in step 162. These new positions are used to generate new positions for the representations 89 of the real vehicles 12 in the virtual racing environment in step 160, which are then presented to the player at the gaming device 2. In response, the gaming device generates gaming control data (virtual race commands) derived from the player's user input 38 to control the virtual vehicles in step 162.These virtual race commands are then compared with the output of the reference black box within the virtual race simulation engine 36 and used to generate the next position of the virtual vehicle as a result of the player's user input in step 166. (It will be understood that real-time black box generation in this embodiment is as described with reference to FIG. 10.) This new position of the virtual vehicle can then be transmitted to the gaming server 4a in step 168, and the positions of the other players' virtual vehicles 90 are received from the gaming server 4a and determined to be relevant by the gaming server 4a. The new positions of the relevant virtual vehicles are then presented to the players on the gaming devices 2 in step 170. This process continues until the end of the race is determined in step 172.

[0130] Finally, it will be apparent that the features of the present invention can be extended to a wide range of sporting events, where the moving objects tracked in real time on a delimited playing field are not automobiles, but rather players on a soccer pitch or basketball court, competitors in a downhill skiing event, or hounds on a race track. These are all examples of non-track competition areas and non-vehicle moving objects. The positional data capture system would be sensors, computer equipment, and communication equipment with an appropriate architecture. Non-infrared-emitting fabrics or small infrared-emitting devices (capable of emitting constant or dimmed light and possibly triggered) could be incorporated into the clothing of real competitors to provide cold or hot spots suitable for tracking and identification. While interactive competition between real players or teams and virtual players or teams may not be as practical as in motorsports, the immersive entertainment and spectator enhancement features described in 18) above are practical. The collection of sports statistics and analytics can be automated in a straightforward manner. Furthermore, for example, soccer balls or basketballs could be equipped with infrared-reflecting or infrared-transmitting materials invisible to the human eye. It may also be possible to develop technology incorporating non-radiative markers, enabling the tracking device's capabilities to continuously measure the ball's rotation. This could also be applied to, for example, the ball in a snooker or billiard game, where a single sensor is placed on the table. A common distinctive and novel feature is the tracking of moving objects (vehicles, players, balls, etc.) at relatively short ranges (<100 m) using wide-angle infrared sensors and illuminators, and associated computing and communication equipment, to detect passive or active infrared markers, reflectors, absorbers, or emitters. The real-time data thus obtained can be used to enhance a wide range of interactive entertainment and gaming experiences.

[0131] The features of this embodiment will be described below in itemized form. 1) A system and method are described for computer game players to interact with real, live races involving drivered or autonomous real cars, whereby a single player can start a race by associating their simulated car with one of the real cars, and can progress by remaining associated with the initial car or by associating from real car to real car based on defined proximity parameters, thereby competitively competing as a virtual driver against the real drivers in the real race. 2) A system and method are described in which, if a computer game player's simulated car is not associated with a single real car, all real cars are displayed in the race, and the computer game player can choose, at his discretion, to avoid the real cars and race as an independent moving object, or to associate with a single real car based on specified proximity parameters. If a player's simulated car is associated with a single real car, that real car will not appear as a competing car in the player's computer game. 3) Systems and methods are described in which, when a computer game player's car is associated with a real car, the simulation of the game player's car is a digital counterpart of the real car, meaning that the performance of the game player's car in its environment is close enough to the simulation of the performance of the real car in its environment to create fair competition between the game player and a real driver or autonomous vehicle. 4) A system and method is described in which digital pairing is based on highly accurate measured real vehicle kinematic performance in its environment and dynamic black-box simulation of real driver inputs to the real vehicle. 5) A system and method are described in which a computer game dynamically associates a virtual zone with each real vehicle, and when a computer game player's vehicle enters the zone, it becomes the digital counterpart of the real vehicle or takes on certain performance characteristics associated with the real vehicle. 6) Systems and methods are described in which, if a computer game player's car is associated with a real car and the real car takes a race timeout for maintenance or modification operations (such as troubleshooting, refueling, or putting on new tires), the computer game player can choose to take a race timeout as well (which may be delayed by one lap), in which case their simulated car will take on the performance characteristics of the modified real car. 7) Systems and methods are described whereby if a computer game player's car is not associated with a real car, the computer game player can choose to take a timeout of the race for simulated maintenance or modification operations, which will qualify the computer game player, upon returning to the actual race, to associate the computer game player's car with any real car that has undergone similar maintenance or modification operations. 8) A system and method are described in which multiple computer game players, up to the number of real cars in the race, can be associated with the real cars during the race, thereby competing against each other and the real cars in a single, real, live race. 9) Real-world live races involving driverless or autonomous vehicles, and numerous A system and method for collective interaction with computer game players is described, whereby each player can associate their simulated car with one of the real cars to start a race, and progress by associating from real car to real car based on defined parameters, thereby competitively competing as a virtual driver in a real race against the real drivers and all other computer game players. 10) A system and method are described for game players to compete in mass e-sports races, where each player is required to interact with a real-life race in a realistic, engaging, and challenging manner, and where a fair method is required to rank a large number of players based on their performance and final placement in an e-sports event, where the method is based on highly accurate positions of the computer game players' simulated cars relative to one another. 11) A system and method are described in which a computer game player's interaction is with a recorded version of a past race, the recording consisting of highly accurately measured dynamic performance of a real car in the environment of the past race and the inputs of a real driver to the real car during that race. 12) Alternatively, systems and methods are described in which the computer game player's interaction is with a recorded version of a past race, the recording consisting solely of highly accurately measured dynamic performance of real cars in that environment, where the digital pairing is based on any other method that ensures that the simulated performance of the game player's car in that environment, associated with the digital record of the dynamic performance of the real cars in those environments, is sufficiently close to a representation of that performance to create fair competition between the game player and real drivers or autonomous vehicles. 13) Systems and methods are described in which, when a computer game player's simulated car is replaced by a real car, there is an exchange of video, audio, or any other technical data between the game player's computer system and the real car and / or its associated team and their facilities in any direction and for any purpose. 14) A system and method are described in which each computer game player is assigned a handicap based on a fair assessment of their past experience and accumulated skill, and the handicap is used by the computer game to moderate the response of cars in its environment to player inputs, so that players of different experience and skill levels have approximately equal opportunities to compete against real drivers in real cars and / or against each other and win races. 15) Systems and methods are described that allow car racing fans who are not computer game players to participate in an interactive spectator experience by associating any number of real cars with the computer game player's car, optionally using any of the methods described above, and to receive technical data, video streams, audio streams, or any other data included in the systems and methods described above. 16) A gaming player's virtual vehicle can be coupled with a representation of a real car to receive input position data for the real vehicle in the race and real driver inputs, which can be used to generate a reference black-box representation. A system and method are described that evaluates gaming player inputs during the race against the reference black-box representation to determine a simulated gaming environment that can match the virtual player's vehicle to the representation of the real vehicle and simulate a real racing environment. 17) Systems and methods are described whereby real vehicle dynamic input and output data is captured and analyzed using black box modeling methods, and the results can be fed back to competing teams in real time to aid their understanding of driver and car performance, and in particular provide teams with highly accurate kinematic data about their car's performance that was previously unavailable.

[0132] A system and method for detecting vehicle movement using a group of position detection devices positioned around a racetrack is described. Each detection device includes an IR sensor for detecting IR radiation emitted, reflected, or transmitted from the vehicle to determine the vehicle's track position, and communication means for transmitting the position data to a position data capture system, where the data is collated and provided to a gaming device or server as live position data for a racing event involving those vehicles. However, as noted above, it will be appreciated that the motion tracking system is not limited to vehicle movement. The accuracy of using a group of infrared sensors, suitably configured and connected to other infrared sensors, allows for accurate location detection of moving objects and is applicable to non-motor sports, for example, involving human or animal movement.

[0133] It will also be understood that various variations of the described embodiments are possible, and that elements of one embodiment can be readily combined with elements of other embodiments. Accordingly, it will be understood that the methods and systems described herein are non-limiting examples of how different aspects of the invention may be implemented, and that the invention is defined by the spirit and scope of this disclosure.

Claims

1. 1. A computer-implemented method for controlling an interactive hybrid environment representing a motorsport event on a track, the interactive hybrid environment including representations of real and virtual vehicles on the track, the method comprising: receiving a stream of real sensor data, the real sensor data comprising: a. Actual kinematic data of a real vehicle on the track, captured by an infrared sensor on the track; b. real control data captured by vehicle sensors and obtained from the real vehicle via a telemetry system relating to a driver's control of the real vehicle; using the real-world kinematic data to determine the position and kinematic behavior of a representation of the real vehicle within the interactive hybrid environment; using the real control data and the real kinematic data to generate a black-box determination of the position of the real vehicle on the track based on the real control data; receiving a stream of computer-generated control data resulting from user interaction with a computer that presents the interactive hybrid environment to the user and captures user input for controlling the kinematic behavior of a representation of a virtual vehicle; and determining the position and kinematic behavior of the representation of the virtual vehicle within the interactive hybrid environment by using the black-box determination and the computer-generated control data.

2. The computer-implemented method of claim 1 , wherein the real sensor data includes real kinematic data of a plurality of real vehicles on the track and real control data relating to driver control of each of the plurality of real vehicles.

3. The computer-implemented method of claim 2 , wherein the real sensor data for each real vehicle of the plurality of real vehicles includes a vehicle identifier.

4. 4. The computer-implemented method of claim 2 or 3, wherein the stream of computer-generated control data comprises multiple streams of computer-generated data, each stream generated by a different user interaction with a respective computer and capture of a respective user input.

5. The computer-implemented method of claim 4 , wherein each stream of computer-generated control data in the plurality of streams of computer-generated data includes a computer device identifier.

6. 5. The computer-implemented method of claim 2, wherein the plurality of real vehicles is less than the plurality of streams of computer-generated data, and the method further comprises associating a subset of the plurality of representations of the virtual vehicle with a representation of a real vehicle to generate an associated representation.

7. 7. The computer-implemented method of claim 6, further comprising using the associated representations to represent a subset of the plurality of representations of the virtual vehicle within the interactive hybrid environment while positions of the subset of virtual vehicles are within tolerance limits of the real vehicle.

8. 8. The computer-implemented method of claim 6 or 7, wherein the plurality of streams of computer-generated data are multiple times more numerous than the plurality of real vehicles, and the associating step includes associating each of the plurality of computer-generated data with a plurality of representations of the real vehicles with an average distribution.

9. 8. The computer-implemented method of claim 6 or 7, wherein the plurality of streams of computer-generated data are multiple times more numerous than the plurality of real vehicles, and the associating step includes associating each of the plurality of computer-generated data with a plurality of representations of the real vehicles in a logarithmic distribution.

10. updating the interactive hybrid environment with new positions of the representations of the real and virtual vehicles as determined by the received real sensor data and the computer-generated data; generating an updated interactive hybrid environment; The computer-implemented method of any one of claims 1 to 9, further comprising broadcasting the updated interactive hybrid environment from a central server to a plurality of remotely located computers.

11. broadcasting the black box determinations and the real-world sensor data from a central server to a plurality of remotely located computers; generating the interactive hybrid environment on each remotely located computer; updating the interactive hybrid environment with new positions of the representations of the real and virtual vehicles as determined by the received real sensor data and the computer-generated data; The computer-implemented method of any one of claims 1 to 9, further comprising the step of: transmitting the new position of the representation of the virtual vehicle to the central server.

12. 12. The computer-implemented method of claim 1, further comprising using an artificial intelligence engine that references the black-box determination to vary the association between the computer-generated control data and the resulting position of the virtual vehicle.

13. The computer-implemented method of any one of claims 1 to 12, wherein the received real kinematic data comprises longitudinal position data relative to the truck, lateral position data relative to the truck, and heading data relative to the truck.

14. The computer-implemented method of any one of claims 1 to 13, wherein the real control data comprises one or more of steering wheel position, accelerator position, brake pedal position, and gear selection of the real vehicle.

15. 15. The computer-implemented method of claim 1, further comprising retrieving the real sensor data from a data store that stored a copy of the real sensor data at the time the real sensor data was generated.

16. The computer-implemented method of any one of claims 1 to 14, wherein the receiving step comprises receiving the real-world sensor data in near real-time as the sporting event is taking place.

17. The real-world sensor data stream has a sampling rate of at least 25 Hz. and wherein the position of the real vehicle at a point in time is captured and provided to the interactive hybrid environment within 40 milliseconds of being captured.

18. 17. The computer-implemented method of claim 1, wherein the stream of real sensor data has a sampling rate of at least 60 Hz, and wherein the position of the real vehicle at a point in time is captured and provided to the interactive hybrid environment within 16.7 milliseconds of being captured.

19. The computer-implemented method of any one of claims 1 to 18, further comprising using a stored data model to generate the interactive hybrid environment.

20. The computer-implemented method of any one of claims 1 to 19, further comprising receiving a stream of video data from the real vehicle and including the stream of video data in the interactive hybrid environment.

21. The computer-implemented method of any one of claims 1 to 20, further comprising receiving a stream of acoustic data from the real vehicle and including the stream of video data in the interactive hybrid environment.

22. 22. The computer-implemented method of claim 2, or any one of claims 3 to 21 dependent on claim 2, further comprising associating a representation of a virtual vehicle of the plurality of virtual vehicles with a representation of a real vehicle of the plurality of real vehicles at a time when the position of the representation of the virtual vehicle is within a predetermined threshold of the position of the representation of the real vehicle, and using the representation of the real vehicle as the representation of the virtual vehicle in the interactive hybrid environment.

23. 23. The computer-implemented method of claim 22 when dependent on claim 20 or 21, wherein the associating step operates to provide the stream of audio data or video data received from the real vehicle to a computer that presents the interactive hybrid environment to a user.

24. 24. The computer-implemented method of claim 22 or 23, further comprising disassociating a representation of a virtual vehicle of the plurality of virtual vehicles from a representation of a real vehicle of the plurality of real vehicles when the position of the representation of the virtual vehicle is outside a predetermined threshold of the position of the representation of the real vehicle, and presenting the representation of the virtual vehicle separately from the representation of the real vehicle within the interactive hybrid environment.

25. The computer-implemented method of any one of claims 22 to 24, further comprising providing details of any virtual vehicle associated with the representation of the real vehicle to a remotely located third party computer.

26. 26. A computer-implemented method according to claim 2, or any one of claims 3 to 25 dependent on claim 2, wherein each real vehicle has a set of performance characteristics, and the method further comprises determining a representation of a real vehicle from the plurality of real vehicles that is located closest to the representation of the virtual vehicle, and introducing the set of performance characteristics of the representation of the closest real vehicle as performance characteristics of the virtual vehicle.

27. using the infrared sensor to capture position data of the actual vehicle on the track; 27. The computer-implemented method of any one of claims 1 to 26, further comprising converting the position data into a stream of real-world kinematic data over time and transmitting the stream of real-world kinematic data in real-time to a central server.

28. 28. The computer-implemented method of claim 27, wherein the capturing step includes capturing the position data using a group of sensors monitoring different locations of the track, each sensor in each group of sensors detecting infrared radiation emitted, reflected, or transmitted from one or more vehicles operating on the track within a field of view (FOV) of the sensor.

29. 30. The computer-implemented method of claim 28, further comprising processing the infrared emissions detected by an infrared sensor to determine kinematic data of one or more real vehicles operating on the track.

30. 1. A computer system for controlling an interactive hybrid environment representing a motorsport event on a track, the interactive hybrid environment including representations of real and virtual vehicles on the track, the system comprising: a receiver for receiving a stream of real sensor data, the real sensor data including real kinematic data of a real vehicle on the track and real control data relating to control of the real vehicle by a driver, the real kinematic data being captured by infrared sensors on the track and the real control data being captured by vehicle sensors and obtained from the real vehicle via a telemetry system; a virtual race command processor configured to receive a stream of computer-generated control data resulting from user interaction with a computer that presents the interactive hybrid environment to a user and captures user input for controlling the kinematic behavior of the virtual vehicle representation; a virtual racing simulation engine; The virtual racing simulation engine a racing simulation output engine for using the real-world kinematic data to determine the position and kinematic behavior of a representation of a real vehicle within the interactive hybrid environment; a reference black-box model generator configured to use the real control data and the real kinematic data to generate a black-box determination of the position of the real vehicle on the track based on the real control data; a gaming black box implementation engine configured to determine the position and kinematic behavior of a representation of the virtual vehicle within the interactive hybrid environment by using the black box determination and the computer-generated control data.

31. 31. The computer system of claim 30, further comprising an artificial intelligence engine configured to change an association between the virtual vehicle's final position and the final position of the virtual vehicle.

32. 32. The computer system of claim 31, wherein the artificial intelligence engine is configured to scale a threshold required from the received computer-generated control data to generate a predetermined position for the virtual vehicle.

33. 1. A detection system for providing location data of one or more mobile objects operating in a playing area to a central server, the detection system including a plurality of sensor groups, each of the sensor groups configured to monitor a portion of the playing area, each of the sensor groups comprising: The present invention provides a method for monitoring a plurality of position detection devices arranged around the playing area, each of which is configured to monitor a portion of the playing area different from an upper position, and each of which: an infrared sensor having a field of view (FOV) for detecting infrared radiation emitted, reflected, or transmitted from one or more moving objects operating over the competition area within the FOV and generating a sensor output; a transmitter configured to transmit the sensor output of the infrared sensor or information derived therefrom to another one of the plurality of position detection devices in the sensor group of the position detection device, the position detection device acting as a communication node for the sensor group; a communication facility communicatively coupled to the location detection device operating as the communication node in one sensor group, the communication facility configured to transmit the sensor output of each infrared sensor of the sensor group or information derived therefrom to a central matching server.

34. 34. The sensing system of claim 33, wherein each of the sensor groups includes 10 or fewer position sensing devices.

35. 35. The detection system of claim 33 or 34, wherein at least one of the position sensing devices of the sensor group includes a processor configured to determine current kinematic data in at least two dimensions of the one or more moving bodies operating over the competition area within the FOV based on the sensor output or information derived therefrom.

36. 36. The detection system of claim 33, wherein a first group of sensors of the plurality of groups of sensors is configured to relay the sensor output or information derived therefrom determined by the first group of sensors to a second group of sensors of the plurality of groups of sensors.

37. 37. The detection system of any one of claims 33 to 36, wherein one or more of the plurality of detection devices comprises a long wave infrared (LWIR) microbolometer or a mid wave infrared (MWIR) photon detection camera configured to detect thermal IR emitted by one or more real moving objects on the playing area.

38. 38. A detection system according to any one of claims 33 to 37, wherein one or more of the plurality of detection devices comprises a short wave infrared (SWIR) or near infrared (NIR) photon detection camera for detecting broadband or narrowband light emitted, reflected or transmitted from the moving object.

39. A sensing system according to any one of claims 33 to 38, wherein one or more of the plurality of position sensing devices has a frame rate of at least 60 Hz.

40. 40. The detection system of claim 33, wherein one or more of the plurality of position detection devices includes an LED floodlight directed toward a portion of the playing area and is configured to detect reflected light from the LED floodlight.

41. A detection system according to any one of claims 33 to 40, wherein one or more of the plurality of position sensing devices are configured to detect an infrared signature of the moving object, the infrared signature consisting of modulated infrared light.

42. 42. The detection system of claim 41, wherein one or more of the plurality of location detection devices are configured to detect a unique identifier of the mobile object based on the infrared signature. Hmm.

43. A detection system as described in any one of claims 33 to 42, wherein one or more of the plurality of position detection devices are configured to detect infrared radiation reflected or emitted at the edge of the playing area, and the system uses the detected information as a reference system to determine the lateral position of the moving object.

44. 44. A detection system as claimed in any one of claims 33 to 43, wherein at least some of the position detection devices have a boresight with a field of view (FOV) that is at an acute angle to the horizontal plane and are positioned, in use, to face an approaching moving object as the moving object advances within the competition area.

45. 45. The sensing system of any one of claims 33 to 44, wherein at least some of the plurality of position sensing devices include a FOV of 20 to 30 degrees and a detection range of up to 50 meters.

46. 46. ​​The sensing system of any one of claims 33 to 45, wherein at least some of the plurality of position sensing devices include a 70 degree FOV and a detection range of up to 15 meters.

47. 47. The sensing system of any one of claims 33 to 46, further comprising a GPS receiver providing a time stamp for the sensor output or information derived therefrom, the system being configured to use the time stamp to establish a common time reference for sensor output or data derived therefrom from at least some of the location sensing devices.

48. A detection system as described in any one of claims 33 to 47, wherein the communication equipment is configured to operate at a refresh rate of at least 25 Hz and provide sensor output or information derived therefrom of the one or more mobile objects operating on the competition area to the central server.

49. A detection system as described in any one of claims 33 to 47, wherein the communication equipment is configured to operate at a refresh rate of at least 60 Hz and provide sensor output or information derived therefrom of the one or more mobile objects operating on the competition area to the central server.

50. 49. The detection system of any one of claims 33 to 48, wherein the system is configured to determine a longitudinal position along the playing area, a lateral position across the playing area, and a rotational orientation of the moving object.

51. A detection system according to any one of claims 33 to 50, wherein a plurality of position detection devices of the sensor group are arranged in series, and a position detection device located at an intermediate point of the series arrangement functions as a communication node of the sensor group.

52. The detection system according to any one of claims 33 to 51, wherein the moving object includes a vehicle and the competition area includes a track.

Images