System and method for controlling an interactive hybrid environment representing a motor sports event on a track

Abstract

A computer-implemented method for controlling an interactive hybrid environment representing a motorsports event on a track is described. The interactive hybrid environment includes representations of real and virtual vehicles on the track. The method includes receiving streams of real sensor data, the real sensor data including real kinematic data of a real vehicle on the track captured by infrared sensors on the track and real control data related to a driver's control of the real vehicle captured by vehicle sensors and obtained from the real vehicle via a telemetry system. Using the real kinematic data, the method determines the position and kinematic behavior of the representation of the real vehicle within the interactive hybrid environment. Using the real control data and the real kinematic data to generate a black-box determination of the position of the real vehicle on the track based on the real control data. Receiving streams of computer-generated control data resulting from user interaction with a computer that presents the interactive hybrid environment to the user and captures user inputs to control the kinematic behavior of a virtual vehicle representation. Using the black-box determination and the computer-generated control data, the method determines the position and kinematic behavior of the virtual vehicle representation within the interactive hybrid environment.

Description

Technical Field

[0001] The present invention relates to a system and method for controlling an interactive hybrid environment representing a motor sports event in a track. More specifically, the present invention relates to a method and system for a remotely located gaming computer or other possible entertainment device to compete in a group against a real motor sports event, but is not limited thereto. The present invention also captures and live broadcasts high-precision real-time vehicle tracking and vehicle control data from a live motor sports event in real time, and uses the data to enable a remotely located gamer to compete with actual participants in a motor sports event or a remotely located viewer to interactively compete with the event, and also to provide a new form of gaming or viewing experience. By capturing accurate kinematic data, the present system and method can be applied to group competitions in the scope of other types of indoor and outdoor sports events such as soccer, basketball, cycling, and skiing.

Background Art

[0002] To provide a physical replica system and method for virtual representation of motor sports events controlled by a player, various approaches have been used. Most of them are completely virtual and use complex models that attempt to provide apparent realism by determining the kinematic behavior of virtual vehicles in response to user input to actuators. This realism is often provided by constructing a model that uses real kinematic data obtained by recording the movement of vehicles during a race. However, such models are not only complex and difficult to construct, but are often based on inaccurate kinematic data. Also, such prior art approaches are often directed at single or a small number of players, which means that the scope of interaction with other players is very limited. All of these result in only non-scalable and unrealistic simulations and gaming experiences being achievable.

[0003] None of the known prior art attempts to define a system and method for simulation for a player to interact (dialogue) with a real live race has been extended to a practical method that can prove to a large number of players and followers around the world that they can engage in group battles simultaneously with real drivers, cars, racing teams, tracks, off-track competition courses, and event environments, and be satisfying and challenging for computer game players, support group competition in e-sports tournaments, and provide this feature to followers who want to participate in games with a greater desire to immerse themselves and enjoy live motor sports events. Achieving any of these objectives would, for example, expand and enhance the appeal of automobile racing to fan segments that include from traditional television viewers or fans of famous drivers, through core automobile racing computer game fans, to consumers of "over the top" racing and technical data services.

[0004] The inventor of the present application is the author of the prior art UK Patent No. 2518602 B, which describes a system, method, and technology for accurately tracking vehicles. This patent provides the ability to accurately track vehicles, particularly in a real live motor race (e.g., Formula 1), by replacing one of these vehicles with a virtual vehicle for a computer gamer to compete in a virtual race, except in passive and worst weather conditions. The data provided by UK Patent No. 2518602 B relates to the position information of vehicles on a track, and is non-invasively detected by a single IR tracking sensor placed at a relatively high altitude (1 - 2 km) above a racing circuit, such as on a helicopter, drone, or light aircraft. This is based on the field of view (FOV) of a single sensor covering the entire combat area. Taking Formula 1 as an example, this approach cannot be implemented when the FOV is obstructed by plants (tall trees) or grandstand spectators, or in the case of a street racing circuit where it is obstructed by all buildings and other structures. Therefore, this prior art approach is somewhat limited in practical applications.

[0005] Video games that interact with live events are known, which describe in-game methods where a player-controlled virtual vehicle interacts with the representation of a real vehicle in a limited and somewhat artificial manner via data distributed from a live event to a conventional computer racing game (see, for example, US Patent Application Publication No. 2010 / 0271367). The performance of the virtual vehicle is determined by a combination of the player's input, the vehicle's software model, and its environment. This physics-based model, which is part of the video game software, has many limitations in achieving the fidelity required to simulate the very complex and dynamic scenarios typical of an automobile racing event. This results in a poor quality of interaction between the virtual vehicle and the representation of the real vehicle.

[0006] Other prior art documents (e.g., UK Patent Application Publication No. 2365360 A) attempt to solve this major shortcoming in that the physical model simulation of the dynamics of a virtual vehicle in a computer game and its environment can be pre-specified by data collected from actual driving, or specified in real time from performance data transmitted live from an actual vehicle and its environment, and then generate an "optimal physical model" of the vehicle and its environment that can be controlled by the game player. This approach suffers from the same basic constraints in that there are no realistic means described to achieve extremely high performance tracking. Also, since the software model of a realistic scenario involving a very complex vehicle operating in a very complex environment includes a very large number of variables and extremely complex relationships between variables, even when some of the more obvious variables are continuously measured and used, this model will, a) Due to many approximations and deficiencies in the model, deviate very quickly from reality, or b) Require an enormous amount of computing power and may not be able to operate in real time even on the largest computers in some cases.

[0007] Still other prior art documents (e.g., U.S. Patent No. 6155927) describe in an abstract form a system that enables a computer game player to compete in live races and recorded real-world races, but do not describe anything about the specific methods and systems described in UK Patent No. 2518602 B that would actually make this abstract concept realizable.

[0008] UK Patent Application Publication No. 2585165 A, in which the inventor of the present application is also a co-author, describes an infrared (IR) tracking approach for ordinary vehicles, large trucks, etc. on the transportation networks of highways, lanes, and streets. Here, the tracking device is arranged on infrastructure at a low altitude such as a street lamp post, and an IR radiator is arranged on the vehicle to enable the detection of the vehicle, or alternatively, an IR reflector is arranged on the vehicle, and an IR lamp is selectively provided in the tracking device. Subsequently, high-precision real-time tracking data for the vehicle and its vicinity is communicated to the vehicle by the tracking device to assist in vehicle navigation and autonomous or semi-autonomous driving. The tracking devices are connected to form a highly reliable linear network, enabling the vehicle to be tracked along a continuously and uniformly extended lane. The arrangement of the tracking device described in UK Patent Application Publication No. 2585165 has features not related to the present invention (for example, the aggregation and distribution of highly reliable data from the tracking device to the vehicle itself). Furthermore, motorsport events typically require tracking high-performance vehicles such as cars and motorcycles along a closed and heterogeneous circuit with challenging shapes (sharp curves, obstacles, inclines, bumps, pit stops, etc.) where the vehicles travel at extremely high accelerations, speeds, cornering speeds, etc. Therefore, the system and method described in UK Patent Application Publication No. 2585165, which is generally configured to monitor slow-moving vehicles, would not be able to handle it well.

[0009] Accordingly, an object of the present invention is to overcome the limitations of the above-described prior art documents. Also, in different embodiments, it is an object to overcome the limitations regarding the accurate tracking ability as defined in UK Patent No. 2518602 B and described above, and to provide, for high-performance motor sports events in complex environments, for the first time a practically realizable improved system and method for a group battle of interactive virtual-reality car races in which a large number of players and followers worldwide can simultaneously participate. Also, in different embodiments, it is an object to identify how a virtual driver can interact with a real race and a real driver to enhance the player's experience, and to enable a local, regional, national, or worldwide e-sports or automotive racing event to be conducted fairly while being highly integrated with real motor sports events and competitions. The present invention is also applicable to the generation and use of recorded data for a real motor sports event in which kinematic data and vehicle control data (optionally driver input) of competing vehicles are faithfully recorded or reconstructed to meet the accuracy and latency requirements for computer gaming as defined in UK Patent No. 2518602 B. SUMMARY OF THE INVENTION

[0010] According to one aspect of the present invention, there is provided a computer-implemented method for controlling an interactive hybrid environment representing a motorsport event on a track. The interactive hybrid environment includes representations of real and virtual vehicles on the track, and the method includes receiving a stream of real data, the real data including real kinematic data of a real vehicle on the track captured by an infrared sensor on the track and real control data regarding the control of the real vehicle by a driver captured by a vehicle sensor and obtained from the real vehicle via a telemetry system; determining the position and kinematic behavior of the representation of the real vehicle within the interactive hybrid environment using the real kinematic data; using the real control data and the real kinematic data to generate a black box determination of the position of the real vehicle on the track based on the real control data; receiving a stream of computer-generated control data obtained by user interaction with a computer that presents the interactive hybrid environment to the user and captures user input for controlling the kinematic behavior of the representation of the virtual vehicle; and determining the position and kinematic behavior of the representation of the virtual vehicle within the interactive hybrid environment by using the black box determination and the computer-generated control data.

[0011] The use of real data that includes both kinematic data and control data in an interactive hybrid environment enables the generation of accurate black box determinations. This in turn provides a reference for the received computer-generated control data and enables accurate kinematic control of a virtual vehicle that is an exact replica in relation to the representation of a real vehicle on a track. The degree to which a user operates an actuator is accurately reflected, for example, in the effect on the virtual vehicle, in the same way that driver operation of controls in a real vehicle affects the kinematic data of the real vehicle (e.g., speed, orientation, acceleration, and position). This provides a realism that was not possible with prior art approaches and enables the movement in both the virtual domain and the real domain to be accurately reflected. Further, this combination of features overcomes the interoperability issue where vehicle representations from different domains are provided in the same realistic and accurately identical hybrid environment. When the real data is live data, i.e., the data is streamed from events that are occurring simultaneously with the control of the interactive hybrid environment, for example, the present invention enables a virtual driver in the virtual domain to compete with a real driver in the real domain in real time. This was not previously possible.

[0012] Preferably, the real sensor data includes the real kinematic data of a plurality of real vehicles on a track and the real control data regarding the control by the respective drivers of the plurality of real vehicles. Embodiments of the present invention are configured to enable the competitive behavior of a number of vehicles to be captured overall, for example, as recognized in a Formula 1 race. Having a stream of real data representing the behavior of a plurality of real vehicles enables the generation of such a competitive environment. Each real data stream can be processed separately and can include a unique vehicle identifier.

[0013] Similarly, a stream of computer-generated control data may include a plurality of streams of computer-generated data, each stream being generated by different user interactions with each computer and the capture of each user input. This enables a large number of users to be associated with a sports event and, advantageously, allows for the realization of group games. Each computer-generated data stream can be processed separately and may include a unique computer device identifier.

[0014] Preferably, the plurality of real vehicles is less than the plurality of streams of computer-generated data, and the method further includes associating a subset of the plurality of representations of virtual vehicles with the representation of a single real vehicle to generate the associated representation. Clearly, in a group gaming environment, the number of users participating via gaming computers / devices far exceeds the number of vehicles in a race (because typically there are physical limitations on the number of vehicles that can participate in a race for safety reasons). Thus, by associating the representations of two or more virtual vehicles with the representation of a single real vehicle, it becomes possible to accommodate any number of users. This enables the method to be extended to group game scenarios where hundreds of thousands of users in the virtual domain can compete simultaneously with drivers in the real domain.

[0015] In some embodiments, the method further includes using this associated representation to represent a subset of the plurality of representations of virtual vehicles within an interactive hybrid environment while the positions of the subset of virtual vehicles are within the tolerances of the real vehicle.

[0016] Typically, in many embodiments, there are multiple times more streams of computer-generated data than there are multiple real vehicles. The associating step may include associating each of the multiple streams of computer-generated data with multiple representations of a real vehicle in an even distribution. In other embodiments where there are multiple times more streams of computer-generated data than there are multiple real vehicles, the associating step may include associating each of the multiple computer-generated data with multiple representations of a real vehicle in a logarithmic distribution. In either case, in an interactive hybrid environment with a small number of real vehicle representations, it is possible to accommodate a huge number of players / users.

[0017] In one embodiment, it is possible to have a central gaming server that generates an interactive hybrid environment and then provides this to all gaming devices via a communication network. In this case, the method may further include updating the interactive hybrid environment with new positions of representations of real and virtual vehicles as determined by received real sensor data and computer-generated data, generating an updated interactive hybrid environment, and broadcasting the updated interactive hybrid environment from the central server to a plurality of remotely located computers. The main solution may require greater processing power, but the updating and control are relatively easy.

[0018] In an alternative embodiment, each user / player's gaming device locally generates an interactive hybrid environment, and there are multiple such generated local environments. In this case, the method may further include the steps of broadcasting black box determination and real sensor data from a central server to a plurality of remotely located computers, generating an interactive hybrid environment at each remotely located computer, updating the interactive hybrid environment with a new position of the real vehicle and virtual vehicle representations as determined by the received real sensor data and computer-generated data, and transmitting the new position of the virtual vehicle representation to the central server. Such a distributed solution may require more management but is not troubled by potential obstacles and generally is not affected by time delays or lags in the generation of the interactive hybrid environment.

[0019] In some embodiments, the method further includes the step of varying the relevance between computer-generated control data and the resulting virtual vehicle position using an artificial intelligence engine that references the black box determination. The use of such an artificial intelligence engine enables assistance to be provided to each player in controlling the player's virtual vehicle. Such assistance can provide handicap factors that enable weaker virtual gamers to compete fairly with professional drivers in the real domain.

[0020] In most embodiments, the received real kinematic data includes longitudinal position data relative to the track, lateral position data relative to the track, and vehicle orientation data relative to the track. These types of data enable the kinematic behavior of the vehicle to be accurately mapped in the virtual domain.

[0021] In some embodiments, the real control data includes the position of one or more steering wheels, the position of the accelerator, the position of the brake pedal, and gear selection of the actual vehicle. These are typical control data, and the information is provided by the telematics system, and the control data helps to determine the driver control input for determining the track position of the actual vehicle. Also advantageously, these data can be easily associated with corresponding actuators that can be controlled by the player in the virtual world.

[0022] Different embodiments generate an interactive hybrid environment from real data from different sources. In one embodiment, this source is a storage device for previously recorded real data. Thus, the method may further include the step of retrieving real sensor data from a data storage device that stores a copy of the real sensor data when the real sensor data was generated. In an alternative embodiment, this source is the sports event itself, in which case the receiving step includes receiving real sensor data at approximately real-time when the sports event is taking place.

[0023] In one embodiment, the stream of real sensor data has a sampling rate of at least 25 Hz, the position of the actual vehicle is captured at a certain point in time, and is provided to the interactive hybrid environment within 40 milliseconds after being captured. This enables the real-time realization of the real domain within the virtual domain operating at a minimum refresh rate of 25 Hz. More specifically, in some embodiments, the stream of real sensor data has a sampling rate of at least 60 Hz, the position of the actual vehicle is captured at a certain point in time, and is provided to the interactive hybrid environment within 16.7 milliseconds after being captured. This refresh rate is typically what is provided in most computer monitors, and thus helps to support the high-quality representation of real-time events within the virtual domain.

[0024] To assist in generating an interactive virtual environment, the method may further include the step of using a stored data model. This makes it possible to make the virtual environment more realistic for the player.

[0025] Furthermore, it is possible to receive video and audio data streams from a real vehicle to enhance the interactive hybrid environment. In this case, the method further includes the steps of receiving a stream of video data or audio data from the real vehicle and including the stream of video data or audio data in the interactive hybrid environment.

[0026] To facilitate a group game, in some embodiments, the method associates the representation of one virtual vehicle among a plurality of virtual vehicles with the representation of one real vehicle among a plurality of real vehicles when the position of the representation of the virtual vehicle is within a predetermined threshold of the position of the representation of the real vehicle, and uses the representation of the real vehicle as the representation of the virtual vehicle in the interactive hybrid environment. Advantageously, this enables a vast number of virtual vehicles to be included in the interactive hybrid environment without cluttering the screen with representations of a vast number of virtual vehicles. In fact, this form of representation solves the technical problem of how to present a motorsport event that, in some cases, includes millions of players within a limited screen size. Furthermore, the challenge of live associating millions of computer game players and other fans around the world with a motorsport event is solved by some of these embodiments. This provides an interactive experience that is challenging, satisfying, and interesting for gamers and viewers alike, and is important for e-sports events that increasingly attract professional gamers, making it possible to fairly manage and rank all players as an integrated part of a real-virtual car racing event.

[0027] In some embodiments where audio and / or video data is provided from an actual vehicle, the step of associating (also referred to herein as "snapping") may operate to provide a stream of audio data or video data received from the actual vehicle to a computer that presents an interactive hybrid environment to the user. This makes it possible to provide the scenery and sounds experienced by a particular actual vehicle into the virtual domain, making the interactive hybrid environment more realistic.

[0028] In some embodiments, the method further includes the step of disassociating the representation of one virtual vehicle among a plurality of virtual vehicles from the representation of one actual vehicle among a plurality of actual vehicles when the position of the representation of the virtual vehicle is outside a predetermined threshold of the position of the representation of the actual vehicle, and presenting the representation of the virtual vehicle separately from the representation of the actual vehicle within an interactive hybrid environment. This enables the representation of the virtual vehicle to be displayed when the representation of the virtual vehicle does not match the actual vehicle representation, thereby making the actual position of the virtual vehicle relative to the representation of the actual vehicle visible to the player and enabling transitions between representations of actual vehicles within the interactive hybrid environment.

[0029] In some embodiments, aspects of the interactive hybrid environment can also be relayed to a team associated with the actual vehicle, as described below. In this case, the method may further include the step of providing details of any virtual vehicle associated with the representation of the actual vehicle to a remotely located third-party computer.

[0030] In some embodiments, it is possible to match the performance of the virtual vehicle with the performance of the representation of the actual vehicle arranged in the vicinity. If each actual vehicle has a different set of performance characteristics, a fairer game becomes possible. In this embodiment, the method further includes the step of determining the representation of the actual vehicle closest to the representation of the virtual vehicle among the plurality of actual vehicles, and introducing a set of performance characteristics of the representation of the closest actual vehicle as the performance characteristics of the virtual vehicle.

[0031] In some embodiments, the method further includes capturing position data of a real vehicle on a track using an infrared sensor, converting the position data into a stream of real kinematic data over time, and transmitting the stream of real kinematic data to a central server in real time.

[0032] Preferably, capturing the position data includes using a group of sensors that monitor different positions of the track, where each sensor in each group of sensors detects infrared radiation reflected or transmitted from one or more vehicles operating on the track within the field of view (FOV) of the sensor. This configuration is particularly advantageous for providing the accurate real-time information described below for an interactive hybrid environment.

[0033] In some embodiments, the method further includes processing the infrared radiation detected by the infrared sensor to determine kinematic data of one or more real vehicles operating on the track. Preferably, this processing is performed at each sensor, enabling a small amount of data to be transmitted for use in an interactive hybrid environment.

[0034] According to another aspect of the present invention, there is provided a computer system for controlling an interactive hybrid environment representing a motor sports event on a track. The interactive hybrid environment includes representations of real and virtual vehicles on the track, and the system is a receiver for receiving a stream of real sensor data, where the real sensor data includes real kinematic data of the real vehicle on the track and real control data regarding the control of the real vehicle by the driver, the real kinematic data being captured by infrared sensors on the track and the real control data being captured by vehicle sensors and obtained from the real vehicle via a telemetry system, a virtual race command processor configured to receive a stream of computer-generated control data obtained by user interaction with a computer for presenting the interactive hybrid environment to the user and capturing user input for controlling the kinematic behavior of the representation of the virtual vehicle. A virtual race simulation engine, and the virtual race simulation engine includes a race simulation output engine for determining the position and kinematic behavior of the representation of the real vehicle within the interactive hybrid environment using the real kinematic data, a reference black box model generator configured to use the real control data and the real kinematic data to generate a black box determination of the position of the real vehicle on the track based on the real control data, and a gaming black box implementation engine configured to determine the position and kinematic behavior of the representation of the virtual vehicle within the interactive hybrid environment by using the black box determination and the computer-generated control data.

[0035] In some embodiments, this computer system may further include an artificial intelligence engine configured to vary the relevance between the virtual vehicle and its final position. In some embodiments, the artificial intelligence engine may be configured to expand the required thresholds from the received computer-generated control data to generate a predetermined position of the virtual vehicle.

[0036] As will be described in detail below, some embodiments of the present invention relate to systems and methods in which millions of players of motorsport computer games around the world interact for some entertainment purpose simultaneously with a real live motorsport event. Some entertainment includes, but is not limited to, computer gaming, e-sports tournaments, streaming, viewing, gambling, and enhancing fan engagement with motorsport in general. One embodiment enables a computer game player to start a race by forming a digital pair with one of the real cars and moving from car to car or operating as an additional car according to certain parameters. The terms "car" and "vehicle" are used interchangeably herein and are understood to have the broader meaning of any vehicle. When forming a pair with one car, it is possible to closely and reliably match the performance of the virtual car in that environment using a dynamic black box simulation method, thus providing realistic, challenging, fun, and fair content between the computer game player and the real driver. Subsequently, the system and method for one player form the basis of the expanded system and method for millions of players to interact simultaneously with and in competition with the real event. The system and method may be used to enable participants who are simply watching a motor race rather than playing a computer game to interact more strongly with the motorsport event. The system and method may be used to enhance the engagement of viewers and computer gamers with a variety of sports.

[0037] Generally, embodiments of the invention relate to improvements over known prior art for capturing and live-broadcasting high-precision real-time vehicle tracking data in motor car races (e.g., Formula 1), thereby enabling a truly fair, competitive, and enjoyable race between computer gamers and professional drivers, as well as enabling a wide range of other spectator and entertainment enhancement features. This improvement involves capturing and live-broadcasting tracking data from any motor sports venue for a significant number of high-performance vehicles competing in a live event, such that the broadcast data is sufficiently accurate in real-time and in a format suitable for computer games and other entertainment media to integrate with and utilize this data, whether it is live data or previously recorded data, thereby enhancing the computer gaming experience and / or providing additional benefits such as selectable viewpoints for the user, customer-tailored streaming, user-focused advertising, live gambling, etc. to motor sports organizers and fans. In certain embodiments, while competing in a live-associated race, the gamer can change the actual vehicle selected at the start of the race to a different actual vehicle during the race, or operate as an additional vehicle with certain specific parameters, thereby allowing the gamer to match their capabilities with the drivers of other vehicles at any position where they realize they are during the race. When selecting an option to change to a new actual vehicle, the capabilities of the gamer's vehicle are matched to the new actual vehicle, and again, a fair and competitive race with new real drivers is established. And the systems and methods described herein are extended to enable an unlimited number of gamers to participate in an attractive, fair, and competitive manner. In certain situations, the gamer can also interact by viewing other virtual vehicles, but in all situations, the systems and methods enable a fair and competitive race between an unlimited number of gamers in a virtual environment and real drivers in a real environment. The combination of these is referred to as an interactive hybrid environment.The systems and methods described herein are applicable to other sports events and describe numerous representative examples.

[0038] Thus, embodiments of the present invention advantageously provide a system and method of operation for an interactive virtual motorsports event. Thereby, when using the system and method, any number of participants using a virtual environment (an interactive hybrid environment) can a) Compete in a realistic, satisfying, challenging, and fair manner as an individual virtual driver of all experience and skill levels, and, in some cases, with a limited number of other virtual drivers (any number where the total number of virtual drivers is at most the number of real drivers) and a real driver in a compatible live motorsports event, or b) Compete in a live competitive motorsports event, a highly integrated, large-scale multiplayer e-sports computer gaming event held in a realistic, satisfying, challenging, and fair manner, or any other (often self-managed among friends and relatives) computer gaming event or activity, or c) Be included as an observer using the system and method of this embodiment to enhance the live viewing or live streaming experience, and be able to interact with real live events and real drivers.

[0039] According to another aspect of the present invention, a detection system for providing position data of one or more moving bodies operating in a battle area to a central server is provided. The detection system includes a plurality of sensor groups, each sensor group being configured to monitor a part of the battle area, each sensor group including a plurality of position detection devices arranged around the battle area, each position detection device being configured to monitor a part of the battle area different from the upper position, each position detection device being an infrared sensor having a field of view (FOV), the infrared sensor being configured to detect infrared radiation emitted, reflected, or transmitted from one or more moving bodies operating on the battle area within the FOV and generating a sensor output, a transmitter configured to transmit the sensor output of the infrared sensor or information derived therefrom to another position detection device operating as a communication node of the plurality of position detection devices in the sensor group of the position detection device, and a communication facility communicably coupled to the position detection device operating as a communication node in one sensor group, the communication facility being configured to transmit the sensor output of each infrared sensor in the sensor group or information derived therefrom to a central collation server.

[0040] In some embodiments, each sensor group includes 10 or fewer position detection devices. Minimizing the number of position detection devices advantageously ensures an optimal balance between data transmission delay and system complexity due to the need for additional communication paths.

[0041] In some embodiments of a group, at least one of the position detection devices in the sensor group includes a processor configured to determine current kinematic data of one or more moving bodies operating on the battle area within the FOV in at least two dimensions based on the sensor output or information derived therefrom. This feature can greatly improve the reduction of the amount of data to be transmitted around the system, and since the sensor output is processed before being transmitted to the communication node, it is possible to increase the information transmission speed.

[0042] In some embodiments, the first sensor group among the plurality of sensor groups is configured to relay the sensor output determined by the first sensor group or information derived therefrom to the second sensor group among the plurality of sensor groups.

[0043] In various embodiments, one or more of the plurality of position detection devices may include a long-wave infrared (LWIR) microbolometer or a mid-wave infrared (MWIR) photon detection camera configured to detect thermal IR radiated by one or more real moving bodies on the combat area. Also, one or more of the plurality of detection devices may include a short-wave infrared (SWIR) or near-infrared (NIR) photon detection camera for detecting broadband or narrowband light radiated, reflected, or transmitted from the moving body.

[0044] Preferably, in order to accurately detect vehicles moving very fast and quickly, one or more of the plurality of position detection devices may have a frame rate of at least 60 Hz, more preferably at least 100 Hz. This is typically comparable to the minimum gaming refresh rate of 25 Hz to 60 Hz.

[0045] Preferably, in some embodiments, one or more of the plurality of position detection devices are configured to detect a unique identifier of the moving body based on an infrared signature. This is extremely useful for tracking purposes where there are a number of moving bodies being tracked within the same FOV of the position detection device. The unique identifier name may be a modulated IR signal from a radiator on the moving body, and each different moving body has a different modulation signal.

[0046] One or more other selectable features of the plurality of position detection devices include LED projection lighting directed toward a portion of the combat area and are configured to detect reflected light from the LED projection lighting. Such lighting of the actual vehicle provides more resilience in dark and inclement weather conditions. In some embodiments, one or more of the plurality of position detection devices are configured to detect an infrared signature of the vehicle consisting of modulated infrared light. Such modulation provides additional resilience against changing environmental factors and, advantageously, also enables more accurate distance measurement.

[0047] To assist in relative position determination, one or more of the plurality of position detection devices may be configured to detect infrared radiation reflected or emitted at the edge of the combat area, and the system can use the detected information as a reference system to determine the lateral position of the moving body.

[0048] In some embodiments, the position detection devices may be oriented to face an approaching actual vehicle. To capture the infrared signature of the actual vehicle, they may be arranged at an angle with respect to the horizontal and vertical directions. More specifically, in these embodiments, at least some of the position detection devices have a bore sight (center line) of the field of view (FOV) that is an acute angle with respect to the horizontal and vertical planes and, in use, are arranged to face an approaching moving body as the moving body advances within the combat area. Different configurations are possible, and thus, in some embodiments, at least some of the plurality of position detection devices include an FOV of 20 - 30 degrees and a detection range of up to 50 meters. In other embodiments, at least some of the plurality of position detection devices include an FOV of 70 degrees and a detection range of up to 15 meters.

[0049] In some embodiments, the system further comprises a GPS receiver. The GPS receiver provides a time stamp for sensor output or information derived therefrom, and the system is configured to use the time stamp to establish a common time reference for sensor output or data derived therefrom from at least some of the position detection devices.

[0050] As described above, in some embodiments, the communication facility is configured to operate at a refresh rate of at least 25 Hz to provide sensor output or information derived therefrom of one or more moving objects operating in the combat area to the central server. Using this refresh rate provides data of sufficient resolution to enable tracking of high-speed vehicles (e.g., operating at up to 220 mph), and these kinematic data are provided in an interactive hybrid environment. However, in a more preferred embodiment, the communication facility is configured to operate at a refresh rate of at least 60 Hz to provide sensor output or information derived therefrom of one or more moving objects operating in the combat area to the central server. Using this refresh rate is consistent with the use of most computer gaming monitors and thus helps to provide a near-photorealistic representation of the movement of moving objects in, for example, an interactive hybrid environment.

[0051] In some embodiments, the processor of each position detection device is configured to determine a longitudinal position along the combat area, a lateral position across the combat area, and the rotational orientation of the moving object.

[0052] In some embodiments, a plurality of position detection devices of the sensor group are arranged continuously, and the position detection device located at the midpoint of the continuous arrangement functions as a communication node of the sensor group. This configuration shortens the communication hop between position detection devices to the communication node. Thus, for example, in a group of 9 sensors, the 5th sensor is the communication node, and for the sensor output or information derived therefrom of any position detection device, the maximum number of hops to reach the communication node is 4 hops.

[0053] In an embodiment of the motor sports event, the moving body includes a vehicle and the competition area includes a track.

[0054] According to another aspect of the present invention, there is provided a system for generating and controlling an interactive hybrid environment representing a motor sports event on a track. The interactive hybrid environment includes representations of real vehicles and virtual vehicles on the track, and the system includes a combination of the computer system and the detection system described above.

[0055] According to another aspect of the present invention, there is provided a computer-implemented method for determining an updated position of a virtual vehicle operated by one or more users on a virtual representation of a track using data from one or more physical vehicles on the track. The method includes, in a processor, associating each of the virtual vehicles operated by one or more users with one of the one or more physical vehicles; receiving, in the processor, initial position data indicating a position of the one or more physical vehicles on the track at a first time; determining, in the processor, for each of the virtual vehicles operated by one or more users on the virtual representation of the track, initial position data based on the initial position data of the physical vehicle with which it is associated; continuously receiving, in the processor, position data indicating a position of the one or more physical vehicles on the track at a second time, driver input data for each of the one or more physical vehicles, and user input for controlling the operation of the virtual vehicles operated by one or more users; and determining, based on the continuously received position data, driver input data, and user input, a position of the virtual vehicles operated by one or more users at a third time.

[0056] Some embodiments of the present invention particularly provide the tracking ability for data latency and reconciliation of data from a number of vehicles, which is required by vehicles moving extremely fast around heterogeneous racing circuits in any type of environment, regardless of whether it is an urban, metropolitan, or megalopolitan area. Specifically, these embodiments identify how it is possible to simultaneously achieve the tracking of real drivers around complex and cluttered urban-based tracks, around tracks where tunnels, adjacent buildings, and other obstacles interfere with tracking by a single line of sight and high-altitude IR sensors. The system for implementing the present invention features a specific architecture of sensor hardware including computers, communications, and computer software. All of these are either individually configured or can be configured and advantageously arranged in a wide range of motorsport venues, such that all competing vehicles can simultaneously track around complex racing circuits in various configurations and environments where the vehicles are pushed to their performance limits. Each architecture of the equipment and software configured according to the individual racing circuit is capable of delivering a single real-time dynamic data stream indicating the exact positions of all real competing vehicles within a very representative computer-based model, at least in terms of shape, of the real racing circuit. This data stream is suitable in terms of accuracy, latency, and, in some cases, other aspects of representational faithfulness when being simultaneously delivered to millions of computer gaming devices via the Internet or other communication technologies. The computer gaming devices include any range of computer hardware that is conventionally used by participants to implement some or all of the methods described below, typically to play computer games related to motorsports or to experience motorsport events or recordings via digital data streams provided by commercial or other suppliers.

[0057] This embodiment provides enhancements to the technology described in the following UK Patent No. 2,585,165A. This enables addressing the difficult task of tracking high-performance motorsport vehicles in complex environments, and is configured in an architecture that constructs a single integrated real-time data stream that simultaneously includes high-precision and real-time tracking data for all competitors, suitable for gaming and other entertainment environments suitable for broadcast and use. The IR detection capabilities of the tracking device described herein include those described in UK Patent No. 2,585,165A, but the inclusion of the thermal IR tracking technology described in UK Patent No. 2,585,165A in this application exceeds the scope of the technology described in UK Patent No. 2,518,602B and enables the ability to track motorsport vehicles based solely on their thermal IR signatures.

[0058] The features of the above-described embodiments can be combined in various ways and can be added, in particular, to the specific descriptions of the embodiments of the present invention shown below, unless otherwise stated.

Brief Description of the Drawings

[0059] For the present invention to be more easily understood, the attached drawings are described below as examples.

Figure 1

Figure 2

Figure 3a

Figure 3b

Figure 4

Figure 5

Figure 6

Figure 7a

Figure 7b

Figure 8

Figure 9

Figure 10a

Figure 10b

Figure 11a

Figure 11b

Figure 11c

Figure 12a

Figure 12b

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18a

Figure 18b

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0060] All of the computer gaming methods described below assume that each computer gaming element of the system (including the player's computer / tablet / phone, etc., and / or remote server hardware, and all associated software) can provide conventional functions typical of games related to motorsports. Thus, the player can provide input to the computer system to control the simulated automobile to move around a detailed data model of a real racing track. The player's simulated vehicle can interact with the track model and other simulated vehicles. The other simulated vehicles are controlled by the game's physics model and artificial intelligence (AI) functions, or by other types of simulations, or by other players. These other players can be present and provide input to the same computer system, or can use a remote computer system elsewhere. The remote computer system communicates with the original player's computer system via the Internet or other suitable network (which may include a game server) to provide a realistic interaction between all vehicles and between all vehicles and the track. All of the methods of the present embodiment described below are additions, changes, or replacements to these conventional functions.

[0061] Overall, this new environment in which a system for implementing the present invention exists can be considered as a multi-component infrastructure that provides support for computer game players and other interactive viewers to compete in groups against live motor sports and other types of sports events. The new environment in which the present embodiment exists includes the following three elements. a. A number of devices used by players and viewers, such as gaming consoles, personal computers, simulators, arcade gaming devices, satellite reception and processing facilities, smartphones, tablets, etc., and in some cases, devices including cameras and acoustic capture devices for distribution from participants anywhere in the world. b. The Internet for supporting the real-time reception, operation, mixing, storage, and dissemination of data from players and viewers, and associated computer facilities such as servers, real-time data centers, real-time server clouds (and in some cases, dedicated game and entertainment servers), including but not limited to the Internet and associated computer facilities including acoustics, video game data, and control inputs when driving a virtual vehicle. c. Real live motor sports or other sports events, including the systems of UK Patent No. 2518602 B and UK Patent Application Publication No. 2365360 A, and other systems for collecting and providing other data about real live events when they occur, and systems for storing data related to live events for future use, particularly the enhanced and developed forms described herein, real live motor sports or other sports events.

[0062] The following describes specific embodiments with reference to the accompanying drawings. First, referring to FIG. 1, the above three basic elements encompassing non-limiting embodiments of the present invention are schematically shown. More specifically, FIG. 1 shows several different gaming devices 2 connected to a central gaming server 4 via a communication network 6. The gaming server also includes a local data storage device 5 for collating different types of kinematic control data. FIG. 1 also shows a track 8 for a motor sports event 10, which includes real vehicles 12 arranged at different positions around the track 8. Each vehicle is equipped with a vehicle telemetry system. The vehicle telemetry system wirelessly provides a stream of data via the communication infrastructure around the circuit to the corresponding team's telemetry data capture computer 14, enabling the analysis of control data (the control of the driver of the vehicle) and other data related to the operation of the vehicle 12. This stream of telemetry data can include live video data from an in-vehicle camera of the vehicle 12 and acoustic data from a microphone provided in the vehicle 12. The entire telemetry system ensures that the data of each team is a. not publicly available outside the team, b. a predetermined portion is available to the organization that manages and governs, c. maintainable such that a predetermined portion is available for broadcast on TV or other media streams.

[0063] Track 8 is also provided with a plurality of sensor groups 16 (although schematically nine groups are illustrated, this is not limiting and different numbers of groups may be provided in other embodiments). Each sensor group 16 consists of a plurality of track-side infrared sensors 18. These sensors are configured to detect infrared radiation emitted therefrom when vehicle 12 is competing around track 8 and convert the sensor data into real kinematic data for each vehicle 12 in the field of view (FOV) of the sensor. Each sensor group 16 is communicable with a position data capture system 20 for this real kinematic data. The position data capture system 20 collates individual streams of kinematic data and provides a stream of kinematic data 22 for all vehicles 12 on track 8 to a live event data capture server 24. A stream of live telemetry data 26 from the team telemetry data capture system 14 is also provided to the live event data capture server 24.

[0064] The live event data capture server 24 collates information received from a position data capture system 20 that includes a plurality of tracking sensor groups 16 with the team telemetry system 14. All of this information (the real sensor data stream 28) is provided in this embodiment to a gaming server 4 that provides data compilation and generates a virtual race simulation. The virtual race simulation is provided in this embodiment to all gaming devices. However, in another further embodiment (described later with reference to FIGS. 16 - 20), the virtual race simulation is generated on each gaming device / computer 2 and the gaming server 4 merely orchestrates the overall gaming process.

[0065] In a further embodiment described below with reference to FIG. 5, the live event data capture system also collates other information received from the team, including all data generated by the team, and has a general event data capture system. This covers all other data related to motorsport events that may be related to entertainment and gaming functions. In this further embodiment, all of the information is provided to an entertainment and gaming server. This server provides an entertainment feed for fans who are not playing the game via an authorized distributor, and this whole ensures that the need for data confidentiality protection of all stakeholders, as exemplified for telemetry data in the above paragraph

[0062] , is satisfied as will be described later.

[0066] The elements that form the gaming mode of the gaming server of the system shown in FIG. 6 are schematically shown in FIG. 2. The gaming server has a real data processor 30 for processing the real data (sensor data stream) 28 received from the live event data capture server 24. The live event data capture server 24 ensures the synchronization of all contributing data sources (the stream of kinematic data 22 and the stream of live telemetry data 26), which will be described in detail below. This received live event data 28 is stored in the data storage device 5 as the real data currently recorded from the live event 32 and is also transmitted to the virtual race simulation engine 36. Then, it is used to generate a reference black box model generator (described later) that maps the input (control data 26) of the actual driver of each vehicle 12 to the kinematic position data 22 of the vehicle 12. The term "black box" is used in its conventional sense, that is, it is used in the technology of systems or engines characterized by the response to signals applied to the input ports. Thereafter, the output of this reference black box model generator is used within the virtual race engine 36 to access the player virtual race commands (driving inputs) 38 received from the gaming device 2 via the communication network 6, and as a result, to determine where each virtual vehicle should be located. In this embodiment, a virtual race command processing engine 40 is provided to receive these player virtual race command data streams 38, convert them to a common format as necessary, and provide them to the virtual race simulation engine 36.

[0067] Instead of the real-time live race data 28, the virtual black box function may be generated using the previously recorded real data 42 from the live event stored in the data storage device 5. The virtual race simulation engine 36 may also generate a virtual race simulation 46 using the data model 44 stored in the data storage device. Finally, the AI engine 48 is optionally provided to assist in adjusting the response of the virtual vehicle to the driver's input (virtual race command 38) with reference to the black box model generated by the reference black box model generator (described below), and in some cases, to assist the virtual vehicle driver (player) in changing the degree with reference to the handicap obtained by the virtual driver to, for example, equalize the racing between real drivers and virtual drivers with different skill and experience levels. For example, this assistance widens the threshold around each player's input (virtual race command 38) to enable adjustment to the optimal position for the virtual vehicle. Thus, while using the reference black box model generator to determine the optimal set of control data inputs that result in a defined vehicle position, the player control input 38 can be within the threshold of these specific control data inputs 26 that result in the defined position of the vehicle. The degree to which these thresholds are adjusted can determine the handicap applied by the AI engine 48 for a defined player and the corresponding control of its virtual vehicle. It will be understood that the AI engine 48 is used because it can be trained to perform a specific vehicle position output on an array of complex combinations of control data inputs.

[0068] In this embodiment, the real vehicle data 28 received from the live event data capture server 24 is schematically shown in FIG. 3a. Here, specific vehicle data 28 is associated with a vehicle ID 50 and includes kinematic real vehicle data 22 (e.g., longitudinal position 52b, lateral position 52c, and azimuth data 52a), and driver input data (control data) 26 such as steering position 54a, brake position 54b, accelerator position 54c, gear selection 54d, and optionally other actuator control inputs (not shown). The specific vehicle data 28 is provided to each real vehicle 12 competing around the track 8.

[0069] In this embodiment, the virtual vehicle control data (virtual race commands) 38 received at the gaming server 4 from the gaming device / simulator / computer 2 is schematically shown in FIG. 3b. This virtual control data 38 includes a gaming computer ID 56 and player actuator inputs 58 for controlling the virtual vehicle and optionally other data. The vehicle control player inputs 58 include steering position data 58a, brake position data 58b, accelerator position data 58c, gear selection data 58d, and optionally other actuator control inputs (not shown), and typically either match those of an actual driver or differ only by measured amounts, such that the virtual vehicle position can be determined as a result of these inputs using a reference black box.

[0070] The general operating method of the system in the gaming server 4 is shown in the schematic flow diagram of FIG. 4. The method 60 starts with a setup phase at step 62, in which in particular, all virtual participants are assigned to one of the real vehicles 12 participating in the race. One real vehicle 12 can have a number of players assigned to it, which will be described in detail later. After this, at step 64, a virtual race environment model 46 is generated, which includes a race circuit data model and the initial positions of the virtual representations of the real vehicles. As mentioned above, this can be based on live data 28 from the live event 10 or pre-recorded data 42 from a previously recorded live event. The circuit data model and possibly other data models 44 are stored in the local data storage device 5 to generate the virtual race environment 46. When the race starts at step 66, a stream of data 28, 42 from a live event or a pre-recorded event is received at step 68, including accurate kinematic data 22 for all real vehicles 12 measured and synchronized at a series of precise points in time over the duration of the race, and this can be used to change the position of the representation of the real race vehicles 12. These new positions are used at step 68 to generate new positions for the representation of the real vehicles in the virtual race environment 46. This new position is then sent (not shown) to each gaming device 2 registered for this race and displayed to the associated player. In response to this, the gaming server 4 receives gaming control data 38 indicating the player's user input at step 70 and controls their virtual vehicles. This player's user input 38 is then compared with the input of a reference black box model generator, which is used at step 72 to generate the next position of the virtual vehicle as a result of the player's user input 38. In some embodiments (see the detailed description of other embodiments below), these new positions of the virtual vehicles can then be sent (not shown) to each gaming device 2. This process of steps 68 to 74 continues until the end of the race, which is determined at step 76.

[0071] The operating method of the system of this embodiment is characterized by the following advantageous features. That is, a) Providing accurate and timely live data from real motorsports events in any type of environment to millions of players and followers around the world simultaneously, b) Providing a realistic, challenging, satisfying, and fair simulation of participating in a real race to computer game players of all skill and experience levels, i. As a single player competing against a real driver in a race, and as the limited number (at most the real driver minus 1) of other players all represented in the game, or ii. As a single player and competitor in an organized esports event or other computer gaming event including any number (potentially millions) of other players and possibly real drivers, some or all of whom are represented in the game, c) Optionally, during a race event, providing data from any number of players and followers around the world to the actual race team and real driver for any purpose, where any purpose includes, but is not limited to, presenting in real time information about the players participating in the event to the team (or also to the driver), and d) Optionally, providing any data from real events and / or all players and / or all esports competitors associated with live events to motorsports followers, enabling them to interact and compete against the event to enhance their enjoyment.

[0072] In yet another embodiment, as shown in FIG. 5, the gaming server 4 described above with reference to FIG. 2 can be replaced with a gaming and entertainment server 4a. This gaming and entertainment server 4a operates in the same manner as the gaming server but has additional functions. Therefore, for the sake of brevity, only the differences between the gaming server 4 in FIG. 2 and the gaming and entertainment server 4a in FIG. 5 will be described here.

[0073] Referring to FIG. 5, the components of the gaming and entertainment server 4a are the same as those of the gaming server in FIG. 2, and the same reference numerals are used. New reference numerals are introduced for the different parts. Thus, in this embodiment, it can be seen that the gaming and entertainment server 4a further includes a race simulation and entertainment engine 36a that receives virtual race commands, viewer commands, and other data 38a from the gaming device 4 and other entertainment devices (not shown). This data is collected and transmitted to the race simulation and entertainment engine 36a via the entertainment / race command and data processing engine 40a. Furthermore, a gambling engine 78 and an advertising engine that provide data to the race simulation and entertainment engine 36a are also provided.

[0074] There are many entertainment enhancement features that can be provided based on the availability of accurate tracking data from a location data capture system. Only three of them will be described as examples below. The first example is a roaming perspective. This allows, based on viewer commands from an entertainment device, for example, to switch or pan without interruption to a perspective generated by software functionality in an entertainment engine that is common or very similar to the virtual race simulation engine of Figure 2, from the driver's perspective in one of the real cars, for example, to a perspective generated by a computer above the car. The second example is to extend the live video stream with a display or overlay generated by a computer integrated into the live video in a convincing manner, for example, based on live technical data from sensors installed in the car or other sources, to extend it with the color of the temperature-dependent tires or an "under-the-hood" image of the car's mechanism. This can be extended, for example, to extend the live video stream from the driver's perspective in one of the real cars with an advertising billboard on the track side or a bridge, or an advertising image integrated convincingly and seamlessly on the car side. The advertisement generated by the computer from the advertising engine 80 moves dynamically with the scenery on the track side and the car and is indistinguishable from the advertisement that appears on the live video stream. Since this advertisement is combined for streaming to a specific known user, it is possible to tailor the advertisement to the user based on the data received in the entertainment commands, and the data processing functions, as well as the algorithms included in the advertising engine 80. The third example is the ability for an entertainment viewer to bet money live on an event as it unfolds in a real race, for example, to bet money on "Car 1 overtaking Car 2 within 20 seconds on the inside line". The gaming engine enables such a function and provides this to the race simulation and entertainment engine 36a.

[0075] Next, as an example of the system and method provided by the present invention, a specific operation of the position data capture system 20 including the tracking sensor group 16 as introduced in the above non-limiting embodiments will be described. To more clearly understand the position data capture system 20 of the above embodiments, first, a high-difficulty racing track will be referred to.

[0076] Referring to FIG. 6, an example of a particularly high-difficulty racing circuit 8 is shown. This cannot be monitored from above with a single high-performance IR sensor and requires a new system and communication equipment for the sensors 18 to provide the data required for real-time and interactive gaming. An exemplary example of a live motor sports event 10 is the Formula 1 race including 24 actual cars in 12 teams each having a pit, a garage, and other facilities. The race location is a particularly complex circuit shown in FIG. 6. This is in an assembled location including tall buildings, bridges, tunnels, and has various permanent streets and traffic infrastructure, as well as temporary infrastructure such as spectator stands. This is affected by the dominant range of the atmosphere, weather, and lighting conditions according to the time and place where the event is held.

[0077] Figures 7a and 7b show two possible strategies used in this embodiment for generating a highly representative computer-based data model 44 of the racing circuit 8. These strategies are suitable for ensuring that real-time association is possible between the measured kinematic data of the actual vehicle 12 and the virtual kinematic data of the virtual vehicle of the gamer, and are capable of overcoming the problem of verifying that the faithfulness of the circuit representation is sufficiently accurate and normal. The model 44 is a biolink between the actual environment and the actual event 10 and the computer gaming environment and the computer gaming event 46. Because each tracking sensor 18 in the network of tracking sensors has a relationship between a part of the actual field of view (FOV) of a part of the circuit 8 and the part of the data model 44 of the racing circuit 8 in its memory. Thus, an image of an IR reflector or IR emitter of the vehicle, or an image of natural thermal infrared radiation from the vehicle, is converted into kinematic data represented with respect to the racing circuit data model 44 within the tracking sensor 18. In another embodiment, it is also possible for the raw data from the tracking sensor 18 to be first communicated to the components of the overall system. Therefore, the relationship of the raw data with the circuit data model is calculated before being transferred to the gaming server 4 for processing.

[0078] Referring more specifically to FIG. 7a, an example underlying the data model 44 including an orthogonal grid 82 of horizontal and vertical lines arranged at narrow intervals is shown. The interval between the grid lines is adapted to the desired tracking accuracy in the computer gaming software, and this tracking accuracy is achievable by the tracking sensor 18. The kinematic data of this model typically includes at least longitudinal data 52b, lateral data 52c, and rotational orientation data 52a for each vehicle 12. The circuit model may include accurate altitude data for each cell in the horizontal and vertical grid 82 that enables a virtual interactive hybrid environment to simulate vertical G forces, accelerations, etc.

[0079] FIG. 7b is a diagram showing an alternative model for promoting computational accuracy. This circuit is modeled as a series of horizontal strips, each horizontal strip having the geographical position (horizontal line, longitude and altitude) of the horizontal center point of the horizontal strip, the azimuth, the width, and, optionally (in some embodiments), the horizontal tilt angle or altitude profile. And the kinematic data of each vehicle includes, as reflected in FIG. 3a, at least the longitudinal position 52b along the track 8, the lateral position 52c across the width of the track 8, and the rotational azimuth 52a.

[0080] Conventionally, it has been surprisingly difficult to establish a high-precision relationship between the computer game data model of a circuit in the real world and the topography of the circuit. This can be resolved by the overall system of this embodiment by, for example, placing stationary IR reflection / radiation markers (not shown) at accurate measurement positions along the periphery of the race track 8 within the FOV of each camera during the system setup of each sensor 18, and performing calibration by generating extremely accurate objects for data model mapping. In some embodiments, dynamic calibration is achieved by, for example, driving a special-purpose calibration vehicle along each edge of the race track 8 or moving it stepwise to adjust the tracking performance and achieve continuity between the sensors 18.

[0081] The inventor of the present application excludes connecting more than 100 tracking sensors required in a "star" configuration, all of which communicate in parallel, to a central position data capture system for such a complex environmental shape as shown in FIG. 6. In the case of wireless communication, it is probably impossible to find 100 independent reliable signal paths, and a wired connection would involve cables of about 100 km extending radially across the urban landscape, which is a major drawback. The only reliable path for any technology is to connect the tracking sensors 18 in a closed-loop shape around the circuit 8. In this closed loop, at least one point on or near the circuit needs to accumulate data in real time. In the case of only one point, there are at least 50 individual data hops from device to device required from some sensors, and the accumulated data transfer delay would be unacceptable.

[0082] Real-time interactive gaming between two geographically separated players / computers A and B requires that player input to gaming device A be received and processed by gaming device B within 20 - 40 ms as measured by a typical computer game update rate of 25 - 50 Hz, and vice versa. The term "computer game" is understood to have the same meaning as a simulated environment and to include an interactive hybrid environment. In this embodiment, the entire position data capture system including tracking sensor 18 can be considered to provide the equivalent of player input for all 24 real vehicles on the circuit, and this data needs to be supplied to remote gaming device 2 within a similar latency delay. By using data time tagging to ensure synchronization (explained in more detail below as one alternative), it is possible to introduce an overall time lag, for example 1 second, into the data stream, but this would result in not considering other information streams such as live video for the entire live gaming experience, which is not desirable. Therefore, by allowing an additional 5 ms for original image signal processing with an infrared camera frame rate of 100 Hz (maximum 10 ms delay), only approximately 20 - 30 ms is allowed for the aggregation of data from sensor 18 around circuit 8 and its forward communication. For wired or wireless technologies, the inventors have considered this and determined that in this embodiment, 50 hops around the tracking sensor circuit are not acceptable, and even considering the expected development of communication technologies, only 10 hops around the tracking sensor circuit are essential.

[0083] Accordingly, the exemplary sensor group 16 shown in FIG. 1 is transformed into an IR sensor network architecture comprising a small group 16 of sensors arranged in a high-performance local network, but is adapted to the actual circuit of FIG. 6 while considering a range of other factors. These other factors include IR sensor capabilities such as viewing angle shape, line of sight, obscuration (between vehicles 12 and between vehicle and infrastructure), available mounting points, image resolution (typically 640×512 or 1280×1024) and frame rate (typically 100 Hz), image processing parameters such as the number of "pixels-on-target" (typically a number greater than 4 is required), the required tracking accuracy (fast but much lower relative vehicle speed, 10 cm or better laterally, less critical longitudinally), the maximum vehicle speed in the FOV of each sensor, the number of worst-case processing executions, etc. All aims are to have the output from the placement data capture system 20 in FIG. 1. This output includes, for example, the kinematic data of FIG. 3a for each of the 24 vehicles 12 synchronized to a common accurate time reference at an update frequency of at least 25 Hz and includes a total real-time delay of only 1 s if a real-time lag is found to be necessary. (Note that the actual driver input data of FIG. 3a, which is sensitive to the team and confidential outside the team, is protected by fixed or dynamic encryption when transmitted to a gaming device or when used in a central gaming server and not transmitted to the gaming device at all.)

[0084] To explain a method for defining an architecture for a particular circuit 8, exemplary approaches to a plurality of possible sensor groups 16 are described in more detail below. These examples are based to some extent on the capabilities of currently available technologies such as IR camera image resolution, frame rate, LED intensity, etc., and it should be noted that these technologies are evolving and improving rapidly.

[0085] Figure 8 is a diagram showing a representative configuration of a specific sensor array 18 suitable for monitoring a specific type of racing circuit area of a specific class of racing car 12. These specific (F1) cars 12 have an aerodynamic and forward-tilted low profile. This means that when a tracking sensor 18 provided at a position higher than a truck is directed at an approaching car, that is, when the FOV has a bore sight at an acute angle to the horizontal plane when looking towards the approaching car, these cars are most likely to be detected reliably at high speed. The bore sight of the FOV is considered to be the central axis of the FOV. This configuration also means that, conveniently, the sensor 18 can be placed closer to the ground and can have a larger FOV compared to the simply downward-facing sensors of UK Patent Application Publication No. 2585165A. In some situations, extremely high poles or other structural mounting points 84 may be preferred. In this case, fewer sensors 18 are required to monitor the truck, and the bore sight angle with respect to the horizontal will increase. Alternative configurations can be defined in the same way as for other types of positions and other types of motorsport events, such as motorboat racing or NASCAR auto racing.

[0086] With particular reference to the circuit of the example shown in FIG. 6 in particular, a method of arranging the sensors 18 to provide a position data capture system 20 will be described below.

[0087] First, for the entire 2 km area of the circuit in FIG. 6 where standard street light poles are provided, FIG. 8 shows one of many possible arrangements of a group 16 of IR sensors 18 attached to the street light poles 84 in this area of the track 8. Each sensor (detection device) has an exemplary resolution of 640 x 512 pixels, a field of view of approximately 20 degrees to 30 degrees (see Views A and B), and a maximum detection range of 50 m. With this shape, it is considered possible to achieve reliable detection with a resolution of up to 10 cm in any of the following arrangements of the sensor devices 18 for implementation. a) A long-wave infrared (LWIR) microbolometer or mid-wave infrared (MWIR) photon detection (or other suitable technology) camera that detects the thermal IR emitted by the vehicle (its tires, engine, exhaust, etc.) or the thermal IR "cold spot" generated by applying or adhering an IR non-radiating marker or material to the vehicle. b) A short-wave infrared (SWIR) or near-infrared (NIR) photon detection (or other suitable technology) camera that detects broadband (or narrowband) light emitted by at least two small LEDs or LED clusters attached to the vehicle, or natural (sunlight-derived) IR radiation from the vehicle's reflectors, or the cold spot from the vehicle's IR non-radiator. The light emitted by the LEDs may be constant or dimmed over time. c) A SWIR or NIR camera that detects broadband (or narrowband) SWIR or NIR light provided by LED floodlighting placed near the IR camera and reflected by at least two small reflectors attached to the vehicle, or the cold spot in the reflected image from the vehicle's IR non-radiator. This enables night racing, and the LED floodlighting is pulsed and synchronized with imaging in the camera to improve detection performance in extreme weather conditions.

[0088] As described above, the positions of all individual targets (LED radiators, reflectors, non-radiators, or vehicle / tire) in the original image on the camera focal plane array are converted to exact positions in the circuit model by signal processing and geometric calculations tailored to the specific location, configuration, or orientation of the tracking sensor. The specific location, configuration, or orientation of the tracking sensor may, in some cases, include the relative positions measured for the outer physical limits of each of the two radiators (or reflectors or non-radiators) of vehicle 12. Such techniques are well within the capabilities of those skilled in the art and thus need not be described further below.

[0089] Continuously identifying and tracking each individual vehicle 12 can be achieved in UK Patent Application Publication No. 2585165A by each tracking sensor communicating the position of the vehicle to the next sensor ahead as the vehicle leaves its FOV. Additionally, in relation to the arrangement of the SWIR or NIR cameras as described in b) above, the LED light may be constant or may be dimmed to emit a pattern (or IR signature) characteristic of each vehicle 12. This enables each tracking sensor to simultaneously track and identify, eliminating the need for the tracking sensor to "pass on" the vehicle identifier to the next tracking sensor.

[0090] Next, around the circuit 8, there may be a position (e.g., the black square 81 in FIG. 6) useful for a single IR camera 18 mounted on the upper part of the building or a mobile boom lift (up to 45 m) or a work platform (up to 100 m), or carried by an aircraft such as a drone or a tethered drone (thereby enabling power supply for extended flight time or for data delivered to a ground station) at a higher altitude. At a height of 100 m, a wide-angle lens that expands to approximately 140 m × 140 m (the black square 81 in FIG. 6 is a typical example at scale) is attached to the FOV of the single IR sensor 18. In some circuits 8, it may be preferable to arrange sensors 18 with some drones attached around the circuit to form a complete closed-loop network. Each drone will stay at the station using its own on-board GPS and provide an accurate reference image by placing fixed IR wireless markers on the ground, from which absolute tracking detection accuracy can be achieved. This is an efficient arrangement because fewer sensors are needed to cover the entire circuit and it is very convenient for circuits where the sensors are not fixedly attached because the deployment of this arrangement is easy. It is possible to achieve a safe state for drones flying above high-performance motor races because the drones do not need to be directly above the circuit or the spectator stands. Tethered drones can have particular advantages because the drone will circle and fall into a defined crash zone around the base station immediately if it malfunctions.

[0091] Thirdly, a wide FOV (70 degrees) IR sensor may be attached to the short street light pole 84a in Fig. 8. Each IR sensor covers a short lane section about 15 m in length. The same applies to the tunnel 83, where the sensor 18 is attached to the roof of the tunnel. In another alternative embodiment, instead of using the short street light pole 84a, a movable pole up to 30 m in height that covers lane sections up to 50 m in length each may be arranged, which can be more cost-effective. In summary, the 3.3 km long complex circuit in this example requires 50 standard sensors, 50 wide-angle sensors, and sensors with 2 long-distance high platforms or drones attached.

[0092] Note that on many racing circuits, street light poles may not be provided or may be more widely spaced for safety reasons, or may need to be protected so as not to pose a safety risk to vehicles. In different embodiments, it may be necessary or preferable to temporarily attach the tracking sensors to buildings or appropriately arranged movable poles.

[0093] Finally, in order to achieve the above-described real-time data streaming performance criteria, the ground tracking sensors of this example and / or the tracking sensors attached to the drones are grouped into a plurality of groups 16. Generally, this is as shown in FIG. 1, but more specifically, it is as shown in FIG. 9. FIG. 9 shows how the sensors 18 are arranged in the group 16 in order to achieve the overall real-time performance of the position data capture system 20, and how the sensors 18 and the group 16 can communicate data in order to provide a single, fully and continuously synchronized data stream 28 with a suitable delay suitable for transmission to the real-time interactive gaming environment 46. FIG. 19 shows one group 16 consisting of seven sensors 18 attached to a street light pole, but other configurations are possible in different embodiments. A plurality of groups 16 are generated around the track 8 to provide the required complete reception range. In this way, while also achieving the low latency and simultaneity required for interactive gaming, it is possible to operate simultaneously as a single group that generates a single data stream for all real vehicles for the complete set of tracking sensors 18 arranged around the racing circuit 8.

[0094] Referring to FIG. 9 in more detail, the sensors of each group are connected to a high-performance and low-latency communication link 86. Such a communication link 86 uses a suitable communication technology (this is a technology for ground sensors, which may be a technology using a wired connection such as Gigabit Ethernet with a data transfer rate higher than 900 Mbit / s and a latency of 100 - 500 μs, or a technology for sensors attached to ground sensors or drones, which may be a technology using a wireless connection such as dual-band Wi-Fi with a data transfer rate higher than 90 Mb / s and a latency of 1 - 2 ms). Each group 16 of sensors is provided with a forward communication node 88, which is typically used in optical fiber ("wired") or wireless networks and is often used for streaming telemetry data in automobile racing sports. This forward communication node 88 is configured to communicate with a live event data capture server either wired or wirelessly, where the position data of all 24 cars are collected at a refresh rate of at least 25 Hz in the case of forward transmission. Additionally, and depending on the communication time delay around the network, in fact from circuit to circuit, all sets of sensors or each sensor group 16, or in some embodiments each sensor 18, may be equipped with a GPS receiver so as to be able to establish a common accurate time standard across the entire sensor network. Even when an accurate position measurement of the vehicle 12 is subsequently time-tagged and a small time lag (<1 second) is introduced into the forward transmission to the gaming server 4, it is possible for the live event data capture server 24 to obtain the position data of the vehicle as a simultaneous set.

[0095] Next, as an example of a system and method for an interactive hybrid (augmented reality) gaming environment 46 provided by the present invention, the specific operations of the above non-limiting embodiments will be described in more detail with reference to FIGS. 10a - 14.

[0096] In this example, FIG. 10a shows the starting configuration of a Formula 1 motor race including 24 real cars and drivers. The representations (89a, 89b, 89c, 89d, …, 89x) of these real cars 12 are associated with a computer game for only one player, where the real car representation 89f is assigned to the player or selected by the player as its starting position such that the representation of the simulated car 90 exactly overlaps the selected real car representation 89f.

[0097] According to FIG. 10a, the first case described above includes one computer game player who controls a simulated car 90 that is in the same position in the computer game as the real car 12f in a real event at the start of the race. After the start of the race, the advancement of the simulated car 90 is determined by the input made by the player to its computer device (gaming device / computer 2) and by the simulation of the car and its environment by the gaming server 4. The advancement of the real car representation 89 is determined by the input made by the driver to the real car 12 and by the physical behavior of the extremely complex car 12 in its extremely complex environment. Due to either i) the lack of fidelity in the simulation of the car and its environment or ii) the difference in input between the player and the driver, the advancement of the simulated car 90 is different from that of the real car 12f. This embodiment minimizes the former reason, thereby making the competition between the real driver and the virtual player as fair, realistic, and enjoyable as possible.

[0098] One of the main features of the method of the described embodiment is that the computer game exhibits particular mobility and increases the interaction with the real car 12 while maintaining the state where the simulated car 90 is sufficiently close to the real car representation 89f to the tolerance that the simulated car 90 can be defined in many ways (the following description refers to the simulated car 90 “snapped” to the real car representation 89). This includes, but is not limited to, the following. a) The real car representation 89 is not displayed and does not interact with the player's simulated car 90 as part of the snapped player's game. b) The snapped player's car 90 is "digitally paired" with the real car representation 89, and the simulated car 90 replicates the performance of the real car representation 89 so that the competition between the driver and the player is close enough to a fair competition. c) Inputs (control data 26) from a real driver (steering, accelerating, gear selection, etc.) can be streamed and are available for an interactive hybrid environment generated in real time by the gaming server. d) Acoustic, video, and other data feeds from the real car 12, its driver, and a wide range of racing teams can be streamed to a computer game (interactive hybrid environment) generated in real time by the gaming server. e) Acoustic, video, and other data feeds from the snapped player can be made available in the reverse direction to the real car 12 and / or the team and / or other computer gamers.

[0099] Figure 10b is a diagram showing a racing situation where the player's simulated car 90 deviates from the real car representation 89f by a known amount in terms of the instantaneous position and, optionally, other attributes such as speed, velocity, and acceleration. Here, a non-limiting and simple method of defining when the player's car 90 is snapped to the real car representation 89f is described. Within the computer game software operating on the gaming server 4, an area of the close contact point 96f, in this case a rectangular box, is defined using the position of the reference point 94f on the race track of the real car representation 89f. When the position of the equivalent reference point 91 on the simulated car 90 falls within the area of the close contact point 96f, the player's car is "snapped" to the real car representation 89f.

[0100] There are also many other conceivable snapping methods. For example, there are snapping methods that may depend on other parameters such as how far ahead or behind the player's car representation 90 is in time relative to the real car representation 89 (a method such as that used in F1's "Drag Reduction System (DRS)" where a chasing car can achieve "boost" when it is within a specific range of the car in front). Other snapping methods may include speed, velocity vector, angular momentum, etc., but the principle is clear. In addition, the computer game AI engine 48 can use algorithmic or other strategies to implement the degree of "stickiness" that provides a more flexible tolerance so that the snap is not subject to jitter. Here, a handicapping system is used, and higher-ranked players may be given lower "AI assistance" than lower-ranked players. Also, when "unsnapping" from the real car representation 89, the game software operating on the gaming server can consider the proximity points of the virtual cars of other players.

[0101] If the player's simulated car 90 is not snapped to any of the representations of the 24 real cars in the race, all the car representations are displayed in the computer game in the conventional way. If the player's simulated car 90 is snapped to the real car representation 89, only the other 23 real car representations 89 are displayed to that player, and that player is likely to feel as if they are interacting very closely with the real driver and their race team, especially if a live audio feed is provided to that player from the team.

[0102] This snapshot method enables the player to move forward during the race by avoiding the real car representation 89 or by moving from the real car representation 89 to the real car representation 89 and snapping them (not as selected by the player's operation). Figure 11a is a diagram showing the racing situation immediately after the start of the race. Here, the simulated car 90 is somewhat between the three closest real car representations 89c, 89b, and 89e. The player's simulated car 90 is unsnapped from its original pair (race car representation 89f) by improving its performance more than this, and is here in front of it. At this time, the real car representations closest to the player's virtual car are the real car representations 89b, 89c, and 89e. Depending on the skills and actions of the player and the three real drivers, the positional relationships 98b, 98c, and 98e will change rapidly. At the time of Figure 11a, the computer game (virtual race simulation) displays the real car representation 89c in front of the player's car 90 and the real car representation 89e in its rearview mirror.

[0103] Figure 11b shows a continuation of the racing situation in Figure 11a, at a point slightly later when the simulated car 90 has advanced and approached the real car representation 89c. It is shown that the simulated car 90 has entered the close contact zone 96c of the real car representation 89c and is snapped to it here. At this time, the gaming server prevents the real car representation 89c from being displayed on the player's screen, and the features as described above become effective.

[0104] Figure 11c shows a further continuation of the racing situation of Figure 11b after a longer time interval, and shows another part of the circuit 8 where the simulated car 90 is designated and approaching on the leading real car representation 89a. Here, the simulated car 90 of the computer game player has moved clearly ahead of the real car representation 89c and can be seen to have advanced more than halfway between the real car representation 89c and the real car representation 89a of the race leader. However, the player's virtual car 90 has not yet reached the snap zone 96a of the real car representation 89a and is chasing after it. This figure introduces two further zones around each real car representation 89. The first zone 98a is in front of the real car representation 89a and is called the "post-overtake zone", and the second zone 100a is behind the real car representation 89a and is called the "chasing zone". These three zones 96a, 98a, 100a can together form a continuous set as shown in Figure 11c. Thus, when the simulated car 100 enters any of these three zones associated with the real car representation 89, the performance of the simulated car 90, which closely matches the performance of the real car representation, is digitally paired. This feature equalizes the opportunity for the virtual car 90 to track the real car representation 89 or, conversely, enables it to integrate naturally with real features such as DRS. Therefore, the simulated car 90 is here in the chasing zone 100a of the real car representation 89a. As soon as the simulated car 90 moves from the post-overtake zone 98c of the real car representation 96c to the chasing zone 100a of the real car 89a, its performance characteristics are switched between the performance characteristics of the real car representation 89c and the performance characteristics of the real car representation 89a. This means that the performance of the simulated car 90 always conforms to the real car representation 89 with which it is most closely interacting and competing, ensuring a fair competition based only on the driving skills between the computer game player and the real driver. This zoning method is an example of the principle of zoning, and there are several possible variations of this method.

[0105] As described in a) of the previous paragraph

[0038] , there may be two or more computer game players included in a race simulation associated with a live or recorded race. In conventional motorsport computer games, it is common for a large number of players (from one to the maximum number of actual regular participants in a real race) to play together online from wherever they are located around the world. Thus, in the above example of Formula 1, which is a conventional comparative example, it would be normal for up to 24 players to play, i.e., for each person to occupy a starting position and for any empty space to be occupied by a simulated car controlled by the game software's algorithm or AI. This kind of limited player gaming mode would also be possible in the present embodiment. However, in other embodiments described below, it is possible to implement other gaming modes that are not restricted by the number of players. According to the embodiment of the present invention described above, it is possible to implement the above-described limited player gaming mode, i.e., the gaming mode in which empty player positions are assigned to the real car representations 89 in an actual race and the data streamed from the live event controls their behavior in the game. In this case, there will always be a large number of, i.e., "n", computer players such that the 24 drivers and / or players displayed in the computer game will compete with "(24 - n)" real cars. All aspects of the above-described "chase and snap" method are applicable, and the performance of the simulation 90 of each car is paired with the performance of the real car representation 89 that is replaced at the starting position, except when each car is in the "chase and snap" zone of the real car representation 89 displayed in the race.

[0106] According to an alternative embodiment for implementing a gaming mode for restricted players, there are 24 game players and 24 real drivers. Whenever the virtual cars 90 of the players are not snapped to the real car representation 89, those virtual cars 90 will appear in the computer game environment, be visible to all other players' virtual cars 90, and interact with them. Therefore, there will always be 24 real car representations visible in the computer race simulation and about 0 to 24 unsnapped simulated cars 90. These two method variations in this gaming mode are just examples, and there are many other possible variations, but the principle of this method is clear.

[0107] In many types of motorsports (Formula 1 is just an example of a very general application of the present invention), real cars can take a timeout during the race for maintenance or modification operations such as repair of a breakdown, refueling, or replacement of new tires. Therefore, in this embodiment, a computer game player can choose to take a timeout during the race for the same maintenance or modification operations as the real car in the simulated version. In this case, the simulated car 90 of the game player inherits the changed performance characteristics of the real car representation 89. Alternatively, if the car 90 of the computer game player is not snapped to the real car representation 89, the computer game player can choose to take a timeout during the race for simulated maintenance or modification operations. This will qualify the car 90 of the computer game player to be associated with any real car representation 89 that has performed a similar or compatible maintenance or modification operation when returning to the actual race. This is one of many possible examples of a way to ensure fair competition when real cars can be maintained or modified during the race.

[0108] As described above, it is possible to implement other gaming modes that are not restricted by the number of players. An embodiment that conforms to the situation described in paragraph

[0038] b) above is described below, in which a very large number of computer game players participate in the game, and at the same time the game is associated with a live or recorded real event. A group e-sports race is just one example of such a situation, where each player is required to interact with the real race in a realistic, attractive, and challenging way, and a fair method is required to rank a very large number of players according to their performance and final position in the e-sports event.

[0109] Figures 12a and 12b show two examples of possible start configurations for a race involving 24 real cars that are live-associated and interacting in e-sports, or an online gaming race involving tens of millions of game players whose start positions are distributed among the start positions of the real cars. More specifically, Figure 12a shows the start configuration of a race in which 24 real cars are associated with 10 million video game players evenly distributed among the start positions. Figure 12b shows an alternative configuration to the start configuration shown in Figure 12a, which is a start configuration by 100 million video game players logarithmically distributed over 24 real car representations, but only the top virtual car 90 is associated with the leading real car representation 89. These gaming modes are called unrestricted player gaming modes. These examples are just two examples out of many similar options, and for the purpose of the method of this embodiment, it is only necessary that each participating player in the computer game is assigned to one of these cars. Thus, each player starts a race snapped to one of the real car representations. Each player snapped to the same real car representation 89 is digitally paired with that real car and has the same performance model as the real car representation 89 in that environment.

[0110] FIG. 13 shows the racing situation up to the middle of a race between 24 real cars and a number of computer game players, where each computer game player has a virtual car 90 associated with one real car representation 89. The actual race situation in FIG. 13 is similar to the situation in FIG. 11c where real car representation 89a is in the lead, followed by real car representation 89c, and then by real car representation 89b. Each computer game player experiences the race in substantially the same way as when one player, as described above, interacts with the representations of 24 real cars. The situation shown in FIG. 13 is where the simulated cars 90 of player 102a(3) are in zone 98a in front of the lead (real car representation 89a). This is shown in an enlarged view as one dashed-line car on the track and specifically as three individual simulated cars 90c, 90f, 90z in accurately and closely separated positions. In this lead zone, each player can be presented with information and / or representations regarding several or all of the other players within the zone and / or can be promoted to engage in an interactive competition to take the lead position. Four cars 104a(4) are within the snap zone 96a of real car representation 89a and are snapped to real car representation 89a. These are shown in FIG. 13 as one dashed-line car on track 8 and in the enlarged view as four individual dashed-line car representations 90b, 90m, 90w, 90s in their exact positions relative to that real car representation 89a within the game. Nine players 106a(9) are in the chase zone 100a of real car representation 89a. All the simulated cars within zones 96a, 98a, and 100a are digitally paired with real car representation 89a. Real car representation 89c has a snap zone 96c where 25 cars 104c(25) are currently snapped. This post-overtake zone 98c includes 18 players 102c(18), and the chase zone 100c includes 57 players 106c(57). All players are digitally paired with real car representation 89c.In the overtaking and chasing zones, the virtual cars 90 of each player are displayed together with information and / or representations regarding several or all other players having their virtual cars 90 in the same zone, facilitating interactive competition among virtual players in the overtaking and chasing zones 98, 100. Advantageously, this display method enables a large amount of player information to be provided within a small screen size, allowing gaming devices with small screen sizes to maximize the large amount of information being displayed. When a real car crash occurs in an actual race, if the player's input commands are not such that they reduce and avoid the crash as determined with reference to the black box model, the snapped virtual cars 90 will also follow. Otherwise, it will continue from the point where it has strayed off the track as an unsnapped car 90.

[0111] Thus, in this unrestricted player mode, the method described above conveniently enables any number of computer game players to interact with a live race, providing each player with the experience of event 10 and engaging in attractive and exciting competition with real cars across the entire area. This is made possible by each player's gaming device 2 receiving a live data stream of the real car moving around the circuit and, where appropriate, the driver's input (into the virtual race simulation) from other data streams such as the team's audio and driver video. Additionally, when a player is participating in a race associated live with a team battle, typically, it is communicated that each player is moving around the circuit, and this is aggregated at the central gaming server 4. This enables a perspective that combines the entire circuit 8 shown in FIG. 13 and is analyzed in real time such that the relative positions of the virtual cars 90 of a large number of players in the race can be monitored continuously and with a certain degree of accuracy (for example, a computer game software simulation can present the instantaneous position of any simulated car at any time with any desired accuracy). This then enables the following. a) At the end of the race, the final positions of all players are determined unambiguously and fairly, and, b) In an environment where competition among computer game players can be enhanced, each individual player can be presented with an image of the cars of other relevant players in the vicinity, as described above.

[0112] Also, the computer game for each player displays behavior and interactions with real-world events 10, but is enhanced by group battles among many players. For example, not only are live audio feeds of real drivers and teams streamed to all the players who have been snapped, but the number of players who have been snapped, chased, and overtaken can also be presented to real teams or a large number of spectators or real drivers (which could be the names of individual players, etc.). It is possible for real drivers to comment on several or all of the drivers who have been snapped, and for the selected snapped driver to reply to that comment (of course, subject to an assessment of what is acceptable from a safety perspective). There are potentially many other immersive entertainment functions made possible by this embodiment.

[0113] In event 10 associated with the group battle and live of computer game players, depending on how it is organized, there may be a wide range of skill levels in the game player population. This embodiment can be easily extended to allow for a "for all applicants" or "open" event by including handicap parameters for each player as described above (based on a fair assessment of previous experience, like all handicap systems). For example, each player might be able to achieve a handicap from 0 to 100. Here, 0 represents the professional gamer status, and 100 represents the beginner gamer status. A handicap of 0 means that when the gaming server 4's AI engine 48 receives the player's input and converts these into forward movement around the circuit 8 of their virtual cars, it provides no assistance to the game. Conversely, a handicap of 5 might mean that if the player's data input 38 is within 5% of the input of the real driver 26, these are considered to match.

[0114] All of the above methods can advantageously exhibit the following features. That is, it is possible to exhibit the feature that computer game simulation of the performance of a real vehicle in its environment can present a fair and enjoyable competition not only between game players but also between a game player and a real driver. In other words, when a game player makes an equivalent simulated input 38 to a simulated vehicle as a real driver makes a real control input 26 to a real vehicle, the forward movement of the simulated vehicle 90 will match the forward movement of the real vehicle representation 89 with acceptable accuracy. Achieving a high level of fidelity has been the goal of motor racing computer game development and professional simulators for many years, and there is already a fair amount of capability. However, in the case of events associated with group battles in real time, there will be many factors in the play for any event. These factors need to be measured and communicated to the computer game, and realistically explained in computer game algorithms and AI. As described above, there is prior art suggesting sending such a large number of measured values to a computer game in real time and enhancing a conventional physical model based on the simulation. An alternative and preferred approach is realized in the last embodiment described below and can be summarized as a method of "real-time dynamic black box simulation for real-virtual interaction".

[0115] This embodiment enables the implementation of a dynamic black box simulation model within a computer game associated with a live event. In these embodiments, in addition to the accurate position data 22 of the real vehicle being provided to the computer game in real time via the position data capture system 20 shown in FIG. 1, the control input 26 made by the real driver to the real vehicle 12 is also measured and sent to the gaming server 4 in real time. FIG. 14 is a graph showing various parameters 28 transmitted by the live event data capture server 24 over time to explain the principle of the black box dynamic simulation method.

[0116] Referring to FIG. 14, this embodiment uses in real time a dynamic black box simulation model for an exemplary and simplified set of driver inputs (control inputs) to a real vehicle, namely steering wheel position 54a, accelerator position 54c, and brake pedal position 54b. In reality, there may be more driver inputs such as gear changes (not shown in FIG. 14). These are dynamic data inputs to the black box model. The dynamic output from the black box model is the forward movement of the real vehicle 89 over time. That is, in the model of this example, these are shown as the longitudinal distance 52b and lateral position 52c traveled on the track 8. FIG. 5 shows that variations in the driver's input can correspond to variations in the output of the real vehicle 89's track position / location over time. Recording and correlating the time histories of the inputs and outputs over one lap provides the simplest version of the black box dynamic model. This is then used to determine the possible track positions of the virtual vehicle 90 based on the input 38 generated on the user's computer.

[0117] This embodiment is based on a race simulation that occurs in the central server 4. In this case, an example of how the virtual race simulation engine 36 performs dynamic black box simulation will be described below and illustrated in FIG. 15. The kinematic race position data 22 is consistent with the actual driver input 26 from the event 10 that is live or recorded by the (above-mentioned) reference black box model generator 110. The output of this data is provided to the gaming black box implementation generator 112, and the gaming black box implementation generator 112 refers to this to determine whether the player input data (driving commands) 38 is acting on the position of the virtual vehicle 90 being controlled in the race. In fact, the reference black box generator defines a function that transmits the input to the output, which can be used by the gaming black box implementation engine 112. As described above, an AI engine 48 is also provided to assist the player in competing with a professional driver of a real vehicle. Once the position of the virtual vehicle 90 is determined by the gaming black box implementation engine 112, this is sent to the race simulation output engine 114 and used to generate the representation 90 of the vehicle in the race. The interactive hybrid race environment of all the vehicles 89, 90 participating in the race is generated by the race simulation output engine 114, which outputs the virtual race environment 46 to all the gaming devices 2 related to the stored data model 44.

[0118] The black box dynamic model is a very complex vehicle, minimizing the need to generate a highly accurate "white box" model of the vehicle with its very complex and dynamically changing environment (i.e., a model that combines more detailed models based on complex physics, and more detailed models also combine more detailed models, …). Thus, complex factors that are impossible to measure and model dynamically, such as wear and tear of the body, deposits of tire compounds on the track surface acting on wheel adhesion, gusts acting on the aerodynamics of the vehicle's forward movement (only 3 of many are listed here), are all incorporated into the reality of the measured black box model. The measured black box model generated by the reference black box generator 110 then functions as a reference model for the AI algorithm of the AI engine 48 in the game, and when different inputs 38 are given by the computer game player than the reference model input, the output will change consistently, proportionally, and realistically.

[0119] The black box model of a single vehicle will vary from lap to lap due to large variations in driver input per lap and variations in other physical factors (usually continuous and stepwise, and in some cases individual). However, there are many detailed strategies and techniques available for use in this embodiment to ensure that this method provides an optimal dynamic and accurate simulation during the race. These include, but are not limited to, the following. That is, i. Averaging the black box model data from a number of laps through practice sessions and real races, or otherwise combining to generate a reference black box lap model that best represents the driver, vehicle, and track on that day. ii. Do the same as (a), but for shorter sections of the track, such as a specific corner or a specific straight. iii. Take a lap with the best lap time as the reference black box model for the final performance of the driver, vehicle, and track on that day. Do the same as in iv.c), but in a shorter section of the track, e.g., at a specific corner or a specific straight line. v. Build a black box model every second, compare the gamer input 38 with the real driver input 26, and adjust the output with a direct and immediate reaction (this method will be particularly useful in the first lap when no history is available, or in the first lap after a "pit stop" and thus after the real car has been modified). vi. Supplement the measured black box model with the modeling of known physical laws that are known to apply continuity and relative simplicity. vii. Model known limitations on the performance of the car, such as maximum straight-line acceleration, top straight-line speed, etc.

[0120] There are many other techniques, but the gist of this method is that it is much more practical to model and simulate changes in one or more dynamic black box reference models known to be effective than to generate a white box model of an extremely complex physical system in real time.

[0121] Not only is it associated live with an actual motorsport event 10, but other embodiments of the present invention are also applicable to situations where the interaction of a computer gamer is an interaction with a recorded version 42 of a past live race. This record 42 consists of the motion performance 22 of a real car 89 measured extremely accurately in the environment of a past race and the input 26 of a real driver to the real car during that race. In these embodiments, all of the above descriptions are applicable to a battle between the recorded race and the computer gamer.

[0122] Alternatively, in another embodiment where the actual race record 42 consists only of the measured in this environment highly accurate performance 22 of the real vehicle 89 and there is no record of the input 26 of a real driver, digitally pairing can be done based on any other method ensuring that the simulated performance of the player's car 90 in that environment is close enough to the representation of the performance of the real vehicle 89 to create a fair competition between the player and a real driver or a self-driving vehicle. This can include, for example, reverse engineering of the driver input 26 using a high-fidelity racing car simulator. Here, the driver rehearses and then records the input history that enables the driver to reproduce the performance of each real vehicle 89 in the recorded race. This is an exemplary example of a method for reconstructing the input 26 into a dynamic black box model when data from the original event is not available.

[0123] Referring to FIGS. 16 to 20, a further embodiment of the present invention is described which is mainly different from the above-described embodiment in that it is implemented in a more decentralized manner. This further embodiment has many components that function in the same way as the components of the embodiments of FIGS. 1 to 4, and only the differences will be described below to avoid unnecessary repetition. Referring to FIG. 16, the configuration of the gaming device 2 is shown. Here, the gaming device 2 is provided with a virtual race simulation engine 36 instead of the central server 4, and the player input command 38 derived from the player input can be directly input into the virtual race simulation engine 36 and does not need to be transmitted to the central server 4. This greatly improves efficiency and bandwidth consumption. The virtual race simulation engine 36 is provided with at least the position data 22 and driver input data 26 of the real vehicle 89 from the live event 10, the recorded reference black box data 120, and the supplementary data 122 from the gaming server 4 via the communication engine 124. This received data is stored in the local data storage device 5 and is used by the virtual race simulation engine 36 in the same way as described above in the embodiment of FIG. 2. Similarly, the data model 44 provides further assistance for the generation of the simulated race environment 46. As already described, an AI engine 48 is provided to provide assistance for helping the player compete with the real driver. As a result of having a more decentralized approach to the virtual race simulation, the central server 4 needs to be updated at the position of the virtual vehicle 90 of each player. Therefore, the communication engine 124 is arranged to transmit not only the race car selection 126 for the start of the race and the changes thereto during the race, but also the virtual vehicle position data 126 as determined by the virtual race simulation engine 36 to the central gaming server 4 during the race.

[0124] Referring to FIG. 17, the gaming server 4a of this distributed embodiment is shown. The gaming server 4a has a virtual vehicle position processor 128 that collates each virtual vehicle position 90 and provides this to the virtual race management engine 130. Thereafter, the virtual race management engine 130 provides each gaming device 2 with information regarding their positions and relative performance during the race, and, optionally (if it is determined that the race between virtual players is to be enhanced or made more competitive), the positions of other virtual vehicles 90 during the race, to complete the racing environment. The virtual race management engine 130 is also capable of generating a reference black box for the previously recorded data 42 and providing this to all gaming devices 2. Since this is pre-recorded data 42, the information is not provided in a time-critical manner that would be associated with real-time racing. Thereafter, the gaming server 4a only handles the monitoring of the competition as it is aware of the positions of all competitors, and optionally supplies virtual and real, and in some cases telemetry and other data, to the "snapped" cars (thus requiring a link from the live event data capture server 24), optionally supplies selected competition images to each virtual competitor, and optionally supplies statistics regarding the virtual competition to the real teams.

[0125] Also, in this embodiment, the real live event data 28 is sent from the live event data capture server 24 to the gaming server 4a and distributed from the gaming server 4a to the gaming devices 2. This is convenient for synchronization. However, in another embodiment, the real live event data can be sent directly to each gaming device 2, which has the advantage of reducing the latency time and reducing the processing power required at the central server.

[0126] FIG. 18a is a diagram showing data from the live event data capture server 24 and virtual vehicle position data received from the gaming device 2 at the gaming server 4a. Although FIGS. 8a and 13a are very similar, in this embodiment, other data 132 is also provided to the central server 4a. This other data 132 can be live telemetry data or wireless communication data between a team and a driver, etc. These additional data streams 132 can make the gaming more realistic or attractive as described above.

[0127] FIG. 18b is a diagram showing virtual vehicle position data 134 transmitted from the gaming device 2 to the gaming server 4a. This data further includes a gaming computer identifier 56. Using this position data 134, the position of the virtual vehicle 90 is updated so that the position can be communicated to other participants in the same race using other gaming devices 2 as determined by the gaming server 4a. Other data fields 136 are related to the selection of the actual vehicle to be associated or other data such as handicap data.

[0128] Referring now to FIG. 19, the virtual race management engine 130 within the gaming server 4a is shown. The virtual race management engine 130 determines which gaming device 2 will receive which updated position data 38. Central to this is the race engine management processor 140, which collects race pictures provided to the central server 4a from different data streams. Further, the virtual race management engine 130 includes a reference black box model generator 142. This is the same as that described with reference to FIG. 15 in the above-described embodiment, except that it operates only on the past recorded event data 42. The generation of the live data reference black box model is transferred to each gaming device 2. The communication engine 144 provides the position data 38 and the reference black box model data 146 to the respective appropriate gaming devices 2. Further, the communication engine 144 can provide other race data 132 (as described above) to each gaming device 2 and can transmit live event data if it is not provided directly to each gaming device 2.

[0129] Referring to FIG. 20, an operating method 150 in the gaming device 2 of the system of this embodiment is shown. This method 150 starts with the gaming device transmitting a participation request for the race to the gaming server 4a in step 152. This request includes details of a specific configuration, which enables the game server 4a to interact with the gaming device 2 and also enables identification of the actual vehicle 89 to which the player of the device 2 will be associated. For example, if the gaming device 2 requires direct feeds of live data from the live event data capture server 24 and / or the live telematics feed 26, these gaming options can be specified. In the next step, the gaming server, in step 154, provides the requested information, such as a link to the live data fed from the live event data capture server 24 and the positions of other virtual players 90 during the race determined by the virtual race management engine 130. Although not shown, the gaming server 4a evaluated with this information can generate an overall picture of the active gaming devices 2 assigned to a specific event 10. Similar to the embodiment described above, a single actual vehicle 89 can have a number of players associated with it. Thereafter, in step 156, a virtual race environment is locally generated on the gaming device including the initial position of the virtual representation 89 of the actual vehicle. As described above, this is based on the live data 28 from the live event or the pre-recorded data 42 from a previously recorded live event. The virtual race environment is generated using the data model 44 stored in the local data storage device 5. When the race starts in step 158, a data stream 28, 42 from the live event or the pre-recorded event is received in step 160, and using this, the position of the representation of the actual race vehicle can be changed in step 162. Using these new positions, in step 160, new positions for the representation 89 of the actual vehicle 12 in the virtual race environment are generated. This is then presented to the player on the gaming device 2.In contrast, in step 162, the gaming device generates gaming control data (virtual race commands) derived from the player's user input 38 to control those virtual vehicles. After that, these virtual race commands are compared with the output of the reference black box within the virtual race simulation engine 36, and in step 166, they are used to generate, as a result of the player's user input, the next position of the virtual vehicle. (It will be understood that the real-time black box generation of this embodiment is as described with reference to FIG. 10.) This new position of the virtual vehicle can then be transmitted in step 168 to the gaming server 4a, and the positions of the virtual vehicles 90 of other players are received from the gaming server 4a and determined to be relevant by the gaming server 4a. Then, in step 170, the new positions of the relevant virtual vehicles are presented to the player on the gaming device 2. This process continues until the end of the race is determined in step 172.

[0130] Finally, it will be apparent that the features of the present invention can be extended to a wide range of sports events where the moving object tracked in real time in a segmented playing field is not an automobile, but rather a player on a soccer pitch or basketball court, a competitor in a downhill skiing event, or a hunting dog on a race track. All of these are examples of a combat area that is not a track and a moving object that is not a vehicle. The position data capture system will consist of sensors, computer equipment, and communication equipment of a compatible architecture. It is also possible to incorporate into the clothing of a real competitor fibers that do not emit infrared radiation or small infrared emitting devices (capable of emitting constant or dimming light and possibly being triggered in some cases) to provide cold spots or hot spots suitable for tracking and identification. Although an interactive competition between a real player or team and a virtual player or team may not be as practical as in the case of motorsports, the immersive entertainment and spectator enhancement functions described in 18) above are practical. It is possible to automate in an easy way the collection of sports statistics and analysis. Furthermore, it may also be possible to develop a technology to incorporate into, for example, a soccer ball or basketball an infrared reflecting or non-emitting marker invisible to the human eye, enabling the performance of the tracking device to continuously measure the rotation of the ball. This is also applicable, for example, to the cue ball in a snooker or billiards game where one sensor is placed on the table. The common characteristic and novel feature is to track a moving object (vehicle, player, ball, etc.) in a relatively short range (<100 m) using wide-angle infrared sensors, illuminators, and associated computer equipment and communication equipment to detect passive or active infrared markers, reflectors, absorbers, or emitters. Using the real-time data thus obtained, a wide range of interactive entertainment and gaming experiences are enhanced.

[0131] The features of this embodiment will be presented and described below in a numbered list. 1) A system and method are described for a computer game player to interact with a live race that includes a real vehicle with a driver or an autonomous driving vehicle. Thereby, a single player can start a race by associating their simulated vehicle with one of the real vehicles, advance by maintaining the state associated with the initial vehicle, or by associating from one real vehicle to another based on a defined proximity parameter, thereby competing competitively against the real driver as a virtual driver in the real race. 2) If the computer game player's simulated vehicle is not associated with a real vehicle, all real vehicles are displayed in the race and the computer game player can choose to avoid the real vehicles and compete as an independent moving entity at their own discretion, or to associate with one real vehicle based on a defined proximity parameter. When the player's simulated vehicle is associated with a real vehicle, that real vehicle is not displayed as a competing vehicle in the player's computer game. 3) When the computer game player's vehicle is associated with a real vehicle, the simulation of the player's vehicle is a digital twin of the real vehicle, meaning that the performance of the player's vehicle in that environment is close enough to the simulation of the performance of the real vehicle in that environment to create a fair competition between the player and the real driver or autonomous vehicle. A system and method are described. 4) A system and method are described where the digital pairing is done based on the kinematic performance of the real vehicle measured extremely accurately in that environment and the dynamic black box simulation of the real driver's input to the real vehicle. 5) A system and method are described where the computer game dynamically links a virtual zone with each real vehicle, and when the computer game player's vehicle enters that zone, it becomes a digital twin of the real vehicle or inherits specific performance characteristics associated with the real vehicle. 6) If the computer game player's vehicle is associated with a single real vehicle and that real vehicle takes a timeout during the race for maintenance or modification operations (repairing a breakdown, refueling, or changing to new tires), the computer game player can likewise choose to take a timeout for the race (which can be delayed by only one lap). In this case, a system and method are described where these simulated vehicles inherit the performance characteristics of the modified real vehicle. 7) If the computer game player's vehicle is not associated with a single real vehicle, the computer game player can choose to take a timeout for the simulated maintenance or modification operation, which, when returning to the actual race, gives the computer game player's vehicle the qualification to be associated with any real vehicle that has performed a similar maintenance or modification operation. A system and method are described. 8) A system and method are described where a number of computer game players, including a maximum number of real vehicles during the race, can be associated with real vehicles during the race, thereby enabling them to compete with each other and against the real vehicles in a single actual live race. 9) A system and method are described for the collective interaction between an actual live race, including a real vehicle with a driver or an autonomous driving vehicle, and a number of computer game players. Thereby, each player can start the race by associating their simulated vehicle with one of the real vehicles, and can advance by associating from real vehicle to real vehicle based on defined parameters, thereby competing competitively against the real driver and all other computer game players as a virtual driver in the actual race. 10) A system and method are described for a gamer to compete in a group e-sports race. Here, each player is required to interact with the real race in a realistic, engaging, and challenging manner, and a fair method is required to rank a very large number of players based on their performance and final position in the e-sports event. This method is based on the extremely accurate relative positions of the simulated cars of the computer game players. 11) A system and method are described where the interaction of the computer game player is with a recorded version of a past race, and this record consists of the motion performance of a real car measured extremely accurately in the environment of the past race and the input of a real driver to the real car during that race. 12) Alternatively, a system and method are described where the interaction of the computer game player is with a recorded version of a past race, and this record consists only of the motion performance of a real car measured extremely accurately in this environment. In this case, digitally pairing is based on any other method that ensures that the simulated performance of the gamer's car in that environment, associated with the digital record of the motion performance of the real car in those environments, is close enough to the representation of that performance to create a fair competition between the gamer and a real driver or a self-driving car. 13) A system and method are described where when the simulated car of the computer game player is replaced by a real car, there is an exchange of video, audio, or any other technical data between the gamer's computer system and the real car and / or its associated team and their facilities for any direction and any purpose. 14) A system and method are described in which each computer game player is assigned a handicap based on a fair assessment of their past experience and accumulated skills, and the handicap is used by the computer game to moderate the response of the vehicle in that environment to the player's input, substantially equalizing the opportunity for players with different levels of experience and skill to compete against a real driver in a real vehicle and / or to compete against each other to win the race. 15) Optionally, by associating any number of real vehicles with the computer game player's vehicle using any of the methods described above, a non - computer game player automobile racing fan can participate in an interactive spectator experience and can receive technical data, video streams, audio streams, or any other data included in the system and method described above. 16) A system and method are described in which a gaming player's virtual vehicle can be associated with a representation of a real vehicle and received as input position data regarding the real vehicle in a race and as input from a real driver, and using these, a reference black box representation can be generated. For the reference black box representation, a gaming player input during the race can be evaluated to match the virtual player's vehicle to the representation of the real vehicle and determine a simulated gaming environment capable of simulating a real racing environment. 17) A system and method are described in which dynamic input and output data of a real vehicle are captured, analyzed using a black box modeling method, and the results can be fed back in real time to competing teams, assisting their understanding of driver and vehicle performance and, in particular, providing the teams with highly accurate kinematic data regarding the performance of those vehicles that could not be obtained previously.

[0132] A system and method are described for detecting the movement of a vehicle by a group of position detection devices arranged around a racetrack. Each detection device includes an IR sensor for detecting IR radiation emitted, reflected, or transmitted from the vehicle to determine the vehicle's track position, and communication means for transmitting the position data to a position data capture system. The data is collated as live position data regarding a racing event including these vehicles and provided to a gaming device or server. However, as described above, it will be understood that the movement tracking system is not limited to the movement of vehicles. The accuracy in using a group of infrared sensors having a suitable configuration and connection to other groups of infrared sensors enables accurate position detection of a moving object and is applicable, for example, to non-motor sports including the movement of humans or animals.

[0133] It will also be understood that various modifications of the described embodiments are possible and that elements of one embodiment can be readily combined with elements of other embodiments. Accordingly, the methods and systems described herein are non-limiting examples showing how different aspects of the invention can be implemented, and it will be understood that the invention is defined by the gist and scope of this disclosure.

Claims

Claim 1 A computer-implemented method for controlling an interactive hybrid environment representing a motorsport event on a track, the interactive hybrid environment including representations of real and virtual vehicles on the track, the method comprising: Receiving a stream of real sensor data, the real sensor data including: a. Real kinematic data of a real vehicle on the track captured by an infrared sensor on the track; and b. Real control data regarding the control of the real vehicle by a driver, captured by a vehicle sensor and obtained via a telemetry system from the real vehicle; Determining the position and kinematic behavior of the representation of the real vehicle within the interactive hybrid environment using the real kinematic data; Using the real control data and the real kinematic data to generate a black box determination of the position of the real vehicle on the track based on the real control data; Receiving a stream of computer-generated control data obtained by user interaction with a computer for presenting the interactive hybrid environment to a user and capturing user input for controlling the kinematic behavior of the representation of the virtual vehicle; Determining the position and kinematic behavior of the representation of the virtual vehicle within the interactive hybrid environment by using the black box determination and the computer-generated control data. Claim 2 The computer-implemented method according to claim 1, wherein the real sensor data includes real kinematic data of a plurality of real vehicles on the track and real control data regarding the control of each of the plurality of real vehicles by its respective driver. Claim 3 The computer-implemented method according to claim 2, wherein the real sensor data of each real vehicle among the plurality of real vehicles includes a vehicle identifier. Claim 4 The computer-implemented method according to claim 2, wherein the stream of computer-generated control data includes a plurality of streams of computer-generated data, each stream being generated by a different user interaction with a computer and capturing a different user input. Claim 5 The computer-implemented method according to claim 4, wherein each of the plurality of streams of the computer-generated data includes a computer device identifier.

6. The plurality of physical vehicles is less than the plurality of streams of the computer-generated data, and the method further includes associating a subset of the plurality of representations of the virtual vehicle with a representation of one physical vehicle to generate an associated representation, the computer-implemented method according to claim 4.

7. The computer-implemented method according to claim 6, further including representing, within the interactive hybrid environment, a subset of the plurality of representations of the virtual vehicle while the position of the subset of virtual vehicles is within the tolerance of the physical vehicle, using the associated representation.

8. The plurality of streams of the computer-generated data is multiple times more than the plurality of physical vehicles, and the associating step includes associating each of the plurality of streams of the computer-generated data with the plurality of representations of the physical vehicle in an even distribution, the computer-implemented method according to claim 6.

9. The plurality of streams of the computer-generated data is multiple times more than the plurality of physical vehicles, and the associating step includes associating each of the plurality of streams of the computer-generated data with the plurality of representations of the physical vehicle in a logarithmic distribution, the computer-implemented method according to claim 6.

10. Updating the interactive hybrid environment with new positions of the physical and virtual vehicle representations as determined by the received real sensor data and the plurality of streams of computer-generated data; Generating the updated interactive hybrid environment; The computer-implemented method according to claim 4, further including broadcasting the updated interactive hybrid environment from a central server to a plurality of remotely located computers.

11. Broadcasting the black box determination and the real sensor data from a central server to a plurality of remotely located computers; Generating the interactive hybrid environment at each of the remotely located computers; Updating the interactive hybrid environment with a new position of the representation of the real vehicle and the virtual vehicle as determined by the received real sensor data and the computer-generated data of the plurality of streams; Transmitting the new position of the representation of the virtual vehicle to the central server; The computer-implemented method according to claim 4, further comprising: **Claim 12** The computer-implemented method according to claim 1, further comprising varying the relevance between the computer-generated control data and the resulting position of the virtual vehicle using an artificial intelligence engine that refers to the black box determination. **Claim 13** The computer-implemented method according to claim 1, wherein the received real kinematic data includes longitudinal position data with respect to the track, lateral position data with respect to the track, and vehicle orientation data with respect to the track. **Claim 14** The computer-implemented method according to claim 1, wherein the real control data includes one or more steering wheel positions, accelerator positions, brake pedal positions, and gear selections of the real vehicle. **Claim 15** The computer-implemented method according to claim 1, further comprising retrieving the real sensor data from a data storage device that stores a copy of the real sensor data when the real sensor data was generated. **Claim 16** The computer-implemented method according to claim 1, wherein the receiving step includes receiving the real sensor data substantially in real time when a sports event is taking place. **Claim 17** The computer-implemented method according to claim 1, wherein the stream of real sensor data has a sampling rate of at least 25 Hz, the position of the real vehicle is captured at a point in time, and provided to the interactive hybrid environment within 40 milliseconds after being captured. **Claim 18** The computer-implemented method according to claim 1, wherein the stream of real sensor data has a sampling rate of at least 60 Hz, the position of the real vehicle is captured at a point in time, and provided to the interactive hybrid environment within 16.7 milliseconds after being captured. **Claim 19** The computer-implemented method according to claim 1, further comprising the step of using a stored data model to generate the interactive hybrid environment.

20. The computer-implemented method according to claim 1, further comprising the step of receiving a stream of video data from the actual vehicle and including the stream of video data in the interactive hybrid environment.

21. The computer-implemented method according to claim 1, further comprising the step of receiving a stream of acoustic data from the actual vehicle and including the stream of acoustic data in the interactive hybrid environment.

22. At a time when the position of the representation of the virtual vehicle is within a predetermined threshold of the position of the representation of the actual vehicle, associating the representation of one of the plurality of virtual vehicles with the representation of one of the plurality of actual vehicles, and using the representation of the actual vehicle as the representation of the virtual vehicle in the interactive hybrid environment. The computer-implemented method according to claim 2, further comprising the step of:

23. Receiving a stream of acoustic data or video data from the actual vehicle, further comprising the step of including the stream of acoustic data or the stream of video data in the interactive hybrid environment, and the associating step comprises operating to provide the stream of acoustic data or the stream of video data received from the actual vehicle to a computer that presents the interactive hybrid environment to a user. The computer-implemented method according to claim 22.

24. At a time when the position of the representation of the virtual vehicle is outside a predetermined threshold of the position of the representation of the actual vehicle, releasing the association between the representation of one of the plurality of virtual vehicles and the representation of one of the plurality of actual vehicles, and presenting the representation of the virtual vehicle separately from the representation of the actual vehicle within the interactive hybrid environment. The computer-implemented method according to claim 22, further comprising the step of:

25. The computer-implemented method according to claim 22, further comprising the step of providing details of any virtual vehicle associated with the representation of the actual vehicle to a computer of a third party located remotely.

26. Each real vehicle has a set of performance characteristics, and the method further includes determining the representation of one real vehicle among the plurality of real vehicles that is disposed closest to the representation of the virtual vehicle, and introducing the set of performance characteristics of the representation of the closest real vehicle as the performance characteristics of the virtual vehicle. The computer-implemented method according to claim 2.

27. The computer-implemented method according to claim 1, further including capturing position data of the real vehicle on the track using the infrared sensor, converting the position data into a stream of real kinematic data over time, and transmitting the stream of real kinematic data to a central server in real time.

28. The capturing step includes capturing the position data using a group of sensors that monitor different positions on the track, and each sensor in each group of sensors detects infrared radiation that is emitted, reflected, or transmitted from one or more vehicles operating on the track within the field of view (FOV) of the sensor. The computer-implemented method according to claim 27.

29. The computer-implemented method according to claim 28, further including processing the infrared radiation detected by the infrared sensor to determine kinematic data of one or more real vehicles operating on the track.

30. A computer system for controlling an interactive hybrid environment representing a motor sports event on a track, the interactive hybrid environment including representations of real vehicles and virtual vehicles on the track, the system including: A receiver for receiving a stream of real sensor data, the real sensor data including real kinematic data of a real vehicle on the track and real control data regarding the control of the real vehicle by a driver, the real kinematic data being captured by an infrared sensor in the track, and the real control data being captured by a vehicle sensor and obtained from the real vehicle via a telemetry system. A receiver. A virtual race command processor configured to receive a stream of computer-generated control data obtained by user interaction with a computer that presents the interactive hybrid environment to a user and captures user input for controlling the kinematic behavior of the representation of the virtual vehicle. A virtual race simulation engine. The virtual race simulation engine. A race simulation output engine for determining the position and kinematic behavior of the representation of the real vehicle in the interactive hybrid environment using the real kinematic data. A reference black box model generator configured to use the real control data and the real kinematic data to generate a black box determination of the position of the real vehicle on the track based on the real control data. A computer system including a gaming black box implementation engine configured to determine the position and kinematic behavior of the representation of the virtual vehicle in the interactive hybrid environment by using the black box determination and the computer-generated control data.

31. The computer system according to claim 30, further including an artificial intelligence engine configured to vary the relevance to the final position of the virtual vehicle.

32. The computer system according to claim 31, wherein the artificial intelligence engine is configured to expand a threshold required from the received computer-generated control data to generate a predetermined position of the virtual vehicle.

Images