Ai based monitoring of race tracks

An AI-based racetrack monitoring system using cameras and image processing techniques addresses the limitations of existing systems by providing flexible, cost-effective detection and response to critical situations without vehicle-mounted equipment, suitable for various races.

JP2024156219AActive Publication Date: 2024-11-01FUJITSU TECHNOLOGY SOLUTIONS GMBH

Patent Information

Application Number
JP2024118361
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-04
Filing Date
2024-07-24
Publication Date
2024-11-01
Estimated Expiration
2042-06-27

AI Technical Summary

Technical Problem

Existing race track monitoring systems require specialized equipment installed within vehicles, are expensive, and are not suitable for all types of races or racing vehicles, making them difficult to implement, especially on large tracks.

Method used

An AI-based monitoring system using cameras and image processing techniques for racetrack monitoring, including image segmentation, object detection, and rule-based alert triggering, which does not require specialized vehicle-mounted equipment.

Benefits of technology

Enables automatic detection and response to critical situations, tracks vehicle positions, and provides flexible, cost-effective monitoring suitable for various races, including amateur events.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024156219000001_ABST
    Figure 2024156219000001_ABST
Patent Text Reader

Abstract

To provide an improved monitoring system and method for race tracks, which is easy to implement and cost effective.SOLUTION: A novel AI based monitoring system for race tracks used for car racing enables: automatic detection and reaction to critical situations, including a deviation of a vehicle from the race track and / or collision with a guide plank, a loss of oil, a person or other object on the race track or the like; rule based definition and association of the automatic detection and / or automatic reactions; tracking of vehicles along the race track, including storage of the driven track; automatic mapping of detected critical situations to one or more tracked vehicles; and / or automatic generation and cutting of video footage for a tracked vehicle.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to novel AI-based monitoring systems and methods for race tracks, such as those used for professional and amateur car racing. [Background technology]

[0002] Race tracks, especially car race tracks such as the Nürburgring's North Loop, often feature many turns, including blind corners, drops, and significant elevation changes, making the race track difficult and therefore potentially dangerous for high speed driving. Historically, in the event of an incident such as a vehicle leaving the track, race marshals relied on radio communication and communications with other marshals on the track to relay such information to one or more race controllers and make decisions regarding track safety. In the absence of a direct line of sight, it was often difficult to quickly assess and respond to an incident. With hundreds of vehicles on the track at the same time, the speed and correctness of safety decisions is essential to protect drivers and spectators.

[0003] US Patent Publication No. 2005 / 0133963 discloses a track monitoring device and system. Specifically, it relates to a system and method for detecting whether one or more participating vehicles in a race are on or have left a predetermined route or track. The system includes a set of indicating means provided in or on the track and a detection means mounted on the vehicle. If the indicating means is detected by the detection means, this is taken as an indication that the vehicle has left the track, and an alert or alarm can be generated and a penalty may be assigned to the identified vehicle.

[0004] No. 5,999,333 discloses another race monitoring system. The disclosed auto race monitoring system provides the race track with a ground positioning system that includes at least three transmitters that transmit signals to be received by at least a pair of receivers on each race car. The receivers instantaneously determine the position and, correspondingly, the exact location and attitude of the race car on the race track. This information, along with data regarding the parameters of the race cars, such as speed, engine temperature, oil pressure, etc., are transmitted by the transmitters to receivers interconnected with a mainframe computer so that a viewer can select any particular race car that he or she wishes to monitor at any particular time during the race.

[0005] US Patent No. 5,399,633 discloses a method and system for automatically tracking and analyzing image data of at least one vehicle on a race track, where a video event management system with multiple video cameras positioned around the race track determines the presence of at least one vehicle, captures video images and still images based on weighted event scores corresponding to dynamics for the at least one vehicle and other objects, and generates at least one sub-frame, where redundant video image data and redundant still image data are discarded based on metadata of the linked sub-frame. Summary of the Invention [Problem to be solved by the invention]

[0006] While the above monitoring systems can be useful in race monitoring and control, they require specialized equipment to be installed in the vehicle and therefore are not suitable for all types of races or race vehicles. Moreover, because of the specialized sensing technology used, they are relatively expensive and difficult to implement, especially on large race tracks. Thus, there is a need to provide an improved monitoring system and method for race tracks that is suitable for many types of races or race participants and is preferably simple and cost-effective to implement. [Means for solving the problem]

[0007] According to a first aspect, a method for monitoring a racetrack comprises the following steps: - acquiring at least one sequence of images from a camera capturing at least one section of a race track; - segmenting images of the sequence of images into different areas associated with a race track; - detecting at least one vehicle in the sequence of images by means of automatic pattern recognition, in particular automatic object recognition; - mapping at least one detected vehicle to at least one of the different areas associated with the race track; - activating at least one warning device based on a first set of rules, the first set of rules including at least one first rule that triggers a first warning if at least one detected vehicle is mapped to a first predetermined area of ​​the race track, such as a guardrail or boundary area.

[0008] In particular, the inventors have recognized that the detection of incidents on a race track can be automated and therefore accelerated using artificial intelligence (AI) based techniques such as image segmentation, object detection and localization. Moreover, providing a set of rules that associate detection results with relevant actions can greatly increase the flexibility and effectiveness of race track monitoring.

[0009] According to a second aspect, a monitoring system for a race track includes one or more cameras, each camera having a field of view covering at least one section of the race track, an image capture system configured to acquire at least one sequence of images from at least one of the cameras, one or more warning devices configured to be activated when a first warning is triggered, and an image processing system including at least one processor, the processor configured to segment images of the sequence of images into different areas associated with the race track, detect at least one vehicle in the sequence of images using automatic pattern recognition, in particular automatic object recognition, map the at least one detected vehicle to at least one of the different areas associated with the race track, and trigger the first warning if the at least one detected vehicle is mapped to a first predetermined area of ​​the race track based on a first set of rules.

[0010] The disclosed monitoring systems and methods enable: - Automatic detection of critical situations along the track, including vehicle departure from the race track and / or collision with guide plates, loss of oil, people or other objects on the race track, etc. - automatic reactions to critical situations, including activating warning and control signals, outputting messages to panels positioned along the track, selecting, outputting and / or recording video footage of relevant sections of the race track; - Rule-based definition and association of automatic detection and / or automatic reactions; - Tracking of vehicles along the race track, including storage of the track as it is driven; and - automatically mapping the detected critical situation to one or more tracked vehicles involved in the critical situation; and / or - Automatic generation and cutting of video footage about the tracked vehicle.

[0011] The disclosed system implements the above configuration based on a combination of one or more of the following electronic processing techniques: At least a portion of the race track is monitored by a series of video cameras to generate a sequence of images. - Predetermined classes of objects, such as vehicles or vehicle parts, oil and / or (unacceptable) anomalies, such as gravel, are detected in image sequences using AI-based object identification algorithms, e.g. Convolutional Neural Networks (CNNs) and in particular Deep Learning Networks (DLNs). The sequence of images is analysed using segmentation, e.g. using a segmentation mask and / or a segmentation network, to identify different parts of or along the race track, e.g. the running surface, lanes, track boundaries, guard rails, escape areas, out-of-boundary areas, pit areas, view areas, tarmac areas, gravel areas, dirt areas, grass areas and forested areas. - Detected objects are mapped to different parts of the race track. - Based on one or more sets of rules, warning devices such as flags or lights along the race track are activated fully automatically or semi-automatically to alert race track officials or race management to control potentially dangerous situations, such as a race car crashing into a guardrail or an oil spill on the running surface. - The real-world positions of detected and optionally identified vehicles and anomalies are calculated based on triangulation and interpolation of known reference points in the captured images. - Detected vehicles are identified based on the training data, e.g. by computing an embedding vector. - A digital twin of the vehicles is maintained in a digital model of the race track, including data such as the position and speed of each vehicle on the race track. - the detected anomalies are associated with the vehicles on the track, for example by comparing the trajectory of each vehicle with the location and first occurrence of the detected anomalies; and / or - Image sequences relating to selected events, such as crashes or other incidents, or selected vehicles may be collected and cut automatically, or provided to race management on request.

[0012] In the following, specific embodiments of the disclosed monitoring systems and methods are described with reference to the accompanying figures. [Brief description of the drawings]

[0013] [Figure 1] 1 shows in a schematic manner an image of a race track captured by a trackside camera. [Diagram 2] 1 shows a flow chart of a method for racetrack monitoring. [Figure 3A] 1 shows a sequence of images taken before, during and after detecting a vehicle on a segment of a race track. [Figure 3B] 1 shows a sequence of images taken before, during and after detecting a vehicle on a segment of a race track. [Figure 4] 4 illustrates in a schematic manner the detection of anomalies on the racetrack segments of FIG. 3; [Diagram 5] 2 illustrates in a schematic manner a triangulation method for locating a detected object within a scene; [Figure 6A] 1 illustrates in a schematic manner training and inference using embedding vectors for object re-identification. [Figure 6B] 1 illustrates in a schematic manner training and inference using embedding vectors for object re-identification. [Figure 6C] 1 illustrates in a schematic manner training and inference using embedding vectors for object re-identification. [Figure 6D] 1 illustrates in a schematic manner training and inference using embedding vectors for object re-identification. [Figure 7] 1 illustrates in a schematic manner the parameter space of embedding vectors used for vehicle re-identification. [Figure 8A] 1 illustrates in a schematic manner the collaboration between an inference server and a digital twin server based on images from a single camera. [Figure 8B] 1 illustrates in a schematic manner the collaboration between an inference server and a digital twin server based on images from a single camera. [Figure 9A] 1 shows in a schematic way the collaboration between an inference server and a digital twin server based on images from two cameras. [Figure 9B] 1 shows in a schematic way the collaboration between an inference server and a digital twin server based on images from two cameras. [Figure 10] 1 illustrates in a schematic manner the system architecture of a monitoring system for a race track. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Below, a particular system architecture and method of operation for a racetrack monitoring system is described.

[0015] In the described system, one or more cameras provide video footage of all or part of a race track 100. As shown in Figure 1, an artificial intelligence (AI) system, based on predefined rules or suitable training, recognizes different areas of the captured footage, such as the running surface 102, grass areas 104, gravel areas 106 or crash barriers 108, and divides images from the footage into corresponding segments.

[0016] In the described embodiment, the individual segments are automatically detected using a segmentation network, for example based on the so-called U-Net architecture or another type of convolutional neural network (CNN) pre-trained using images of the race track taken by the trackside cameras. In comparison with a potential static segmentation of the camera images, this allows to cover the movements, panning and zooming of the trackside cameras. Optionally, feedback information from the camera control device may be used to trigger a renewed segmentation of the image stream taken by a given camera whenever the camera moves, pans, zooms, etc. While the segmentation is being performed, the output from the respective camera may not be used by the monitoring system. Moreover, to improve reliability, an initial segmentation may be performed for each camera at a known default position. The output of the segmentation stage is a pixel mask. For each pixel, the probability of belonging to one of several possible classes of segments is calculated. The pixel mask indicates for each pixel the most likely segments of the race track 100, such as the driving surface 102, grass areas 104, gravel areas 106 and guard rails 108 (crash barrier).

[0017] Alternatively, segmentation may be performed manually, i.e. statically, before the monitoring system is started, especially if one or more cameras are mounted in a fixed position with a fixed field of view. Moreover, it is also possible to provide a manual mask for only a portion of the captured images, for example for parts known to be located outside the racetrack 100. Masking such parts may speed up subsequent segmentation or improve the quality of the subsequent segmentation. This is also called semi-automatic segmentation.

[0018] Vehicles 110 on the race track are also recognized based on artificial intelligence. For example, known object detection algorithms based on publicly available training data may be used to detect vehicles 110. For example, deep learning networks (DLNs) or other types of convolutional neural networks (CNNs) may be used to detect vehicles, people, animals, and other objects on the racing track 100. In the described system, the YOLO real-time object detection system is used for object detection. Detection accuracy can be improved by using additional training data captured on the relevant race track showing previous racing events and manually tagging vehicles 110 in the training data.

[0019] Based on the above steps, the disclosed system analyzes which segment the detected vehicle 110 is in. This data is passed to a rule-based system, which can trigger events and alarms based on predefined rules. For example, LED panels and warnings emitted along the track 100 can be generated if a blocked vehicle, foreign object on the race track, dirt or oil on the race track is detected.

[0020] Preferably, the system can distinguish between acceptable and unacceptable anomalies. Acceptable anomalies include deviations in the captured video footage caused, for example, by reflections, rain, shadows and / or lighting beams. Unacceptable anomalies relate to missing parts of the vehicle 110, for example, the spread of oil and other operating fluids or gravel or soil onto the race track from adjacent areas. As described below, convolutional neural networks (CNNs) may be used to detect the entry and exit of known types of objects, such as the vehicle 110, at a given section of the race track.

[0021] FIG. 2 illustrates, in flow chart form, the steps of a method 120 for monitoring a racetrack 100.

[0022] In step 121, at least one sequence of images is acquired from a camera capturing at least one section of the race track 100. In step 122, images of the sequence of images are segmented into different areas 102, 104, 106 and 108 associated with the race track 100. The segmentation is performed based on a manual segmentation mask provided before the race starts or based on an automatically or semi-automatically generated segmentation mask calculated during initialization, after camera movement or in real time. In step 123, at least one vehicle 110 is detected in the sequence of images using automatic object recognition. The order of steps 122 and 123 may be reversed, i.e., potentially moving objects may be detected first and used to assist the image segmentation process. In step 124, at least one detected vehicle 110 is mapped to at least one of the different areas 102, 104, 106 or 108 associated with the race track 100. In decision step 125, it is determined whether at least one rule from the set of rules has been triggered. The set of rules includes at least one first rule that triggers a first warning if at least one detected vehicle 110 is mapped to a first predefined area of ​​the race track 100, such as the guard rail 108. If not, the method proceeds to step 121 to acquire and process a next image. However, if at least one rule from the set of rules is triggered in step 125, at least one warning device is activated in step 126 based on the set of rules.

[0023] In the following, different aspects of the disclosed monitoring system and method are described in more detail.

[0024] Starting with a set of rules, which may include multiple predefined individual rules for triggering an alert. Each rule may include one or more conditions that determine when the rule is triggered, and one or more actions that are performed when the rule is triggered. Note that not all rules are segmentation dependent. For example, if an animal is detected anywhere in the image, an alert may be issued regardless of whether the animal is on the driving surface 102 or not. As a specific example, the set of rules may include the following rules: [Table 1]

[0025] As shown in Figures 3A, 3B and 4, detection of anomalies may be triggered whenever the vehicle 110 enters a monitored section 130 of the track 100. A differential analysis of the race track 100 before and after the vehicle 110 passes through the relevant section 130 of the track 100 can then be performed. Again, this detection may be based on artificial intelligence and appropriate training data. For example, the system may be trained with acceptable and unacceptable anomalies based on previous race events and manual tagging of acceptable and unacceptable deviations in historical footage.

[0026] In addition to historical footage tagging, anomalies may be artificially created, for example, by placing a foreign object on the racetrack 100 and marking it as an unacceptable anomaly. To improve detection rates, images of both acceptable and unacceptable anomalies should be taken in different weather conditions.

[0027] As shown in FIG. 3A, a sequence of images 132, 134, and 136 show the same section 130 of the race track 100 before, during, and after the passage of the vehicle 110 as detected by the object detection system. The process may be triggered by the successful detection of the car or other vehicle 110 in an image 134 provided by a camera and captured by a frame grabber. Once the vehicle 110 is detected, the images 132 and 136 may be selected from the captured video material. For example, the last image in the time series of images taken before a bounding box 138 surrounding the detected vehicle 110 enters the section 130 may be selected as the first image 132. Similarly, the first image in the time series of images taken after a bounding box 138 surrounding the detected vehicle 110 leaves the section 130 may be selected as the third image 136.

[0028] 3B shows the same sequence of images 132, 134 and 136 after segmentation. In the example shown, the segmentation algorithm used has extracted only a single segment 140, which corresponds to the racetrack running surface 102. As shown in the bottom center, the detected vehicle 110 is located within segment 140, i.e. on the racetrack 100. Hence, no alarm is triggered at this stage.

[0029] 4, once the detected vehicle 110 leaves the section 130, a comparison may be made between the segments of interest. In particular, a delta image 142 may be calculated between a segment 140 in the first image 132 and the same segment 140 in the third image 136. The delta image 142 is classified using a deep learning classifier to determine whether the delta image corresponds to or contains at least one known anomaly 144.

[0030] In the presented example, an oil spill left by a passing vehicle 110 is detected. A corresponding alert may therefore be activated for all or at least the section 130 of the race track 100 containing the anomaly 144. Depending on the severity of the detected anomaly and / or the associated uncertainty of the detection algorithm, the alert may be triggered automatically by the monitoring system, i.e. without human intervention, or may be displayed to a race official or similar official for verification, with the option to trigger or suppress the alert.

[0031] As a further feature, the position of the vehicle 110 on the race track 100 may be determined and, optionally, tracked along the course. For this purpose, real-world positions, e.g., GPS coordinates of reference points visible in the monitored section 130 of the track 100, may be recorded. Then, based on triangulation and similar interpolation techniques, detected objects such as the vehicle 110 and anomalies 144 along the monitored section 130 of the track 100 can be associated to the previously recorded positions of the reference points. That is, the reference points map image locations defining single pixels or groups of pixels (collectively referred to as pixel areas) of the digital image of the race track 100 to corresponding real-world positions on the race track 100.

[0032] This is shown in detail in FIG. 5. A total of 15 reference points 150 are provided within another section 130 of the race track 100. In the illustrated example, the reference points 150 form a fixed grid of 3×5 reference points 150. This may be useful, for example, when the position, orientation and angle of the camera capturing the section 130 is fixed. In this case, a GPS or similar location device with a visible marker may be placed at each reference point 150 in turn during set-up or training to determine its exact real-world location. Alternatively, the reference points 150 may coincide with salient, high-contrast features within the section 130, i.e. edges or corners of segment boundaries, special objects such as flagpoles or the like. Such reference points can be easily identified in the video image of the section 130 even if the camera moves, pans, zooms in and out. Again, the real-world locations of such salient features are measured and stored during set-up or training and used for triangulation during normal operation of the system.

[0033] In the situation illustrated in Fig. 5, two vehicles 110a and 110b are detected in the section 130. To determine their positions, three or more reference points 150 in their vicinity are used to estimate their current positions. As illustrated, the reference points 150i, 150j, and 150o closest to the center of the first bounding box 138a of the first vehicle 110a are used to estimate the current position of the first vehicle 110a by triangulation. Similarly, the reference points 150b, 150f, and 150g closest to the center of the second bounding box 138b of the second vehicle 110b are used to estimate the current position of the second vehicle 110b.

[0034] Thus, the locations of the detected vehicles 110 and anomalies 144 can be displayed on a visual representation of the race track 100, for example, at race control where race officials work and make decisions about track warnings and potential interruptions to the race.

[0035] Thus far, detection and optional location of a particular class of objects, e.g., vehicles 110 and anomalies 144, have been described. As detailed above, this is sufficient to generate a safety alert for the monitored race track 100. However, the described AI-based monitoring system also allows for some more advanced features, which are described below. In particular, it allows for re-identification of individual objects on the race track, e.g., a particular vehicle 110. For clarity, the terms "object detection" or "object recognition" are used to describe the mere detection of a particular type of object in a captured image, such as the presence of any race cars on the monitored section 130 of the race track 100. In contrast, the term "object re-identification" is used to describe the identification of a unique entity, such as the pre-registered race cars of individual race participants currently on the track.

[0036] As an additional configuration, if a digital model of the race track 100 is maintained by the monitoring system, the position of each vehicle 110 may be continuously updated to create a so-called digital twin of the monitored vehicle 110, for example as shown in FIG. 8 below. In the described monitoring system, this is based on re-identification of the individual vehicles 110, rather than just detection of any vehicle 110 on the race track 100. The vehicles 110 may be differentiated based on extractable features, such as shape, color, or other visible marks, such as logos or symbols printed on the vehicles 110. Suitable training data for automatic identification may be captured at well-defined points of the race track 100. Images of the individual vehicles from different positions, for example from the front, rear or sides of the vehicle 110, may be taken when the vehicle 110 enters the race track 100 for the first time at the entrance lane. If only a portion of the race track 100 is monitored, the training images may be taken by one or more cameras monitoring the first section 130 of the race track 100. Based on this information, whenever a vehicle 110 is detected along the race track 100, the detected object can be compared to previously captured image data, for example by inference and calculation of an embedding vector. When a registered vehicle 110 is racing on the race track 100, the additional information may be used, for example, to improve the OCR detection rate of the registration plate.

[0037] In the described embodiment, object re-identification is implemented using a neural network 160 trained offline, i.e., before using the monitoring system in an actual race, using an encoder / decoder model to identify specific instances from a given class of objects, e.g., individual racing cars participating in a currently running race. Different training sets may be used to train different instances of the corresponding neural network for different classes of objects, e.g., Formula 1 cars, regular road cars, motorcycles, etc. Also, training of the neural network 160 may be repeated if the specific characteristics of the objects to be identified change, e.g., due to rule changes for racing competitors.

[0038] The training of a neural network 160 used for object identification is conceptually illustrated in Figures 6A and 6C. During the training phase, images of different vehicles 110 are provided to the input 162 or encoder of the neural network 160. To achieve the desired accuracy, a large number of training images are used, manually selected or verified and taken at the actual race track 100. For the described monitoring system, a set of about 17,000 vehicles extracted from archive footage of races at the Nürburgring was used to train the neural network 160 after human review of the material.

[0039] During the actual training, the information received at the input side is simplified by the nodes 164 of the neural network 160 to form or encode an embedding vector 166. The information of the embedding vector 166 is decoded to recreate the image at the output side 168 or decoder of the neural network 160. The weights and other settings of the neural network 160 are changed until the image at the output side 168 is highly similar to the image provided at the input side 162, for example until the difference between them is very small or minimal. At that stage, the neural network has learned the characteristic features of the vehicle 110 or a particular subclass of vehicles 110, such as a racing car. The comparison of the input side 162 and the output side 168 can be done automatically using an appropriate similarity metric. Thus, the training of the neural network 160 may be performed unsupervised.

[0040] In the example of Figure 6A, the embedding vector 166 has only a single node 164 that corresponds to a single dimension of information, e.g., the shape of the object. Figure 6C shows another neural network 160 under training with two nodes 164 forming the embedding vector 166, e.g., corresponding to the shape and color of the object of interest.

[0041] In practice, many more dimensions may be used to reliably identify an object such as a vehicle 110. In this context, attention is drawn to the fact that in a race, a relatively large number of very similar cars, e.g. cars of the same model, often race against each other, and they may differ only in relatively small details, such as color, the presence and shape of auxiliary parts such as advertisements, spoilers, etc. Thus, a multidimensional embedding vector 166 in a multidimensional feature space 170, having, for example, 168 independent dimensions, is used in the described system. During unsupervised training, the most significant characteristics for re-identifying a particular vehicle 110, corresponding to each of the nodes 164 representing the dimensions of the embedding vector 166, are automatically determined by the training algorithm. Thus, they may not correspond to high-level characteristics such as color or shape and may not be easily understandable by humans.

[0042] Once the neural network 160 is trained, it can be used to extract or infer previously unknown characteristic properties of the vehicle 110 online, e.g., in real time during a race. This process is illustrated in Figures 6B and 6D, which show the classification of input images to determine the shape, or shape and color, of the vehicle 110. This inference is used for object re-identification during track monitoring.

[0043] The process of initial registration and subsequent identification of a particular vehicle 110 is also illustrated in Figure 7. Before the vehicle 110 enters the race track 100, one or more images of the vehicle 110 are taken and processed by a pre-trained neural network 160 to generate one or more corresponding embedding vectors 166. For example, in a multi-dimensional feature space 170, a first vehicle 110a may correspond to a first embedding vector 172 and a second vehicle 110b may correspond to a second embedding vector 174. In the described system, multiple images of the same vehicle 110 from different perspectives, e.g., from the front, rear and sides, are used to create an array of possible embedding vectors, as described below.

[0044] Then, once the vehicle 110 is on the track, the portion of the image corresponding to the detected vehicle 110 can be sent to the neural network to determine a new embedding vector 176. The new embedding vector 176 may then be compared to the previously registered vectors 172 and 174. The vehicle may then be identified as the vehicle corresponding to the closest previously known embedding vector, e.g., the first vehicle 110a corresponding to the first vector 172. Specifically, the angle between the new embedding vector and all the previously registered embedding vectors 172 and 174 is calculated, and the vector with the smallest angle, e.g., the first embedding vector 172, is selected as the best matching vector.

[0045] Optionally, if the new embedding vector 176 differs from the closest pre-registered vector 172 or 174 by more than a first preset threshold, the new vector may be stored in the system in an array of vectors 178 corresponding to the given vehicle 110. This may be used to improve future object identification, for example, by adding embedding vectors 176 corresponding to images of the same vehicle 110 taken from different angles or under different environmental conditions, such as lighting or weather conditions. In the described system, up to 500 different embedding vectors can be stored for each registered vehicle. In general, this makes vehicle detection more likely and more reliable.

[0046] Alternatively or additionally, if the new embedding vector 176 differs from each one of the previously registered vectors 172 and 174 by more than a second pre-set threshold, then the identification may fail and / or the new embedding vector 176 is not included in the array of vectors 130. This may be used to exclude uncertain matches and / or to avoid degradation of the array of vectors 178.

[0047] The vehicle detection and identification can be further improved if the entire track 100 is covered by video cameras. In this case, continuous tracking of each moving object allows its detection even if some or all of its characteristic features are temporarily blocked. If not all parts of the race track 100 are covered by the respective cameras and / or if movable cameras are used and do not cover the vehicle 110 of interest at any given moment, certain logical assumptions about the likelihood that the vehicle 110 will appear in certain areas of the race track 100, based on a digital model of the race track 100, may be used to improve detection accuracy.

[0048] 8 illustrates an exemplary setup and operation of a monitoring system 180. The system 180 includes a camera 182 that captures a corresponding section 130 of the race track 100. The system 180 further includes an inference server 184 and a digital twin server 186.

[0049] The inference server 184 captures digital images or video frames provided by the camera 182 and stores them for later use. At least a subset of the captured images are fed to an AI-based object detection 188 unit to identify objects of a given type, such as vehicles 110. As shown in FIG. 8, each identified object is surrounded by a corresponding bounding box 138.

[0050] In addition, the detection unit 188 also segments the received camera images. For example, the image of the track section 130 may be subdivided into areas corresponding to the running surface 102, the grass area 104, and the gravel area 106, respectively.

[0051] The inference server 184 further includes an inference unit 190. The inference unit 190 is configured to determine a position of the detected vehicle based on GPS position interpolation using the known reference points 150 in the section 130.

[0052] The inference unit 190 is further configured to generate an embedding vector for each detected vehicle 110 using the neural network 160. The determined embedding vector, together with the real-world position on the race track 100 of the detected vehicle 110 and / or an identifier of the camera that captured the image, are passed to the digital twin server 186 for identification.

[0053] The digital twin server may use multiple plausibility checks to identify each of the vehicles 110a and 110b. In general, the set of embedding vectors used for matching can be limited to vehicles 110 known to be on the track 100. Moreover, in step 1, based on the camera's identifier, it can select only a subset of vehicles 110 that are likely to be in that camera's view. This can be achieved, for example, by considering only vehicles that have already passed another camera located in an earlier section of the race track 100. For example, the subset of vehicles used in the identification process may be limited to vehicles that were last successfully identified in any one of two or three sections of the track covered by the other cameras. The inclusion of more than one up-track camera addresses the fact that some vehicles 110 may be covered in a particular camera setting or may not be successfully identified due to difficult light conditions or camera angles.

[0054] Alternatively or additionally, in step 2, the selection may be refined based on real-world positions provided by inference server 184. For example, based on the estimated speed of vehicle 110 in the digital model, only vehicles that could plausibly have reached a particular real-world position in the captured image when the image was taken may be considered for matching.

[0055] Reducing the number of vehicles 110 used during identification to a predetermined subset may reduce the required processing power, thereby enabling real-time identification at relatively high resolution of the input image during high speed racing. The higher resolution and reduced plausible matching also improves detection accuracy. For example, when two very similar vehicles of the same model and color are participating in the same race, but are located in different sections 140 of the race track 100, object identification is significantly improved compared to typical real-time identification engines.

[0056] In step 3, after such filtering, the digital twin server selects the embedding vector in the multidimensional feature space 170 that is closest to the new embedding vector 176 (see FIG. 7 ) provided by the inference server 184 to identify the first vehicle 110a as a racing car with a given identifier, i.e., CarID 11, and identify the second vehicle 110b as another racing car with a given identifier, i.e., CarID 34.

[0057] Figure 9 shows another configuration of a detection system 180 including two cameras 182a and 182b corresponding to two different track sections 130a and 130b. For each location, essentially the same steps are performed as detailed above with respect to Figure 8; the details will not be repeated here.

[0058] In addition, after a vehicle 110 is successfully identified, in step 4, the array of vectors 178 used to represent the identified vehicle is updated as described above with reference to Figure 7. A new embedding vector 176 is added to the array 178, particularly if it is significantly different from all vectors in the array. If the array 178 reaches a maximum size, another embedding vector may be removed from the array, e.g., a vector that is very similar to another vector in the array, or a vector that lies entirely within the vector space defined by other vectors in the array 178.

[0059] Moreover, in step 5, the digital twin of each vehicle is then updated using the information passed from the inference server 184. For example, its last known location, speed, position in the race order, etc. may be updated based on the updated identification of the vehicle at its new current location. In the described system, all positions, estimated speeds, and other relevant information are persistently stored along with a timestamp corresponding to the moment when the corresponding digital image is taken. In this way, the estimated positions and speeds of the vehicles can be displayed live in the digital model. Moreover, the digital model can be used to reconstruct the race, for example, for race analysis, identification of drivers responsible for accidents or anomalies 144 along the race track 100.

[0060] The above technique allows further functionality to be implemented. For example, if both anomalies 144 and vehicle trajectories are detected and maintained by the system 180, it is possible to identify the vehicle 110 that caused the anomaly 144, such as a lost part or oil spill on the race track, and hold the driver or owner of the vehicle 110 responsible.

[0061] Moreover, if a driver or other person is interested in video footage of a particular event, such as a crash, or all available footage of a selected car on the race track 100, such video sequences can be automatically selected and cut and provided to the interested parties.

[0062] Figure 10 shows a potential architecture for implementing the above functionality. However, as detailed above, each of the described functions can be implemented and used separately, if desired.

[0063] The monitoring system 180 of FIG. 10 includes two camera clusters 202a and 202b of four cameras 182a-d each. The cameras of each camera cluster 202 are connected to a corresponding rack workstation 204a and 204b, which includes one or more frame grabbing cards 206. Among other things, the rack workstation 204 is responsible for grabbing essentially all of the video output of the respective camera cluster 202. In the described system, video footage is stored for three days for analysis and post-processing, as described in more detail below. Additionally, the rack workstation 204 performs further different tasks during race monitoring and at other times when a race is not currently taking place.

[0064] During the race, the rack workstation 204 performs anomaly detection based on image segmentation, delta image computation and anomaly classification as detailed above, as well as vehicle tracking based on vehicle detection, vehicle embedding, image segmentation and location estimation as detailed above.

[0065] At other times, e.g. overnight, it performs computationally expensive tasks such as batch processing for video generation, cutting video footage for individual vehicles or drivers, pixelating other vehicles or their registration numbers, e.g. for data protection.

[0066] The digital twin processing unit 208 maps the inferences and other information provided by the rack workstations 204 to a corresponding digital twin of the race track 100 and / or the individual vehicles 110. In particular, it tracks the speed estimates and current position of each vehicle 110. It further records the location, time, and corresponding images of any detected incidents, in particular crashes of any vehicle 110. The digital twin processing unit 208 further monitors the condition of the race track 100 and the positions of the vehicles 110 on the track.

[0067] In the described embodiment, the digital twin processing unit 208 also runs a rules engine. As previously described, the rules engine may generate alerts corresponding to detected incidents, such as vehicle crashes, and control flags, displays, and other notifications along the track. Additional vehicle information may be provided to the rules engine from the digital twin. This allows rules to be formulated based on the dynamic characteristics of the monitored system. For example, a rule may be triggered if the calculated speed of the vehicle falls below or exceeds a certain threshold.

[0068] The monitored track conditions and car positions are provided from the digital twin processing unit 208 to a track and car monitoring unit 210, for example for the benefit of race officials or as information for spectators.

[0069] A system monitoring unit 212 is provided to control and view the operation of the monitoring system 180 itself. It may indicate the operational status of the software and hardware components of the monitoring system.

[0070] Finally, a video post-production unit 214 is provided that is configured to collect all or selective footage from all cameras 182, build a targeted footage of a particular vehicle 110 or group of vehicles, for example a vehicle leading a race or involved in a particular incident, and control the video post-production. The video post-production unit 214 also provides a customer API or web interface to allow registered users to request and retrieve video material related to their own vehicles 110.

[0071] Compared to previous solutions, the described monitoring system is more flexible because both the training data and the rules set for detection, classification and triggering can be continuously updated. Moreover, it does not require specific hardware, such as RFID-based transponders as used in professional racing, to be installed on or in the vehicles participating in the race. Therefore, the described monitoring system is particularly useful for amateur racing using privately owned vehicles rather than professional racing. [Prior art documents] [Patent documents]

[0072] [Patent Document 1] International Publication No. 2017 / 212232A1 [Patent Document 2] U.S. Patent No. 6,020,851A [Patent Document 3] US Patent Application Publication No. 2018 / 341812A1 [Explanation of symbols]

[0073] 100 Race Tracks 102 Running surface 104 Grassland Area 106 Gravel Area 108 Guardrail 110 vehicles 120 Monitoring Methods 121~126 Method steps Section 130 (of a race track) 132 Bounding Box 134 Bounding Box 136 Bounding Box 140 (image) segments 142 Delta Images 144 Abnormality 150 reference points 160 Neural Networks 162 Input side 164 nodes 166 embedding vectors 168 Output side 170 Multidimensional feature space 172 First Embedding Vector 174 Second Embedding Vector 176 new embedding vectors Array of 178 Vectors 180 Monitoring System 182 Cameras 184 Inference Server 186 Digital Twin Server 188 Detection Unit 190 Inference Units 202 Camera Cluster 204 Rack Workstation 206 Frame Grabbing Card 208 Digital Twin Processing Unit 210 Truck and Vehicle Monitoring Unit 212 System Monitoring Unit 214 Video Post-Production Unit

Claims

1. 1. A method for identifying individual vehicles on a race track, comprising: taking and processing at least one image of a plurality of vehicles on the race track and processing the at least one image using a neural network to calculate at least one corresponding reference embedding vector for each vehicle of the plurality of vehicles, the at least one image being taken during an initial registration process as each vehicle enters the race track or a monitored portion of the race track; acquiring at least one sequence of images from a camera capturing at least one section of the race track; detecting at least one vehicle in the sequence of images using automatic object recognition; sending a portion of an image in the sequence of images corresponding to the detected vehicle to the neural network to determine a new embedding vector; comparing the new embedding vector to a set of embedding vectors to be enrolled, the set including the at least one reference embedding vector calculated for each vehicle of a predetermined set of vehicles during the initial enrollment; and identifying the at least one vehicle as the vehicle corresponding to a closest embedding vector from the set of registered embedding vectors. method.

2. 2. The method of claim 1, wherein an angle between the new embedding vector and all embedding vectors in the set of registered embedding vectors is calculated, and the embedding vector with the smallest angle is selected as the closest embedding vector for identifying the at least one vehicle.

3. storing the new embedding vector in an array of vectors corresponding to the identified vehicle if the new embedding vector differs from the closest embedding vector by more than a first preset threshold; and / or if the new embedding vector differs from each one of the set of registered embedding vectors by more than a second preset threshold, not including the new embedding vector in the array of vectors corresponding to the identified vehicle and / or failing to identify the at least one vehicle. The method of claim 1.

4. 2. The method of claim 1, wherein the neural network is pre-trained during a training phase using images of different vehicles to learn characteristic configurations of vehicles or of particular subclasses of vehicles, and is configured to extract or infer in real time during a race characteristics of vehicles that are not known beforehand.

5. The training step comprises: providing the images of the different vehicles to an input of the neural network; simplifying information received at the input by nodes of the neural network to form embedding vectors; Decoding the information of the embedding vector to recreate the image at the output of the neural network; and varying weights and other settings of the neural network based on an automatic comparison of the input and output sides using a similarity metric until the image at the output side highly resembles the image provided at the input side. The method according to claim 4.

6. 2. The method of claim 1, further comprising extracting at least one characteristic feature, in particular a license plate or other registration number, of the at least one vehicle from the at least one image taken as the at least one vehicle enters the race track.

7. The method of claim 1 , further comprising mapping the identified vehicles to corresponding digital twins in a digital representation of the track race.

8. 8. The method of claim 7, wherein the predetermined set of vehicles corresponds to a subset of all vehicles having corresponding digital twins in the digital representation of the race track, the subset being selected based on a set of rules that provide a possibility of identifying a given vehicle in the sequence of images corresponding to at least one section of the race track based on data of the corresponding digital twin.

9. 9. The method of claim 8, wherein the subset of vehicles used in the identification process is limited to vehicles that have already passed another camera located on an earlier section of the race track, specifically to vehicles that were last successfully identified in one of the two or three sections of the race track covered by the other camera.

10. The method of claim 1 , wherein the entire race track is covered by a video camera and the at least one vehicle is tracked continuously.

11. selecting a plurality of sequences of images from a plurality of cameras capturing different sections of the race track based on an identification of at least one particular vehicle in each one of the plurality of sequences; and cutting the plurality of sequences to generate an image of at least one particular vehicle driving along the race track. The method of claim 1.

12. determining a first real world position of the at least one identified vehicle based on a mapping relationship, the mapping relationship mapping a plurality of pixel areas in images of the at least one sequence of images to a corresponding plurality of real world positions of the corresponding section of the race track captured by the camera; and adding first position and timestamp information to the corresponding digital twin each time a first real-world position of a vehicle to be re-identified is determined to add a tracking history of each vehicle into the digital representation of the race track. The method of claim 1.

13. detecting foreign objects along the racetrack based on a comparison of at least two images of the at least one sequence of images; classifying the detected foreign objects as acceptable or unacceptable based on automatic pattern recognition; adding second location and time stamp information to a corresponding digital representation of at least one foreign object detected along the racetrack and classified as unacceptable; and correlating the first and second location and time stamp information to identify vehicles that are likely causing at least one unacceptable foreign object. The method of claim 12.

14. the detected foreign object is classified as acceptable if it is classified as one or more of a raindrop, a leaf, a reflection, a shadow, and / or a light beam; and / or The detected anomaly is classified as unacceptable if it is classified as a vehicle part, oil, and / or gravel; The method of claim 13.

15. 1. A monitoring system for a race track, comprising: one or more cameras, each camera having a field of view covering at least one section of the racetrack; an image capture system configured to acquire a sequence of images from at least one of the cameras; an image processing system including at least one processor; The processor, taking at least one image of a plurality of vehicles on the race track, and processing the at least one image using a neural network during an initial enrollment process to calculate at least one corresponding reference embedding vector for each vehicle of the vehicles; Detecting at least one vehicle in one of the sequences of images using automatic object recognition; sending a portion of an image in the sequence of images corresponding to the detected vehicle to the neural network to determine a new embedding vector; comparing the new embedding vector to a set of embedding vectors to be enrolled, the set including the at least one reference embedding vector calculated for each vehicle of the predetermined set of vehicles during the initial enrollment; identifying the at least one vehicle as the vehicle corresponding to a closest embedding vector from the set of registered embedding vectors; It is configured as follows: Monitoring system.

Citation Information

Patent Citations

  • System and method for OCR output verification

    JP2014232533A

  • Virtual object movement control method, movement control device, terminal, and computer program

    JP2022527157A

  • Automatic determination and monitoring of vehicles on a racetrack with corresponding imagery data for broadcast

    US20180341812A1

  • Vehicle re-identification method, apparatus, device and storage medium

    US20210192214A1

  • Image search system, image search method, and program

    WO2018134964A1

Cited By

  • Communication problem identification device and method

    US20250390527A1