AI-based monitoring of racetracks

Abstract

To provide an improved monitoring system and method for race tracks, which is easy to implement and cost effective.SOLUTION: A novel AI based monitoring system for race tracks used for car racing enables: automatic detection and reaction to critical situations, including a deviation of a vehicle from the race track and / or collision with a guide plank, a loss of oil, a person or other object on the race track or the like; rule based definition and association of the automatic detection and / or automatic reactions; tracking of vehicles along the race track, including storage of the driven track; automatic mapping of detected critical situations to one or more tracked vehicles; and / or automatic generation and cutting of video footage for a tracked vehicle.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a new AI-based monitoring system and method for a racetrack such as a racetrack used for professional and amateur car races.

Background Art

[0002] Racetracks, especially car race tracks such as the North Loop of the Nürburgring, often feature many corners, including blind corners, drops, and significant elevation changes, which make the racetrack difficult and thus potentially dangerous for high-speed driving. Historically, in situations such as when a vehicle leaves the track, race marshals have relied on wireless communication and communication with other marshals on the track to relay such information to one or more race stewards and make decisions regarding track safety. It has often been difficult to quickly assess and respond to an incident when there is no direct line of sight. If hundreds of vehicles are on the track at the same time, the speed and accuracy of safety decisions are essential to protect drivers and spectators.

[0003] Patent Document 1 discloses a track monitoring device and system. Specifically, it relates to a system and method for detecting whether one or more participating vehicles in a race are on a predetermined route or track or have left the predetermined route or track. The system includes a series of indicating means provided within or on the track and detecting means mounted on the vehicle. If the indicating means is detected by the detecting means, this is understood as an indication that the vehicle has left the track, and an alert or alarm can be generated and may be assigned to the vehicle for which a penalty has been specified.

[0004] Patent Document 2 discloses another race monitoring system. The disclosed auto race monitoring system provides a ground positioning system to a race track, which includes at least three transmitters that transmit signals to be received by at least one pair of receivers for each race car. These receivers instantaneously determine their position and, accordingly, the precise position and attitude of the race car on the race track. This information, along with data on race car parameters such as vehicle speed, engine temperature, and oil pressure, is transmitted by the transmitters to receivers interconnected with a mainframe computer, so that a viewer can select any particular race car they wish to monitor at any particular time during the race.

[0005] Patent Document 3 discloses a method and system for automatically tracking and analyzing image data of at least one vehicle on a race track, wherein a video event management system, comprising multiple video cameras positioned around the race track, determines the presence of at least one vehicle, captures video and still images based on weighted event scores corresponding to the dynamics of at least one vehicle and other objects, and generates at least one subframe. Excess video and still image data is discarded based on the metadata of the linked subframe. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] While the above monitoring systems may be beneficial in race monitoring and control, they require specialized equipment to be installed inside the vehicle and are therefore not suitable for all types of races or race cars. Furthermore, due to the specialized sensing technologies used, they are relatively expensive and difficult to implement, especially on large race tracks. Therefore, there is a need to provide improved monitoring systems and methods for race tracks that are suitable for many types of races or race participants, and are preferably easy to implement and cost-effective. [Means for solving the problem]

[0007] According to the first embodiment, a method for monitoring a race track is the following steps: - A step of obtaining at least one sequence of images from a camera that captures at least one section of the race track, - A step of segmenting the images in a sequence of images into different areas that can be associated with the racetrack, - A step of detecting at least one vehicle in a sequence of images using automatic pattern recognition, particularly automatic object recognition, - A step of mapping at least one detected vehicle to at least one of the different areas associated with the racetrack, - The activation step includes activating at least one warning device based on a first set of rules, the first set of rules including at least one first rule that triggers a first warning if at least one detected vehicle is mapped to a first predetermined area of ​​a race track, such as a guardrail or boundary area.

[0008] In particular, the inventors recognized that incident detection on racetracks can be automated and thus accelerated by using artificial intelligence (AI)-based techniques such as image segmentation, object detection, and positioning. Furthermore, providing a set of rules that associate detection results with relevant actions can significantly enhance the flexibility and effectiveness of racetrack monitoring.

[0009] According to a second aspect, a monitoring system for a race track includes one or more cameras, each having a field of view covering at least one section of the race track; an image capture system configured to acquire a sequence of at least one image from at least one of the cameras; one or more warning devices configured to be activated when a first warning is triggered; and an image processing system including at least one processor. The processor is configured to segment the images of the sequence of images into different areas associated with the race track, to detect at least one vehicle in the sequence of images using automatic pattern recognition, in particular automatic object recognition, to map the at least one detected vehicle to at least one of the different areas associated with the race track, and to trigger a first warning if, based on a first set of rules, the at least one detected vehicle is mapped to a first predetermined area of ​​the race track.

[0010] The disclosed monitoring systems and methods enable the following: - Automatic detection of critical situations along the track, including vehicle departure from the racetrack and / or collision with guide markers, oil loss, people or other objects on the racetrack, etc. - Automatic response to critical situations, including activating warnings and control signals, outputting messages to panels positioned along the track, selecting, outputting, and / or recording video footage of relevant sections of the race track. - Rule-based definition and association of automatic detection and / or automatic response, - Tracking of vehicles along the race track, including storage of the tracks being run on, and - Automatically map detected critical situations to one or more tracked vehicles involved in the critical situation, and / or - Automatic generation and editing of video footage of tracked vehicles.

[0011] The disclosed system implements the above configuration based on a combination of one or more of the following electronic processing techniques. - At least a portion of the racetrack is monitored by a series of video cameras to generate a sequence of images. - A given class of objects, such as a vehicle or vehicle parts, oil, and / or gravel (unacceptable) anomalies, is detected within an image sequence using an AI-based object recognition algorithm, such as a convolutional neural network (CNN), particularly a deep learning network (DLN). - The sequence of images is analyzed using segmentation, for example, using a segmentation mask and / or a segmentation network, to identify different parts of the racetrack or along the racetrack, such as the running surface, lanes, track boundaries, guardrails, escape areas, off-boundary areas, pit areas, scenic areas, tarmac areas, gravel areas, filth areas, grass areas, and planted areas. - Detected objects are mapped to different parts of the racetrack. - Based on one or more sets of rules, warning devices such as flags or lights along the racetrack are activated fully automatically or semi-automatically by alerting racetrack personnel or race management to control potentially dangerous situations, such as a race car crashing into a guardrail or an oil spill on the track surface. - The real-world locations of detected and arbitrarily identified vehicles and anomalies are calculated based on triangulation and interpolation of known reference points within the captured images. - Detected vehicles are identified based on training data, for example, by calculating embedding vectors. - A digital twin of the vehicles is maintained in a digital model of the racetrack, including data such as the position or speed of each vehicle on the racetrack. - Detected anomalies are associated with vehicles on the track, for example, by comparing the track of each vehicle with the location of the detected anomaly and / or - Image sequences related to selected events or selected vehicles, such as crashes or other incidents, may be automatically collected and edited, or provided to race management upon request.

[0012] Specific embodiments of the disclosed monitoring system and method are described below with reference to the attached figures. [Brief explanation of the drawing]

[0013] [Figure 1] This shows a schematic representation of the racetrack as captured by trackside cameras. [Figure 2] This shows a flowchart of methods for monitoring a race track. [Figure 3A] This shows a sequence of images taken before, during, and after vehicle detection on a segment of the racetrack. [Figure 3B] This shows a sequence of images taken before, during, and after vehicle detection on a segment of the racetrack. [Figure 4] Figure 3 schematically illustrates the detection of anomalies on a segment of the racetrack. [Figure 5] This provides a schematic representation of a triangulation method used to position detected objects within the scene. [Figure 6A] This provides a schematic demonstration of training and inference using embedding vectors for object re-identification. [Figure 6B] This provides a schematic demonstration of training and inference using embedding vectors for object re-identification. [Figure 6C] This provides a schematic demonstration of training and inference using embedding vectors for object re-identification. [Figure 6D] Training and inference using embedded vectors for object re-identification are shown in a schematic way. [Figure 7] The parameter space of the embedded vectors used for vehicle re-identification is shown in a schematic way. [Figure 8A] Collaboration between an inference server and a digital twin server based on images from a single camera is shown in a schematic way. [Figure 8B] Collaboration between an inference server and a digital twin server based on images from a single camera is shown in a schematic way. [Figure 9A] Collaboration between an inference server and a digital twin server based on images from two cameras is shown in a schematic way. [Figure 9B] Collaboration between an inference server and a digital twin server based on images from two cameras is shown in a schematic way. [Figure 10] The system architecture of a monitoring system for a race track is shown in a schematic way.

Best Mode for Carrying Out the Invention

[0014] Hereinafter, a specific system architecture and an operation method for a race track monitoring system will be described.

[0015] In the system described, one or more cameras provide video footage of all or part of the race track 100. As shown in FIG. 1, an artificial intelligence (AI) system recognizes different areas of the captured (acquired) video, such as the driving surface 102, the grass area 104, the gravel area 106, or the crash barrier 108, based on predefined rules or appropriate training, and divides the images from the video into corresponding segments.

[0016] In the embodiments described, individual segments are automatically detected using a segmentation network based on, for example, a so-called U-Net architecture or another type of convolutional neural network (CNN) pre-trained using images of the racetrack taken by trackside cameras. Compared to the potential static segmentation of camera images, this allows for the covering of movements, panning, and zooming of trackside cameras. Optionally, feedback information from the camera control device may be used to trigger newly segmented image streams taken by a given camera whenever the camera moves, pans, zooms, etc. During segmentation, the output from each camera may not be used by the monitoring system. Furthermore, for improved reliability, initial segmentation may be performed for each camera at known default positions. The output of the segmentation stage is a pixel mask. For each pixel, the probability of belonging to one of several possible classes of segments is calculated. The pixel mask indicates, for each pixel, the most likely segments of the race track 100, such as the driving surface 102, grass area 104, gravel area 106, and guardrail 108 (crash barrier).

[0017] Alternatively, especially when one or more cameras are mounted in fixed positions with a fixed field of view, segmentation may be performed manually, i.e., statically, before the monitoring system is started. Furthermore, it is also possible to manually mask only a portion of the captured image for a known portion located outside the racetrack 100, for example. Masking such portions may speed up or improve the quality of subsequent segmentation. This is also called semi-automatic segmentation.

[0018] Vehicles 110 on the racetrack are also recognized based on artificial intelligence. For example, known object detection algorithms based on publicly available training data may be used to detect vehicles 110. For example, deep learning networks (DLNs) or other types of convolutional neural networks (CNNs) may be used to detect vehicles, people, animals, and other objects on the racetrack 100. In the system described, the YOLO real-time object detection system is used for object detection. Detection accuracy can be improved by using additional training data captured on relevant racetracks showing previous race events and by manually tagging vehicles 110 in the training data.

[0019] Based on the steps described above, the disclosed system analyzes which segment the detected vehicle 110 is in. This data is passed to a rule-based system, which can trigger events and alarms based on predefined rules. For example, LED panels and warnings emitted along track 100 may be generated if a blocked vehicle, foreign object on the racetrack, or dirt or oil on the racetrack is detected.

[0020] Preferably, the system can distinguish between acceptable and unacceptable anomalies. Acceptable anomalies include, for example, deviations in the captured video footage caused by reflections, rain, shadows, and / or lighting beams. Unacceptable anomalies relate to the loss of parts of the vehicle 110, for example, the spread of oil and other working fluids or gravel or soil from an adjacent area onto the racetrack. As described below, a convolutional neural network (CNN) may be used to detect the entry and exit of objects of known types, such as the vehicle 110, into a given section (partition) of the racetrack.

[0021] Figure 2 shows the steps of a method 120 for monitoring the racetrack 100 in the form of a flowchart.

[0022] In step 121, at least one sequence of images is acquired from a camera capturing at least one section of the racetrack 100. In step 122, the images from the sequence of images are segmented into different areas 102, 104, 106, and 108 associated with the racetrack 100. Segmentation is performed based on a manual segmentation mask provided before the race starts, or based on an automatically or semi-automatically generated segmentation mask calculated during initialization, after camera movement, or in real time. In step 123, at least one vehicle 110 is detected in the sequence of images using automatic object recognition. The order of steps 122 and 123 may be reversed; that is, potentially moving objects may be detected first and used to assist the image segmentation process. In step 124, the at least one detected vehicle 110 is mapped to at least one of the different areas 102, 104, 106, or 108 associated with the racetrack 100. In determination step 125, it is determined whether at least one rule from a set of rules has been triggered. The set of rules includes at least one first rule that triggers a first warning if at least one detected vehicle 110 is mapped to a first predefined area of ​​the racetrack 100, for example, a guardrail 108. Otherwise, the method proceeds to step 121 to acquire and process the next image. However, if at least one rule from the set of rules is triggered in step 125, at least one warning device is activated in step 126 based on the set of rules.

[0023] The following sections provide a more detailed description of the different aspects of the monitoring systems and monitoring methods that will be disclosed.

[0024] Starting with a set of rules, this may include several predefined, separate rules for triggering alerts. Each rule may include one or more conditions that determine when the rule is triggered, and one or more actions that are performed when the rule is triggered. Note that not all rules depend on segmentation. For example, if an animal is detected somewhere in the image, an alert may be issued regardless of whether the animal is on the running surface 102. As a specific example, a set of rules may include the following rules: [Table 1]

[0025] As shown in Figures 3A, 3B, and 4, anomaly detection may be triggered whenever vehicle 110 enters the monitored section 130 of track 100. Subsequently, a differential analysis of the race track 100 before and after vehicle 110 passes through the relevant section 130 of track 100 can be performed. Again, this detection may be based on artificial intelligence and appropriate training data. For example, the system may be trained on the above acceptable and unacceptable anomalies based on manual tagging of acceptable and unacceptable deviations in previous race events and historical footage.

[0026] In addition to tagging historical footage, anomalies may be artificially created, for example, by placing a foreign object on racetrack 100 and marking it as an unacceptable anomaly. To improve detection rates, images of both acceptable and unacceptable anomalies must be taken under different weather conditions.

[0027] As shown in Figure 3A, the sequence of images 132, 134, and 136 shows the same section 130 of the racetrack 100 before, during, and after the passage of a vehicle 110, as detected by an object detection system. Processing may be triggered by the successful detection of the car or other vehicle 110 in image 134 provided by the camera and captured by a frame grabber. Once the vehicle 110 is detected, images 132 and 136 may be selected from the captured video footage. For example, the last image in the time series of images taken before the bounding box 138 surrounding the detected vehicle 110 enters section 130 may be selected as the first image 132. Similarly, the first image in the time series of images taken after the bounding box 138 surrounding the detected vehicle 110 leaves section 130 may be selected as the third image 136.

[0028] Figure 3B shows the same sequence of images 132, 134, and 136 after segmentation. In the illustrated example, the segmentation algorithm used extracts only a single segment 140 corresponding to the racetrack's running surface 102. As shown in the lower center, the detected vehicle 110 is positioned within segment 140, i.e., on the racetrack 100. Therefore, no alarm is triggered at this stage.

[0029] As shown in Figure 4, once the detected vehicle 110 has left section 130, a comparison may be made between the segments of interest. In particular, a delta image 142 may be calculated between segment 140 in the first image 132 and the same segment 140 in the third image 136. The delta image 142 is classified using a deep learning classifier to determine whether the delta image corresponds to or contains at least one known anomaly 144.

[0030] In the example presented, an oil spill left behind by a passing vehicle 110 is detected. Accordingly, a corresponding warning may be activated for all or at least section 130 of the racetrack 100 containing the anomaly 144. Depending on the severity of the detected anomaly and / or the associated uncertainties of the detection algorithm, the warning may be triggered automatically by the monitoring system, i.e., without human intervention, or displayed to a race director or similar officer for verification, with the option of triggering or suppressing the warning.

[0031] A further feature is that the position of a vehicle 110 on the racetrack 100 may be determined and optionally tracked along the course. For this purpose, real-world positions, such as the GPS coordinates of reference points visible in the monitored section 130 of the track 100, may be recorded. Then, based on triangulation and similar interpolation techniques, detected objects such as the vehicle 110 and anomalies 144 along the monitored section 130 of the track 100 can be associated with the previously recorded positions of the reference points. That is, the reference points map image positions that define a single pixel or group of pixels (collectively referred to as a pixel area) in the digital image of the racetrack 100 to the corresponding real-world positions on the racetrack 100.

[0032] This is illustrated in detail in Figure 5. Within another section 130 of the racetrack 100, a total of 15 reference points 150 are provided. In the illustrated example, the reference points 150 form a fixed grid of 3 x 5 reference points 150. This can be useful, for example, when the position, orientation, and angle of the camera capturing section 130 are fixed. In this case, during setup or training, a GPS or similar positioning device with visible markers may be placed sequentially at each reference point 150 to determine its exact real-world position. Alternatively, the reference points 150 may coincide with prominent, high-contrast features within section 130, i.e., edges or corners of segment boundaries, special objects such as flagpoles, or equivalent. Such reference points can be easily identified in the video images of section 130, even if the camera moves, pans, zooms in and out. Again, the real-world positions of such prominent features are measured and stored during setup or training and used for triangulation during normal operation of the system.

[0033] In the situation illustrated in Figure 5, two vehicles 110a and 110b are detected in section 130. To determine their positions, their current positions are estimated using three or more reference points 150 located near them. As shown in the figure, the current position of the first vehicle 110a is estimated by triangulation using reference points 150i, 150j, and 150o that are closest to the center of the first boundary box 138a of the first vehicle 110a. Similarly, the current position of the second vehicle 110b is estimated using reference points 150b, 150f, and 150g that are closest to the center of the second boundary box 138b of the second vehicle 110b.

[0034] Therefore, the locations of the detected vehicles 110 and anomalies 144 can be displayed on a visual representation of the racetrack 100 in race control, for example, where race personnel work and make decisions regarding track warnings and potential race interruptions.

[0035] To date, the detection and arbitrary location of objects of a specific class, such as vehicles 110 and anomalies 144, has been described. As detailed above, this is sufficient to generate safety alerts for the monitored racetrack 100. However, the AI-based monitoring system described also enables several more advanced configurations (features) described below. In particular, it enables the re-identification of individual objects on the racetrack, such as a specific vehicle 110. For clarity, the terms “object detection” or “object recognition” are used to describe the mere detection of a particular type of object in the captured image, such as the presence of any race car on the monitored section 130 of the racetrack 100. In contrast, the term “object re-identification” is used to describe the identification of a unique entity, such as a pre-registered race car of an individual race participant currently on the track.

[0036] As an additional configuration, if a digital model of the racetrack 100 is maintained by a monitoring system, the position of each vehicle 110 may be continuously updated to create a so-called digital twin of the monitored vehicle 110, for example, as shown in Figure 8 below. In the monitoring system described, this is based on the re-identification of individual vehicles 110 rather than merely the detection of any vehicle 110 on the racetrack 100. Vehicles 110 may be distinguished based on extractable features such as shape, color, or other visible marks such as logos or symbols printed on the vehicle 110. Appropriate training data for automatic identification may be captured at clearly defined points on the racetrack 100. Images of individual vehicles from different positions, for example, from the front, rear, or side of the vehicle 110, may be taken when the vehicle 110 first enters the racetrack 100 in the entrance lane. If only a portion of the racetrack 100 is monitored, training images may be taken by one or more cameras monitoring a first section 130 of the racetrack 100. Based on this information, whenever vehicle 110 is detected along the racetrack 100, the detected object can be compared with previously captured image data, for example, by inference and calculation of embedding vectors. If the registered vehicle 110 is racing on the racetrack 100, additional information may be used, for example, to improve the OCR detection rate of the registration plate.

[0037] In the embodiments described, object re-identification is implemented using a neural network 160 trained offline, i.e., before the monitoring system is used in an actual race, using an encoder / decoder model to identify a specific instance from a given class of objects, e.g., individual racing cars participating in a currently running race. Different training sets may be used to train different instances of the corresponding neural network to represent different classes of objects, e.g., Formula 1 cars, regular road cars, motorcycles, etc. Furthermore, if certain characteristics of the objects to be identified change due to, for example, rule changes regarding racers, the training of the neural network 160 may be repeated.

[0038] The training of the neural network 160 used for object recognition is conceptually shown in Figures 6A and 6C. During the training phase, images of different vehicles 110 are provided to the input side 162 or encoder of the neural network 160. To achieve the desired accuracy, a large number of training images are used, which are manually selected or verified and taken at the actual race track 100. For the monitoring system described, a set of approximately 17,000 vehicles extracted from archival footage of races at the Nürburgring was used to train the neural network 160 after human review of the material.

[0039] During actual training, the information received on the input side is simplified by node 164 of the neural network 160 to form or encode an embedding vector 166. The information in the embedding vector 166 is decoded by the output side 168 or decoder of the neural network 160 to recreate the image. The weights and other settings of the neural network 160 are changed until the image at output side 168 becomes highly similar to the image provided to input side 162, for example, until the difference between them becomes very small or minimal. At that stage, the neural network is learning characteristic features of a particular subclass of vehicle 110, such as a racing car. The comparison between input side 162 and output side 168 can be done automatically using an appropriate similarity metric. Thus, training of the neural network 160 can be performed unsupervised.

[0040] In the example in Figure 6A, the embedding vector 166 has only one node 164, which corresponds to single-dimensional information, for example, the shape of the object. Figure 6C shows another neural network 160 under training, which has two nodes 164 that form the embedding vector 166, for example, corresponding to the shape and color of the object of interest.

[0041] In practice, even more dimensions may be used to reliably identify objects such as vehicle 110. In this context, it is worth noting that in racing, a relatively large number of very similar cars, for example, cars of the same model, often race against each other, and they may differ only in relatively small details such as the presence and shape of auxiliary parts such as color, advertising, and spoilers. Thus, a multidimensional embedding vector 166 in a multidimensional feature space 170 having, for example, 168 independent dimensions, is used in the system described. During unsupervised training, the most significant features for re-identifying a particular vehicle 110 corresponding to each node 164 representing the dimensions of the embedding vector 166 are automatically determined by the training algorithm. Thus, they may not correspond to high-level features such as color or shape, and may not be easily understood by humans.

[0042] Once the neural network 160 is trained, it can be used to extract or infer previously unknown characteristic properties of vehicle 110 online, for example, in real time during a race. This process is illustrated in Figures 6B and 6D, which show the classification of an input image to determine the shape, or shape and color, of vehicle 110. This inference is used for re-identifying objects during track monitoring.

[0043] The process for the initial registration and subsequent identification of a specific vehicle 110 is also shown in Figure 7. Before a vehicle 110 enters the racetrack 100, one or more images of the vehicle 110 are taken and processed by a pre-trained neural network 160 to generate one or more corresponding embedding vectors 166. For example, in a multidimensional feature space 170, a first vehicle 110a may correspond to a first embedding vector 172, and a second vehicle 110b may correspond to a second embedding vector 174. In the described system, multiple images of the same vehicle 110 from different viewpoints, for example, from the front, rear, and side, are used to create an array of possible embedding vectors, as described later.

[0044] Next, once vehicle 110 is on the track, the portion of the image corresponding to the detected vehicle 110 can be sent to the neural network to determine a new embedding vector 176. The new embedding vector 176 may then be compared with previously registered vectors 172 and 174. The vehicle may then be identified as the vehicle corresponding to the nearest previously known embedding vector, for example, the first vehicle 110a corresponding to the first vector 172. Specifically, the angle between the new embedding vector and all the previously registered embedding vectors 172 and 174 is calculated, and the vector with the smallest angle, for example, the first embedding vector 172, is selected as the best matching vector.

[0045] Optionally, if a new embedding vector 176 differs from the nearest pre-registered vector 172 or 174 by a first preset threshold, the new vector may be stored in the system within an array of vectors 178 corresponding to a given vehicle 110. This may be used, for example, to improve future object identification by adding embedding vectors 176 corresponding to images of the same vehicle 110 taken from different angles or under different environmental conditions such as lighting or weather conditions. The system described can store up to 500 different embedding vectors for each registered vehicle. Generally, this makes vehicle detection more likely and reliable.

[0046] Alternatively or additionally, if the new embedding vector 176 differs from one of the previously registered vectors 172 and 174 by a second predetermined threshold, identification may fail and / or the new embedding vector 176 is not included in the array of vectors 130. This may be used to exclude uncertain matchings and / or to avoid degradation of the array of vectors 178.

[0047] If the entire track 100 is covered by video cameras, the vehicle detection and identification described above can be further improved. In this case, continuously tracking each moving object allows for its detection even if some or all of its characteristic configuration is temporarily obscured. If not all parts of the racetrack 100 are covered by each camera, and / or if a movable camera is used and does not cover the vehicle of interest 110 at any given moment, detection accuracy may be improved by using certain logical assumptions about the likelihood of the vehicle 110 appearing in a particular area of ​​the racetrack 100, based on a digital model of the racetrack 100.

[0048] Figure 8 shows the specific configuration and operation of the monitoring system 180. The system 180 includes a camera 182 that captures the corresponding section 130 of the racetrack 100. The system 180 further includes an inference server 184 and a digital twin server 186.

[0049] The inference server 184 captures digital images or video frames provided by the camera 182 and stores them for later use. At least a subset of the captured images is fed to the AI-based object detection unit 188 to identify objects of a predetermined type, such as the vehicle 110. As shown in Figure 8, each identified object is enclosed by a corresponding bounding box 138.

[0050] In addition, the detection unit 188 also segments the received camera image. For example, the image of the track section 130 may be subdivided into areas corresponding to the driving surface 102, the grass area 104, and the gravel area 106.

[0051] The inference server 184 further includes an inference unit 190. The inference unit 190 is configured to determine the position of a vehicle to be detected based on GPS position interpolation using known reference points 150 in section 130.

[0052] The inference unit 190 is further configured to use the neural network 160 to generate embedding vectors for each detected vehicle 110. The determined embedding vectors, along with the real-world position of the detected vehicle 110 on the racetrack 100 and / or the identifier of the camera that captured the image, are passed to the digital twin server 186 for identification.

[0053] The digital twin server may use multiple plausibility checks to identify each of the vehicles 110a and 110b. Generally, the set of embedding vectors used for matching can be limited to vehicles 110 that are known to be on track 100. Furthermore, in step 1, based on the camera identifier, it can select only a subset of vehicles 110 that are likely to be in the view of that camera. This can be achieved, for example, by considering only vehicles that have already passed another camera positioned in an earlier section of the racetrack 100. For example, the subset of vehicles used in the identification process may be limited to vehicles that were last successfully identified in any one of two or three sections of the track covered by other cameras. Including more than one up-track camera addresses the fact that some vehicles 110 may be covered in a particular camera setup or may not be successfully identified due to difficult lighting conditions or camera angles.

[0054] Alternatively or additionally, in step 2, the selection may be refined based on real-world locations provided by the inference server 184. For example, based on the estimated speed of the vehicle 110 in the digital model, only vehicles that could most likely have reached a specific real-world location in the captured image at the time the image was taken may be considered for matching.

[0055] Reducing the number of vehicles 110 used during identification to a predetermined subset can lower the required processing power, thereby enabling real-time identification at relatively high resolution of input images during high-speed races. Higher resolution and reduced plausibility of matching also improve detection accuracy. For example, if two very similar vehicles of the same model and color are participating in the same race but are located in different sections 140 of the race track 100, object identification is significantly improved compared to a typical real-time identification engine.

[0056] In step 3, after such filtering, the digital twin server selects the embedding vector that is closest to the new embedding vector 176 (see Figure 7) provided by the inference server 184 in the multidimensional feature space 170 to identify the first vehicle 110a as a racing car with a given identifier, i.e., CarID 11, and the second vehicle 110b as another racing car with a given identifier, i.e., CarID 34.

[0057] Figure 9 shows another configuration of the detection system 180, including two cameras 182a and 182b corresponding to two different track sections 130a and 130b. For each location, essentially the same steps as detailed above with respect to Figure 8 are performed. Details are not repeated here.

[0058] In addition, after successfully identifying vehicle 110, in step 4, the array of vectors 178 used to represent the identified vehicle is updated as previously described with reference to Figure 7. In particular, a new embedding vector 176 is added to array 178 if it is significantly different from all the vectors in the array. If array 178 reaches its maximum size, another embedding vector, such as a vector that is very similar to another vector in the array, or a vector that is entirely located within the vector space defined by the other vectors in array 178, may be removed from the array.

[0059] Furthermore, in step 5, the digital twin of each vehicle is updated using the information passed from the inference server 184. For example, its last known location, speed, and position in the race order may be updated based on the updated identification of the vehicle at its new current location. In the system described, all location, estimated speed, and other relevant information are permanently stored along with a timestamp corresponding to the moment when the corresponding digital image is taken. In this way, the estimated location and speed of the vehicles can be displayed live in the digital model. Moreover, the race can be reconstructed using the digital model, for example, for race analysis, identification of drivers responsible for accidents, or anomalies 144 along the race track 100.

[0060] The above technique allows for the implementation of further functionality. For example, if both the anomaly 144 and the vehicle trajectory are detected and maintained by the system 180, it is possible to identify the vehicle 110 that caused the anomaly 144, such as a lost part or an oil spill on the racetrack, and hold the driver or owner of vehicle 110 responsible.

[0061] Furthermore, if a driver or other person is interested in video footage of a specific event, such as a crash, or all available footage of selected cars on Racetrack 100, such video sequences can be automatically selected and cut and provided to the relevant party.

[0062] Figure 10 shows a potential architecture for implementing the above functionality. However, as detailed above, each of the described functions can be implemented and used individually if desired.

[0063] The monitoring system 180 in Figure 10 includes two camera clusters 202a and 202b, each containing four cameras 182a to 182d. The cameras in each camera cluster 202 are connected to corresponding rack workstations 204a and 204b, which include one or more frame grabbing cards 206. In particular, rack workstation 204 is involved in grabbing essentially all video output from each camera cluster 202. In the described system, video footage is stored for three days for analysis and post-processing, as detailed below. In addition, rack workstation 204 performs further different tasks during race monitoring and at other times when races are not currently taking place.

[0064] During a race, the rack workstation 204 performs anomaly detection based on image segmentation, delta image calculation, and anomaly classification, as detailed above. In addition, it also performs vehicle tracking based on vehicle detection, vehicle embedding, image segmentation, and location estimation, as detailed above.

[0065] At other times, for example at night, computationally intensive tasks are performed, such as batch processing for video generation, cutting video footage for individual vehicles or drivers, and pixelating other vehicles or their registration numbers for data protection.

[0066] The digital twin processing unit 208 maps inferences and other information provided by the rack workstation 204 to corresponding digital twins of the racetrack 100 and / or individual vehicles 110. In particular, it tracks the speed estimate and current position of each vehicle 110. It further records detected incidents, especially the location, time, and corresponding images of any vehicle 110 crashes. The digital twin processing unit 208 further monitors the state of the racetrack 100 and the positions of the vehicles 110 on the track.

[0067] In the embodiments described, the digital twin processing unit 208 also runs a rule engine. As previously described, the rule engine may generate alerts in response to detected incidents such as vehicle crashes and control flags, displays, and other notifications along the track. Further vehicle information may be provided to the rule engine from the digital twin, which enables the formulation of rules based on the dynamic characteristics of the monitored system. For example, a rule may be triggered if the calculated speed of a vehicle falls below or exceeds a certain threshold.

[0068] The monitored track conditions and vehicle positions are provided from the digital twin processing unit 208 to the track and vehicle monitoring unit 210, for example, for the benefit of race officials or as information for spectators.

[0069] The system monitoring unit 212 is provided to control and monitor the operation of the monitoring system 180 itself. It may indicate the operating status of the software and hardware components of the monitoring system.

[0070] Finally, a video post-production unit 214 is provided, which is configured to collect all or selective footage from all cameras 182, construct targeted footage of a specific vehicle 110 or group of vehicles, for example, a vehicle leading a race or involved in a specific incident, and control video post-production. The video post-production unit 214 also provides a customer API or web interface, which allows registered users to request and retrieve video material related to their own vehicles 110.

[0071] Compared to previous solutions, the described monitoring system is more flexible because both the training data and the rules set for detection, classification, and triggering can be continuously updated. Moreover, it does not require the installation of specific hardware, such as RFID-based transponders used in professional racing, on or inside the racing vehicles. Therefore, the described monitoring system is particularly useful for amateur racing using privately owned vehicles rather than professional racing. [Prior art documents] [Patent Documents]

[0072] [Patent Document 1] International Publication No. 2017 / 212232A1 [Patent Document 2] U.S. Patent No. 6,020,851A [Patent Document 3] U.S. Patent Application Publication No. 2018 / 341812A1 [Explanation of symbols]

[0073] 100 Race Tracks 102 Running surface 104 Grassland Area 106 Gravel Area 108 Guardrail 110 vehicles 120 Monitoring Methods Steps 121-126 130 (Race Track) Sections 132 Boundary Box 134 Boundary Box 136 Boundary Box 140 (image) segments 142 Delta image 144 Abnormality 150 reference points 160 Neural Networks 162 Input side 164 nodes 166 Embedding Vectors 168 Output side 170 Multidimensional Feature Space 172 First embedding vector 174 Second embedding vector 176 New embedding vectors 178 vector arrays 180 Monitoring System 182 Cameras 184 Inference Servers 186 Digital Twin Server 188 detection units 190 inference units 202 Camera Cluster 204 Rack Workstations 206 Frame Grabbing Cards 208 Digital Twin Processing Unit 210 Truck and Automotive Monitoring Units 212 System Monitoring Unit 214 Video Post-Production Unit

Claims

1. A method for identifying individual vehicles on a race track, The processor of the image processing system captures and processes at least one image of a plurality of vehicles on the racetrack and processes the at least one image using a neural network to compute a corresponding at least one reference embedding vector for each of the plurality of vehicles, wherein the at least one image is captured and processed during the initial registration process when each vehicle enters the racetrack or the monitored portion of the racetrack. The processor acquires at least one sequence of images from a camera that captures at least one section of the race track, The processor detects at least one vehicle in the sequence of images using automatic object recognition, The processor sends a portion of the image in the sequence of images corresponding to the detected vehicle to the neural network in order to determine a new embedding vector, The processor compares the new embedding vector with a set of registered embedding vectors, which includes the at least one reference embedding vector calculated for each vehicle of a predetermined set of vehicles during the initial registration. The processor includes identifying the at least one vehicle as the vehicle corresponding to the closest embedding vector among the set of registered embedding vectors, The angle between the new embedding vector and all the embedding vectors in the set of registered embedding vectors is calculated, and the embedding vector with the smallest angle is selected as the closest embedding vector for identifying the at least one vehicle. The method further includes the processor mapping the identified vehicle to a corresponding digital twin in a digital representation of the racetrack. method.

2. If the new embedding vector differs from the nearest embedding vector by a value greater than a first preset threshold, the processor stores the new embedding vector in the array of vectors corresponding to the identified vehicle, and / or If the new embedding vector differs from each of the set of registered embedding vectors by a value greater than a second preset threshold, the processor further includes not including the new embedding vector in the array of vectors corresponding to the identified vehicle and / or failing to identify at least one vehicle. The method according to claim 1.

3. The method according to claim 1, wherein the neural network is pre-trained during a training phase using images of different vehicles to learn characteristic configurations of a vehicle or a particular subclass of a vehicle, and is configured to extract or infer previously unknown vehicle characteristics in real time during a race.

4. The aforementioned training stage is The processor provides the images of the different vehicles to the input side of the neural network, The processor simplifies the information received on the input side by the nodes of the neural network in order to form an embedding vector, The processor decodes the information of the embedding vector in order to regenerate the image on the output side of the neural network, The processor includes changing the weights and other settings of the neural network based on an automatic comparison of the input and output sides using a similarity metric until the image on the output side becomes highly similar to the image provided on the input side. The method according to claim 3.

5. The method according to claim 1, further comprising the processor extracting at least one characteristic configuration, specifically, the license plate or other registration number of the at least one vehicle, from the at least one image taken when the at least one vehicle enters the race track.

6. The method according to claim 1, wherein a predetermined set of vehicles corresponds to a subset of all vehicles having corresponding digital twins in the digital representation of the racetrack, and the subset is selected based on a set of rules that provide the possibility of identifying a given vehicle in the sequence of images corresponding to at least one section of the racetrack based on data from the corresponding digital twins.

7. The method according to claim 6, wherein the subset of vehicles used in the identification process is limited to vehicles that have already passed another camera located in an earlier section of the racetrack, specifically, vehicles that were last successfully identified in one of two or three sections of the racetrack covered by the other camera.

8. The method according to claim 1, wherein the entire race track is covered by video cameras and at least one vehicle is continuously tracked.

9. The processor selects multiple sequences of images from multiple cameras capturing different sections of the racetrack based on the identification of at least one specific vehicle in each of the multiple sequences, The processor further includes disconnecting the plurality of sequences to generate video of at least one specific vehicle driving along the race track, The method according to claim 1.

10. The processor determines a first real-world location of at least one identified vehicle based on a mapping relationship, wherein the mapping relationship determines that a plurality of pixel areas in an image of at least one sequence of images are mapped to a plurality of corresponding real-world locations of the corresponding sections of the racetrack captured by the camera. The processor further includes adding first location and timestamp information to the corresponding digital twin whenever a first real-world location of a vehicle to be re-identified is determined, in order to add the tracking history of each vehicle to the digital representation of the racetrack, The method according to claim 1.

11. The processor detects foreign objects along the racetrack based on a comparison of at least two images from at least one sequence of images, The processor classifies the detected foreign objects as acceptable or unacceptable based on recognition of the automatic pattern, The processor adds the second position and timestamp information to the corresponding digital representation of at least one foreign object detected along the racetrack and classified as unacceptable. The processor further includes correlating the first and second location and timestamp information to identify a vehicle that is likely to be generating at least one unacceptable foreign object, The method according to claim 10.

12. The detected foreign matter is classified as acceptable if it is classified as one or more raindrops, leaves, reflections, shadows, and / or light beams, and / or The detected abnormality is classified as unacceptable if it is classified as a vehicle part, oil, and / or gravel. The method according to claim 11.

13. A monitoring system for racetracks, Each camera has a field of view that covers at least one section of the race track, and one or more cameras are provided. An image capture system configured to acquire a sequence of images from at least one of the aforementioned cameras, An image processing system including at least one processor, The aforementioned processor, During the initial registration process, a neural network is used to process the at least one image of a plurality of vehicles on the race track and to calculate a corresponding reference embedding vector for each of the vehicles. Automatic object recognition is used to detect at least one vehicle in one of the sequences of images, To determine a new embedding vector, a portion of the image in the sequence of images corresponding to the detected vehicle is sent to the neural network. The new embedding vector is compared with a set of registered embedding vectors, which includes the at least one reference embedding vector calculated for each vehicle in the predetermined set of vehicles being initially registered. Identify the at least one vehicle as the vehicle corresponding to the closest embedding vector among the set of registered embedding vectors. It is configured in such a way, The angle between the new embedding vector and all the embedding vectors in the set of registered embedding vectors is calculated, and the embedding vector with the smallest angle is selected as the closest embedding vector for identifying the at least one vehicle. The processor is further configured to map the identified vehicle to a corresponding digital twin in the digital representation of the racetrack. Monitoring system.

Images