Sports timing based on camera systems

Abstract

A method and system for calibrating a timing system configured to determine a time of passage of an object passing a timing line across a sports track is described, the method comprising: receiving video frames captured by a calibrated camera system, where each video frame represents a picture of a scene including one or more objects moving along the track, each video frame being associated with a time instance; determining or receiving calibration data, where the calibration data defines a virtual timing zone located at a predetermined distance from the calibrated camera system, the virtual timing zone extending across and along the track and including a virtual timing line that crosses the track; determining or receiving calibration data, where the calibration data defines a virtual timing zone located at a predetermined distance from the calibrated camera system, the virtual timing zone extending across and along the track and including a virtual timing line that crosses the track; The method may include detecting an object in the video frames using a rhythm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least a portion of the video frames, where the depth information includes information regarding a distance between the detected object and the calibrated camera system; determining whether the detected object is located inside the virtual timing zone based on the calibration data and the depth information; and determining a passing time for the detected object in the virtual timing zone based on one or more time instances of one or more video frames including the at least one detected object passing through the timing line.

Description

[Technical field]

[0001] The present disclosure relates to camera system based sports timing, and in particular, but not exclusively, to methods and systems for camera system based sports timing, and to computer program products enabling a computer system to carry out such methods. [Background technology]

[0002] Sporting events, such as car or motor racing, cycling, athletics, drone matches and ice skating, typically require accurate and fast time registration to track objects (people or vehicles) during the event. Such timing systems are usually based on RFID systems, where each participant at the event is provided with an RFID transponder, for example a UHF back scattering tag or an LF tag based on magnetic induction, which can be read by RFID readers located along the track. Such readers can be implemented in the form of an antenna mat, a side antenna and / or an antenna mounted on a frame above the track. Each transponder is configured to transmit a packet at a certain frequency and to insert a unique identifier in the packet, so that a detector can associate the packet with a certain transmitter.

[0003] Disadvantages with timing systems based on RFID technology include the need to provide every participant with a UHF tag, the sensitivity of the UHF signal to environmental influences such as moisture and rain, detuning when the UHF tag is located close to the human body, reflection of the UHF signal by "hard" objects such as the road surface and walls, and collision of the UHF signal when multiple participants simultaneously pass RFID detectors, e.g. RFID antenna mats, provided along the track.

[0004] WO 2021 / 048446 describes an example of a vision-based timing system for measuring the passing times of participants in a large-scale sporting event, where a large number of participants may pass a virtual timing line at the same time or almost at the same time. The system comprises one or more cameras for capturing images of participants passing the virtual timing line. The system may further comprise a processor configured to analyze the images, i.e. to detect objects, such as participants, in the image at the sporting event, and to determine the passing times for the detected objects passing the virtual timing line. Furthermore, based on the captured images, the processor is configured to detect identification numbers and / or symbols (such as QR codes) printed on identification carriers, such as so-called bib numbers (BIBs) attached to the participants' clothing, and to associate the detected identification information with the passing times of the participants. The advantage of such a vision-based timing system is that it does not exhibit the disadvantages of the RFID-based timing systems described above.

[0005] Although vision-based timing systems make it possible to accurately determine the time of passage of participants across the virtual timing line, several challenges still exist to accurately time and identify in real time all participants who pass the virtual timing line. In particular, a camera positioned to capture the area around the timing line, e.g. the finish line, typically captures a scene that may significantly extend beyond the timing line, e.g. an area including the area around the timing line or especially the areas on either side of the timing line, e.g. an area including not only athletes but also other people not participating in the sporting event, e.g. spectators and officials.

[0006] This makes accurate, reliable and rapid determination and identification of participants crossing a timing line, e.g., a start line or a finish line, particularly challenging when dealing with large-scale sporting events (e.g., marathons, etc.) that require high detection accuracy, e.g., detection accuracy of more than 99.9%. In addition, various privacy regulations further increase the technical challenge that data, especially biometric data, can only be used under strict conditions. Summary of the Invention [Problem to be solved by the invention]

[0007] From the above it can therefore be seen that there exists a need in the art for improved vision-based timing of sporting events, particularly large-scale sporting events, that allows for highly accurate and reliable determination of passing times and identification of participants in sporting events. [Means for solving the problem]

[0008] As will be appreciated by those skilled in the art, aspects of the invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Moreover, aspects of the invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon, e.g., stored thereon.

[0009] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device.

[0010] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium, including but not limited to a computer-readable storage medium, and which is capable of communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0011] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages ​​(e.g., Java, Scala, C++, Python, etc.), and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partly on the user's computer as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer, server, or virtualized server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including any of the types of networks listed above, such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider).

[0012] Aspects of the present embodiments are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor, in particular a microprocessor or central processing unit (CPU) or graphics processing unit (GPU), of a general-purpose computer, special-purpose computer, or other programmable data processing device to generate a machine such that the instructions executed by the processor of the computer, other programmable data processing device, or other device create means for implementing the function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0013] These computer program instructions may also be stored in a computer-readable medium, the computer program instructions may direct the computer, other programmable data processing device, or other apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions that implement a function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0014] The computer program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause the computer, other programmable processing device, or other device to perform a series of operating steps, creating a computer-implemented process, such that the instructions executing on the computer or other programmable processing device provide a process for implementing a function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0015] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs the specified functions or acts, or a combination of special purpose hardware and computer instructions.

[0016] It is an object of embodiments of the present disclosure to reduce or eliminate at least one of the disadvantages known in the prior art. In one aspect, the embodiments may relate to a method for determining a passing time of an object passing a virtual timing line.

[0017] In one embodiment, the method includes receiving video frames captured by a calibrated camera system, where each video frame represents a picture of a scene including one or more objects moving along a track, each video frame associated with a time instance; determining or receiving calibration data, where the calibration data defines a virtual timing zone located a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and along the sides of the track and defining a virtual timing line or plane that crosses the track. detecting an object in the video frames using an object detection algorithm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least a portion of the video frames, where the depth information includes information about a distance between the detected object and a calibrated camera system; determining a detected object located within the virtual clocking zone based on calibration data and the depth information; and determining a pass time for a detected object in the virtual clocking zone that has passed the virtual clocking line based on one or more time instances of one or more video frames that include at least one detected object passing the clocking line.

[0018] In another embodiment, the method includes receiving video frames captured by a calibrated camera system, where each video frame represents a picture of a scene including one or more objects moving on a track and objects moving outside the track, each video frame being associated with a time instance, the calibrated camera system including calibration data defining a virtual timing zone within the video frames located at a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and extending along a side of the track; detecting an object within the video frames using an object detection algorithm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least a portion of the video frames, where the depth information is based on the calibration data. the calibration data including information regarding a position of the detected object relative to a camera system configured to measure the virtual time zone; determining whether the detected object is located inside the virtual time zone or outside the virtual time zone based on the calibration data and the depth information; applying a feature analysis algorithm to at least a portion of the detected object in the video frames determined to be inside the virtual time zone, where the feature analysis algorithm determines identifying features for the one or more detected objects inside the virtual time zone; and determining a pass time for the detected object determined to be inside the virtual time zone and pass a virtual time line crossing the track based on the calibration data, the depth information, and one or more time instances in the one or more video frames of the detected object passing the virtual time line.

[0019] Thus, the timing system uses depth information to detect objects in the image and determine whether the detected objects are inside or outside the virtual timing zone, i.e., a zone located on the track where the sides of the timing zone substantially coincide with the sides of the track. The computer may track the detected objects inside the virtual timing zone and execute an identification algorithm to identify these objects (athletes on the track) and / or determine the passing times for these objects, while the computer will not process objects that are outside the virtual timing zone. In this way, the computer can process and identify only athletes who will pass the virtual timing line within the virtual timing zone. Furthermore, the computer can ignore objects outside the virtual timing zone, such as spectators and / or officials. In this way, the privacy of the spectators is respected, while athletes participating in the sporting event (athletes who have given their consent to be timed by a vision-based timing system) are timed and identified without wasting computer resources. The virtual timing zone effectively enables spatial filtering based on depth information to filter out detected objects located outside the timing zone before determining a passage time for the detected object and / or before executing an identification algorithm to identify the timed object.

[0020] The timing system is calibrated such that the camera system not only contains information, e.g. coordinates, of a virtual timing line across a track located at a predetermined distance from the camera system, but also information, e.g. coordinates, of a zone, e.g. a 2D area or a 3D volume, located on the track and having sides extending along the track. Preferably, the sides of the timing and identification zone substantially coincide with the edges or boundaries of the track. The virtual timing line may be located within the virtual timing zone. In the case where the coordinates of the virtual timing zone define a 3D volume, the virtual timing and identification zone may define a volume of interest (VOI) located on the track and having sides extending along the track. Preferably, the sides of the timing and identification zone substantially coincide with the edges or boundaries of the track.

[0021] In one embodiment, the width of the virtual timing zone may be equal to or greater than the width of the track, and the length of the virtual timing zone is equal to or greater than the width of the track.

[0022] In one embodiment, the virtual timing zone may be substantially rectangular, where a longer side of the rectangular virtual timing zone is aligned with a side of the track.

[0023] In this way, the timing system can efficiently determine whether an object is an athlete or not because the virtual timing zone defines a portion on the track where only athletes are allowed to be. Objects within the virtual timing zone are monitored, identified, and timed as they pass a virtual timing line (or plane). As long as the virtual timing zone identifies an area where athletes (only) move and includes a virtual timing line, the virtual timing zone can have a variety of shapes depending on the situation.

[0024] In one embodiment, the method may further include applying a feature analysis algorithm to detected objects in video frames within the virtual timing zone, where the feature analysis algorithm determines identifying features for one or more detected objects in the video frames; and determining the identity of the detected object whose passage time has been determined based on the identifying features of the detected object that has passed the timing line.

[0025] In one embodiment, the identifying features of the detected object may include one or more optically readable identifying markers, such as a race bib or printed markings, and / or one or more characteristics related to the shape and / or color of the detected object, and / or if the detected object is an animal or human, one or more biometric identifiers of the detected object.

[0026] In one embodiment, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, e.g., a deep learning algorithm, e.g., a convolutional deep neural network system, trained to detect one or more objects in a video frame and determine identifying features associated with the detected objects.

[0027] In one embodiment, objects can be detected in the image using an instance object segmentation algorithm based on deep neural networks. In one embodiment, the instance object segmentation algorithm can be based on R-CNN, fast R-CNN, faster R-CNN, mask R-CCN ​​algorithms, or related algorithms. Such algorithms can detect and classify objects in images in (near) real time.

[0028] In one embodiment, identifying the object may be based on an optical character recognition (OCR) algorithm, or an optical code recognition algorithm may be used to recognize a (printed) identification marker within the ROI. In one embodiment, the identification marker may include printed text. In one embodiment, the printed text may be only visible using an infrared camera.

[0029] In a further embodiment, identifying the object may be based on an algorithm configured to identify the object based on biometric features, for example a facial recognition algorithm configured to identify a person based on facial features.

[0030] In one embodiment, the calibration data may include coordinates defining the virtual timing zone, preferably the coordinates defining a 2D area on the track, where the sides of the 2D area are substantially parallel to the sides of the track; or the coordinates defining a 3D area on the track, where the sides of the 3D area are substantially parallel to the sides of the track.

[0031] In one embodiment, determining a detected object located within the virtual clocking zone includes: determining a position of the object using the depth information; and determining whether the object is located within the virtual clocking zone or outside the virtual clocking zone based on the position and the coordinates of the virtual clocking zone.

[0032] In one embodiment, detecting one or more objects within the video frame may include determining one or more regions of interest (ROI) within the video frame, where each ROI includes pixels representing an object; determining identifying features in one of the one or more ROIs; and determining the object within the ROI based on the determined identifying features.

[0033] In one embodiment, the camera system may comprise a plurality of camera modules, preferably two camera modules forming a stereo camera, configured to generate at least a first video frame and a second video frame of the scene at each time instance, and a depth map is determined based on a disparity mapping algorithm configured to determine a disparity between pixels of the first video frame and pixels of the second video frame.

[0034] In one embodiment, the passage time may be determined based on at least one video frame of a scene in which a predetermined portion of the detected object passes through the virtual surface.

[0035] In a further aspect, the embodiment provides a method of calibrating a timing system configured to determine a time of passage of an object passing a timing line across a sports track, comprising: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of the timing system, where each video frame represents a picture of a scene including the track and one or more calibration markers, preferably at least two calibration markers defining positions on opposite sides of the track; determining depth information, e.g. one or more depth maps, based on the video frames, where the depth information is based on the pictures of the video frames. determining a relative position between the one or more calibration markers and the camera system using the depth information; determining calibration data, where the calibration data includes position information of a virtual timing line oriented to cross the track at a predetermined position relative to the positions of the one or more calibration markers, and the position information is associated with a virtual timing zone on the track at a predetermined position relative to the positions of the one or more calibration markers, the timing zone extending across and along the track; and storing the coordinates of the virtual surface in a memory of the timing system.

[0036] In one embodiment, the calibration data may include coordinates defining the virtual timing zone, preferably a 2D area on the track, e.g., a rectangular area, where the sides of the 2D area are substantially parallel to and / or substantially coincide with the sides of the track; or the coordinates define a 3D volume on the track, e.g., a rectangular volume, where the sides of the 3D volume are substantially parallel to and / or substantially coincide with the sides of the track. The sides of the virtual timing zone may extend a substantial length along the sides of the track so that the system can distinguish athletes from spectators.

[0037] In another aspect, the embodiment provides a system for determining a time of passage of an object passing a timing line across a sports track, comprising at least one camera system coupled to a computer comprising a computer readable storage medium having computer readable program code embodied therein, and a processor, preferably a microprocessor, coupled to the computer readable storage medium, wherein in response to executing the computer readable program code, the processor receives video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving along a track, each video frame being associated with a time instance; determining or receiving calibration data, where the calibration data defines a virtual timing zone located a predetermined distance from the calibrated camera system, the virtual timing zone including a virtual timing line extending across and along and across the track; detecting an object in the video frames using an object detection algorithm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least a portion of the video frames, where the depth information includes information regarding a distance between a detected object and the calibrated camera system; determining a detected object located within the virtual timing zone based on the calibration data and the depth information; and determining a pass time for a detected object passing through the virtual timing zone within the virtual timing zone based on the calibration data, the depth information, and one or more time instances of one or more video frames of the detected object.

[0038] In further embodiments, the executable instructions may further include any of the method steps described above.

[0039] In yet another aspect, the embodiment provides a calibration module for a timing system configured to determine a time of passage of an object passing a timing line across a sports track, comprising: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of the timing system, where each video frame represents a picture of a scene including the track and one or more calibration markers, preferably at least two calibration markers defining positions on opposite sides of the track; determining depth information, e.g. one or more depth maps, based on the video frames, where the depth information is based on the pixel positions of the video frames; the depth information including information about a distance between one or more objects within a structure; using the depth information to determine a relative position between the one or more calibration markers and the camera system; determining calibration data, where the calibration data includes position information of a virtual timing line oriented to cross the track at a predetermined position relative to the positions of the one or more calibration markers, and the position information is associated with a virtual timing zone on the track at a predetermined position relative to the positions of the one or more calibration markers, the timing zone extending across and along the track; and storing the coordinates of the virtual surface in a memory of the timing system.

[0040] The embodiments may also relate to a computer program or computer program suite comprising at least one software code portion, or a computer program product storing at least one software code portion, which is configured to perform any of the method steps described above when executed on a computer system.

[0041] An embodiment may also further relate to a non-transitory computer-readable storage medium storing at least one software code portion, where the software code portion is configured to perform any of the method steps described above when executed or processed by a computer.

[0042] The invention will be further described with reference to the attached drawings, which show, in a schematic manner, embodiments according to the invention, it being understood that the invention is in no way restricted to those particular embodiments. [Brief description of the drawings]

[0043] [Figure 1] FIG. 1 illustrates a schematic diagram of a vision-based sports timing system. [Diagram 2] FIG. 2 illustrates a 3D camera system configured to generate depth information. [Diagram 3] FIG. 3 illustrates the principle of the disparity mapping technique for determining depth information. [Figure 4] FIG. 4 illustrates the detection, classification and identification of athletes based on images captured by a camera. [Diagram 5] FIG. 5 illustrates a vision-based timing system for determining the crossing time of an athlete crossing a virtual timing line. [Figure 6] FIG. 6 illustrates a vision-based timing system for determining an athlete's crossing time across a virtual timing line, according to an embodiment. [Figure 7A] FIG. 7A illustrates the calibration and use of a calibrated vision-based timing system according to an embodiment. [Figure 7B] FIG. 7B illustrates the calibration and use of a calibrated vision-based timing system according to an embodiment. [Figure 8] FIG. 8 illustrates a flow diagram of a method for calibrating a timing system according to an embodiment. [Figure 9A] FIG. 9A illustrates a method for determining a transit time according to an embodiment. [Figure 9B] FIG. 9B illustrates a method for determining a passage time according to an embodiment. [Figure 10] FIG. 10 illustrates a method for identifying objects passing a virtual timing line, according to an embodiment. [Figure 11] FIG. 11 illustrates a flow diagram of a method for determining a passage time according to an embodiment. [Figure 12] FIG. 12 illustrates a system for determining transit time in accordance with an embodiment of the present invention. [Figure 13] FIG. 13 is a block diagram illustrating an exemplary data processing system that may be used to implement the methods and software products described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] FIG. 1 illustrates a computer 104 1,2 One or more camera systems 102 controlled by 1,2 1 illustrates a vision-based sports timing system 100 comprising: a camera system 106 and a camera system 108; each camera system 106 may be configured to capture a scene of the sports track 106 and determine depth information associated with the captured scene. For example, in one embodiment, the depth information may comprise a so-called depth map of a video frame generated by an image sensor of the camera system. A depth map of a video frame, e.g., an RGB video frame, may be represented as a pixelated image including pixel values ​​representing a distance value for each pixel of a video frame. The distance values ​​may define a distance between the camera (the imaging plane of the camera) and an object in the video frame.

[0045] For example, a group of pixels in a video frame may be part of an object in a scene imaged by the camera system. The depth map may then indicate the relative distance between the camera (viewpoint) and the surface of the object in the scene. Thus, during capture of a sequence of time-stamped video frames of an object, such as an athlete or vehicle, moving along a sports track, the associated depth map may provide information about the distance between the moving object in the video frames and the (static) camera system as a function of time.

[0046] Camera systems capable of generating a depth-map are known. For example, in one embodiment, the camera may be implemented as a 3D camera system, e.g., a stereo camera, comprising two or more camera modules, where each camera module has its own lens system. An example of a top view of such a 3D imaging system is illustrated in FIG. 2. As shown in this figure, a stereo camera 202 may be positioned along the side of a track 204. The stereo camera has parallel optical axes 206 (as illustrated in FIG. 1) for observing a scene of a race track including objects, e.g., athletes. 1,2 At least two camera modules 204 1,2 The scene also includes calibration markers 210 1,2 In this way, two images from different viewpoints of the same scene are acquired. Such a stereo camera can be used to determine a depth map and / or a 3D picture. A 3D camera system may have two or more camera modules so that multiple images can be used to calculate a depth map. In this way, the accuracy of the depth map can be improved.

[0047] The 3D camera system (or a computer system controlling the camera system) may comprise a module for computing a depth map based on video frames captured by two (or more than two) camera modules. In one embodiment, the module may use a disparity mapping technique to compute a depth map based on images produced by two image sensors. Figure 3 illustrates the principle of disparity mapping diagrammatically. The 3D camera system comprises two camera modules separated by a distance referred to as baseline 308. The two camera modules capture, at each time instance, two video frames 302 of the same object from different viewpoints. 1,2 can be synchronized so that

[0048] To calculate a depth map based on the video frames, a matching algorithm may be implemented to match corresponding pixels in the left and right video frames. Thus, an object 300 imaged by two synchronized camera modules will appear at the same position 304. 1,2 , but separated by a baseline distance 308. In this case, the object will appear at a similar location in both images. The distance between the object in the left and right images is known as the disparity 306. The algorithm for constructing a disparity map based on the two images is known as a stereo matching algorithm. Various stereo matching algorithms exist, which need to be accurate and fast for real-time applications.

[0049] The camera system used in the embodiments of the present application is not limited to stereo-based imaging technology, and other imaging technologies may be used as well. For example, the depth map may be generated based on RGB / IR technology (as used by Kinect), or 3D time-of-flight (TOF) technology, LIDAR, or a combination thereof, where infrared radiation, such as an IR camera or IR laser, may be used to determine depth information of the scene imaged by the camera. In some embodiments, the camera system may be a color-infrared camera configured to capture images in the visible spectrum (e.g., RGB) and infrared (IR) spectrum to capture images of objects. The use of infrared images may be advantageous when light conditions are not optimal, for example, during weather conditions (e.g., fog, rain, etc.) or low light.

[0050] In order to increase the viewing angle of the camera system, in some embodiments, one or more wide-angle camera modules, such as 180-degree cameras or 360-degree cameras, may be used. Also, in the case of such types of video formats, such as 360-video or immersive video generated using a special 360 camera system, the video may be projected onto a 2D video frame using, for example, equirectangular projection, and a depth map may be generated. To enable accurate image processing, the camera used in the timing system may have a frame rate of at least 30 frames / second or more. Depending on the application, the frame rate may be at least 60 frames / second or more. In addition, the camera may be a high-resolution camera with an image resolution of 4K or more.

[0051] As shown in FIG. 1, the camera system includes one or more camera modules 102 disposed on one or more sides of and / or above the sports track.1,2 The camera modules may be aligned and calibrated so that each camera module captures the same scene 114 of the sports track, including objects moving along the track. To that end, one or more calibration markers 110 may be provided. 1,2 may be used. In one embodiment, such calibration markers may be located above the track and / or along one or both sides of the track. The one or more calibration markers may be used by the timing system to determine the coordinates of a virtual timing reference 112. Calibration markers may be physical objects placed along the track, as shown in FIG. 1. Alternatively, in one embodiment, calibration markers may be geo-coordinates that define the coordinates of a virtual timing reference.

[0052] Depending on the implementation, the virtual timing reference may define a virtual timing line or a virtual timing plane located at a distance z from the camera system. The virtual timing line may be used to determine a passage based on two coordinates, e.g., an object passing the time line at a position x, or based on three coordinates, e.g., a portion of an object passing the time line at a position x and a height y. The virtual timing plane may be used, for example, to determine a passage time when a particular portion of an object passes the virtual timing plane. For example, in one use case, it is desired to determine a passage time when a particular portion of an object, e.g., an athlete's chest constituting a bib (BIB), passes the virtual timing plane.

[0053] In the following, whenever a virtual timing line is mentioned, other kinds of virtual timing references, such as virtual timing planes, are also included. The timing line may be located across the track at (or relative to) one or more calibration markers. For example, when two calibration markers are used, the virtual timing line may be located between the two calibration markers such that the virtual timing line is located across the track between (or relative to) the two calibration markers.

[0054] The calibration process requires a 3D camera system to accurately detect the position and orientation of the calibration marker under all outdoor conditions. Therefore, the calibration marker can be designed to have a predefined well-defined shape and / or color combination so that an object detection program can easily and accurately determine the position of the marker (edge) in the video frame during calibration and the coordinates of the virtual surface can be accurately determined. When the (calibrated) timing system is in operation, the 3D camera system can capture a video frame that includes the athlete passing through the virtual surface. Although the figure shows a camera system along the side of a racing track, in other embodiments, one or more camera systems may be mounted above the sports track using a suitable mounting structure and / or integrated into a mat placed across the sports track.

[0055] As described in more detail herein below, the timing system illustrated in FIG. 1 is configured to determine a time of passage of an object passing through a virtual surface. Furthermore, the timing system is configured to identify the object whose time of passage has been determined. Identification may be based on an identification marker associated with the object. For example, in the case of an athlete, the identification marker may include (but is not limited to) a race bib (BIB), color, biometric information, etc. Identification markers of a vehicle may include characteristics of the vehicle, such as color, shape, branding, etc.

[0056] The computer for controlling one or more 3D camera systems and for carrying out the calibration and timing methods may be implemented as a stand-alone computer or a set of (wirelessly) connected computers, for example a 3D camera system used to determine the passage time based on a virtual surface placed across the track may be controlled by a computer equipped with a wireless interface for wireless communication with computers controlling other 3D camera systems.

[0057] The timing systems illustrated in Fig. 1 may be placed at different positions along the track. In this way, each timing system may determine a passing time based on a video frame of an object passing through the virtual surface. The timing systems may support a wireless protocol that allows setting up a mesh network of timing systems along the track. In this way, information determined by one timing system may be used by one or more other timing systems that are part of the mesh network.

[0058] Because the data processing methods used by the timing system to calculate the depth map and analyze the video frames may require real-time image processing, in some embodiments, the computer system 104 1,2 may include one or more special purpose processors, such as GPUs, to perform the computationally intensive parts of the calibration and timing processes. In other embodiments, one or more 3D camera systems may be connected to cloud resources that may perform the computationally intensive parts of these processes. A CPU clock or a GPS clock may be used to link the video frames with time information. For example, in one embodiment, each or at least a portion of the video frames may be linked to a time instance by stamping the video frames with a time stamp.

[0059] The timing system in Fig. 1 may be operated based on one 3D camera. Alternatively, two or more 3D camera systems with different viewing angles may be used to capture video frames of a scene. In that case, the 3D camera systems may be time-synchronized so that time-stamped video frames of different 3D camera systems can be easily combined and analyzed. Multiple 3D camera systems may be used to handle collisions, i.e. events where multiple objects pass through the virtual surface.

[0060] The computer system of the timing system may include one or more image processing units configured to execute various image processing algorithms. For example, in one embodiment, the computer system may be configured to execute an object detection algorithm to detect objects in an image captured by the camera system. In particular, an instance object segmentation algorithm may be used to detect and separate objects in the image from other objects in the image. As shown in FIG. 4, instance segmentation allows for the separation of multiple objects 400 of the same class. 1~6(e.g., an athlete or a vehicle) and each detected object may be treated as a separate instance. The instance segmentation algorithm may determine a bounding box that defines a region of interest ROI 401 in the image that contains the detected objects (for clarity, only the ROI of the first object is shown in the figure). The ROI containing the detected object is labeled with an object identifier ID, e.g., a unique number assigned to the object by the algorithm, which is used by the system to distinguish between different detected objects of the same class in one image.

[0061] In this way, objects in an image, such as athletes, cars, and bikes, can be accurately and efficiently detected and classified, and represented as labeled ROIs in the image. In one embodiment, information about the ROI and ID may be added or linked to the image as metadata. Efficient instance object segmentation algorithms based on deep learning may be used, including but not limited to algorithms that can detect and classify objects in an image in (near) real time, such as R-CNN, fast R-CNN, faster R-CNN, or related algorithms. An overview of state-of-the-art instance segmentation algorithms is provided in A. Hafiz et al, A survey on Instance Segmentation: State of The Art, International Journal of Multimedia Information Retrieval volume 9, pages 171-189 (2020), which is incorporated herein by reference. These algorithms provide very efficient runtime performance, such that they can be run locally using the computer of the timekeeping system.

[0062] To speed up the object detection and identification process, the timing system's local CPU-based computer, which executes the image processing algorithms, may comprise one or more special purpose processors, such as one or more graphical processing units (GPUs), tensor processing units (TPUs) or field programmable gate arrays (FPGAs), which may be specifically adapted to speed up the calculations performed by the various image processing algorithms, in particular those associated with neural networks.

[0063] Furthermore, in one embodiment, the computer system may be configured to execute an object identification algorithm. Here, object identification refers to the process of assigning an identity, typically personal data (e.g., the name of a person, animal, vehicle, or team, stored in a database before the event), to detected objects based on their object features. An example of an object feature may be a number or code (e.g., a QR code) on a participant's bib (BIB). When a participant registers for a sports event, the participant receives an identification information, such as an identification number, e.g., a bib (BIB) or code, which may be used by the timing system to link data, such as information about the passing time, to personal data stored in a central database, such as the name of the registered participant. Various object features may be used to link an object to an identity. For example, if the object represents a person, the object features may include (but are not limited to) (part of) a race bib (BIB), a color, biometric information of the person, a visual identification code (e.g. QR code), an object item (e.g. shoes, shirt, brand mark), etc. If the object represents a vehicle, the object features may include features of the vehicle, such as number, color, shape, brand mark, license plate or registration plate, etc. The computer may comprise or be connected to a wireless module for wirelessly connecting the timing system to a further computer system (not shown), such as a remote computer system, such as a server system, configured for centrally processing the determined passage times of the detected and identified objects.

[0064] As shown in FIG. 4, a detected object in an image may be identified by one or more object features, such as an identification marker 404 that is specifically used to identify the object. 1~3(e.g., bib number (BIB), QR code, printed text, symbol, etc.), biometric information 408 1~6 (e.g. facial features, age, gender, length, hair color, etc.) or object attributes 406 that are unique for a particular object 1~6 (e.g., color and / or design clothes and / or shoes, or color and / or shape of a vehicle), i.e., associated with an identity. These object features may define visual identification information that can be used by the timing system to identify objects passing a virtual timing line.

[0065] The ROI may be analyzed by one or more algorithms configured to detect object features within the ROI and extract identifying information from the ROI. For example, an optical character recognition (OCR) algorithm may be used to recognize (printed) identifying markers within the ROI. Typically, an OCR algorithm may include a text detection algorithm and a text recognition algorithm. These printed identifying markers may be based on characters and / or symbols, such as illustrative numbers and / or letters, on the BIB or vehicle. Alternatively and / or additionally, a code detection algorithm may be used to detect and encode visual codes, typically standardized codes, within the ROI. For example, the athlete's BIB or clothing may include a QR code or a variation thereof in which identifying information is encoded as (for example) a geometric structure. Furthermore, face detection and recognition algorithms may be used to identify people within the ROI based on facial features.

[0066] In some embodiments, a pose estimation algorithm is used that is configured to identify the pose of the object. For example, in one embodiment, the pose estimation algorithm may determine the key joints of the object (shoulder / hip / knee / head), which may be used to identify possible locations within the ROI that may represent object identification features, i.e., features of the object that can be used for identification. For example, based on the locations of the key joints, a particular part of the body, such as chest, arm, head, leg, or foot, may be determined. Based on the locations of the key joints, locations within the ROI that have a high probability of containing one or more object features, such as face, chest, or foot, may be determined.

[0067] When using a vision-based timing system to time a mass event, such as a marathon, the timing area will typically be very crowded with both athletes and others not participating in the sporting event, such as spectators or officials. Figure 5 illustrates a computer-controlled camera system 504 that captures images of the track area, including a virtual timing line 506. 1,2 5 shows a schematic diagram of a calibrated timing system comprising: a camera system having a virtual timing line 510 and a timing camera 512; a timing camera 514 for detecting athletes approaching the timing line 510; a timing camera 516 for detecting athletes approaching the timing line 510; a timing camera 518 for detecting athletes approaching the timing line 510; a timing camera 519 for detecting athletes approaching the timing line 520; a timing camera 520 for detecting athletes approaching the timing line 520; a timing camera 520 for detecting athletes approaching the timing line 530; a timing camera 530 for detecting athletes approaching the timing line 540; a timing camera 540 for detecting athletes approaching the timing line 540; a timing camera 540 for detecting athletes approaching the timing line 550; a timing camera 550 for detecting athletes approaching the timing line 560; a timing camera 550 for detecting athletes approaching the timing line 570; a timing camera 560 for detecting athletes approaching the timing line 580; a timing camera 570 for detecting athletes approaching the timing line 590; a timing camera 580 for detecting athletes approaching the timing line 590; a timing camera 59 ... 1~5 Not only that, but also for example, spectators and related parties508 1~8In such a situation, a significant portion of the computer resources will be used to detect objects that do not participate in the timing process. In addition, the image processing system will apply object detection and identification algorithms to people who have not consented to being captured and processed by image processing algorithms. Depending on the country in which the timing system is used, privacy issues may arise.

[0068] To address this issue, spatial filtering techniques based on depth information are used to filter detected objects. This is shown diagrammatically in FIG. 6, which illustrates a computer-controlled camera system 604. 1,2 6 illustrates a calibrated timing system with a camera system 606, in which the system is calibrated to include not only the coordinates of a virtual timing line 610 across a track located at a predetermined distance from the camera system, but also the coordinates of a zone, e.g., a 2D area or a 3D volume, located on and extending along the track. Typically, the virtual timing line may be located within this zone, which may be referred to hereinafter as a virtual timing zone 606. If the coordinates of the virtual timing zone define a 3D volume, the virtual timing zone may define a volume of interest (VOI) located on the track. Thus, during calibration of the timing system, position information, e.g., coordinates or information for determining coordinates, of a virtual timing zone may be determined, the position information extending across the rack and along the track. The position information may be stored in a memory of the timing system, so that the timing system can use this information to determine the time of passage of an object (e.g., an athlete) passing the virtual timing line. Preferably, the virtual timing zone is located within the athlete 610. 1~5The virtual timing zone defines a zone on the track where only athlete runners are allowed to enter. Thus, if the timing system detects an object within the virtual timing zone, there is a high probability that the object is an athlete. Thus, using the virtual timing zone, the timing system can efficiently determine whether a detected object in an image is an athlete or not.

[0069] As will be described in more detail below, the calibrated timing system is configured to capture an image and depth information associated with the image. It is further configured to detect an object in the image and use the depth information to determine whether the detected object is within the virtual timing zone. The computer would then track the detected object within the virtual timing zone and execute an identification algorithm to identify the detected object (athletes on a track) within the virtual timing zone, while the computer would not process the detected object located outside the virtual timing zone. In this way, the computer would process (e.g., time and identify) only athletes who would pass the virtual timing line within the virtual timing zone. The computer would ignore objects, such as spectators and / or officials, that are outside the virtual timing zone. In this way, the privacy of spectators is respected, while athletes participating in a sporting event (athletes who have given consent to be timed by a vision-based timing system) are timed without wasting computer resources.

[0070] 7A and 7B illustrate the calibration of the timing system and the operation of the calibrated timing system in accordance with an embodiment of the present invention. As shown in FIG. 7A, a camera system 702 is positioned along the side of a truck 700. At a predetermined distance 706, one or more calibration markers 710 are positioned along the side of the truck 700. 1,27. A camera 700 may be positioned on one side of the truck or on both sides of the truck 700. The field of view (FOV) of the camera may be positioned towards the truck to capture a scene including one or more calibration markers. A calibration program is then executed by the computer 704 of the camera system. During calibration, the camera generates video frames, and based on these generated video frames, depth information, e.g., a depth map, of the scene including one or more calibration markers is determined (e.g., as described with reference to Figures 2 and 3). Furthermore, an object detection program may be used to detect the one or more calibration markers in a video frame (e.g., an RGB picture).

[0071] Any kind of marker may be used for calibration. For example, existing objects on the side of the track, such as trees, traffic signs or poles, may be used. In a further embodiment, geographic coordinates may be used as calibration markers. In such a case, the camera may be associated with geographic coordinates using GPS or the like, and a line or plane across the track that is visible in the camera's FOV may be defined by one or more geographic coordinates relative to the camera's geographic coordinates.

[0072] Alternatively and / or additionally, specially designed calibration markers may be used. For example, in one embodiment, the calibration markers may be designed to have features that allow accurate calibration under different outdoor conditions. For example, the shape, edge, and / or color of the markers may be designed to allow accurate detection in the picture. A depth map associated with the video frame may be used to determine the distance between the camera and the detected calibration marker. Alternatively, if a sufficiently accurate depth map can be constructed, the object detection program may also directly determine the position of the calibration marker based on the depth map.

[0073] Once the position of one or more markers is detected by the computer, the computer may determine a virtual timing line across the track 700. The virtual timing line may be a virtual line 709 or virtual surface 711 across the track located at a predetermined position relative to the position of the one or more markers. In one embodiment, the virtual timing line may be a line extending from a marker located on a first side of the track across the track to the other side of the track. In another embodiment, the virtual timing line may extend from a first marker on one side of the track to a second marker on the other side of the track. The virtual timing line or virtual timing surface may be used as a location from which the timing system determines the passing time. In one embodiment, the virtual line or virtual surface may be located within a rectangular 3D volume 712 in space, where the width of the volume may be determined by calibration markers, and the height and depth of the volume may be determined by the computer. The 3D volume may define a 3D detection zone within which the timing system acquires video data (e.g., multiple video frames) to precisely determine a passage time and to identify the object associated with the passage time.

[0074] In addition to the timing lines, the calibration process may also be used to determine virtual timing zones 706 that traverse the track and are positioned along the track at predetermined positions relative to one or more markers. In one embodiment, the virtual timing zones may define a 3D volume of interest VOI in which the virtual timing lines or planes are positioned. In another embodiment, the virtual timing zones may be 2D virtual timing areas in which the virtual timing lines or planes are positioned. In one embodiment, the one or more markers may determine the location and / or (part of) the dimensions of the virtual timing zone. For example, after positioning and detecting the markers, the calibration program may generate a virtual timing zone of a certain width and length, and optionally a height, based on the location of one or more markers. Because the markers are positioned along the edges of the track, the virtual timing zone may have sides that run substantially parallel to the edges of the track. In this manner, position information, e.g., coordinates, of a virtual timing zone located on the track can be determined during calibration using one or more markers and stored in the memory of the timing system. In this manner, the timing systems 702 and 704 calibrated as shown in FIG. 7B can detect detected objects 716 in an image that are outside the virtual timing zone. 1~3 , and objects 714 within the virtual time zone. 1,2 The virtual timing zone may be used to track, identify and time.

[0075] FIG. 8 illustrates a flow diagram of a method for calibrating a timing system according to an embodiment of the present invention. This method may be used to calibrate a timing system including calibration markers as described with reference to FIG. 7A. As shown in the flow diagram, the method may begin with aiming a 3D camera system of the timing system at one or more calibration markers located on opposite sides of a track. The calibration markers may indicate the position of a virtual surface located between the calibration markers and across the sports track (step 802). In a next step 804, a sequence of video frames may be captured by the 3D camera system, where each video frame may represent a picture of a scene including the track and the one or more calibration markers.

[0076] A depth map may then be determined based on the video frames, where the depth map may include information about the distance between the one or more objects in the picture and the 3D camera system (step 806). For example, a depth map may be generated based on two video frames generated by two camera modules in a stereo camera, and disparity mapping may be used to generate the depth map, similar to that described with reference to Figures 2 and 3 above. The depth information may include, for each video frame (or at least a substantial portion thereof), a depth map that includes information about the distance between an object in the captured video frame and the 3D camera system. The depth information may be used to determine a relative position (distance) between each of one or more calibration markers and the 3D camera (step 808).

[0077] Next, coordinates of a virtual timing line across the track may be calculated to be located relative to one or more calibration markers, for example relative to between opposing calibration markers of the track. The virtual timing line may be used by the timing system to determine the time of passage of an object passing through a virtual surface (step 810). Furthermore, based on the position of the calibration markers, position information, for example coordinates or information for calculating coordinates, of a virtual timing zone, for example a 2D area or a 3D volume, oriented on the sports track may be calculated (step 812). In one embodiment, the virtual timing line may be located within the virtual timing zone, which defines a zone within which objects are tracked, identified and timed when passing through the virtual timing line. The virtual timing line and the coordinates of the virtual timing zone may be stored in a memory of the timing system (step 814).

[0078] In operation, the calibrated timing system no longer requires the calibration markers. The calibrated camera system uses the calibration data defining the virtual timing line and the virtual timing zone to efficiently determine the passage time and one or more identifying features of an object passing through the virtual timing line.

[0079] 8 is contemplated to be just one of many different ways to calibrate the camera with respect to the line or plane used to determine passage time. For example, calibration of the camera system may be accomplished based on a geographically positioned camera system and one or more geographic coordinates that define a virtual line or plane that intersects the track at a predetermined distance from the camera.

[0080] A calibration process may be used to install and calibrate one or more further camera systems along the track, so that each of these camera systems can capture a video frame of an object passing through the timing line. The camera systems may communicate with each other (wirelessly) so that the video capture process can be time-synchronized. In this way, at a certain time instance, each of the camera systems will process one or more time-stamped video frames of the sports track. The time-stamped video frames (and associated depth maps) of different viewing angles may be used to determine the passage times of objects passing through the virtual surface, and to identify the objects whose passage times have been determined.

[0081] Figures 9A and 9B illustrate the operation of a timing system according to an embodiment of the present invention. Figure 9A shows three snapshots of a moving object (in this case an athlete) passing through a 3D detection zone of a timing system described with reference to an embodiment of the present application. The 3D detection zone can be set using the calibration method described above with reference to Figure 7A. As shown in Figure 9B, a 3D detection zone 912 may include a virtual surface 914 located across the track, where the normal of the virtual surface is substantially parallel to the direction of the sports track. The virtual surface divides the 3D detection zone into a first portion 9161 where an object moves towards and across the virtual surface, and a second portion 9162 where a moving object crosses the virtual surface and moves away from the virtual surface.

[0082] Thus, when an object moves along the track, the 3D camera system will capture an image (a pair of images in case of a stereo camera) of the scene including the 3D detection zone. For each image (video frame), the 3D imaging system may compute a depth map. Object detection and tracking algorithms may be used to detect and track a given object, e.g. a human object, or an object representing an object, in subsequent video frames.

[0083] Based on the depth map, the computer may determine that the detected object has entered a first portion of the 3D detection zone. In that case, the computer may begin saving video frames and associated depth maps in a buffer until the object exits the 3D detection zone via a second portion. In another embodiment, only a pair of video frames is saved and the depth map is determined at a later time. These video frames and depth maps may be used by the computer to determine a passage time and to identify an object associated with the passage time.

[0084] FIG. 9A shows three samples 902 of a time-stamped video frame sequence as the athlete moves through the 3D detection zone. 1~3 9B illustrates a video frame captured by the camera system and stored in the computer. As shown in FIG. 9B, the first video frame 9021 is captured at time instance T1, a time when most of the athlete's body is still in the first portion 9161 of the 3D detection zone. At time instance T2, the athlete moves further and a second video frame 9022 is captured as he passes through the virtual surface 914 of the 3D detection zone. Finally, at time instance T3, a third video frame 9023 is captured as the athlete moves into the second portion 9162 of the 3D detection zone.

[0085] A passage time module in the computer of the timing system may analyze the time-stamped video frame sequence to determine at which time instance the athlete passed the virtual surface. To do so, an object detection and classification algorithm may be applied to each video frame. The algorithm may, for example, detect regions of interest 908 in a video frame that belong to an object. 1~3(ROI). Further, for each of these ROIs, the algorithm may classify pixels as belonging to the athlete or not (background).

[0086] Furthermore, the depth map associated with each of the video frames may be used to determine distance values ​​that belong to pixels classified as belonging to the object. These distance values ​​may be compared to the distance between the camera and the virtual surface. In this way, if the 3D camera system captures an object crossing the virtual surface, the portion of pixels of the object that crossed the virtual surface can be determined for each video frame. This can be seen in the video frames of Figure 9A, where the grey areas define the pixels of the object that crossed the virtual surface.

[0087] For the video frame at time instance T1, only pixel 9041 representing a portion of the athlete's hand and pixel 9061 representing the shoe are associated with a distance value that is less than the distance between the virtual surface and the 3D camera system. Similarly, for the video frame at time instance T2, pixel 9042 representing a portion of the upper body and pixel 9062 representing a portion of the leg are associated with a distance value that is less than the distance between the virtual surface and the 3D camera system. Finally, for the video frame at time instance T3, all pixels 908 representing the athlete are associated with a distance value that is less than the distance between the virtual surface and the 3D camera system. Based on this analysis, the computer may determine that at T2, a substantial portion of the athlete's body crossed the virtual surface. For example, the computer may determine that the athlete crossed a plane if the portion of the object that crossed the virtual surface is greater than a certain threshold. Thus, timestamp T2 may then define a crossing time 910, in this example, 2:34. Different rules may be defined to determine if an object crossed the virtual surface.

[0088] FIG. 10 illustrates a method for timed object identification according to an embodiment of the present invention. Once the passage time for an object has been established, the object's identity may be established. To that end, stored video frames may be analyzed by an object identification module. Regions of interest (ROIs) within the video frames may be subjected to a feature analysis process that searches for predefined features within the ROI. For example, as shown in FIG. 10, the ROI 1003 of video frame T3 1002 from FIG. 9A may be subjected to a feature analysis algorithm that may be configured to identify predefined features of a predefined marker, e.g. a bib 10041 or a code, e.g. a QR code 10042. Any unique identifying feature may be used, e.g. shoe type 10043, biometric marker, colored marker or specially shaped marker, or a combination thereof. The analysis may be performed based on a set of ROIs from different video frames during the passage through the 3D detection zone. It may frequently happen that the video frames used to determine the passage time are not suitable for identification purposes. Thus, the object identification module may determine that a time-stamped video frame (e.g., video frame at T2 in the example of FIG. 9A) representing a picture of an object crossing the virtual surface is not suitable for reliable identification of the object. In that case, one or more other video frames (e.g., video frame at T3 in the example of FIG. 9A) may be suitable for reliable identification of the timed object. Any known object detection algorithm may be used to identify features within the ROI, such as conventional computer vision algorithms, machine learning, and deep learning techniques. For example, in one embodiment, one or more deep neural networks may be used to determine an ROI within a video frame that includes the object and to determine certain features associated with the object.

[0089] 11 illustrates a flow diagram for determining passage times using a calibrated vision-based timing system according to embodiments described herein. As shown in this figure, the method may begin with receiving a sequence of video frames captured by at least one camera system, where each video frame represents a picture of a scene of an object, e.g., a person, animal, or vehicle, moving along a track (step 1102). Each of these video frames may be associated with a real-time clock. For example, each video frame may be time-stamped using the clock of the timing system.

[0090] In a further step 1104, the method may include determining or receiving calibration data, e.g. predefined calibration data, which may define a virtual timing zone located on the track at a predefined distance from the calibrated camera system, where the virtual timing zone includes a virtual timing line. In one embodiment, the calibration data may include position information, e.g. coordinates, of the virtual timing zone, e.g. a 2D area located on and along the track, or a 3D volume located on and along the track. The calibration data is stored in a memory of the timing system and is retrieved when the timing process is performed.

[0091] Objects within the video frames may be detected using an object detection algorithm, and the objects detected by the detection algorithm define one or more detected objects (step 1106). Thus, within each video frame, multiple objects may be identified using known object detection algorithms, and the relative positions between the identified objects and the camera system may be determined. In one embodiment, the object detection step may include determining a region of interest ROI that contains the object, and for each ROI, subjecting the pixels within the ROI to a classification algorithm to classify whether the pixels represent part of the object or part of a background.

[0092] Further, depth information, e.g., one or more depth maps, may be determined for the video frame sequence, where the depth information may include information about the position of detected objects in the video frames relative to the camera system (step 1108). Typically, the 3D camera system may be a stereo camera to generate pairs of video frames that can be used to determine the depth maps. Alternatively, other camera systems capable of generating depth information associated with captured video frames may be used.

[0093] The calibration data and the depth information may then be used to determine if the detected object is located within the virtual timing zone (step 1110). Finally, a passage time for a detected object passing within the virtual timing zone may be determined based on the calibration data (particularly position information of the timing line), the depth information, and one or more time instances of one or more video frames of the detected object (step 1112).

[0094] The method provides an efficient way to determine passing times at crowded timing locations, such as the finish of a large sporting event, e.g. a marathon, using a vision-based timing system. The virtual timing zone is used to spatially filter out a substantial portion of detected objects that are not participants of the timed sporting event, such that computational resources are allocated only to the detection, timing and identification of athletes captured by the 3D camera of the timing system. The method further provides a vision-based timing scheme that is privacy-aware in the sense that it applies an identification algorithm only to those detected objects in the video frames that represent athletes.

[0095] The passage of an object on the virtual clock line may be determined by comparing the distance between the 3D camera system and the virtual clock line with the distance between the 3D camera system and a detected object in the virtual clock zone. A time of passage may then be determined based on time information, such as a time instance, such as a timestamp, associated with a video frame that includes the detected object passing the virtual clock line. For example, for that purpose, one or more video frames may be determined in which a part of the object that has passed the virtual plane has a certain dimension.

[0096] Thus, a rule may be used to determine whether the object has passed through the virtual surface. A time instance, e.g., a timestamp, associated with a video frame depicting the situation defines the passage time. A feature analysis algorithm may then be applied to one or more detected objects in the video frames, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames, and an identity of the detected object from which the passage time may be determined may be determined based on the identifying features of the detected object that has passed the timing line.

[0097] In one embodiment, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, trained to detect one or more objects in a video frame and determine discriminatory features associated with the detected objects.

[0098] Thus, various pictures from a video frame sequence may be used to identify the object that crossed the virtual surface at the passage time, and thus the video frame used to determine the passage time of an object may be different from one or more video frames used to determine the identity of the object.

[0099] FIG. 12 illustrates a system for determining transit time according to one embodiment of the present invention. As shown in the figure, the system includes one or more camera systems 1201. 1,2 The one or more camera systems 1201 1,2 1201 includes one or more camera systems 1,2 and the video frame 1203 generated by the camera system. 1,2Each of the camera systems is calibrated according to the calibration method described herein to capture a track scene including a moving object passing through an autocline represented by a virtual surface 1202. The computer system may include an event analyzer 1204 for performing object detection and feature analysis of objects in the video frames using a suitable algorithm. The event analyzer may further determine which detected objects are within a virtual timing zone location on the track. The calibration data stored in the system's memory may include position information associated with the virtual timing zone. Furthermore, the event analyzer may generate depth information, e.g., a depth map, associated with the video frames. A pass time module 1206 may determine a pass time for a detected object in the virtual timing zone based on the depth information and the position information associated with the virtual timing line. Furthermore, an object identification module 1208 may identify an object detected by the pass time module and associated with the pass time. The pass time and the ID of the passed object may be stored in the computer's memory. In addition, a picture of the timed passage of the object may be stored. This information may be communicated to the network server 1214 via the (wireless) interface 1212.

[0100] 13 is a block diagram illustrating an exemplary data processing system as described herein. The data processing system 1300 may include at least one processor 1302 coupled to memory elements 1304 through a system bus 1306. As such, the data processing system may store program code in the memory elements 1304. Furthermore, the processor 1302 may execute the program code accessed from the memory elements 1304 via the system bus 1306. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be appreciated that the data processing system 1300 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0101] The memory elements 1304 may include one or more physical memory devices, such as a local memory 1308 and one or more mass storage devices 1310. A local memory may refer to a random access memory or other non-persistent memory device that is typically used during the actual execution of the program code. A mass storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some of the program code to reduce the number of times the program code must be retrieved from the mass storage device 1310 during execution.

[0102] Input / output (I / O) devices depicted as input devices 1312 and output devices 1314 may optionally be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device, such as a mouse, etc. Examples of output devices may include, but are not limited to, for example, a monitor or display, speakers, etc. The input and / or output devices may be coupled to the data processing system either directly or through an intervening I / O controller. A network adapter 1316 may also be coupled to the data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving the data transmitted by a network to the systems, devices, and / or data, and a data transmitter for transmitting data to the systems, devices, and / or networks. A modem, a cable modem, and an Ethernet card are examples of different types of network adapters that may be used with the data processing system 1100.

[0103] 13, memory element 1304 may store application 1318. It should be appreciated that data processing system 1300 may further execute an operating system (not shown) that may facilitate execution of the application. Applications implemented in the form of executable program code may be executed by data processing system 1300, such as by processor 1302. In response to executing an application, the data processing system may be configured to perform one or more operations described in further detail herein.

[0104] In one aspect, for example, data processing system 1300 may represent a client data processing system, in which case application 1318 may represent a client application that, when executed, configures data processing system 1300 to perform various functions described herein with reference to a "client." Examples of clients may include, but are not limited to, personal computers, portable computers, mobile phones, and the like.

[0105] In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 1318, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function as referred to herein.

[0106] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to encompass the plural forms unless the context clearly indicates otherwise. It will be further understood that the words "comprising" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0107] The corresponding structures, materials, acts, and equivalents of any means or step-function-added elements in the following claims are intended to encompass any structures, materials, or acts for performing that function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been chosen and described to best explain the principles and practical application of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments, with various modifications suitable for the particular uses contemplated.

[0108] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to encompass the plural forms unless the context clearly indicates otherwise. It will be further understood that the words "comprising" and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0109] The corresponding structures, materials, acts, and equivalents of any means or step-function-added elements in the following claims are intended to encompass any structures, materials, or acts for performing that function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been chosen and described to best explain the principles and practical application of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments, with various modifications suitable for the particular uses contemplated.

Claims

1. 1. A method for determining a time of passage of an object through a virtual timing line, comprising the steps of: receiving video frames captured by a calibrated camera system, where each video frame represents a picture of a scene including one or more objects moving on a track and objects outside the track, where each video frame is associated with a time instance, where the calibrated camera system includes calibration data defining a virtual timing zone within the video frame located at a predetermined distance from the calibrated camera system, where the virtual timing zone extends across the track and extends along a side of the track; detecting an object in the video frame using an object detection algorithm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least some of the video frames, where the depth information includes information regarding a position of the detected object relative to the calibrated camera system; determining whether a detected object is located inside the virtual timing zone or outside the virtual timing zone based on the calibration data and the depth information; applying a feature analysis algorithm to at least a portion of the detected objects in the video frames determined to be within the virtual timing zone, wherein the feature analysis algorithm determines distinguishing features for the one or more detected objects that are within the virtual timing zone; and determining a passing time for the detected object determined to be located inside the virtual timing zone and passing a virtual timing line across the track based on the calibration data, the depth information, and one or more time instances in one or more video frames of the detected object passing the virtual timing line. The method comprising:

2. 2. The method of claim 1, wherein the width of the virtual timing zone is equal to or greater than the width of the track, and the length of the virtual timing zone is equal to or greater than the width of the track, and / or the virtual timing zone is substantially rectangular, where a longer side of the rectangular virtual timing zone is aligned with a side of the track.

3. determining the identity of the detected object based on the identifying features of the detected object, preferably based on the identifying features of a detected object that has passed the timing line; The method of claim 2 or 3, further comprising:

4. 4. The method according to claim 2 or 3, wherein the identification features of the detected object comprise one or more identification markers, preferably optically readable identification markers, such as a race bib or printed marks and / or codes, and / or one or more characteristics relating to the shape and / or colour of the detected object.

5. A method according to any one of claims 2 to 4, wherein the identifying features of a detected object comprise one or more biometric identifiers of the detected object.

6. 6. The method according to any one of claims 1 to 5, wherein the object detection algorithm, a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, is trained to detect one or more objects in one or more video frames.

7. 7. The method of any one of claims 2 to 6, wherein the feature analysis algorithm is part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, that is trained to determine discriminatory features associated with detected objects.

8. A method according to any preceding claim, wherein the calibration data comprises coordinates of a 2D area on the track, the 2D area defining a 2D virtual timing zone, wherein a side of the 2D virtual timing zone extends substantially parallel to the side of the track, preferably wherein the side of the 2D virtual timing zone substantially coincides with the side of the track.

9. A method according to any preceding claim, wherein the calibration data comprises coordinates of a 3D area on the track, the 3D area defining a 3D virtual timing zone, wherein a side of the 3D virtual timing zone extends substantially parallel to the side of the track, preferably wherein the side of the 3D virtual timing zone substantially coincides with the side of the track.

10. Determining whether a detected object is located within the virtual timing zone using the depth information to determine a position of the object relative to the camera system; determining whether an object is located within the virtual time zone or outside the virtual time zone based on the position and the coordinates of the virtual time zone; The method according to any one of claims 1 to 9, comprising:

11. Detecting one or more objects within the video frames, determining one or more regions of interest (ROIs) within the video frames, where each ROI includes pixels representing an object; determining object features in one of the one or more ROIs; and determining an object within the ROI based on the determined object features; The method according to any one of claims 1 to 10, comprising:

12. 12. The method according to claim 1, wherein the camera system comprises a plurality of camera modules, preferably two camera modules forming a stereo camera, the stereo camera being configured to generate at each time instance at least a first and a second video frame of the scene, and a depth map is determined based on a disparity mapping algorithm configured to determine a disparity between pixels of the first and second video frames.

13. The method according to any one of claims 1 to 12, wherein the passage time is determined based on at least one video frame of a scene in which a predetermined part of the detected object passes through the virtual surface.

14. 1. A method of calibrating a timing system configured to determine a time of passage of an object across a timing line across a sports track, comprising the steps of: receiving video frames, preferably a sequence of video frames, captured by a camera system of the timing system, where each video frame represents a picture of a scene including said track and one or more calibration markers, preferably at least two calibration markers located on opposite sides of said track; determining depth information, e.g., one or more depth maps, based on the video frames, where the depth information includes information regarding distances between one or more objects within the pictures of the video frames; using the depth information to determine a relative position between the one or more calibration markers and the camera system; determining calibration data, where the calibration data includes position information of a virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration markers, and the position information is associated with a virtual timing zone on the track at a predetermined position relative to the position of the one or more calibration markers, the timing zone extending across and along the track; and storing said coordinates of said virtual surface in a memory of said timing system; The method comprising:

15. 15. The method of claim 14, wherein the calibration data includes coordinates defining the virtual timing zone, preferably the coordinates defining a 2D area on the track, where the sides of the 2D area are substantially parallel to the sides of the track; or the coordinates defining a 3D area on the track, where the sides of the 3D area are substantially parallel to the sides of the track.

16. 1. A system for determining a time of passage of an object across a timing line across a sports track, comprising: at least one camera system connected to a computer; Equipped with The computer comprises a computer-readable storage medium having computer-readable program code embedded therein, and a processor, preferably a microprocessor, connected to the computer-readable storage medium; wherein in response to executing the computer readable program code, the processor: receiving video frames captured by a calibrated camera system, where each video frame represents a picture of a scene including one or more objects moving on a track and objects moving outside the track, each video frame being associated with a time instance, the calibrated camera system including calibration data defining a virtual timing zone within the video frame located at a predetermined distance from the calibrated camera system, the virtual timing zone extending across the track and extending along a side of the track; detecting an object in the video frame using an object detection algorithm, where the object detected by the detection algorithm defines a detected object; determining depth information for at least some of the video frames, where the depth information includes information regarding a position of the detected object relative to the calibrated camera system; determining whether a detected object is located inside the virtual timing zone or outside the virtual timing zone based on the calibration data and the depth information; applying a feature analysis algorithm to at least a portion of the detected objects in the video frames determined to be within the virtual timing zone, wherein the feature analysis algorithm determines distinguishing features for the one or more detected objects that are within the virtual timing zone; and determining a passing time for the detected object determined to be located inside the virtual timing zone and passing a virtual timing line across the track based on the calibration data, the depth information, and one or more time instances in one or more video frames of the detected object passing the virtual timing line. configured to execute executable instructions including: The system.

17. The system of claim 16, wherein the executable instructions further comprise any of the steps recited in any one of claims 2 to 9.

18. 1. A calibration module for a timing system configured to determine a time of passage of an object across a timing line across a sports track, comprising: receiving video frames, preferably a sequence of video frames, captured by a camera system of the timing system, where each video frame represents a picture of a scene including said track and one or more calibration markers, preferably at least two calibration markers located on opposite sides of said track; determining depth information, e.g., one or more depth maps, based on the video frames, where the depth information includes information regarding distances between one or more objects within the pictures of the video frames; using the depth information to determine a relative position between the one or more calibration markers and the camera system; determining calibration data, where the calibration data includes position information of a virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration markers, and the position information is associated with a virtual timing zone on the track at a predetermined position relative to the position of the one or more calibration markers, the timing zone extending across and along the track; and storing said coordinates of said virtual surface in a memory of said timing system; The calibration module.

19. A non-transitory computer readable storage medium storing at least one software code portion, the software code portion being configured to perform the steps of the method according to any one of claims 1 to 15 when executed or processed by a computer.

Images