Vision-Based Sports Timing and Identification System

Abstract

A method of timing and identifying objects participating in a sporting event is described, the method comprising: receiving first image information associated with one or more first images captured by a first camera system of a first timing system positioned at a first location along a sports track, the one or more first images including the objects participating in the sporting event passing a virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but whose identification cannot be made based on the one or more first images; ... receiving or acquiring second image information associated with one or more second images including objects participating in the sporting event, the one or more second images including visual information regarding one or more objects that can be identified based on the one or more second images; and identifying the first object, where the identifying includes determining a second object in the one or more second images that matches the first object using the first image information and the second image information; and, once the second object is determined, identifying the first object based on the visual information of the second object.

Description

[Technical field]

[0001] The present disclosure relates to vision-based sports timing, and in particular, but not exclusively, to methods and systems for vision-based sports timing and identification systems, and computer program products that enable a computer system to carry out such methods. [Background technology]

[0002] Sporting events, such as car or motor racing, cycling, athletics, drones and ice skating, typically require accurate and fast time registration to track objects (people or vehicles) during the event. Such timing systems are usually based on RFID systems, where each participant at the event is provided with an RFID transponder, such as a UHF backscatter tag or an LF tag based on magnetic induction, which can be read by RFID readers placed along the track. Such readers can be implemented in the form of antenna mats, side antennas and / or antennas mounted on a frame above the track. Each transponder is configured to transmit packets at a certain frequency and to insert a unique identifier into the packets so that a detector can associate the packets with a certain transmitter. Disadvantages with timing systems based on RFID technology include the need to provide every participant with a UHF tag, the sensitivity of the UHF signal to environmental influences such as moisture and rain, detuning when the UHF tag is placed close to the human body, reflection of the UHF signal by "hard" objects such as the road surface and walls, and collision of the UHF signal when multiple participants simultaneously pass RFID detectors, e.g. RFID antenna mats, provided along the track.

[0003] International Patent Publication WO2021 / 048446 describes an example of a vision-based timing system for measuring the passing times of participants in a large-scale sporting event, where a large number of participants may pass a virtual timing line at the same time or almost at the same time. The system comprises one or more cameras for capturing images of participants passing the virtual timing line. The system may further comprise a processor configured to analyze the images, i.e. to detect objects, such as participants, in the image at the sporting event, and to determine the passing times for the detected objects passing the virtual timing line. Furthermore, based on the captured images, the processor is configured to detect identification numbers and / or symbols (such as QR codes) printed on an identification carrier, for example a so-called BIB attached to the participant's clothing, and to associate the detected identification with the passing times of the participants. An advantage of such a vision-based timing system is that it does not exhibit the disadvantages of the RFID-based timing system described above.

[0004] Although vision-based timing systems allow for accurate determination of the passing time of participants passing the virtual timing line, there are still some challenges to accurately time and accurately identify all participants passing the virtual timing line. In particular, although a camera placed at the timing line can accurately capture images of participants passing the line, there is a non-zero probability that not all detected participants can be identified based on the captured images because their BIB numbers are visually blocked or at least partially blocked by the passing participants. This makes accurate and reliable determination and identification of participants passing a timing line, e.g., a start line or a finish line, particularly challenging when dealing with large sporting events (e.g., marathons, etc.), which require high detection accuracy, e.g., detection accuracy of more than 99.9%. In addition, various privacy regulations further increase the technical challenge in that data, especially biometric data, can only be used under strict conditions. Summary of the Invention [Problem to be solved by the invention]

[0005] From the above it can therefore be seen that there exists a need in the art for improved visual-based timing of sporting events, in particular large-scale sporting events, which allows for highly accurate and reliable determination of passing times and identification of participants in sporting events. [Means for solving the problem]

[0006] As will be appreciated by those skilled in the art, aspects of the invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Moreover, aspects of the invention may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon, e.g., stored thereon.

[0007] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device.

[0008] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium, including but not limited to a computer-readable storage medium, and which is capable of communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0009] Program code embodied on a computer readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including functional or object-oriented programming languages ​​(e.g., Java, Scala, C++, Python, etc.), and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer, server or virtualized server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, such as any of the above types of networks including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider).

[0010] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor, in particular a microprocessor or central processing unit (CPU) or graphics processing unit (GPU), of a general-purpose computer, special-purpose computer, or other programmable data processing device to generate a machine such that the instructions executed by the processor of the computer, other programmable data processing device, or other device create means for implementing the function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0011] These computer program instructions may also be stored in a computer-readable medium, the computer program instructions may direct the computer, other programmable data processing device, or other apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions that implement a function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0012] The computer program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause the computer, other programmable processing device, or other device to perform a series of operating steps, creating a computer-implemented process, such that the instructions executing on the computer or other programmable processing device provide a process for implementing a function / act specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0013] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of code, including one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs the specified functions or acts, or a combination of special purpose hardware and computer instructions.

[0014] It is an object of embodiments of the present disclosure to reduce or eliminate at least one of the drawbacks known in the prior art. In one aspect, the present invention may relate to a method of timing and identifying objects participating in a sporting event. In one embodiment, the method includes receiving first image information associated with one or more first images captured by a first camera system of a first timing system positioned at a first location along a sports track, where the one or more first images include objects participating in the sporting event passing a virtual timing line, the first image information including visual information regarding at least one first object whose passing time is determined based on the one or more first images but cannot be identified based on the one or more first images; receiving one or more second images associated with one or more second images captured by a second camera system positioned at a location different from the first location; The method may include receiving or acquiring associated second image information, where the one or more second images include objects participating in the sporting event, the second image information including visual information regarding one or more objects that can be identified based on the one or more second images; and identifying the first object, where the identifying includes determining a second object in the one or more second images that matches the first object using the first image information and the second image information; and when the second object is determined, identifying the first object based on the visual information of the second object.

[0015] The method thus provides an accurate vision-based timing and identification process for timing and identifying objects in sporting events, such as athletes and vehicles passing a virtual timing line. The method uses visual information from multiple timing systems or one timing system and one or more camera systems to time and identify mass events with an accuracy similar to that provided by state-of-the-art RFID systems. The virtual timing line may be easily installed at multiple locations along the track and can be easily connected to a central server system that registers the passing times of detected objects and re-identifies timed objects that could not be identified by the first timing system using visual information provided from one or more further cameras placed at different positions along the sports track.

[0016] In one embodiment, the first image information may include at least a part of one of the one or more first images including the first object, or at least one picture of a first region of interest (ROI) of one of the one or more first images, the first ROI including at least a part of the first object. Thus, pictures of objects in the one or more images for which a passing time has been determined but which could not be identified by the first timing system based on the one or more first images are sent to a server equipped with an application configured to identify the not-identified object based on further images of the object. In this way, a visual timing system with a very high success rate of timing and identifying objects passing the virtual timing line can be realized.

[0017] In one embodiment, the first image information may further include timing information indicating a time instance at which the first object passes the virtual timing line, depth information indicating a distance between the first camera system and the first object, and / or an identifier associated with the first ROI. Thus, metadata associated with unidentified objects may be sent to a server for use by a server application.

[0018] In one embodiment, determining the second object in the one or more second images may be based on a re-identification algorithm, where the re-identification algorithm is configured to compare the first object with the object in the one or more second images based on object characteristics.

[0019] In one embodiment, the object characteristic is a non-biometric object characteristic, for example the color and / or design of clothing or shoes.

[0020] In one embodiment, determining a second object in the one or more second images may include determining one or more first object characteristics associated with the first object; determining one or more second object characteristics associated with an object in the second image; and determining whether one of the objects in the one or more second images matches the first object in the one or more first images based on the one or more first object characteristics and the one or more second object characteristics.

[0021] In one embodiment, the matching may be based on a distance metric that is calculated based on the first object characteristics and the second object characteristics, where the distance metric indicates the similarity between the first object and the object in the second image. Thus, objects in different images captured by different camera systems are compared and matched based on object characteristics.

[0022] In one embodiment, the objects in the one or more first images and the one or more second images represent people participating in the sporting event, and wherein the first object characteristics and the second object characteristics define characteristics of the people participating in the event, including non-biometric characteristics (e.g., clothing color, long or short sleeves or pants, shoe color) and biometric information (e.g., height, skin color, gender, long or short hair); or wherein the objects in the one or more first images and the second images represent vehicles participating in the sporting event, and wherein the first object characteristics and the second object characteristics define characteristics of the vehicles participating in the event, such as the color of the vehicle, the shape of the vehicle, numbers and / or letters on the vehicle.

[0023] In one embodiment, identifying the first object may further include: searching for a visual identification marker or code based on the visual information of the second object; and if a visual identification marker or code is found, converting the visual identification marker or code into identification information for linking the second object to an identity, such as a name; and associating the first object with the identification information; and storing the identification information and the timing information of the first object in a database. Thus, in this embodiment, the server application may include an identification algorithm, such as a fast OCR algorithm, for receiving visual information related to an identification marker or code, such as a BIB or QR code, and the identification algorithm converts the visual information into identification information, such as a BIB number or vehicle number.

[0024] In one embodiment, the second image information may include identification information associated with the second image; and identifying the first object further includes associating the first object with the identification information; and storing the identification information and the timing information of the first object in a database. Thus, in this embodiment, the object in the second image is identified by an image processing module of the second camera system such that identification information associated with the object in the one or more second images can be transmitted to the server system. In this way, the server system does not need to perform the identification process itself, but can rely on information already determined by the second camera system.

[0025] In one embodiment, the method may further include receiving timing and identification information associated with objects in the one or more first images that have been detected, timed and identified by the first timing system based on the one or more first images, In this embodiment, information regarding the timed and identified objects in the first images is transmitted to the server.

[0026] In one embodiment, the second camera system includes a computer or processor configured to determine visual information regarding one or more objects that can be identified based on the one or more second images.

[0027] In one embodiment, the second camera system may be part of a second timing system configured to determine the passage time of an object participating in the sporting event passing a virtual timing line.

[0028] In one embodiment, the first image information may include an image frame including the detected unidentified object, hi another embodiment, the first image information may include a ROI picture including the detected unidentified object cropped from an image frame produced by the first camera system.

[0029] In this way, only the relevant information is transmitted, thereby significantly reducing the bandwidth required to transmit the image information to the server system. In a further embodiment, the image information includes a timestamp indicating the time of passage of the detected unidentified object.

[0030] In a further aspect, the invention provides a method of timing and identifying objects participating in a sporting event, the method comprising receiving, by a server system, first image information associated with one or more first images taken by a first camera system of a first timing system arranged at a first location along a sports track, the one or more first images including objects participating in the sporting event associated with a visual identification marker or visual identification code passing a virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but which cannot be identified based on the visual identification marker or visual identification code in the one or more first images; receiving, by the server system, second image information associated with one or more second images taken by a second camera system arranged at a location different from the first location. receiving or obtaining, by the server system, one or more second images including objects participating in the sporting event, the second image information including visual information regarding one or more objects that can be identified based on visual identification markers or visual identification codes in the one or more second images; and identifying, by the server system, the first object, where the identifying includes determining, using the first image information and the second image information, a second object in the one or more second images that matches the first object, the determining being based on a first non-biometric object characteristic associated with the first object and a second non-biometric object characteristic associated with the second object; and, once the second object has been determined, identifying the first object based on the visual identification markers or visual identification code of the second object.

[0031] In a further aspect, the present invention may relate to a vision-based sports timing system for timing and identifying objects participating in a sporting event, comprising: a camera system configured to capture images of a scene including an object on a sports track passing a virtual timeline at a first location along the race track; and a computer connected to the camera system, wherein the computer is configured to: detect objects in the images captured by the camera system; determine depth information associated with the images, the depth information defining a relative distance between the camera system and the detected object; determine a passage time at which the detected object passes the virtual timing line based on the timing information and the depth information; and, if an object cannot be detected based on the images, generate image information, the image information including visual information of the object that cannot be identified; and transmit the image information and the passage time associated with the unidentified objects to a server system for further processing.

[0032] In one embodiment, the server system may comprise a server application configured to identify the unidentified object based on the image information and based on further image information associated with one or more further images of objects participating in the sporting event captured by a further camera system positioned at a location different from the first location.

[0033] In one embodiment, objects can be detected in the image using an instance object segmentation algorithm based on deep neural networks. In one embodiment, the instance object segmentation algorithm can be based on R-CNN, fast R-CNN, faster R-CNN, mask R-CCN ​​algorithms, or related algorithms. Such algorithms can detect and classify objects in images in (near) real time.

[0034] In one embodiment, identifying the object may be based on an optical character recognition (OCR) algorithm, or an optical code recognition algorithm may be used to recognize a (printed) identification marker within the ROI. In one embodiment, the identification marker may include printed text. In one embodiment, the printed text may be only visible using an infrared camera.

[0035] In a further embodiment, identifying the object may be based on an algorithm configured to identify the object based on biometric characteristics, for example a facial recognition algorithm configured to identify a person based on facial characteristics.

[0036] In one embodiment, the camera system may be configured as a 3D camera or a stereo camera with two or more camera modules. In one embodiment, the camera system may be configured to capture images in the visible spectrum and images in the infrared spectrum.

[0037] In a further aspect, a system for timing and identifying objects participating in a sporting event comprising a first camera system connected to or comprising a first computer, wherein the first timing system is configured to generate first image information associated with one or more first images taken by the first camera system positioned at a first location along the sports track, the one or more first images comprising an object passing the virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but which cannot be identified based on the one or more first images; and one or more second camera systems, each connected to or comprising a second computer, wherein the one or more second camera systems are configured to generate first image information associated with one or more first images taken by the first camera system positioned at a first location along the sports track, the one or more first images comprising an object passing the virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but which cannot be identified based on the one or more first images. the camera system is configured to generate second image information associated with one or more second images captured by the second camera system disposed at a location different from the first location, the one or more second images comprising objects participating in the sporting event, the second image information including visual information regarding one or more objects that can be identified based on the one or more second images; and the system comprises a server system configured to receive the first image information and the second image information and to identify the first object, where the identifying includes determining a second object in the one or more second images that matches the first object using the first image information and the second image information; and, when the second object is determined, identifying the first object based on the visual information of the second object.

[0038] In one embodiment, the first timing system and the one or more second camera systems are configured to communicate wirelessly with the server system, wherein preferably the first timing system and the one or more second camera systems form a communication network.

[0039] In one embodiment, the one or more second camera systems may form a wireless network, preferably a mesh network, a star network, or a cellular network. In another embodiment, the communication between the timing system and the server system may be based on a wireless communication standard, such as Zigbee, LoRa, LoRaWAN, 5G.

[0040] In one embodiment, each of the one or more second camera systems may be part of a vision-based timing system, preferably a vision-based timing system as described with reference to the embodiments herein.

[0041] In one embodiment, the system may comprise a plurality of timing systems positioned along a track, where the timing systems are wirelessly connected to the server system.

[0042] The invention also relates to a computer program or a suite of computer programs comprising at least one software code portion, or a computer program product storing at least one software code portion, which, when executed on a computer system, is configured to perform the steps of any of the methods described above.

[0043] The invention may further relate to a non-transitory computer-readable storage medium storing at least one software code portion, which, when executed or processed by a computer, is configured to perform the steps of any of the methods described above.

[0044] The invention will be further explained with reference to the attached drawings, which show, in a schematic manner, embodiments according to the invention, it being understood that the invention is in no way limited to these particular embodiments. [Brief description of the drawings]

[0045] [Figure 1] FIG. 1 illustrates a schematic of a portion of a vision-based timing system. [Diagram 2] FIG. 2 illustrates a network of a timing system according to one embodiment. [Diagram 3] FIG. 3 illustrates a high level overview of the timing and identification process according to one embodiment. [Figure 4] FIG. 4 illustrates object detection and identification during the timing and identification process according to one embodiment. [Diagram 5] FIG. 5 illustrates object re-identification during the timing and identification process according to one embodiment. [Figure 6A] FIG. 6A illustrates a process for determining a passage time for an object passing through a virtual timing line according to one embodiment. [Figure 6B] FIG. 6b illustrates a process for determining a passage time for an object passing through a virtual timing line, according to one embodiment. [Figure 7] FIG. 7 illustrates a general process flow of the timing and identification process according to one embodiment. [Figure 8] FIG. 8 illustrates a system for timing and identifying objects passing through a virtual timing line, according to one embodiment. [Figure 9]FIG. 9 is a block diagram illustrating an example data processing system that may be used to implement the methods and software products described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] FIG. 1 illustrates an example of a timing system that may be used in the embodiments described herein. In particular, the figure illustrates a timing system 100 comprising a camera system 101 controlled by a computer system 104. Each camera system may be configured to capture images (video frames) of a scene of a sports track 106, where the scene may include objects, such as people, animals or vehicles participating in a sports event; and to determine depth information associated with the captured images. For example, in one embodiment, the depth information may include one or more so-called depth maps associated with the video frames generated by the image sensors of the camera systems. The depth map of a video frame, e.g. an RGB video frame, may be represented as a pixelated image comprising pixel values ​​representing a distance value for each pixel of the video frame. The distance values ​​may define a distance between (the imaging plane of) the camera and an object in the video frame.

[0047] For example, a group of pixels in a video frame may be part of an object in a scene being imaged by the camera system. A depth map may then indicate the relative distance between the camera (viewpoint) and the surface of the object in the scene. Thus, during the imaging of a series of time-stamped video frames of an object, such as an athlete or a vehicle, moving along a sports track, the associated depth map may provide information about the distance between the moving object in the video frames and the (static) camera system as a function of time.

[0048] Camera systems are known that can generate depth information, e.g. a depth map associated with one or more images. For example, the camera system can be implemented as a 3D camera system, e.g. a stereo camera system. Typically, such a camera system may have two or more camera modules, where each camera module has its own lens system. In this way, the camera system is configured to simultaneously capture multiple (i.e. two or more) images of the same scene from different viewpoints. In one embodiment, the camera system may have two separate camera modules, e.g. as illustrated in FIG. 1. In another embodiment, the camera system may include one housing with two camera modules. Such a stereo camera can be used to determine a depth map and / or a 3D picture. A 3D camera system may have two or more camera modules, so that multiple images can be used to calculate a depth map. In this way, the accuracy of the depth information can be improved.

[0049] The camera system used in the embodiments of the present application is not limited to stereo-based imaging techniques, and other imaging techniques may be used as well. For example, the depth map may be generated based on RGB / infrared (IR) techniques (as used by Kinect), or 3D time-of-flight (TOF) techniques, LIDAR, or a combination thereof, where infrared radiation, e.g., an IR camera or IR laser, may be used to determine depth information of the scene imaged by the camera. In some embodiments, the camera system may be a color infrared camera configured to capture images in the visible spectrum (e.g., RGB) and infrared (IR) spectrum to capture images of objects. For example, the use of infrared images may be advantageous when light conditions are not optimal, such as during weather conditions (e.g., fog, rain, etc.) or twilight.

[0050] In order to increase the viewing angle of the camera system, in some embodiments, one or more wide-angle camera modules, such as 180-degree cameras or 360-degree cameras, may be used. Also, in the case of such types of formats of video, such as 360-video, i.e. immersive video, generated using a special 360 camera system, a depth map may be generated, where the video is projected onto a 2D video frame, for example using equirectangular projection. To enable accurate image processing, the camera used in the timing system has a frame rate of at least 30 frames / second or more. Depending on the application, the frame rate may be at least 60 frames / second or more. In addition, the camera may be a high-resolution camera with an image resolution of 4K or more.

[0051] As shown in FIG. 1, the camera system includes multiple camera modules 102 positioned on one or more sides and / or above the sports truck. 1,2 The camera modules may be aligned and calibrated so that each camera module images the same scene 114 of the sports track, including objects moving along the track, using one or more calibration markers 110 that may be positioned above the track and / or along one or both sides of the track. 1,2may be used. One or more calibration markers may be used by the timing system to determine the coordinates of a virtual timing reference 112. Depending on the implementation, the virtual timing reference may define a virtual timing line or plane located at a distance z from the camera system. A virtual timing line may be used to determine a passage based on two coordinates, e.g., an object passing the time line at a position x, or based on three coordinates, e.g., a portion of an object passing the time line at a position x and a height y. For example, in some use cases, it may be desirable to determine the passage time when a particular portion of an object, e.g., the chest of an athlete equipped with a BIB, passes through the virtual timing plane.

[0052] In the following, whenever a virtual timing line is mentioned, other types of virtual timing references should also be included, such as a virtual timing plane. The timing line may be located across the track at the location of one or more calibration markers. For example, when using two calibration markers, the virtual timing line may be located between the two calibration markers.

[0053] Examples of (the use of) calibration markers and the calibration process are described in detail in International Patent Publication No. WO2021 / 048446, which is incorporated herein by reference. After calibration, the relative distance between the camera system and the virtual timing line is known, and therefore the marker can be removed unless it needs to be used for recalibration, such as when the camera needs to be reset or when the camera is moved. This allows the timing system to determine the time when a moving object, such as a participant in a sports event, passes the virtual timing line based on the time-stamped video frames and the associated depth map. In this way, the exact passing time of the participant in the sports event can be obtained.

[0054] The computer system 104 may be configured to control a camera system to capture images, typically a set of images, of objects passing the virtual timing line. In some embodiments, the computer may comprise an image processing module 103 configured to process and analyze at least some of the images captured by the camera system. In particular, in some embodiments, the image processing module may be configured to analyze the images, e.g., to detect objects in the images, classify the objects in the images, and determine a crossing time, i.e. a time instance at which a detected object crosses the virtual timing line.

[0055] The image processing module may further be configured to identify an object based on one or more object characteristics associated with the object. Object identification here refers to the process of assigning an identity, typically personal data (e.g., the name of a person, animal, vehicle or team, stored in a database before the event), to a detected object based on these object characteristics. An example of an object characteristic is the number on a participant's BIB 109. When a participant registers himself / herself in a sports event, the participant will receive identification information, such as an identification number (e.g., a BIB number). The identification information may be used by the timing system to link data, such as information about the passage time 107, to personal data stored in a central database, such as the name of the registered participant. Different object characteristics may be used to link an object to an identity. For example, if the object represents a person, the object characteristics may include (but are not limited to) a race bib (part of it), a color, biometric information, a visual identification code, e.g. a QR code, an object item, e.g. a shoe, a shirt, a brand mark, etc. If the object represents a vehicle, the object characteristics may include vehicle features, e.g. a number, a color, a shape, a brand mark, a license plate or a registration plate. The computer may comprise or be connected to a wireless module 105 for wirelessly connecting the timing system to a further computer system (not shown), e.g. a remote computer system, e.g. a server system, configured for centrally processing the determined passage times of the detected and identified objects.

[0056] Although the timing system depicted in Fig. 1 is configured to determine the passing time of objects passing the virtual timing line, it is particularly challenging to identify substantially all timed objects passing the virtual timing line with an accurate detection rate, e.g., a correct rate of 99.9% or more, since this makes it quite likely that not all detected objects in the image can be identified by the timing system. Failure to detect an object may be caused by the object characteristics associated with the object, i.e., the object attributes, e.g., BIB number, that are used to associate the detected object with an identity, being visually blocked or at least partially blocked by other objects in the image and / or by image quality problems. Thus, the image processing module of the timing system may not be able to determine sufficient object characteristics to identify all objects in the image, and thus some objects cannot be identified by the timing system. This makes fast, accurate and reliable visual-based timing and the identification by the visual-based timing system of all participants passing a virtual timing line (e.g., a start line or a finish line) particularly difficult.

[0057] Embodiments of the present application relate to a method and system for timing and identifying objects participating in a sports event passing a virtual timing line across the sports track based on visual information in one or more first images taken by a first camera system of a vision-based timing system and based on visual information of objects in one or more further second images taken by one or more further camera systems arranged at positions different from the position of the first camera system. In particular, timed objects in the first images that cannot be identified based on visual information in the one or more first images can be identified based on visual information of objects in the one or more further second images. Identification of non-objects may be performed by an algorithm executed on a further computer system, for example a central server system or a cloud platform. To that end, the timing system and one or more further camera systems may be connected to the further computer system.

[0058] In some embodiments, each or at least some of the additional camera systems may be part of an additional timing system for determining the passage times of objects passing a virtual timing line at different positions on the race track. In this way, a network of timing systems may be formed, as shown in Figure 2. This figure illustrates a sports track 200 and a virtual timing line at different positions 204 along the track. 1~5 Timing system 206 located at 1~5 1 illustrates a network that may be implemented based on a variety of network configurations, including but not limited to a meshwork, star network, or cellular network. Additionally, any suitable communication standard (Zigbee, LoRa, LoRaWAN, 5G, etc.) may be used to operate the network.

[0059] As explained with reference to Fig. 1, each timing system comprises a camera system comprising one or more camera modules and a computer or processor, the camera system comprising an image processing module configured to determine the passage times based on time-stamped images of objects passing the virtual timing line. The timing systems may share a common clock so that the passage times at different positions along the track have a common time reference. In this way, a wireless network of timing systems may be formed, where a further computer system may be used to receive and collect timing information, i.e. the passage times of detected objects, and to receive image information of detected objects that could be identified by the timing system and of objects that could not be identified by the timing system.

[0060] If all objects in the first image of the first timing system cannot be identified by the first timing system, first image information associated with the objects that could not be identified by the timing system based on the visual information in the first image can be transmitted by the timing system to a remote computer system. The first image information can include, for example, one or more pictures of one or more regions of interests (ROIs) in the first image, where each ROI includes at least a part of the object that could not be identified. Such pictures can be cropped or copied from the first image. The first image information can further include information about the picture, such as object or ROI identifiers (IDs), to enable the system to distinguish between different objects in the image, and information about the time the image was taken.

[0061] The further computer system may be configured as a server system including a server application, configured to receive image information from the timing system, for example first image information associated with one or more first images and second image information associated with one or more further second images including objects participating in the event and which can be identified based on visual information in one or more second images, where the second images may be captured by a camera system of another timing system which may be located at another position along the track or a camera system installed at a position where an object enters the track.

[0062] The server application may then be configured to compare the unidentified object in the first image with the objects in one or more second images to determine whether the unidentified object in the first image matches (i.e., is similar or identical to) an object in one of the one or more second images. If there is a match, the server application may identify the unidentified object in the first image based on one or more object characteristics of the object in the second image that matches the unidentified object in the first image.

[0063] In this way, the success rate of identifying objects passing a particular virtual timing line, for example a timing line used to start a sporting event, can be significantly improved. A high-level schematic of the identification process performed by the server application is shown in Fig. 3, which illustrates three different images, each image taken by a different camera system. For example, a first image 312 is taken by a first camera system of a first timing system, for example the first timing system at the start position of the track (pass 1), while a second image 314 and a third image 316 are taken by a second camera system and a third camera system located at a second position (pass 2) and a third position (pass 3) along the track, respectively. The first camera system and the second camera system may be part of the second and third timing systems.

[0064] The image processing module detects objects in the first image, in this example six objects 318 passing through the virtual clock line. 1~3 and 320 1~3 In particular, an instance object segmentation algorithm may be used to detect objects in the image and separate them from other objects in the image. As shown in FIG. 4, instance segmentation separates multiple objects 400 of the same class. 1~6The instance segmentation algorithm allows detection of objects, such as athletes or vehicles, where each detected object can be treated as a separate instance. The instance segmentation algorithm may determine a bounding box that defines a region of interest ROI 401 in the image (in the figure, for clarity, only the ROI of the first object is depicted) that contains the detected object. To distinguish between different detected objects of the same class in one image, the ROI containing the detected object is labeled with an object identifier ID used by the system, e.g., a unique number assigned to the object by the algorithm.

[0065] In this way, objects in an image, such as athletes, cars, bikes, etc., can be accurately and efficiently detected and classified, and represented as labeled ROIs in the image. In one embodiment, information about the ROI and ID may be added or linked to the image as metadata. Efficient instance object segmentation algorithms based on deep learning may be used, including but not limited to algorithms that can detect and classify objects in an image in (near) real time, such as R-CNN, Fast R-CNN, or Faster R-CNN or related algorithms. An overview of state-of-the-art instance segmentation algorithms is provided in the following paper by A. Hafiz et al.: A survey on Instance Segmentation: State of The Art, International Journal of Multimedia Information Retrieval volume 9, pages 171-189 (2020), which is incorporated herein by reference. These algorithms provide very efficient runtime performance, and thus can be executed locally, for example using the computer of the timing system.

[0066] To speed up the object detection and identification process, the CPU-based local computer of the timing system running the image processing module may be equipped with one or more special purpose processors, such as one or more graphical processing units GPUs, tensor processing units TPUs (tensor processing units) or field programmable gate arrays FPGAs (field programmable gate arrays), which may be specifically adapted to accelerate the calculations performed by various image processing algorithms, in particular those associated with neural networks.

[0067] Objects detected in the image are identified using one or more object characteristics, e.g., identification markers 404 that are specifically used to identify the object. 1~3 (e.g., BIB, QR code, printed text, symbols, etc.), biometric information 408 1~6 (e.g., facial features, age, gender, height, hair color, etc.) and / or non-biometric object items 406 specific to a particular object 1~6 Non-biometric object items 406 that may be identified or associated with an identity based on the user's physical appearance (e.g., color and / or design of clothing and / or shoes, or color and / or shape of a vehicle), facial features, age, gender, length, hair color, etc., and / or are characteristic of a particular object. 1~6 (such as the color and / or design of clothing and / or shoes, or the color and / or shape of a vehicle.) These object characteristics may define visual identification information that can be used by the timing system to identify objects passing a virtual timing line.

[0068] The ROI may be analyzed by one or more algorithms configured to detect object characteristics in the ROI and extract identifying information from the ROI. For example, optical character recognition (OCR) algorithms may be used to recognize (printed) identifying markers in the ROI. Typically, OCR algorithms may include text detection and text recognition algorithms. These printed identifying markers may be based on characters and / or symbols, e.g., numbers and / or letters, as exemplified on the BIB or vehicle. Alternatively and / or additionally, code detection algorithms may be used to detect and encode visual codes, typically standardized codes, in the ROI. For example, the BIB or the athlete's clothing may include a QR code or a variation thereof, in which identifying information is encoded as (for example) a geometric structure. Furthermore, face detection and recognition algorithms may be used to identify people in the ROI based on facial features.

[0069] In some embodiments, a pose estimation algorithm configured to identify the pose of the object is used. For example, in one embodiment, the pose estimation algorithm may determine the key joints of the object (shoulders / hips / knees / head), which may be used to identify possible locations in the ROI that may represent object identification features, i.e., object features that can be used for identification. For example, based on the locations of the key joints, a particular part of the body, such as chest, arms, head, legs, or feet, may be determined. Based on the locations of the key joints, locations within the ROI that have a high probability of having one or more object features, such as face, chest, or feet, may be determined.

[0070] If the object characteristics of an object in an image cannot be determined or are not sufficiently determined, it may not be possible to identify it based solely on information in one or more first images. For example, based on the first image in FIG. 3, three objects 318 1~3 may be identified because for these objects sufficient object characteristics can be determined (e.g., in the form of visible BIBs and possibly other information). 1~3 Some objects cannot be identified because sufficient object characteristics cannot be determined to reliably identify these objects. If the timing system is unable to identify all objects in an image, it will transmit image information associated with the unidentified objects to a server application.

[0071] Depending on the implementation and / or use case, the image information may include different types of information. For example, in one embodiment, the image information may include metadata identifying the captured image and objects that could not be identified in the image, e.g., the location of an ROI in the image that contains the unidentified object. Furthermore, in one embodiment, the image information may include an identifier ID associated with the ROI so that the system can distinguish between different ROIs. In another embodiment, instead of the image, a picture of each detected unidentified object may be cropped from the image. In this way, only the relevant information is transmitted, thereby significantly reducing the bandwidth required to transmit the image information to the server system. In a further embodiment, timing information may be transmitted to the server application, e.g., a timestamp indicating the time of passing of an object or the time the picture was captured.

[0072] The server application will attempt to identify the unidentified object based on image information from different image sources, for example a second image 314 and a third image 316 including the objects participating in the event. These images may be captured by one or more further camera systems capturing images of the object, as described with reference to FIG. 2. The identification process performed by the server application may include a re-identification process configured to search for objects in the second image 314 and / or the third image 316 that match (i.e. are similar or identical to) the unidentified object in the first image. For example, the unidentified object 320 in the first image may be identified by a re-identification process configured to search for objects in the second image 314 and / or the third image 316 that match (i.e. are similar or identical to) the unidentified object in the first image. 1~3 By matching the first image with objects in the second and third images that have sufficient object characteristics, it may be possible for the server application to identify objects that are not the object timed by the first timing system. Increasing the number of images of an object captured by different camera systems may further reduce the rate at which the timing system misses an identification.

[0073] While the examples in Figures 2 and 3 illustrate examples of a system configured to improve the identification rate of timed participants at the start, it will be apparent that the scheme may also be used to improve the identification rate of other passing times, for example passing times associated with a timing system at a certain position along the track that is part of a network of timing systems positioned along the race track.

[0074] The re-identification process performed by the server system may be configured to compare objects detected in different non-overlapping images of different camera systems located at different positions, typically pictures of ROIs containing the objects, and to determine a similarity measure between the compared objects. Thus, the object may be imaged by different camera systems under different conditions (light, angle, distance, quality, etc.). The algorithm may be configured to detect objects in different images from different cameras and to determine an object descriptor for each detected object, where the object descriptor may provide a high-level descriptor of the object based on multiple features of the object, in particular object properties. The object descriptor may have the form of a set of values, where each value may indicate a certain confidence value for a certain object feature, e.g. a certain color of the clothes. These values ​​may be arranged in a predefined format, such as a vector.

[0075] To compare two objects in different images, a distance or difference value can be determined based on the object descriptors of the different objects, where the calculated distance or difference value may represent a similarity measure for a high level of similarity of the two objects. Such a comparison is based on the idea that the difference between certain object features of two objects in different images with the same identity is smaller than the difference between certain object characteristics of two objects in different images with different identities. Deep learning-based re-identification algorithms include, but are not limited to, re-identification algorithms based on local information extraction, distance metric learning, or semantic attributes. An overview of state-of-the-art re-identification algorithms is given in the following paper by Hongbo Wang et al.: A comprehensive overview of person re-identification approaches, March 2020, IEEE Access DOI:10.1109 / ACCESS.2020.2978344, which is incorporated herein by reference.

[0076] For example, in the case of a sports player, the object descriptor may represent the sports player in terms of the object characteristics of the person, such as the color of the clothes, long or short sleeves or pants, height, skin color, long or short hair, shoe color, etc. Alternatively, in the case of a vehicle, the object descriptor may represent the vehicle in terms of the object characteristics of the vehicle, such as the color of the vehicle, the shape of the vehicle, numbers and / or letters on the vehicle. Two objects may then be compared by calculating a distance (e.g. a difference) based on the object descriptors of the two objects. For example, a so-called cosine distance may be calculated, which may be a value between 0 (not similar) and 1 (completely identical). The larger the distance, the greater the confidence that the objects are identical.

[0077] FIG. 5 illustrates an object descriptor, e.g., specific facial features 502 1~3 and specific features of the shoe 504 1~3 1 illustrates an example of a re-identification process based on the object descriptors described above, which contain object characteristics related to the first object 5201 in the first picture and the first object 5241 in the second image. When comparing the first unidentified object 5201 in the first picture and the first object 5241 in the second image based on the object descriptors, a high confidence value may indicate that the objects are likely to have the same or similar object characteristics. Similarly, a comparison between objects based on object descriptors may indicate that there is a high similarity between the second unidentified object 5203 in the first image and the second object 5243 in the second image, and between the third unidentified object 5202 in the first image and the third object 5242 in the third image. If the aggregate confidence value of the two objects is above a certain threshold, the system may determine that the objects in the different images are the same. In this way, the same objects in the different images may be matched.

[0078] If the re-identification process determines that there is a match between the unidentified object in the first image and one of the objects in the second image, i.e., if the unidentified object in the first image is similar (or identical) to an object in the second image (or third image), then the object in the second "matched" image can be used to identify the unidentified object. In one embodiment, object identification algorithms executed by the server system can be used to identify the matched object using an object identification scheme that can be similar to that described with reference to FIG. 4. Since these algorithms are executed on the server system, the models can be more extensive than those executed locally on the computer of the timekeeping system, since the limitations on speed and computer resources (memory usage and processor power) are less severe for the server system.

[0079] In another embodiment, instead of the server application running the identification algorithm, the identification may be based on identification information contained in the second image information and transmitted to the server application. This embodiment thus exploits the fact that the second camera system (which may be a second timing system) is capable of identifying detected objects in the images. The second image information transmitted to the server system may then include a picture of the object and identification information of the identified picture.

[0080] Figures 6A and 6B illustrate the operation of the passage time module according to one embodiment. Figure 6A shows three snapshots of a moving object (in this case an athlete) passing through a 3D detection zone of a timing system as described with reference to this embodiment of the present application. The 3D detection zone may be set using a calibration method as described above with reference to Figure 1. As shown in Figure 6B, the 3D detection zone 612 may include a virtual plane 614 located across the track, where the normal of the virtual plane is substantially parallel to the direction of the sports track. The virtual plane divides the 3D detection zone into a first portion 6161 where the object moves towards and crosses the virtual plane, and a second portion 6162 where the moving object crosses the virtual plane and moves away from the virtual plane.

[0081] Thus, as the object moves along the track, the 3D camera system will capture an image (a pair of images in case of a stereo camera) of the scene including the 3D detection zone. For each image (video frame), the 3D imaging system may compute a depth map. Object detection and tracking algorithms may be used to detect and track a given object, e.g. a human object or an object representing an object, in subsequent video frames.

[0082] Based on the depth map, the computer may determine that the detected object has entered a first portion of the 3D detection zone. If so, the computer may begin saving video frames and associated depth maps in a buffer until the object exits the 3D detection zone through a second portion. In another embodiment, only one pair of video frames is saved and the depth map is determined at a later time. These video frames and depth maps may be used by the computer to determine a passage time and to identify the object associated with the passage time.

[0083] FIG. 6A shows three samples 602 of a sequence of time-stamped video frames as an athlete moves through a 3D detection zone. 1~3 6B illustrates a video frame 6021 captured by a camera system, which may be stored in the computer. As shown in FIG. 6B, a first video frame 6021 is captured at time instance T1, a time when most of the athlete's body was still in the first portion 6161 of the 3D detection zone. At time instance T2, a second video frame 6022 is captured where the athlete has moved further and passed through the virtual plane 614 of the 3D detection zone. Finally, at time instance T3, a third video frame 6023 is captured where the athlete has moved into the second portion 6162 of the 3D detection zone.

[0084] A pass time module in the computer of the timing system may analyze the sequence of time-stamped video frames to determine at which time instance the athlete was passing through the virtual plane. To do so, an object detection and classification algorithm may be applied to each video frame. The algorithm may, for example, detect areas of interest 608 in a video frame that belong to an object. 1~3 Additionally, for each of these ROIs, the algorithm may classify pixels as belonging to the athlete or not (background).

[0085] Furthermore, the depth map associated with each of the video frames may be used to determine distance values ​​that belong to pixels classified as belonging to the object. These distance values ​​may be compared to the distance between the camera and the virtual plane. In this way, when a 3D camera system images an object crossing a virtual plane, for each video frame, the portion of pixels of the object that crossed the virtual plane can be determined. This can be seen in the video frames of FIG. 6A, where the grey areas define the pixels of (part of) the object that crossed the virtual plane.

[0086] For the video frame at time instance T1, only pixel 6041 representing a portion of the athlete's hand and pixel 6061 representing the athlete's shoe are associated with a distance value that is less than the distance between the virtual plane and the 3D camera system. Similarly, for the video frame at time instance T2, pixel 6041 representing a portion of the upper body and pixel 6062 representing a portion of the leg are associated with a distance value that is less than the distance between the virtual plane and the 3D camera system. Finally, for the video frame at time instance T3, all pixels 608 representing the athlete are associated with a distance value that is less than the distance between the virtual plane and the 3D camera system. Based on this analysis, the computer may determine that at T2, a substantial portion of the athlete's body crossed the virtual plane. For example, the computer may determine that the athlete crossed the plane if the portion of the object that crossed the virtual plane was greater than a threshold value. Thus, timestamp T2 may then define a crossing time 510, in this example, 2:34. Different rules may be defined for determining if an object has crossed the virtual plane.

[0087] 7 illustrates a general process flow of a method for timing and identifying an object passing a virtual timing line across a sports track that may be performed by embodiments described herein. The process may begin with an application, preferably a server application, receiving first image information associated with one or more first images captured by a first camera system of a first timing system located at a first location along the sports track (step 702). The first images may include an object, such as a person or a vehicle, passing the virtual timing line. Additionally, the first image information may include visual information regarding at least one first object that cannot be identified based on the one or more first images.

[0088] In a similar manner, the server application may receive second image information associated with one or more second images captured by a second camera system located at a location different from the first location (step 704). The one or more second images may include objects participating in the sporting event, and the second image information includes visual information regarding at least some of the objects in the one or more second images that can be identified based on the one or more second images. Thus, the objects in the one or more second images are captured by different camera systems and taken from different viewing angles and different focal lengths, such that images of objects are captured where an identification marker (e.g., a printed marker) or a biometric marker, such as a person's face, is visible. In this way, different images of each object in the event will be captured during the event.

[0089] The server application may receive one or more second images from the second camera system. Alternatively, the server application may obtain one or more second images from a storage medium, such as an image database, configured to store or buffer images of objects participating in the event.

[0090] Based on the first image information and the second image information, the server application may identify a first object that could not be identified, where identifying the first object may include searching for a second object in the one or more second images that matches the first object in the one or more first images; and identifying the first object based on visual information of the second object.

[0091] Here, retrieving the second object may be subject to object properties of objects in the first image and the second image. For example, the server application may include determining one or more first object properties associated with the first object; determining one or more second object properties associated with an object in the second image; and determining whether one of the objects in the one or more second images matches a first object in the one or more first images based on the one or more first object properties and the one or more second object properties.

[0092] FIG. 8 illustrates a system for timing and identifying objects passing through a virtual timing line according to one embodiment. The system may comprise a vision-based first timing system comprising a first camera system 8021 connected to a first computer or processor 8031 ​​comprising an image processing module for a time-stamped first image 8041 captured by the first camera system. The first timing system may be positioned along a track and communicatively connected, for example wirelessly, to a further computer system 822 in a similar manner as described with reference to FIGS. 1 and 2. The further computer system may be configured as a network node, for example a server system or a cloud system. The first camera system may be configured to generate a first image of an object passing through a virtual timing line. Furthermore, the first camera system may be configured to generate depth information (depth map) associated with the first image, providing information regarding the relative distance between the camera system and the object passing through the virtual timing line.

[0093] The image processing modules implemented by the computer of the first timekeeping system may include an object detection module 8061 for detecting objects in the first image, a passage time module 8081 for determining passage times for detected objects, and an object identification module 8101 for identifying detected objects in the first image. Non-limiting examples of image processing schemes implemented by these modules are described in detail with reference to Figures 2-7. In other embodiments, although the modules are illustrated as separate modules each including an algorithm for performing a function, some or all of the functions of the modules may be integrated or combined into one module including an algorithm configured to provide the functions of the integrated or combined modules.

[0094] Based on information in the first image, the image processing module may detect, time and identify at least some of the objects passing the virtual timing line in the image. In case of these objects, the first timing system may locally assign a passing time and an identity, e.g. a BIB number, to the detected objects. The first timing information 8161 may be transmitted by the first computer to a database 812 that collects the timing information of participants taking part in the event monitored by the first timing system. The database may use the received identity to link the elapsed time to personal data, e.g. the name of the participant, which may be stored in a further registration database (not shown) of the server system.

[0095] If the identification module of the first timing system does not succeed in identifying all objects passing the virtual timing line in the image, then the first computer may transmit the first image information associated with the first image 8181 to a further computer system, which may include a storage medium 820 for (temporarily) storing the first image information. The storage medium may be part of an image database for storing images captured by different camera systems connected to a server system. The first image information may include at least visual information of one or more timed timed objects that could not be identified. This visual information may include at least a part of the first image, for example one or more pictures cropped from the one or more first images that include at least a part of the one or more objects that could not be identified.

[0096] The first image information may further comprise metadata associated with the picture of the unidentified object, such as timing information relating to the passage time determined by the passage time module, an identifier to enable distinguishing between the unidentified object and other information that can be used by the server application to identify the unidentified object based on further images captured by a further camera system, which may for example comprise information regarding the resolution and / or format of the picture.

[0097] In some embodiments, the image information may also include pictures of objects identified by the first timing system. These pictures and associated identification information, such as BIB numbers or vehicle numbers, may also be transmitted to storage medium 820 and entered into the image database. In this manner, the first timing system may transmit visual information, such as pictures, of both identified and unidentified objects to the server system.

[0098] The system may comprise at least one further second camera system 8022 connected to a second computer 8032, which may comprise an image processing module. For example, in one embodiment, the image processing module may comprise at least an object detection module 8062 and an object identification module 8102 for processing images 8042 captured by the second camera system. The second computer may then transmit second image information associated with the one or more second images to the server system, where the second image information may include pictures of detected objects and identification information of the detected objects.

[0099] In a further embodiment, the second camera system may be part of a second timing system configured to time and identify objects passing through a second timing line located at a second position along a track different from the position of the first timing system. In that case, the second computer may also include a passing time module 8082, and thus the second timing system may transmit both second image information and timing information to the server system. In this way, image information relating to identified and unidentified objects may be stored in the storage medium 820 of the server system.

[0100] The server application may then include an object identification module 814 for identifying unidentified objects that could not be identified on the one or more first images based on the one or more second images. This identification process may use image re-identification to match an unidentified object in one image captured with a first camera system with an identified object in another image captured with another second camera system. Furthermore, once a second object in a second image is found to match the unidentified first object in the first image, then the second object may be identified based on identification information provided to the server application by a further second camera system or based on an object identification algorithm executed by the server application.

[0101] The system illustrated in FIG. 8 shows a non-limiting example of an architecture in which part of the processing of the images is performed locally by a computer of a timing system placed along the track. Only the processing of unidentifiable objects, which typically requires more computationally intensive image processing, e.g. object re-identification, is performed on the server. It should be noted that other architectures are also possible. For example, in one embodiment, the timing system only comprises a camera system for generating images and depth information of objects passing the timing line. This information is transmitted, e.g. in a stream, to a server system that comprises an image processing module executed in FIG. 8 by a computer of the timing system.

[0102] In further embodiments, at least a part of the server functionality illustrated in FIG. 8 may be implemented on a server associated with one of the timing systems or integrated within one of the timing systems. For example, a computer of a timing system may include a server system running a server application. A timing system with server functionality may define a master timing system. All other timing systems along the track may be connected to the master timing system to perform the object identification process as described with reference to the embodiments of the present application. Such an architecture may be particularly suitable when a network of timing systems may form a mesh network, where timing systems within a certain range may be configured to relay timing information and images for object identification towards the master timing system.

[0103] 9 is a block diagram illustrating an exemplary data processing system that may be used to execute the methods and software products described herein. The data processing system 900 may include at least one processor 902 coupled to memory elements 904 through a system bus 906. As such, the data processing system may store program code in the memory elements 904. Furthermore, the processor 902 may execute program code accessed from the memory elements 904 via the system bus 906. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 900 may be implemented in the form of any system having a processor and memory capable of performing the functions described herein.

[0104] The memory element 904 may include one or more physical memory devices, such as a local memory 908 and one or more bulk storage devices 910. The local memory may refer to a random access memory or one or more other non-persistent memory devices typically used during the actual execution of the program code. The bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 900 may also include one or more cache memories (not shown) that provide temporary storage of at least some of the program code to reduce the number of times the program code is retrieved from the bulk storage device 910 during execution.

[0105] Input / output (I / O) devices, illustrated as input device(s) 912 and output device(s) 914, may optionally be connected to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device (e.g., a mouse), and the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, speakers, and the like. The input and / or output devices may be connected to the data processing system directly or through an intervening I / O controller. A network adapter 916 may also be connected to the data processing system to enable connection to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may include a data receiver for receiving data transmitted to the system, device, and / or network by the system, device, and / or network, and a data transmitter for transmitting data to the system, device, and / or network. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with the data processing system 900.

[0106] 9, the memory element 904 may store an application 918. It should be appreciated that the data processing system 900 may further execute an operating system (not shown), which may facilitate execution of the application. Applications implemented in the form of executable program code may be executed by the data processing system 900, such as the processor 902. In response to executing the application, the data processing system may be configured to perform one or more operations described in further detail herein.

[0107] In one aspect, for example, data processing system 900 may represent a client data processing system, in which case application 918 may represent a client application that, when executed, configures data processing system 900 to perform various functions described herein with reference to a "client." Examples of clients may include, but are not limited to, personal computers, portable computers, mobile phones, etc.

[0108] In another aspect, the data processing system may represent a server. For example, the data processing system may represent an (HTTP) server, in which case the application 918, when executed, may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function, as referred to herein.

[0109] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that as used herein, the words "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0110] Corresponding structures, materials, acts, and equivalents of all means or step-plus-function elements in the appended claims are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described in order to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated.

[0111] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that as used herein, the words "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0112] Corresponding structures, materials, acts, and equivalents of all means or step-plus-function elements in the appended claims are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described in order to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated.

Claims

1. 1. A method of timing and identifying objects participating in a sporting event, comprising: receiving, by a server system, first image information associated with one or more first images captured by a first camera system of a first timing system positioned at a first location along a sports track, wherein the one or more first images include objects participating in the sports event associated with visual identification markers or visual identification codes passing a virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but which cannot be identified based on the one or more first images; receiving or acquiring, by the server system, second image information associated with one or more second images captured by a second camera system located at a location different from the first location, wherein the one or more second images include objects participating in the sporting event, and the second image information includes visual information regarding one or more objects that can be identified based on the one or more second images; and identifying the first object by the server system, wherein the identifying includes determining, using the first image information and the second image information, a second object associated with a visual identification marker or visual identification code in the one or more second images that matches the first object, the determining being based on a first non-biometric object characteristic associated with the first object and a second non-biometric object characteristic associated with the second object; and, once the second object is determined, identifying the first object based on the visual identification marker or visual identification code of the second object. The method comprising:

2. 2. The method of claim 1, wherein the first image information includes at least a portion of one of the one or more first images including the first object, or at least one picture of a first region of interest (ROI) of one of the one or more first images, wherein the first ROI includes at least a portion of the first object.

3. 3. The method of claim 1, wherein the first image information further comprises timing information indicating a time instance at which the first object passes the virtual timing line, depth information indicating a distance between the first camera system and the first object, and / or an identifier associated with the first ROI.

4. 3. The method of claim 1, wherein determining a second object in the one or more second images is based on a re-identification algorithm, wherein the re-identification algorithm is configured to compare the first object with the object in the one or more second images based on object characteristics.

5. determining a second object in the one or more second images; determining one or more first object properties associated with the first object; determining one or more second object characteristics associated with the object in the second image; and determining whether one of the objects in the one or more second images matches the first object in the one or more first images based on the one or more first object characteristics and the one or more second object characteristics; 3. The method of claim 1 or 2, comprising:

6. 5. The method of claim 4, wherein the matching is based on a distance measure calculated based on the first object characteristics and the second object characteristics, wherein the distance measure indicates a similarity between the first object and an object in the second image.

7. 5. The method of claim 4, wherein an object in the one or more first images and an object in the one or more second images represent a person participating in the sporting event, and wherein the first object characteristic and the second object characteristic define a characteristic of the person; or wherein an object in the one or more first images and an object in the one or more second images represent a vehicle participating in the sporting event, and wherein the first object characteristic and the second object characteristic define a characteristic of the vehicle.

8. identifying the first object; retrieving a visual identification marker or a visual identification code based on the visual information of the second object; and If a visual identification marker or a visual identification code is found, converting the visual identification marker or the visual identification code into identification information for linking the second object to an identity; and associating the first object with the identity; and storing the identification information and the timing information of the first object in a database; The method of claim 1 or 2, further comprising:

9. the second image information includes identification information associated with the second image; and identifying the first object associating the first object with the identity; and storing the identification information and the timing information of the first object in a database; The method of claim 1 or 2, further comprising:

10. receiving timing information and identification information associated with objects in the one or more first images that have been detected, timed, and identified by the first timing system based on the one or more first images; The method of claim 1 or 2, further comprising:

11. 3. The method of claim 1 or 2, wherein the second camera system comprises a computer or processor configured to determine visual information about one or more objects that can be identified based on the one or more second images.

12. 3. The method of claim 1 or 2, wherein the second camera system is part of a second timing system configured to determine the passage times of objects participating in the sporting event passing a virtual timing line.

13. 3. The method of claim 1, wherein the first image information comprises an image frame including the detected unidentified object, or a ROI picture including the detected unidentified object cut out from an image frame, and optionally a timestamp indicating the time of passage of the detected unidentified object.

14. 1. A system for timing and identifying objects participating in a sporting event, the system comprising: a camera system configured to capture an image of a scene including an object on a sports track passing through a virtual timeline at a first location along the race track; and a computer coupled to the camera system. wherein the computer comprises: detecting an object associated with a visual identification marker or visual identification code in the image captured by the camera system; determining depth information associated with an image, wherein the depth information defines a relative distance between the camera system and a detected object; determining a passing time at which a detected object passes through the virtual timing line based on the timing information and the depth information; identifying the detected object based on a visual identification marker or a visual identification code in the image; and generating image information if the object cannot be detected based on the image, wherein the image information includes visual information of the object that cannot be identified; and transmitting the image information associated with the unidentified object and the time of passage to a server system, wherein the server system identifies the unidentified object based on the image information and on further image information associated with one or more further images of objects participating in the sporting event associated with visual identification markers or visual identification codes, the objects being captured by a further camera system located at a location different from the first location. The system is configured as follows.

15. 1. A system for timing and identifying objects participating in a sporting event, comprising: a first timing system comprising a first camera system connected to or comprising a first computer, wherein the first timing system is configured to generate first image information associated with one or more first images taken by the first camera system positioned at a first position along the sports track, the one or more first images including an object associated with a visual identification marker or visual identification code passing the virtual timing line, the first image information including visual information of at least one first object whose passing time is determined based on the one or more first images but which cannot be identified based on the one or more first images; one or more second camera systems, each camera system connected to or comprising a second computer, wherein the one or more second camera systems are configured to generate second image information associated with one or more second images captured by the second camera system located at a location different from the first location, the one or more second images comprising objects participating in the sporting event, the second image information including visual information regarding one or more objects that can be identified based on the one or more second images; and a server system configured to receive the first image information and the second image information and to identify the first object, wherein the identifying includes using the first image information and the second image information to determine a second object associated with a visual identification marker or visual identification code in the one or more second images that matches the first object, the determining being based on a first non-biometric object characteristic associated with the first object and a second non-biometric object characteristic associated with the second object; and, when the second object is determined, identifying the first object based on the visual identification marker or visual identification code of the second object. The system comprises:

16. The system of claim 15 , wherein the first timing system and the one or more second camera systems are configured to wirelessly communicate with the server system.

17. A computer program or a suite of computer programs, or a computer program product storing at least one software code portion, comprising at least one software code portion, the computer program or the suite of computer programs or the computer program product being configured to perform the steps of the method according to claim 1 or 2 when the software code portion is executed on a computer system.

Images