Camera system based motion timing

By calibrating the camera system and depth information technology, identifying athletes in the virtual time zone, solving the timing accuracy and privacy issues in large-scale sports events, achieving accurate timing and privacy protection for athletes.

CN119013704APending Publication Date: 2024-11-22MYLAPS BV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380028362.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-18
Filing Date
2023-03-20
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing vision-based sporting event timing systems have challenges in accurately and reliably identifying and identifying participants through virtual timing lines, especially in large events, and require handling of a large number of non-participants, resulting in detection accuracy and privacy issues.

Method used

The calibrated camera system is used to capture video frames, determine objects in the virtual time zone through object detection and depth information, and use feature analysis algorithms to identify athletes, and use virtual time zones to filter out non-participants to ensure privacy protection.

Benefits of technology

It achieves accurate and rapid identification and timing of athletes, improves detection accuracy, respects non-participants' privacy, and is suitable for large-scale sports events.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013704A_ABST
    Figure CN119013704A_ABST
Patent Text Reader

Abstract

Methods and systems are described for determining a passage time of an object through a timeline across a sports track, where the method may include receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving along the track, each video frame being associated with a moment in time; determining or receiving calibration data, the calibration data defining a virtual timing zone located at a predetermined distance from the calibrated camera system, the virtual timing zone spanning and extending along the track and including a virtual timing line spanning the track; detecting an object in the video frame by using an object detection algorithm, wherein the detected object is defined by the object detected by the detection algorithm; determining depth information for at least a portion of the video frame, the depth information including information related to a distance between the inspected object and the calibrated camera system; based on the calibration data and the depth information, a detected object located in the virtual timing area is determined; and determining the passing time of the detected object in the virtual timing area based on one or more moments of one or more video frames including the at least one detected object passing through the timing line.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to camera-based motion timing and, in particular, although not exclusively, to methods and systems for camera-based motion timing and a computer program product for enabling a computer system to perform these methods. Background Art

[0002] Sporting events such as car or racing, cycling, athletics, drone racing and skating often require accurate and fast time recording in order to track objects (people or vehicles) during the event. Such timing systems are usually based on radio frequency identification systems, in which each participant in the event is provided with an RFID transponder, such as a UHF backscatter tag or a low-frequency tag based on magnetic induction, which can be read by an RFID reader placed along the track. Such a reader can be implemented in the form of an antenna pad, a side antenna and / or an antenna mounted on a frame above the track. Each transponder is configured to send a data packet at a specific frequency and insert a unique identifier into the data packet, so that the detector can associate the data packet with a specific transmitter.

[0003] Disadvantages associated with timing systems based on RFID technology include: the need to provide a UHF tag for each participant; the sensitivity of the UHF signal to environmental influences such as humidity and rain; detuning of the UHF tag when it is placed close to the human body; reflections of the UHF signal from "hard" objects such as street surfaces, walls, etc.; and collisions of the UHF signals when a large number of participants pass simultaneously through RFID detectors such as RFID antenna mats provided across the track.

[0004] WO2021 / 048446 describes an example of a vision-based timing system for measuring the passing time of participants in mass sports events, where many participants can pass through a virtual timing line at the same time or almost at the same time. The system includes one or more cameras for capturing images of participants passing through the virtual timing line. The system may also include a processor configured to analyze the image, i.e., detect objects such as participants in a sports event in the image, and determine the passing time of the detected object through the virtual timing line. In addition, based on the captured image, the processor is configured to detect an identification information carrier, such as an identification number and / or symbol (e.g., a QR code, etc.) printed on a so-called number cloth attached to the participant's clothes, and associate the detected identification information with the participant's passing time. The advantage of this vision-based timing system is that it does not exhibit the shortcomings of the above-mentioned RFID-based timing system.

[0005] While vision-based timing systems allow for accurate determination of the time at which a participant crosses a virtual timing line, there are still some challenges in accurately timing and identifying all participants crossing the virtual timing line in real time. In particular, cameras positioned to capture the area around a timing line, such as a finish line, will often capture scenes that may extend far beyond the timing line, including areas around the timing line, and particularly areas on either side of the timing line, which will include a large number of people, not only athletes, but also other people not attending the sporting event, such as spectators and referees.

[0006] This makes it particularly challenging to accurately, reliably and quickly determine and identify participants crossing a timing line such as a start line or a finish line, especially when dealing with large sporting events such as marathons, where high detection accuracy, such as over 99.9% detection accuracy, is required. In addition, different privacy regulations add further technical challenges, as data, especially biometric data, can only be used under strict conditions.

[0007]

[0013] Therefore, from the foregoing it can be seen that there is a need in the art for improved vision-based sporting event timing that allows for highly accurate and reliable determination of the transit times and identities of participants at sporting events, particularly large-scale sporting events. Summary of the invention

[0008] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as systems, methods or computer program products. Therefore, aspects of the present invention may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects, which are generally referred to herein as "circuit", "module" or "system". The functions described in this disclosure may be implemented as algorithms executed by a microprocessor of a computer. In addition, aspects of the present invention may take the form of a computer program product, which is contained in one or more computer-readable media, which contain, for example, computer-readable program code stored thereon.

[0009] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0010] A computer readable signal medium may include a propagated data signal containing a computer readable program code, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium other than a computer readable storage medium that is capable of communicating, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0011] The program code included in the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, cable, radio frequency, etc., or any appropriate combination of the foregoing. The computer program code for performing the operation of various aspects of the present invention can be written in any combination of one or more programming languages, including functional or object-oriented programming languages, such as Java (TM), Scala, C++, Python, etc., and traditional procedural programming languages, such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer, server, or virtualized server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0012] The following describes various aspects of the embodiments with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart and / or block diagram, as well as the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, in particular a microprocessor or a central processing unit (CPU) or a graphics processing unit (GPU), or other programmable data processing devices to produce a machine, so that instructions executed via a processor of a computer, other programmable data processing devices or other devices create a method for implementing the function / action specified in one or more blocks in the flowchart and / or block diagram.

[0013] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce a product including instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0014] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0015] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions shown in the blocks may appear in the order shown in the accompanying drawings. For example, in fact, two blocks shown in succession may be executed substantially simultaneously, or sometimes in reverse order, depending on the functions involved. It will also be noted that each block of the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or behavior, or by a combination of dedicated hardware and computer instructions.

[0016] It is an object of embodiments in the present disclosure to reduce or eliminate at least one disadvantage known in the prior art.In one aspect, embodiments may be directed to a method for determining a transit time of an object passing a virtual timing line.

[0017] In one embodiment, the method may include: receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving along a track, each video frame being associated with a moment; determining or receiving calibration data, the calibration data defining a virtual timing zone located at a predetermined distance from the calibrated camera system, the virtual timing zone spanning the track and extending along the edge of the track, and including a virtual timing line or a virtual timing plane spanning the track; detecting objects in the video frames using an object detection algorithm, the objects detected by the detection algorithm defining an inspected object; determining depth information for at least a portion of the video frames, the depth information including information about the distance between the inspected object and the calibrated camera system; determining an inspected object located within the virtual timing zone based on the calibration data and the depth information; and determining a passage time of an inspected object passing through the virtual timing line within the virtual timing zone based on one or more moments of one or more video frames including at least one inspected object passing through the timing line.

[0018] In another embodiment, the method may include: receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving on a track and objects moving outside the track, each video frame being associated with a time instant, the calibrated camera system including calibration data defining a virtual timing zone in the video frames located at a predetermined distance from the calibrated camera system, the virtual timing zone spanning the track and extending along an edge of the track; detecting objects in the video frames using an object detection algorithm, the objects detected by the detection algorithm defining the detected objects; determining depth information for at least a portion of the video frames, the depth information The information includes information about the position of the inspected object relative to the calibrated camera system; based on the calibration data and the depth information, determining whether the inspected object is located in the virtual timing zone or outside the virtual timing zone; applying a feature analysis algorithm to at least a portion of the inspected objects determined to be located in the virtual timing zone in the video frame, the feature analysis algorithm determining identification features of the one or more inspected objects in the virtual timing zone; and, based on the calibration data, the depth information and one or more moments of one or more video frames of the inspected objects passing the virtual timing line, determining the passing time of the inspected objects determined to be located in the virtual timing zone and passing the virtual timing line across the track.

[0019] Therefore, the timing system detects objects in the image and uses depth information to determine whether the detected object is inside or outside the virtual timing zone, that is, the virtual timing zone is located above the track, wherein the side of the timing zone substantially coincides with the side of the track. The computer can track the detected objects in the virtual timing zone and execute the recognition algorithm to identify these objects (athletes on the track) and / or determine the passing time of these objects, without processing objects outside the virtual timing zone. In this way, the computer can only process and identify athletes in the virtual timing zone that will pass through the virtual timing line. In addition, the computer can also ignore objects outside the virtual timing zone, such as spectators and / or referees. In this way, the privacy of the spectators is respected, and athletes participating in the sports event (who agree to be timed by the vision-based timing system) can also be timed and identified without wasting computer resources. The virtual timing zone effectively implements spatial filtering based on depth information, filtering out detected objects located outside the timing zone before determining the passing time of the detected object and / or executing the recognition algorithm to identify the timing object.

[0020] The timing system is calibrated so that the camera system includes not only information, such as coordinates, of a virtual timing line that spans the track at a predetermined distance from the camera system, but also information, such as coordinates, of an area, such as a two-dimensional area or a three-dimensional volume, that is located above the track and has sides extending along the track. Preferably, the sides of the timing and identification area substantially coincide with the edges or borders of the track. The virtual timing line may be positioned within the virtual timing area. Where the coordinates of the virtual timing area define a 3D volume, the virtual timing and identification area may define a volume of interest (VOI) located above the track and having sides extending along the track. Preferably, the sides of the timing and identification area substantially coincide with the edges or borders of the track.

[0021] In one embodiment, the width of the virtual timing zone may be equal to or greater than the width of the track, wherein the length of the virtual timing zone is equal to or greater than the width of the track.

[0022] In one embodiment, the virtual timing zone may be substantially rectangular, wherein the long sides of the rectangular virtual timing zone are aligned with the sides of the race track.

[0023] Thus, the virtual timing zone defines a portion of the track where only athletes are allowed to enter so that the timing system can effectively determine whether an object is an athlete. When passing through the virtual timing line (or plane), the object in the virtual timing zone is monitored, identified and timed. The virtual timing zone can have many different shapes depending on the specific situation, as long as the virtual timing zone identifies the area in which (only) athletes move and includes the virtual timing line.

[0024] In one embodiment, the method may further include: applying a feature analysis algorithm to the inspected objects in the video frame within the virtual timing zone, the feature analysis algorithm determining identification features of the one or more inspected objects in the video frame; and determining the identity of the inspected objects determined by time based on the identification features of the inspected objects that have passed the timing line.

[0025] In one embodiment, the identification features of the object under test may include one or more optically readable identification marks, such as a race bib or printed markings; and / or, one or more features regarding the shape and / or color of the object under test; and / or, in the case where the object under test is an animal or a human, one or more biometric identification features of the object under test.

[0026] In one embodiment, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, such as a deep learning algorithm, such as a convolutional deep neural network system, which is trained to detect one or more objects in a video frame and determine identification features associated with the detected objects.

[0027] In one embodiment, an instance object segmentation algorithm based on a deep neural network can be used to detect objects in an image. In one embodiment, the instance object segmentation algorithm can be based on R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CCN ​​algorithm, or related algorithms. Such algorithms can detect and classify objects in an image in (almost) real time.

[0028] In one embodiment, the identification of the object can be based on an optical character recognition (OCR) algorithm, or an optical code recognition algorithm is used to identify a (printed) identification mark in the region of interest. In one embodiment, the identification mark may include printed characters. In one embodiment, the printed characters may only be visible using an infrared camera.

[0029] In another embodiment, identification of an object may be based on an algorithm configured to identify an object based on biometric features, such as a face recognition algorithm configured to identify a person based on facial features.

[0030] In one embodiment, the calibration data may include coordinates defining a virtual timing zone, preferably coordinates defining a 2D area above the track, wherein the sides of the 2D area are substantially parallel to the sides of the track; or, coordinates defining a 3D area above the track, wherein the side surfaces of the 3D area are substantially parallel to the sides of the track.

[0031] In one embodiment, determining the detected object located in the virtual timing zone includes: determining the position of the object using depth information; and determining whether the object is located in the virtual timing zone or outside the virtual timing zone according to the position and coordinates of the virtual timing zone.

[0032] In one embodiment, detecting one or more objects in a video frame may include: determining one or more regions of interest (ROIs) in the video frame, each region of interest including pixels representing an object; determining an identification feature in one of the one or more regions of interest; and determining the object in the region of interest based on the determined identification feature.

[0033] In one embodiment, the camera system may include multiple camera modules, preferably two camera modules forming a stereo camera, and the stereo camera is configured to generate at least a first video frame and a second video frame of the scene at each moment, wherein the depth map is determined based on a disparity mapping algorithm, and the disparity mapping algorithm is configured to determine the disparity between pixels of the first video frame and the second video frame.

[0034] In one embodiment, the passage time may be determined based on at least one video frame of a scene in which a predetermined portion of the detected object has passed through the virtual plane.

[0035] In another aspect, an embodiment may relate to a method for calibrating a timing system configured to determine a passage time of an object across a timing line across a sports track, wherein the method comprises: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of the timing system, each video frame representing a picture of a scene including a track and one or more calibration marks, preferably defining at least two calibration marks at positions on opposite sides of the track; determining depth information, such as one or more depth maps, based on the video frames, the depth information including information about distances between one or more objects in the picture of the video frames; using the depth information to determine a relative position between the one or more calibration marks and the camera system; determining calibration data, the calibration data including position information of a virtual timing line, the virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration marks; and, position information associated with a virtual timing zone, the virtual timing zone located above the track at a predetermined position relative to the position of the one or more calibration marks, the timing zone spanning and extending along the track; and storing coordinates of a virtual plane in a memory of the timing system.

[0036] In one embodiment, the calibration data may include coordinates defining a virtual timing zone, preferably coordinates defining a 2D area on the track, such as a rectangular area, wherein the sides of the 2D area are substantially parallel to and / or substantially coincide with the sides of the track; or, coordinates defining a 3D volume on the track, such as a cuboid volume, wherein the side surfaces of the 3D volume are substantially parallel to and / or substantially coincide with the sides of the track. The sides of the virtual timing zone may extend along the sides of the track for a considerable length so that the system can distinguish between athletes and spectators.

[0037] In another aspect, an embodiment may be directed to a system for determining a time of passage of an object across a timing line across a sports track, the system comprising: at least one camera system connected to a computer; the computer comprising a computer-readable storage medium containing computer-readable program code, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to executing the computer-readable program code, the processor is configured to perform executable operations, the executable operations comprising: receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving along the track, each video frame associated with a time instant; determining or receiving calibration data, the calibration data A virtual timing zone is defined at a predetermined distance from a calibrated camera system, the virtual timing zone spans across and extends along the track, and includes a virtual timing line spanning the track; an object in a video frame is detected using an object detection algorithm, the object detected by the detection algorithm defines an inspected object; depth information of at least a portion of the video frame is determined, the depth information including information about the distance between the inspected object and the calibrated camera system; based on the calibration data and the depth information, the inspected object is determined to be within the virtual timing zone; and, based on the calibration data, the depth information and one or more moments of one or more video frames of the inspected object, a passage time of the inspected object passing through the virtual timing zone within the virtual timing zone is determined.

[0038] In another embodiment, the executable operation may further include any of the method steps described above.

[0039] In yet another aspect, an embodiment may relate to a calibration module for a timing system configured to determine a passage time of an object across a timing line across a sports track, wherein the module may include: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of the timing system, each video frame representing a picture of a scene including a track and one or more calibration markers, preferably at least two calibration markers defined at positions on opposite sides of the track; determining depth information, such as one or more depth maps, based on the video frames, the depth information including information about distances between one or more objects in the picture of the video frames; using the depth information to determine a relative position between the one or more calibration markers and the camera system; determining calibration data, the calibration data including position information of a virtual timing line oriented across the track at a predetermined position relative to the position of the one or more calibration markers; and, position information associated with a virtual timing zone, the virtual timing zone being located above the track at a predetermined position relative to the position of the one or more calibration markers, the timing zone spanning and extending along the track; and storing coordinates of a virtual plane in a memory of the timing system.

[0040] Embodiments of the present invention may also relate to a computer program or a computer program suite, which includes at least one software code portion or a computer program product storing at least one software code portion, and when running on a computer system, the software code portion is configured to perform any of the method steps described above.

[0041] Embodiments of the present invention may also relate to a non-transitory computer-readable storage medium storing at least one software code portion, which, when executed or processed by a computer, is configured to perform the steps of any of the methods described above.

[0042] The embodiments will be further described with reference to the accompanying drawings, which schematically show embodiments according to the invention. It should be understood that the invention is not limited in any way to these specific embodiments.

[0043] BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a vision-based motion timing system is shown;

[0045] Figure 2 A 3D camera system configured to generate depth information is shown;

[0046] Figure 3 The principles of the parallax mapping technique for determining depth information are shown;

[0047] Figure 4 Detection, classification and identification of athletes based on images captured by a camera are shown;

[0048] Figure 5 A vision-based timing system is shown for determining the transit time of an athlete across a virtual timing line;

[0049] Figure 6 A vision-based timing system for determining a transit time of an athlete across a virtual timing line is shown according to one embodiment;

[0050] Fig. 7A and Figure 7B illustrates calibration and use of a calibrated vision-based timing system according to one embodiment;

[0051] Figure 8 A flow chart showing a method of calibrating a timing system according to one embodiment;

[0052] Fig. 9A and Fig. 9B A method of determining a transit time according to one embodiment is shown;

[0053] Fig.10A method of identifying an object passing through a virtual timing line according to one embodiment is shown;

[0054] Fig.11 A flow chart showing a method for determining a transit time according to one embodiment;

[0055] Fig.12 A system for determining transit time according to one embodiment of the present invention is shown;

[0056] Fig.13 is a block diagram illustrating an exemplary data processing system that may be used to execute the methods and software products described in this application. DETAILED DESCRIPTION

[0057] Figure 1 A vision-based sports timing system 100 is shown, which includes a computer 104 1,2 Controlling one or more camera systems 102 1,2 . Each camera system can be configured to capture a scene of the sports track 106 and determine depth information associated with the captured system. For example, in one embodiment, the depth information can include a so-called depth map of a video frame generated by an image sensor of the camera system. A video frame, such as an RGB video frame, whose depth map can be represented as a pixelated image including pixel values, which represent a distance value for each pixel of the video frame. The distance value can define the distance between the camera (the imaging plane of the camera) and an object in the video frame.

[0058] For example, a set of pixels in a video frame may be part of an object in a scene imaged by a camera system. In this case, the depth map may indicate the relative distance between the camera (viewpoint) and the surface of the object in the scene. Thus, during a series of time-stamped video frames capturing an object, such as an athlete or a vehicle moving along a sports track, the associated depth map may provide information about the distance between the moving object and the (static) camera system in the video frames as a function of time.

[0059] Camera systems capable of generating depth maps are known. For example, in one embodiment, the camera may be implemented as a 3D camera system, such as a stereo camera comprising two or more camera modules, each of which has its own lens system. Figure 2 204. As shown in the figure, a stereo camera 202 may be positioned along one side of a track 204. The stereo camera has (at least) two camera modules 204. 1,2 , which has a parallel optical axis 206 1,2 For observing objects such as athletes (such as Figure 1 The scene may also include a calibration mark 2101,2 In this way, two images of the same scene are obtained from different viewpoints. Such a stereo camera can be used to determine a depth map and / or a 3D picture. A 3D camera system can have more than two camera modules, so that a depth map can be calculated using multiple images. This can improve the accuracy of the depth map.

[0060] The 3D camera system (or the computer system controlling the camera system) may include a module for calculating a depth map based on video frames captured by two (or more) camera modules. In one embodiment, the module may use parallax mapping techniques to calculate images based on depth maps generated by two image sensors. Figure 3 The principle of parallax mapping is schematically shown. The 3D camera system includes two camera modules separated by a distance, which is called a baseline 308. The two camera modules can be synchronized so as to generate two video frames 302 of the same object from different viewpoints at each moment. 1,2 .

[0061] In order to calculate a depth map based on the video frames, a matching algorithm may be performed to match corresponding pixels of the left and right video frames. Thus, the object 300 imaged by the two synchronized camera modules is located at the same position 304. 1,2 , but separated by a baseline distance 308. In this case, the object will appear in a similar position in both images. The distance between the object in the left and right images is called disparity 306. Algorithms that construct a disparity map based on two images are called stereo matching algorithms. There are various stereo matching algorithms, which need to be both accurate and fast for real-time applications.

[0062] The camera system used in the embodiments of the present application is not limited to stereo-based imaging technology, and other imaging technologies may also be used. For example, a depth map can be generated based on RGB / infrared (IR) technology (such as used by Kinect) or 3D time of flight (TOF) technology, radar, or a combination thereof, where infrared radiation, such as an IR camera or an IR laser, can be used to determine the depth information of the scene captured by the camera. In some embodiments, the camera system can be a color infrared camera that is configured to capture images of the visible spectrum (e.g., RGB) and the infrared (IR) spectrum to capture images of the object. For example, when the lighting conditions are not optimal, such as due to weather conditions (such as fog, rain, etc.) or during dusk, it can be advantageous to use infrared images.

[0063] To increase the viewing angle of the camera system, in some embodiments, one or more wide-angle camera modules, such as a 180-degree camera or a 360-degree camera, can be used. In addition, for this type of video format, such as 360-degree video or immersive video generated using a special 360-degree camera system, where the video is projected onto a 2D video frame using, for example, an equirectangular projection, a depth map can be generated. To enable accurate image processing, the camera used in the timing system can have a frame rate of at least 30 frames per second or higher. Depending on the application, the frame rate can be at least 60 frames per second or higher. Additionally, the camera can be a high-resolution camera with an image resolution of 4K or higher.

[0064] like Figure 1 As shown, the camera system may include one or more camera modules 102 located on the side of the sports track and / or above the sports track. 1,2 The camera modules may be aligned and calibrated so that each camera module captures the same scene 114 of the moving track, including objects moving along the track. To this end, one or more calibration markers 110 may be used. 1,2 In one embodiment, such calibration markers may be located above the track and / or along one or both sides of the track. The one or more calibration markers may be used by the timing system to determine the coordinates of the virtual timing reference 112. The calibration markers may be physical objects arranged along the track, such as Figure 1 Alternatively, in one embodiment, the calibration markers may be geographical coordinates that define the coordinates of the virtual timing reference.

[0065] Based on the implementation, the virtual timing reference may define a virtual timing line or a virtual timing plane arranged at a distance z from the camera system. The virtual timing line may be used when determining passage based on two coordinates, such as an object passing through a timing line at position x, or when determining passage based on three coordinates, such as a portion of an object passing through a timing line at position x and height y. For example, the virtual timing plane may be used, for example, to determine a passage time when a specific portion of an object has passed through the virtual timing plane. For example, for certain use cases, it may be desirable to determine a passage time of a specific portion of an object, such as a chest of an athlete including a bib number, through a virtual timing plane.

[0066] In the following, whenever a virtual timing line is mentioned, other types of virtual timing references, such as a virtual timing plane, shall also be included. The timing line may be arranged across the track at the location of (or relative to) the one or more calibration marks. For example, when two calibration marks are used, the virtual timing line may be located between the two calibration marks such that it is arranged across the track between (or relative to) the two calibration marks.

[0067] The calibration process requires the 3D camera system to accurately detect the position and orientation of the calibration markers in all outdoor environments. Therefore, the calibration markers can be designed to have a predetermined unique shape and / or color combination so that during calibration, the object detection program can easily and accurately determine the position of the (edges of) the markers in the video frame so that the coordinates of the virtual plane can be accurately determined. When the (calibrated) timing system is running, the 3D camera system can capture video frames including athletes crossing the virtual plane. Although the figure shows a camera system along this side of the track, in other embodiments, one or more of the camera systems can be mounted above the sports track using a suitable mounting structure and / or integrated into a mat arranged across the sports track.

[0068] As will be described in more detail below, Figure 1 The timing system shown in is configured to determine the transit time of an object through a virtual plane. In addition, the timing system is configured to identify the object whose transit time is determined. The identification can be based on an identification mark associated with the object. For example, for an athlete, the identification mark may include (but is not limited to) a race bib number, color, biometric information, etc. For a vehicle, the identification mark may include a feature of the vehicle, such as color, shape, brand logo, etc.

[0069] The computer for controlling the one or more 3D camera systems and performing the calibration and timing methods may be implemented as a standalone computer or as a group of (wirelessly) connected computers. For example, a 3D camera system for determining a transit time based on a virtual plane across a racetrack may be controlled by a computer that includes a wireless interface for wireless communication with computers controlling other 3D camera systems.

[0070] like Figure 1 The multiple timing systems shown may be positioned at different locations along the track. In this way, each timing system may determine a transit time based on a video frame of an object passing through the virtual plane. The timing systems may support a wireless protocol that allows a mesh network of timing systems to be established along the track. In this way, information determined by one timing system may be used by one or more other timing systems that are part of the mesh network.

[0071] The data processing methods used by the timing system to calculate depth maps and analyze video frames may require real-time imaging processing, so in some embodiments, the computer system 104 1,2One or more dedicated processors, such as a GPU, may be included to perform the computationally intensive portion of the calibration and timing process. In other embodiments, the one or more 3D camera systems may be connected to a cloud resource that may run the computationally intensive portion of the process. A CPU clock or a GPS clock may be used to link the video frames with the time information. For example, in one embodiment, each or at least a portion of the video frames may be linked to a time instance by time stamping the video frames.

[0072] Figure 1 The timing system in can be run based on a single 3D camera. Alternatively, two or more 3D camera systems at different viewpoints can be used to capture video frames of a scene. In this case, the 3D camera systems can be time synchronized so that the time-stamped video frames of the different 3D camera systems can be easily combined and analyzed. Multiple 3D camera systems can be used to cope with collisions, i.e., events where multiple objects pass through a virtual plane.

[0073] The computer system of the timing system may include one or more image processing units configured to execute various image processing algorithms. For example, in one embodiment, the computer system may be configured to execute an object detection algorithm for detecting objects in images captured by the camera system. In particular, an instance object segmentation algorithm may be used to detect objects in an image and distinguish them from other objects in the image. Figure 4 As shown, instance segmentation allows detecting multiple objects of the same category 400 1-6 , such as athletes or vehicles, where each detected object can be considered as a separate instance. The instance segmentation algorithm can determine a bounding box defining a region of interest 401 in the image including the detected object (for clarity, only the region of interest of the first object is shown in the figure). The region of interest including the detected object can be marked with an object identifier ID, for example, a unique number assigned to the object by the algorithm, which is used by the system to distinguish different detected objects of the same category in one image.

[0074] In this way, objects in the image, such as athletes, cars, bicycles, etc., can be accurately and efficiently detected, classified, and represented as marked regions of interest in the image. In one embodiment, information about the region of interest and ID can be added or linked to the image as metadata. Efficient instance object segmentation algorithms based on deep learning can be used, including but not limited to algorithms such as R-CNN, Fast R-CNN, or Faster R-CNN or related algorithms, which can detect and classify objects in the image in (almost) real time. The article "A survey on Instance Segmentation: State of The Art (Instance Segmentation Research: Current Situation Review)" by A. Hafiz et al., "International Journal of Multimedia Information Retrieval", Vol. 9, pp. 171-189 (2020) reviews the current state-of-the-art instance segmentation algorithms, which are hereby incorporated by reference into this application. These algorithms provide very efficient runtime performance so that they can be executed locally, such as on a computer using a timing system.

[0075] To accelerate the object detection and recognition process, the CPU-based local computer of the timing system running the image processing algorithms may include one or more specialized processors, such as one or more graphics processing units GPUs, tensor processing units TPUs, or field programmable gate arrays FPGAs, which may be specifically adapted to accelerate the calculations performed by various image processing algorithms, particularly those associated with neural networks.

[0076] In addition, in one embodiment, the computer system can be configured to perform an object recognition algorithm. Here, object recognition refers to the process of assigning an identity to the detected object based on these object features, and the identity is typically personal data (for example, the name of a person, animal, vehicle or team stored in a database before the event occurs). The example of an object feature can be a number or code (for example, a part of a QR code) on a participant's bib. When a participant registers in a sports event, he will receive identity information, such as an identification number such as a bib number or code, which can be used by a timing system to link data, such as information about passing time, to personal data stored in a central database, such as the name of the participant who registered. Different object features can be used to link an object to an identity. For example, if an object represents a person, object features can include (but not limited to) (a part of) a race bib, color, biometric information of the person, visual identification code, such as a QR code, object items such as shoes, shirts, brand logos, etc. If an object represents a vehicle, object features can include features of the vehicle, such as number, color, shape, brand logo, license plate or registration plate, etc. The computer may include or be connected to a wireless radio frequency module for wirelessly connecting the timing system to another computer system (not shown), such as a remote computer system, such as a server system, which is configured to centrally process the determined transit times of inspected and identified objects.

[0077] like Figure 4 As shown, the detected object in the image can be identified based on one or more object features, that is, associated with an identity, such as an identification mark 404 specifically used to identify the object. 1-3 (e.g. bib number, QR code, printed content, symbol, etc.), biometric information 408 1-6 (e.g., facial features, age, gender, length, hair color, etc.) or object items 406 that are specific to a particular object 1-6 (eg, color and / or design of clothing and / or shoes or color and / or shape of a vehicle.) These object features may define visual identification information that can be used by the timing system to identify objects passing through the virtual timing line.

[0078] The region of interest can be analyzed by one or more algorithms, which are configured to detect object features in the region of interest and extract identification information from the region of interest. For example, an optical character recognition (OCR) algorithm can be used to identify the (printed) identification mark in the region of interest. Typically, the OCR algorithm can include a text detection algorithm and a text recognition algorithm. These printed identification marks can be based on characters and / or symbols, such as example numbers and / or characters on a number cloth or a vehicle. Alternatively, and / or additionally, a code detection algorithm can be used to detect and encode the visual code in the region of interest, typically a standardized code. For example, an athlete's number cloth or clothes can include a QR code or its variant, in which the identification information is encoded as (for example) a geometric structure. In addition, facial detection and recognition algorithms can be used to identify people in the region of interest based on facial features.

[0079] In some embodiments, the posture estimation algorithm is configured to identify the posture of the object. For example, in one embodiment, the posture estimation algorithm can determine the key joints of the object (shoulders / hips / knees / head), which can be used to identify the positions in the region of interest that may represent object identification features, that is, features of the object that can be used for identification. For example, based on the position of the key joints, specific parts of the body can be determined, such as the chest, arms, head, legs, or feet. Based on the position of the key joints, it can be determined that the region of interest is likely to include one or more object features, such as the position of the face, chest, or feet.

[0080] When using vision-based timing systems to time mass events such as marathons, the timing area is often crowded with athletes and others not attending the sporting event, such as spectators or refereeing staff. Figure 5 A calibrated timing system is schematically shown, the system comprising a computer controlled camera system 504 1,2 , for capturing an image of an area of ​​the track that includes a virtual timing line 506. Typically, the field of view (FOV) 502 of the camera system is selected so that the image includes a large portion of the track behind the virtual timing line, so that the image includes all athletes approaching the timing line. In this way, the object detection algorithm can detect objects in the image and track the objects in time. Typically, the object detection algorithm is a fast object detection and classification algorithm that can detect and classify people in the image. In this case, people include not only athletes approaching the timing line 510 1-5 , including, for example, spectators and referees 508 1-8 In this case, most of the computer resources are used to detect objects that are not involved in the timing process. Additionally, the image processing system will apply object detection and recognition algorithms to people who have not consented to be captured and processed by the image processing algorithm. Depending on the country where the timing system is used, this may pose privacy issues.

[0081] To solve this problem, a spatial filtering technique based on depth information is used to filter out the detected objects. Figure 6 Shown schematically. Figure 6 A camera system 604 including one controlled by a computer is shown. 1,2 A calibrated timing system is provided, but in this case the system is calibrated so that it includes not only the coordinates of a virtual timing line 610 across the track at a predetermined distance from the camera system, but also the coordinates of an area, such as a 2D area or a 3D area, which is located above and extends along the track. Typically, the virtual timing line may be located within this area, which may be referred to as a virtual timing zone 606 hereinafter. Where the coordinates of the virtual timing zone define a 3D volume, the virtual timing zone may define a volume of interest (VOI) located above the track. Thus, during calibration of the timing system, position information, such as coordinates or information determining the coordinates of a virtual timing zone, which spans and extends along the track, may be determined. The position information may be stored in a memory of the timing system so that it may use the information to determine the time at which an object, such as an athlete, passes through the virtual timing line. Preferably, the virtual timing zone defines a portion of the track where only the athlete 610 is allowed to pass. 1-5 Therefore, when the timing system detects an object within the virtual timing zone, the object is likely to be a player. Therefore, using the virtual timing zone, the timing system can effectively determine whether the detected object in the image is a player.

[0082] As will be described in more detail below, the calibrated timing system is configured to capture images and depth information associated with the images. In addition, it is configured to detect objects in the images and use the depth information to determine whether the detected objects are within the virtual timing zone. If so, the computer will track the detected objects within the virtual timing zone and execute a recognition algorithm to identify these objects (athletes on the track) detected within the virtual timing zone, without processing detected objects located outside the virtual timing zone. In this way, the computer will only process, such as timing and identifying, athletes within the virtual timing zone that will pass through the virtual timing line. The computer will ignore objects outside the virtual timing zone, such as spectators and / or referees. In this way, the privacy of spectators is respected, and athletes participating in the sporting event (who agree to be timed by a vision-based timing system) are timed without wasting computer resources.

[0083] Fig. 7A and Figure 7B The calibration of a timing system and the operation of a calibrated timing system according to one embodiment of the present invention are shown. Fig. 7AAs shown in FIG. 7 , the camera system 702 is located on one side of a track 706. At a predetermined distance 706, one or more calibration markers 710 are located. 1,2 The camera may be positioned on one side of the track or on both sides of the track 700. The field of view (FOV) of the camera may be positioned toward the track so as to capture a scene including the one or more calibration marks. A calibration procedure is then performed by the computer 704 of the camera system. During the calibration procedure, the camera generates video frames and, based on the generated video frames, determines depth information, such as a depth map (e.g., a reference image) of the scene including the one or more calibration marks. Figure 2 and Figure 3 Furthermore, an object detection program may be used to detect the one or more calibration markers in a video frame (eg, an RGB picture).

[0084] Any type of marker may be used for calibration, such as existing objects on the side of the track, such as trees, traffic signs or poles. In another embodiment, geographic coordinates may be used as calibration markers. In this case, the camera may be associated with the geographic coordinates using, for example, GPS, and a line or plane across the track visible in the camera's FOV may be defined by one or more geographic coordinates relative to the camera's geographic coordinates.

[0085] Alternatively and / or additionally, specially designed calibration markers may be used. For example, in one embodiment, the calibration markers may be designed with features that allow accurate calibration under different outdoor conditions. For example, the shape, edges, and / or color of the markers may be designed to allow accurate detection in pictures. A depth map associated with a video frame may be used to determine the distance between the camera and the detected calibration markers. Alternatively, if a sufficiently accurate depth map can be constructed, the object detection program may also determine the position of the calibration markers directly based on the depth map.

[0086] Once the computer has detected the location of the one or more markers, it can determine a virtual timing line across the track 700. The virtual timing line can be a virtual line 709 or a virtual plane 711 across the track at a predetermined position relative to the location of the one or more markers. In one embodiment, the virtual timing line can be a line extending from a marker located on a first side of the track in a direction across the track to the other side of the track. In another embodiment, the virtual timing line can extend from a first marker on one side of the track to a second marker on the other side of the track. The virtual timing line or virtual timing plane can be used as a location for the timing system to determine the passing time. In one embodiment, the virtual line or plane can be located within a rectangular 3D volume 712 in space, where the width of the volume can be determined by the calibration markers and the height and depth of the volume can be determined by the computer. The 3D volume can define a 3D detection area in which the timing system will acquire video data (e.g., multiple video frames) to accurately determine the passing time and identify objects associated with the passing time.

[0087] In addition to the timing lines, the calibration process can also be used to determine a virtual timing zone 706 that spans the track and is positioned along the track at a predetermined position relative to the one or more markers. In one embodiment, the virtual timing zone may define a 3D volume of interest VOI where the virtual timing line or plane is located. In another embodiment, the virtual timing zone may be a 2D virtual timing area where the virtual timing line or plane is located. In one embodiment, the one or more markers may define the position and / or (partial) size of the virtual timing zone. For example, after positioning and detecting the markers, the calibration procedure may generate a virtual timing zone having a certain width and length and an optional height based on the positions of the one or more markers. Since the markers are positioned along the edge of the track, the virtual timing zone may have sides that are substantially parallel to the edge of the track. In this way, position information, such as coordinates, of the virtual timing zone located above the track can be determined using the one or more markers during calibration and stored in the memory of the timing system. In this way, Figure 7B The calibrated timing systems 702, 704 shown can use the virtual timing region to spatially filter out detected objects 716 outside the virtual timing region in the image. 1-3 , and for objects within the virtual timing zone 714 1,2 Tracking, identification and timing.

[0088] Figure 8 A flow chart of a method for calibrating a timing system according to an embodiment of the present invention is shown. The method can be used to calibrate a timing system including a reference Fig. 7AAs shown in the flowchart, the method may begin by pointing a 3D camera system of the timing system at one or more calibration marks located on opposite sides of the track. The calibration marks may indicate the location of a virtual plane located between the calibration marks and across the sports track (step 802). In the next step 804, the 3D camera system may capture a sequence of video frames, where each video frame may represent a picture of a scene including the track and the one or more calibration marks.

[0089] Thereafter, a depth map may be determined based on the video frames, and the depth map may include information about the distance between one or more objects in the picture and the 3D camera system (step 806). For example, a depth map may be generated based on two video frames generated by two camera modules in a stereo camera, and parallax mapping may be used to generate the depth map in the same manner as described above with reference to FIG. Figure 2 and Figure 3 The depth information may include a depth map that includes, for each video frame (or at least a substantial portion thereof), information about the distance between an object in the captured video frame and the 3D camera system. The depth information may be used to determine the relative position (distance) between the one or more calibration markers and the 3D camera system (step 808).

[0090] Coordinates of a virtual timing line across the track may then be calculated, the coordinates being positioned relative to the one or more calibration markers, such as between the calibration markers at opposite sides of the track. The virtual timing line may be used by a timing system to determine a transit time of an object through a virtual plane (step 810). Additionally, based on the positions of the calibration markers, position information of a virtual timing zone, such as a 2D region or 3D volume, oriented on the sports track, such as coordinates or information for calculating coordinates, may be calculated (step 812). In one embodiment, the virtual timing line may be located within a virtual timing zone that defines an area in which an object is tracked, identified, and timed as it passes through the virtual timing line. The coordinates of the virtual timing line and the virtual timing zone may be stored in a memory of the timing system (step 814).

[0091] In operation, the calibrated timing system no longer requires calibration marks. The calibrated camera system will use the calibration data defining the virtual timing line and the virtual timing zone to effectively determine the transit time and one or more identification features of an object passing through the virtual timing line.

[0092] It is believed that Figure 8 The calibration process shown is only one of many different ways to calibrate a camera relative to a line or plane used to determine the time of passage. For example, calibration of the camera system may be accomplished based on geo-locating the camera system and one or more geographic coordinates defining an imaginary line or plane that crosses the track at a predetermined distance from the camera.

[0093] The calibration process can be used to install and calibrate one or more other camera systems along the track so that each of these camera systems can capture video frames of objects passing through the timing line. The camera systems can communicate with each other (wirelessly) to enable time synchronization of the video capture process. In this way, at a moment in time, each camera system will process one or more time-stamped video frames of the moving track. The time-stamped video frames (and associated depth maps) of different perspectives can be used to determine the passage time of objects through the virtual plane, and identify objects whose passage time has been determined.

[0094] Fig. 9A and Fig. 9B The operation of a timing system according to one embodiment of the present invention is shown. Fig. 9A Three snapshots are shown of a moving object (in this case an athlete) passing through the 3D detection area of ​​a timing system as described with reference to an embodiment of the present application. Fig. 7A The calibration method described above establishes a 3D detection area. Fig. 9B As shown, the 3D detection zone 912 may include a virtual plane 914 located across the track, wherein the normal of the virtual plane is substantially parallel to the direction of the moving track. The virtual plane divides the 3D detection zone into a first portion 9161 where the object moves toward and through the virtual plane and a second portion 9162 where the moving object passes through the virtual plane and moves away from the virtual plane.

[0095] Thus, as the object moves along the track, the 3D camera system will capture images (pairs of images in the case of a stereo camera) of the scene including the 3D detection zone. For each image (video frame), the 3D imaging system may compute a depth map. Object detection and tracking algorithms may be used to detect and track a predetermined object, such as a human subject or an object representing an object, in subsequent video frames.

[0096] Based on the depth map, the computer can determine that the detected object has entered the first portion of the 3D detection area. In this case, the computer can begin storing video frames and associated depth maps in a buffer until the object leaves the 3D detection area via the second portion. In another embodiment, only the video frame pairs are stored and the depth map is determined later. These video frames and depth maps can be used by the computer to determine the passing time and identify the object associated with the passing time.

[0097] Fig. 9A Three samples 902 of a sequence of time-stamped video frames are shown as an athlete moves through a 3D detection zone 1-3 These video frames can be captured by a camera system and stored in a computer. Fig. 9BAs shown, the first video frame 9021 is captured at time T1, i.e., when the majority of the athlete's body is still in the first portion 9161 of the 3D detection zone. At time T2, a second video frame 9022 is captured, in which the athlete has moved further and passed through the virtual plane 914 of the 3D detection zone. Finally, at time T3, a third video frame 9023 is captured, in which the athlete has moved to the second portion 9162 of the 3D detection zone.

[0098] A passing time module in the timing system's computer can analyze a sequence of time-stamped video frames to determine at what moment the athlete passed through the virtual plane. To this end, an object detection and classification algorithm can be applied to each video frame. The algorithm can determine, for example, regions of interest in the video frame that belong to an object 908 1-3 (ROI). In addition, for each of these regions of interest, the algorithm can classify pixels as belonging to a player (background) or not.

[0099] In addition, the depth map associated with each of the video frames can be used to determine distance values ​​belonging to pixels that are classified as belonging to an object. These distance values ​​can be compared to the distance between the camera and the virtual plane. In this way, when the 3D camera system captures an object that crosses the virtual plane, for each video frame, the portion of pixels of the object that has crossed the virtual plane can be determined. This is Fig. 9A , where the grey areas define the pixels of (parts of) the object that have crossed the virtual plane.

[0100] For the video frame of time T1, only the pixel 9041 representing a part of the hand and the pixel 9061 representing the athlete's shoes are associated with a distance value less than the distance between the virtual plane and the 3D camera system. Similarly, for the video frame of time T2, the pixel 9041 representing a part of the upper body and the pixel 9062 representing a part of the leg are associated with a distance value less than the distance between the virtual plane and the 3D camera system. Finally, for the video frame of T3, all pixels 908 representing the athlete are associated with a distance value less than the distance between the virtual plane and the 3D camera system. Based on this analysis, the computer can determine that at T2, most of the athlete's body has crossed the virtual plane. For example, the computer can determine that if the object has passed through a part of the virtual plane greater than a certain threshold, in this case, it is determined that the athlete has passed through the plane. Therefore, in this case, the timestamp T2 can define the passing time 510, which is 2:34 in this example. Different rules can be defined to determine whether the object has passed through the virtual plane.

[0101] Fig.10A method for identifying a timed object according to one embodiment of the present invention is shown. Once the time of passage of an object has been established, the identity of the object can be established. To this end, the stored video frames can be analyzed by an object recognition module. A region of interest (ROI) in the video frame can be subjected to a feature analysis process that searches for predetermined features in the region of interest. For example, Fig.10 As shown, Fig. 9A The region of interest 1003 of the video frame T3 1002 may be subjected to a feature analysis algorithm which may be configured to recognize predetermined features, such as predetermined markings, e.g. a bib number 10041 or a QR code 10042. Any unique identification feature, including a shoe type 10043, a biometric marker, a colored or specially shaped marker, or a combination thereof may be used. During passage through the 3D detection zone, analysis may be performed based on a set of regions of interest from different video frames. It may often be the case that a video frame used to determine the time of passage is not suitable for recognition purposes. Therefore, the object recognition module may determine a video frame with a timestamp (e.g. Fig. 9A In the example of the video frame at T2 (in the example of the video frame at T2), the picture showing the object crossing the virtual plane is not suitable for reliable object recognition. In this case, it can determine that one or more other video frames (such as Fig. 9A Any known object detection algorithm can be used to identify features in the region of interest, including conventional computer vision algorithms, machine learning, and deep learning techniques. For example, in one embodiment, one or more deep neural networks can be used to determine a region of interest in a video frame that includes an object, and to determine certain features associated with the object.

[0102] Fig.11 A flow chart for determining a passing time using a calibrated vision-based timing system according to an embodiment described in the present application is shown. As shown in the figure, the method can begin by receiving a sequence of video frames captured by at least one camera system, each video frame representing a scene picture of an object, such as a person, animal, or vehicle, moving along a track (step 1102). Each of these video frames can be associated with a real-time clock. For example, each video frame can be time-stamped using the clock of the timing system.

[0103] In a further step 1104, the method may include determining or receiving calibration data, such as predefined calibration data, which may define a virtual timing zone located on the track at a predetermined distance from the calibrated camera system, wherein the virtual timing zone includes a virtual timing line. In one embodiment, the calibration data may include position information, such as coordinates, of the virtual timing zone, the virtual timing zone being, for example, a 2D area above the track or a 3D volume located above and along the track. The calibration data may be stored in a memory of the timing system and retrieved when performing a timing process.

[0104] An object detection algorithm may be used to detect objects in the video frames, with the objects detected by the detection algorithm defining one or more detected objects (step 1106). Thus, within each video frame, a plurality of objects may be identified using a known object detection algorithm, and the relative position between the identified objects and the camera system may be determined. In one embodiment, the object detection step may include determining a region of interest ROI that includes the objects, and for each region of interest, performing a classification algorithm on pixels in the region of interest to classify whether the pixels represent part of an object or part of a background.

[0105] In addition, depth information such as one or more depth maps may be determined for the sequence of video frames, wherein the depth information may include information about the position of detected objects in the video frames relative to the camera system (step 1108). Typically, the 3D camera system may be a stereo camera, thereby generating a pair of video frames that can be used to determine the depth map. Alternatively, any other camera system capable of generating depth information associated with the captured video frames may be used.

[0106] Then, the calibration data and the depth information can be used to determine whether the detected object is located in the virtual timing zone (step 1110). Finally, the passing time of the detected object passing through the virtual timing zone in the virtual timing zone can be determined based on the calibration data (particularly the position information of the timing line), the depth information, and one or more moments of one or more video frames of the detected object (step 1112).

[0107] The method provides an efficient way to determine the passing time using a vision-based timing system for crowded timing locations, such as the finish line of mass sporting events such as marathons. A virtual timing zone is used to spatially filter out most of the detected objects that are not participating in the timed sporting event so that computing resources are only allocated to the detection, timing, and recognition of athletes captured by the timing system's 3D camera. The method also provides a privacy-friendly vision-based timing scheme because it only applies the recognition algorithm to the detected objects that represent athletes in the video frame.

[0108] The object's passage through the virtual timing line can be determined by comparing the distance between the 3D camera system and the virtual timing line with the distance between the 3D camera system and the detected object within the virtual timing zone. The passage time can then be determined based on time information, such as a moment associated with a video frame including the detected object passing through the virtual timing line, such as a timestamp. For example, to this end, one or more video frames can be determined in which a portion of the object that has passed through the virtual plane has a specific size.

[0109] Therefore, specific rules can be used to determine whether an object has passed through the virtual plane. The moment associated with the video frame describing the situation, such as a timestamp, defines the passing time. Thereafter, a feature analysis algorithm can be applied to the one or more detected objects in the video frame, which determines the identification features of the one or more detected objects in the video frame, and determines the identity of the detected object whose passing time has been determined based on the identification features of the detected object that has passed the timing line.

[0110] In one embodiment, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, which is trained to detect one or more objects in a video frame and determine identification features associated with the detected objects.

[0111] Thus, different pictures of the sequence of video frames may be used to identify an object that has passed through the virtual plane at the time of passage.Thus, the video frames used to determine the object's passage time may be different from the one or more video frames used to determine the object's identity.

[0112] Fig.12 A system for determining the time of passage according to an embodiment of the present invention is shown. As shown in the figure, the system includes one or more camera systems 1201 connected to a computer system. 1,2 , the computer system is configured to control the camera system and process the video frames generated by the camera system 1203 1,2. Each of the camera systems is calibrated according to the calibration method described in the present application so that it captures a scene of the track including a timing line represented by a moving object passing through a virtual plane 1202. The computer system may include a race analyzer 1204 for object detection and feature analysis of objects in a video frame using a suitable algorithm. The race analyzer may also determine which detected objects are located within a virtual timing zone position on the track. The calibration data stored in the system's memory may include position information associated with the virtual timing zone. In addition, the race analyzer may generate depth information associated with the video frame, such as a depth map. The passing module 1206 may determine the passing time of the detected object within the virtual timing zone based on the depth information and position information associated with the virtual timing line. In addition, the object recognition module 1208 may identify an object detected by the passing time module and associated with the passing time. The passing time and ID of the object that has passed may be stored in the memory of the computer. Additionally, pictures of the passing of the timing object may be stored. This information may be communicated with the network server 1214 via the (wireless) interface 1212.

[0113] Fig.13 1300 is a block diagram illustrating an exemplary data processing system described in the present disclosure. Data processing system 1300 may include at least one processor 1302 coupled to storage element 1304 via system bus 1306. Therefore, the data processing system may store program codes in storage element 1304. In addition, processor 1302 may execute program codes accessed from storage element 1104 via system bus 1306. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program codes. However, it should be understood that data processing system 1300 may be implemented in the form of any system including a processor and a memory capable of performing the functions described in this specification.

[0114] Storage element 1304 may include one or more physical memory devices, such as local memory 1308, and one or more mass storage devices 1310. Local memory may refer to random access memory or other non-persistent storage devices that are typically used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Processing system 1300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code to reduce the number of times program code must be retrieved from mass storage device 1310 during execution.

[0115] Input / output (I / O) devices, shown as input devices 1312 and output devices 1314, can optionally be coupled to a data processing system. Examples of input devices may include, but are not limited to, for example, keyboards, pointing devices such as mice, etc. Examples of output devices may include, but are not limited to, for example, monitors or displays, speakers, etc. Input devices and / or output devices may be coupled to a data processing system directly or through an intermediate I / O controller. Network adapter 1316 may also be coupled to a data processing system to enable it to be coupled to other systems, computer systems, remote network devices, and / or remote storage devices through an intervening dedicated or public network. A network adapter may include a data receiver for receiving data sent to the data by the system, device, and / or network and a data transmitter for sending data to the system and / or device. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with data processing system 1100.

[0116] like Fig.13 As shown, storage element 1304 can store application 1118. It should be understood that data processing system 1300 can also execute an operating system (not shown) that can facilitate the execution of application programs. Application programs implemented in the form of executable program code can be executed by data processing system 1300, for example, by processor 1302. In response to executing the application program, the data processing system can be configured to perform one or more operations that will be further described in detail herein.

[0117] In one aspect, for example, data processing system 1300 may represent a client data processing system. In this case, application 1318 may represent a client application that, when executed, configures data processing system 1300 to perform various functions described herein with respect to a "client." Examples of a client can include, but are not limited to, a personal computer, a laptop, a mobile phone, etc.

[0118] In another aspect, the data processing system may represent a server. For example, the data processing system may represent a (HTTP) server, in which case, when executed, the application 1318 may configure the data processing system to perform (HTTP) server operations. In another aspect, the data processing system may represent a module, unit, or function mentioned in this specification.

[0119] The terms used herein are only used to describe specific embodiments and are not intended to limit the present invention. Unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" used herein are intended to include plural forms as well. It should be further understood that when the terms "include" and / or "comprise" are used in this specification, the presence of the stated features, wholes, steps, operations, elements and / or parts is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or groups thereof is not excluded.

[0120] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the following claims are intended to include any structure, material, or act that performs the function in combination with other elements specifically claimed. The description of the present invention is presented for the purpose of illustration and description, but is not meant to be exhaustive or limited to the invention in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiments selected and described are intended to best explain the principles and practical applications of the invention and to enable those of ordinary skill in the art to understand the various embodiments of the invention and various modifications suitable for the intended specific use.

[0121] The terms used herein are only used to describe specific embodiments and are not intended to limit the present invention. Unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" used herein are intended to include plural forms as well. It should be further understood that when the terms "include" and / or "comprise" are used in this specification, the presence of the stated features, wholes, steps, operations, elements and / or parts is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or groups thereof is not excluded.

[0122] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the following claims are intended to include any structure, material, or act that performs the function in combination with other elements specifically claimed. The description of the present invention is presented for the purpose of illustration and description, but is not meant to be exhaustive or limited to the invention in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiments selected and described are intended to best explain the principles and practical applications of the invention and to enable those of ordinary skill in the art to understand the various embodiments of the invention and various modifications suitable for the intended specific use.

Claims

1. A method for determining the time at which an object passes through a virtual timing line, the method include: receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving on a racetrack and objects outside the racetrack, each video frame being associated with a time instant, the calibrated camera system including calibration data defining a virtual timing zone in the video frames located at a predetermined distance from the calibrated camera system, the virtual timing zone spanning the racetrack and extending along an edge of the racetrack; Detecting objects in the video frame using an object detection algorithm, wherein the objects detected by the detection algorithm define detected objects; determining depth information for at least a portion of a video frame, the depth information comprising information about a position of the detected object relative to the calibrated camera system; determining, based on the calibration data and the depth information, whether the detected object is located within the virtual timing area or outside the virtual timing area; applying a feature analysis algorithm to at least a portion of the detected objects determined to be located within the virtual timing zone in the video frame, the feature analysis algorithm determining identification features of the one or more detected objects within the virtual timing zone; and A passing time of the detected object determined to be located within the virtual timing zone and passing a virtual timing line across the racetrack is determined based on the calibration data, the depth information, and one or more times of one or more video frames of the detected object passing the virtual timing line.

2. The method according to claim 1, wherein the width of the virtual timing zone is equal to or greater than the width of the track, wherein the length of the virtual timing zone is equal to or greater than the width of the track; and / or, the virtual timing zone is substantially rectangular, wherein the long side of the rectangular virtual timing zone is aligned with the side of the track.

3. The method according to claim 2 or 3, further comprising: include: The identity of the detected object is determined based on the identification feature of the detected object, preferably based on the identification feature of the detected object passing the timing line.

4. A method according to claim 2 or 3, wherein the identification features of the object under inspection include one or more identification marks, preferably optically readable identification marks, such as race bibs or printed marks and / or codes; and / or, one or more features regarding the shape and / or color of the object under inspection.

5. The method according to any one of claims 2 to 4, wherein the identification feature of the detected object comprises one or more biometric identifiers of the detected object.

6. The method according to any one of claims 1-5, wherein the object detection algorithm is part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, which is trained to detect one or more objects in one or more video frames.

7. A method according to any one of claims 2-6, wherein the feature analysis algorithm is part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, which is trained to determine identification features associated with the object being inspected.

8. The method according to any one of claims 1 to 7, wherein the calibration data comprises coordinates of a 2D area on the track, the 2D area defining a 2D virtual timing zone, wherein sides of the 2D virtual timing zone extend substantially parallel to sides of the track, preferably, sides of the 2D virtual timing zone substantially coincide with sides of the track.

9. The method according to any one of claims 1 to 7, wherein the calibration data comprises coordinates of a 3D area on the track, the 3D area defining a 3D virtual timing zone, wherein sides of the 3D virtual timing zone extend substantially parallel to sides of the track, preferably, sides of the 3D virtual timing zone substantially coincide with sides of the track.

10. The method according to any one of claims 1 to 9, wherein determining the detected object located in the virtual timing zone include: determining a position of the object relative to the camera system using the depth information; Whether the object is located in the virtual timing zone or outside the virtual timing zone is determined based on the position and coordinates of the virtual timing zone.

11. The method according to any one of claims 1 to 10, wherein detecting one or more objects in the video frame can be include: Determine one or more regions of interest (ROIs) in a video frame, each region of interest including pixels representing an object; determining an object characteristic in one of the one or more regions of interest; as well as, An object in the region of interest is determined based on the determined object features.

12. The method according to any one of claims 1-11, wherein the camera system comprises a plurality of camera modules, preferably two camera modules forming a stereo camera, the stereo camera being configured to generate at least a first video frame and a second video frame of the scene at each moment, wherein the depth map is determined based on a disparity mapping algorithm, the disparity mapping algorithm being configured to determine the disparity between pixels of the first video frame and the second video frame.

13. The method according to any one of claims 1 to 12, wherein the passage time is determined based on at least one video frame of the scene, wherein a predetermined portion of the detected object in the scene has passed through the virtual plane.

14. A method for calibrating a timing system, the timing system being configured to determine a transit time of an object across a timing line across a sports track, the method include: receiving a video frame, preferably a sequence of video frames, captured by a camera system of the timing system, each video frame representing a picture of a scene including the racetrack and one or more calibration marks, preferably at least two calibration marks located on opposite sides of the racetrack; determining depth information, such as one or more depth maps, based on the video frame, the depth information comprising information about distances between one or more objects in a picture of the video frame; determining a relative position between the one or more calibration markers and the camera system using the depth information; determining calibration data, the calibration data comprising position information of a virtual timing line oriented across the racetrack at a predetermined position relative to one or more calibration marker positions, and position information associated with a virtual timing zone located above the racetrack at a predetermined position relative to the one or more calibration marker positions, the timing zone spanning and extending along the racetrack; as well as, The coordinates of the virtual plane are stored in a memory of the timing system.

15. A method according to claim 14, wherein the calibration data includes coordinates defining the virtual timing zone, preferably coordinates defining a 2D area above the track, wherein the sides of the 2D area are substantially parallel to the sides of the track; or, coordinates defining a 3D area above the track, wherein the side surfaces of the 3D area are substantially parallel to the sides of the track.

16. A system for determining the time at which an object passes a timing line across a sports track, wherein include: at least one camera system connected to the computer; The computer comprises a computer-readable storage medium containing computer-readable program code, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium. In response to executing the computer readable program code, the processor is configured to perform executable operations, the executable operations comprising: receiving video frames captured by a calibrated camera system, each video frame representing a picture of a scene including one or more objects moving on a racetrack and objects moving off the racetrack, each video frame being associated with a time instant, the calibrated camera system including calibration data defining a virtual timing zone in the video frames located at a predetermined distance from the calibrated camera system, the virtual timing zone spanning the racetrack and extending along an edge of the racetrack; Detecting objects in the video frame using an object detection algorithm, wherein the objects detected by the detection algorithm define detected objects; determining depth information for at least a portion of a video frame, the depth information comprising information about a position of the detected object relative to the calibrated camera system; determining, based on the calibration data and the depth information, whether the detected object is located within the virtual timing area or outside the virtual timing area; applying a feature analysis algorithm to at least a portion of the detected objects in the video frame that are determined to be located within the virtual timing zone, the feature analysis algorithm determining identification features of the one or more detected objects within the virtual timing zone; and A passing time of the detected object determined to be located within the virtual timing zone and passing a virtual timing line across the racetrack is determined based on the calibration data, the depth information, and one or more times of one or more video frames of the detected object passing the virtual timing line.

17. The system of claim 115, wherein the executable operations further comprise any of the method steps of claims 2-9.

18. A calibration module for a timing system configured to determine a transit time of an object across a timing line across a sports track, the module include: receiving a video frame, preferably a sequence of video frames, captured by a camera system of the timing system, each video frame representing a picture of a scene including the racetrack and one or more calibration marks, preferably at least two calibration marks located on opposite sides of the racetrack; determining depth information, such as one or more depth maps, based on the video frame, the depth information comprising information about distances between one or more objects in a picture of the video frame; using the depth information to determine a relative position between the one or more calibration markers and the camera system; determining calibration data, the calibration data comprising position information of a virtual timing line oriented across the racetrack at a predetermined position relative to the position of the one or more calibration marks, and position information associated with a virtual timing zone located above the racetrack at a predetermined position relative to the position of the one or more calibration marks, the timing zone spanning and extending along the racetrack; as well as, The coordinates of the virtual plane are stored in a memory of the timing system.

19. A non-transitory computer-readable storage medium storing at least software code portions, which, when executed or processed by a computer, are configured to perform the steps of the method according to any one of claims 1-15.