Method and system for stabilizing an output from a video of a sport event

Abstract

This disclosure relates to computer-implemented method of stabilizing an output from a video of a sport event. The method comprises the steps of: providing raw recordings of the sport event at a sport area from a first and second recording units; providing camera calibration parameters for the recording units; determining first feature points in a first image and second feature points in a second image of the recordings; determining first and second directions based on the camera calibration parameters, the first and second directions indicative of directions towards first and second feature points relative to a common viewpoint of the recording units; calculating an angular displacement between the second directions and the first directions; and establishing the output based on directional tracing of contents in the raw recordings relative to the common viewpoint, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

Description

[0001] METHOD AND SYSTEM FOR STABILIZING AN OUTPUT FROM A VIDEO OF A SPORT EVENT

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to a computer-implemented method of stabilizing an output from a video of a sport event. The invention further relates to a computer program for stabilizing an output from a video of a sport event at a sport area. Moreover, the invention relates to a system for stabilizing an output from a video of a sport event at a sport area.

[0004] BACKGROUND OF THE INVENTION

[0005] Movement of a video camera while the video camera is recording will typically result in a corresponding motion and vibration being visible in the recorded video, which is often unintended and undesired. This can for example take place when recording video with a hand-held camera, or when recording video using recording units on a mount which is prone to, e.g., wind and vibrations from the surrounding environment.

[0006] One context in which this occurs is when recording sport events at a sport area. While some sport events, such as professional sport events, may utilize sturdy, rigorous mounts and cameras, other sport events, such as amateur sport events, rely on recording units elevated above the level of the sport area via a telescopic tripod or the like. To cover large portions of the sport area, multiple recording units and wide-angle lenses are often employed. Recorded videos are then processed to provide an output, such as an output video, to a user.

[0007] However, these types of recording units can be sensitive to the surroundings, and gusts of wind occurring while the recording units are recording will translate into the output.

[0008] Conventional image stabilization typically involves detecting feature points in consecutive image frames and using these points to estimate the motion between frames. Feature point detection identifies distinct and trackable points within an image, such as edges or corners, which can be reliably followed across successive frames. Once these points are identified, the system calculates the relative movement or translation of the feature points between frames in terms of pixels. This translation data is then used to compensate for unintended camera motion, by translating and rotating adjusting the position of the current image frame to align it with the previous one. However, such conventional image stabilization is not generally directly applicable to recording units for sport events such as those exemplified above. In particular, since such recording units often have wide-angle lenses which distort the recorded videos, a straightforward translation and rotation of image frames cannot be implemented without significant artifacts arising due to the distortion from the lenses.

[0009] Hence, there is a need for improved stabilization for outputs, such as output videos, from videos of sport events.

[0010] SUMMARY OF THE INVENTION

[0011] On the above background, it is an object of preferred embodiments of the present disclosure to provide improved stabilization for outputs, such as output videos, from videos of sport events.

[0012] A first aspect of the present disclosure relates to a computer-implemented method of stabilizing an output from a video of a sport event, the method comprising the steps of: providing raw video recordings of the sport event at a sport area, the raw video recordings comprising a first video recording from a first recording unit and a second video recording from a second recording unit, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; providing camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determining first feature points in a first raw image of the raw video recordings; determining second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determining first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determining second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, optionally wherein the common viewpoint is associated with a reference direction, relative to which the first directions and the second directions are provided; calculating an angular displacement between the second directions and the first directions; and establishing the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

[0013] Examples of the present disclosure utilize the particular conditions under which some types of recording units record sport events.

[0014] As an example, a camera arrangement for recording sport events comprises two recording units, is positioned besides a sport field for association football, and is elevated above the sport field via a telescopic tripod such that the two recording units collectively record the entire sport field. Due to the elevation, the camera arrangement is sensitive to wind, resulting in an unstable output video.

[0015] In practice, the recording units of the camera arrangement are both spatially translated and rotated by the wind relative to the sport field. However, examples of the present disclosure use the condition that the spatial translation is often insignificant, in particular in comparison with the rotation exerted onto the recording units by the wind. This is in part due to the placement of the recording units relative to the sport event, which entails that there are rarely any objects close to the recording units within their field of view. Furthermore, the telescopic tripod provides a limitation to the displacement of the recording units, in comparison with, e.g., a handheld camera held by a person moving along the sport field. Given these conditions, it is possible to implement stabilization of the output based on mainly addressing the rotation exerted onto the recording units.

[0016] This rotation is addressed by determining directions respectively associated with feature points, in contrast to conventional image stabilization which determine, e.g., pixel coordinates respectively associated with feature points.

[0017] Given a lens which distorts recorded images, a rotation of a recording unit will not provide a uniform translation of the pixel coordinates across the recorded images. For example, considering barrel distortion, a given rotation of a recording unit may generate a shift of 20 pixels in the centre of the recorded images, while a corresponding shift closer to the edge of the recorded images is only 5 pixels. Thereby, recorded images cannot simply be shifted via pixel coordinates, at least not without providing significant distortion artifacts in the output.

[0018] By instead performing stabilization based on directions relative to a common viewpoint, artifacts from lenses can be reduced significantly in the output.

[0019] Furthermore, aspects of the present disclosure rely on two recording units from which an output is generated based on directional tracing from a common viewpoint. Thereby stabilization as presented herein provides a complimentary effect, since the common viewpoint can both be used as a framework for stabilization, as well as a framework for generating a common output which is based on recorded videos from both the first recording unit and the second recording unit.

[0020] In addition, examples according to the present disclosure can be implemented without the need for additional sensing means, such as an accelerometer.

[0021] A primary application of examples of the present disclosure is to generate output videos in which the influence of unintended camera motion is reduced. Such an output video can be generated by image rectification of images of the raw video recordings in which images from the contents of the first video recordings and the second video recordings are projected onto a plane based on directional tracing of the contents relative to the common viewpoint, based on the camera calibration parameters, and based on the angular displacement. Such projection of contents onto a plane can be performed by means of ray tracing.

[0022] However, the present disclosure is not limited to output videos. An output can also be, e.g., tracking data relating to sport elements obtained by analysis of the video recordings from the recording units. Such tracking data can for example be tracking of one or more players of the sport event. Without image stabilization according to the present disclosure, unintended camera motion can propagate into such tracking data, thereby providing an inaccurate representation of the tracked sport element.

[0023] The common viewpoint may be associated with reference direction, relative to which the first and second directions are provided. Each of the first and second directions may be represented as a polar angle and an azimuthal angle, for example relative to the reference direction.

[0024] The angular displacement between the second directions and the first directions can, for example, be calculated by performing relative rotation of the second directions and / or the first directions to reduce an error between the second directions and the first directions, or by calculating an average difference between the respective polar angles of the second and first directions, and between the respective azimuthal angles of the second and first directions.

[0025] Although principles of the present disclosure are mostly exemplified using two recording units, more than two recording units can also be used, such as three recording units, or more than three recording units.

[0026] Aspects according to the present disclosure may, for example, be implemented via a camera arrangement comprising a first recording unit, a second recording unit, and a mount in which the first recording unit and the second recording unit are fixed, or implemented via a first mobile user device and a second mobile user device releasably mounted in a mount for the mobile user devices, the first mobile user device comprising a first recording unit and the second mobile user device comprising the second recording unit. An example of a mobile user device is a smart phone.

[0027] Method steps may be performed through via a local processing system, such as a processing system of a camera arrangement or of a mobile user device, and / or via a remote processing system, such as via a remove processing server communicatively connected to the recording units via, e.g., an internet connection. The output may be generated while the sport event is taking place to provide a live output, and / or it may be generated at a later stage.

[0028] Generally, any conventional approach for determining feature points may be used as basis for determining the first and second feature points, for example a scale-invariant feature transform algorithm, which is also typically capable of identifying matching feature points in respective images. Other algorithms, such as speeded up robust features, gradient location and orientation histogram, or histogram of oriented gradients can alternatively be used. Examples of sport events at a sport area of relevance to the present disclosure are an association football event at an association football pitch, an American football event at an American football field, a handball event at a handball court, a rugby event at a rugby pitch, a basketball event at a basketball court, and a lacrosse event at a lacrosse field. An event can be a match event or a training event. These examples are non-exhaustive.

[0029] The feature of having the common viewpoint being associated with a reference direction, relative to which the first directions and the second directions are provided, is entirely optional. That is, the first, second, or third aspects according to the present disclosure may be implemented with or without this feature.

[0030] According to examples of the present disclosure, the directional tracing from the common viewpoint is adjusted according to the angular displacement by changing an orientation of the common viewpoint according to the angular displacement and / or by changing an orientation of directions of the directional tracing from the common viewpoint according to the angular displacement.

[0031] According to examples of the present disclosure, the output comprises an output video established based on performing ray tracing from the common view point using the raw video recordings in combination with the camera calibration parameters, for example wherein the output video comprises at least a first processed image based on the first raw image and a second processed image based on the second raw image, wherein the ray tracing from the common viewpoint is adjusted according to the angular displacement between the second processed image and the first processed image.

[0032] According to examples of the present disclosure, the method comprises a step of determining a region of interest in relation to the sport event, wherein the output video corresponds to the region of interest, wherein the directional tracing from the common viewpoint is adjusted at least partly by shifting the region of interest to match the angular displacement.

[0033] Thereby, the stabilization may be implemented through corresponding adjustments to the region of interest, which may reduce the computational power required to implement stabilization through the calculated angular displacement.

[0034] The region of interest may be a moving region of interest, for example a region of interest which moves according to tracking data indicative of sport elements of the sport event.

[0035] According to examples of the present disclosure, the output comprises tracking data, wherein the tracking data is established by tracking one or more sport elements present in the raw video recordings, wherein the one or more sport elements comprise any of: sport players of the sport event, sport objects of the sport event, and game events of the sport event.

[0036] Game events can, for example, be any of shots, passes, tackles, referee-related events, etc. A sport object may, for example, be a ball.

[0037] By utilizing stabilization according to the present disclosure to provide tracking data, tracking data having improved accuracy and precision can be provided. Notably, in case no stabilization is implemented, regular displacement of recording units due to wind may generate a systematic error in tracking data since. This can occur since a vertical rotation which shifts recording units backward relative to a sport area will provide a different absolute displacement (for example, in meters) of a tracked object than a vertical rotation shifting recording unit forward relative to a sport area. Thereby, stabilization according to the present disclosure can reduce systematic errors in tracking data.

[0038] According to examples of the present disclosure, the steps of determining the second feature points, determining the second directions, calculating the angular displacement, and establishing the output video are performed for different instances of second raw images of the raw video recordings such that consecutive sections of the output are stabilized relative to the raw video recordings.

[0039] In other words, aspects of the present disclosure are not limited to merely stabilizing an output based on two raw images, but can be utilized to stabilize an output from, e.g., an entire sport event, such as an entire association football match.

[0040] According to examples of the present disclosure, a baseline between the first recording unit and the second recording unit is at most 1.0 meter, for example at most 0.50 meter, such as at most 0.30 meter.

[0041] According to examples of the present disclosure, the first recording unit and the second recording unit are mounted in a common mount which provide the fixed position and angle that the first recording unit and the second recording unit have relative to each other.

[0042] Hence, some examples according to the present disclosure are directed at stabilization of an output from recording units which have a similar point of view, and not, e.g., from separate recording units placed at entirely different relative to a sport area. According to examples of the present disclosure, the common viewpoint is determined based on the fixed position and angle that the first recording unit and the second recording unit have relative to each other.

[0043] The common viewpoint may, for example, be positioned at one of the recording units or their sensing chips, between the centres of the recording units or their sensing chips, or at an intersection of optical axes of the recording units. For example, if the centres of the respective sensing chips of the recording units are spaced apart by 20 cm, the common viewpoint can be positioned in between having a spacing of 10 cm to each of the two centres of the sensing chips.

[0044] An orientation of the common viewpoint, relative to which directions can be determined, can, for example, be set to an optical axis of one of the recording units, or set to a direction between the optical axes of the recording units.

[0045] According to examples of the present disclosure, the method comprises a step of: determining a three-dimensional world space of the sport area with respect to the first recording unit and the second recording unit based on the raw video recordings, wherein the step of establishing the output based on performing directional tracing from the common viewpoint is further based on the three-dimensional world space of the sport area.

[0046] By determining a three-dimensional world space, directional tracing from a common viewpoint can be improved, in particular in the region in which the field of view from the two image sensors overlap.

[0047] The determination of the three-dimensional world space may in particular be directed at the sport area, such that coordinates of the sport area is provided, for example relative to the common viewpoint.

[0048] According to examples of the present disclosure, the fixed position and / or angle that the first recording unit and the second recording unit have relative to each other are based on image analysis of the three-dimensional world space, and / or based on pre-defined data relating to the fixed position and / or angle that the first recording unit and the second recording unit have relative to each other. Even through a camera arrangement or a mount for the recording units may provide a fixed position and / or angle between the two recording units, this fixed position and / or angle may not necessarily be accurately known to, e.g., a computer processing system configured to carry out parts of aspects of the present disclosure.

[0049] By analysing the tree-dimensional world space, the fixed position and / or angle may however be determined.

[0050] As an example, the fixed position and / or angle may be determined based on stereo image analysis, based on image recognition of the sport area, and / or based on identification of feature points in respective images in an overlapping portion of the fields of view from the respective recording units.

[0051] In some examples, the fixed position and angle may be preprogrammed, for example based on a geometry of a common camera arrangement or of a common mount.

[0052] According to examples of the present disclosure, the camera calibration parameters are based on intrinsic camera parameters and distortion parameters of the first recording unit and of the second recording unit.

[0053] Generally, camera calibration parameters may be provided based on a pinhole camera model, under which a camera matrix can be used to denote a projective mapping from world coordinates to pixel coordinates. Such a camera matrix can be expressed in terms in extrinsic parameters and intrinsic parameters. The extrinsic parameters relate to the position and orientation of the camera relative to world coordinates, and the intrinsic parameters relate to focal length, image sensor format, and camera principal point.

[0054] The intrinsic parameters can also be expressed as a matrix, more specifically a 3x3 matrix, comprising the parameters oxrepresenting focal length along an x axis, oyrepresenting focal length along an y axis, y representing a skew coefficient between the x axis and the y axis, and u0and v0which represent a principal point.

[0055] Distortion parameters of the recording unit may be provided based on the Brown-Conradi distortion model, which can convert coordinates xuand yuof an undistorted image point as projected by an ideal pinhole camera into coordinates Xd and y of a distorted image point as projected on an image plane using a distorting lens by xu= xd+ d - xc~)(Kir2+ K2r4+ ••• ) + represent the distortion centre, Knis the nthradial distortion coefficient, Pnis the nthtangential distortion coefficient, and r = 7 ( ; - xcy + (yd- yc)2is the Euclidian distance between the distorted image point and the distortion centre.

[0056] Thereby, the parameters xc, yc, Kn, and Pnare examples of distortion parameters. Including a greater number of coefficients for Knand Pncan generally increase accuracy, but examples of the present disclosure are not limited to a particular number of set of coefficients.

[0057] For a given recording unit, the lens arrangement of this recording unit, and its position relative to the sensing chip, will determine the relevant distortion parameters. These can either be measured via a calibration procedure of the recording unit, or can be modelled / calculated.

[0058] Barrel distortion, which is relevant in examples of the present disclosure, will typically have a negative term Ki.

[0059] Based on the camera calibration parameters exemplified above, a pixel coordinate can be related to a direction from which light received by this pixel arrives. In turn, this can be used to determine directions associated with feature points, and establishing an output based on directional tracing of contents in the raw video recordings.

[0060] According to examples of the present disclosure, the raw video recordings collectively cover the entire sport area, and / or collectively cover a horizontal angle of at least 140°, for example a horizontal angle of at least 150°, for example a horizontal angle of at least 160 °, such as a horizontal angle of at least 170°.

[0061] It can be desirable to cover the entire sport area and / or cover a broad horizontal angle, which requires recording units capable of doing so. This, in turn, is indicative of recording units having wide-angle lenses.

[0062] Such a horizontal angle may be covered by raw video recordings from only the first recording unit and the second recording unit, or it may be covered based on raw video recordings from more than two recording units.

[0063] According to examples of the present disclosure, an imaging objective of the first recording unit and / or an imaging objective of the second recording unit has a focal length of at most 35 mm, for example at most 24 mm and / or; wherein a ratio of a diagonal of an imaging sensor of the first recording unit to the focal length of the imaging objective of the first recording unit is at least 1.0, for example at least 1.5, such as at least 2.0, and a ratio of a diagonal of an imaging sensor of the second recording unit to the focal length of the imaging objective of the second recording unit is at least 1.0, for example at least 1.5, such as at least 2.0.

[0064] According to examples of the present disclosure, the first feature points and / or the second feature points are determined based on determining a preliminary set of feature points and selecting a subset of the preliminary set of feature points to provide the first feature points and / or the second feature points, wherein feature points of the subset are selected from the preliminary set of feature points based on angular distance among these feature points and / or wherein feature points of the subset are selected from the preliminary set of feature points by excluding feature points close to an edge of the first raw image and / or the second raw image.

[0065] By selecting feature points based on an angular distance among the feature points, it is possible to provide a more robust stabilization.

[0066] Further, by utilizing the angular distance, instead of, e.g., the distance in terms of pixel coordinates, stabilization is more robust in the context of lenses providing distortion.

[0067] As an example, feature points are selected based on selecting a set of feature points which internally have large angular distances rather than a set of feature points which internally has small angular distances. Angular distance among feature points can be determined based on the first directions and / or the second directions, or can be measured based on the common viewpoint and the camera calibration parameters. This angular distance is preferably provided based on a single raw image, or based on raw images acquired simultaneously.

[0068] Alternatively or additionally, feature points can be selected by excluding feature points close the an edge of an image. This is particularly relevant since a rotation of the recording units can result in feature points close to the edge being translated outside the imaging frame of the recording units.

[0069] Generally, providing a particular selection of feature points can ensure that the method can be implemented computationally efficiently. This, in turn, can be advantageous if implementing stabilization via a local processing arrangement, and / or if performing stabilization on a live recording.

[0070] According to examples of the present disclosure, the first feature points and the second feature points are determined such that these feature points are primarily located outside the sport area, preferably primarily at a perimeter of the sport area and / or further away from the first recording unit and the second recording unit than the perimeter of the sport area.

[0071] Such feature points are less sensitive to translation of the first and second recording units, while being indicative of rotation of these recording units. Thereby, these points are advantageous for implementation of the stabilization presented herein.

[0072] Further, by considering primarily or only feature points located outside the sport area, these feature points are at a reduced risk of relating to elements which are poorly suited for stabilization. Preferably, feature points as presented herein relate to, for example, static landmarks.

[0073] Example of a perimeter of a sport area is the touch line and the goal line of an association football field. The relevant touch line or perimeter is the one further away from the first and second recording units.

[0074] Such feature points may be determined by considering specific regions of the sensing chip(s), and / or based on determining a preliminary set of feature points and selecting a subset of the preliminary set of feature points to provide the first feature points and / or the second feature points, for example in combination with image analysis or determination of the three- dimensional world space.

[0075] A second aspect of the present disclosure relates to a computer program for stabilizing an output from a video of a sport event at a sport area, the computer program comprising instruction which, when the program is executed by a computer processing system, cause the computer processing system to carry out the steps of: receive raw video recordings of the sport event, the raw video recordings comprising a first video recording from a first recording unit and a second video recording from a second recording unit, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; receive camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determine first feature points in a first raw image of the raw video recordings; determine second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determine first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determine second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, optionally wherein the common viewpoint is associated with a reference direction, relative to which the first directions and the second directions are provided; calculate an angular displacement between the second directions and the first directions; and establish the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

[0076] According to examples of the present disclosure, the computer program according to the second aspect is configured to perform any method according to the first aspect of the present disclosure.

[0077] Generally, a computer program according to the second aspect may have any of the same effects or advantages as a method according to the first aspect. A computer program may be executed by a local computer processing system, such as a processing system of a camera arrangement or a processing system of a mobile user device, by a remote computer processing system, such as a server system connected to the internet, or by a combination thereof.

[0078] A third aspect of the present disclosure relates to a system for stabilizing an output from a video of a sport event at a sport area, the system comprising: a first recording unit for providing a first video recording of the sport event at the sport area; a second recording unit for providing a second video recording of the sport event at the sport area, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; and a computer processing system communicatively linked to the first recording unit and the second recording unit, wherein the computer processing system is configured to: receive raw video recordings from the first recording unit and the second recording unit, the raw video recordings comprising the first video recording from the first recording unit and the second video recording from the second recording unit; receive camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determine first feature points in a first raw image of the raw video recordings; determine second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determine first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determine second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, optionally wherein the common viewpoint is associated with a reference direction, relative to which the first directions and the second directions are provided; calculate an angular displacement between the second directions and the first directions; and establish the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

[0079] According to examples of the present disclosure, the system according to the third aspect, or the computer processing system of the third aspect, is configured to perform any method according to the first aspect of the present disclosure.

[0080] Generally, a system according to the third aspect may have any of the same effects or advantages as a method according to the first aspect.

[0081] The computer processing system may be a local computer processing system, such as a processing system of a camera arrangement or a processing system of a mobile user device, a remote computer processing system, such as a server system connected to the internet, or a combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Embodiments of the invention will now be further described by reference to the accompanying drawings, in which:

[0083] Fig. 1 illustrates displacement and rotation of a camera arrangement due to wind,

[0084] Fig. 2 illustrates an exemplary camara arrangement,

[0085] Fig. 3 schematically illustrates method steps according to an example of the present disclosure,

[0086] Fig. 4 illustrates a computer program according to an example of the present disclosure,

[0087] Figs. 5a-b illustrate examples of raw images from a raw video recording,

[0088] Figs. 6a-b illustrate zooms of raw images from a raw video recording,

[0089] Fig. 7 illustrates a representation of an output video without stabilization, and

[0090] Fig. 8 illustrates a representation of an output video in which stabilization according to an example of the present disclosure has been applied.

[0091] DETAILED DESCRIPTION

[0092] Fig. 1 illustrates displacement and rotation of a camera arrangement 1 due to wind.

[0093] The camera arrangement 1 is attached to a stand 3 in the form of a telescopic tripod, which elevates the camera arrangement 1 relative to the ground and the relevant sport area.

[0094] Wind may push the camera arrangement 1 relative to an unperturbed resting position. As a result, the camera arrangement 1 is translated and rotated relative to this resting position. In the figure, the unperturbed resting position of the camera arrangement 1 is illustrated by solid lines, whereas exemplary displacements of the camera arrangement 1 due to, e.g., wind is illustrated by dashed lines.

[0095] The figure further illustrates exemplary field of views 4a-c of the recording units of the camera arrangement. A first field of view 4a is illustrated using solid lines and indicates the field of view of the recording units of the camera arrangement when the camera arrangement is in a resting position, and two other fields of view 4b-c are illustrated using dashed lines and indicate the exemplary fields of view when the camera arrangement 1 is displaced due to, e.g., wind.

[0096] As evident from the illustrate, the displacement of the camera arrangement 1 relative to, e.g., a sport area located outside the right-hand boundary of the figure has a relatively large influence on the orientation of the camera arrangement relative to the sport area, but only minimal influence on the translation / position of the camera relative to the sport area.

[0097] This observation compliments the use of stabilization based on determining directions associated with feature points and based on establishing an output by adjusting according to an angular displacement of said directions.

[0098] Fig. 2 illustrates an exemplary camara arrangement 1. The camera arrangement 1 comprises a first recording unit 5a, the first recording unit 5a comprising a first sensing chip 6a and a first lens arrangement 7a aligned along a first optical axis to provide a first field of view 4a of a sport area. The camera arrangement further comprises a second recording unit 5b, the second recording unit 5b comprising a second sensing chip 6b and a second lens arrangement 7b aligned along a second optical axis to provide a second field of view (not illustrated) of the sport area, the first second optical axis and the second optical axis having different orientations. Additionally, the camera arrangement comprises a mount 2 in which the first lens arrangement 7a, the first sensing chip 6a, the second lens arrangement 7b, and the second sensing chip 6b are mounted. The first recording unit 5a and the second recording unit 5b have a fixed position and angle relative to each other, which is provided by the mount 2.

[0099] The first lens arrangement 7a and the first sensing chip 6a collectively provide a relation between a direction of incoming light 9 in the field of view 4a of the first recording unit 5a, and the pixel(s) of the first sensing chip 6a which receives said incoming light 9. Similarly, the second lens arrangement 7b and the second sensing chip 6b collectively provide a relation between a direction of incoming light in the field of view of the second recording unit 5b, and the pixel(s) of the second sensing chip 6b which receives this incoming light.

[0100] The camera arrangement 1 and the two recording units 5a-b are related to a common viewpoint 8 which is of relevance for providing a common output, such as a processed output video, from the camera arrangement. Further, such a common viewpoint is of relevance for preforming stabilization as exemplified herein. The camera arrangement may optionally comprise a computer processing arrangement for processing raw videos recorded by the recording units 5a-b. Further, the camera arrangement may optionally comprise more than two recording units.

[0101] In other examples, the first recording unit is a recording unit of a first mobile user device such as a smartphone, and the second recording unit is a recording unit of a second mobile user device such as a smartphone, and the mount is a smartphone mount.

[0102] Fig. 3 schematically illustrates method steps according to an example of the present disclosure.

[0103] In the top left-hand corner of the figure, a camera arrangement 1 comprising a first recording unit and a second recording unit is illustrated. These recording unit provide a first raw video recording 10a and a second raw video recording 10b, respectively. These are recordings of a sport event at a sport area, for example of an association football match at an association football field.

[0104] In the bottom right-hand corner of the figure, the camera calibration parameters 11 are illustrated, the camera calibration parameters comprising intrinsic camera parameters 16 and distortion parameters 17. These parameters are provided for each of the first recording unit and the second recording unit, and provide a relationship between directions of incoming light and respective pixels of the first recording unit and / or of the second recording unit which receive this incoming light.

[0105] The method comprises a step of determining first feature points 14aa, 14ab, 14ac in a first raw image 20a and a step of determining second feature points 14ba, 14bb, 14bc in a second raw image 20b. The two raw images 20a, 20b are recorded at different times of the sport event. In the present example, each of these raw images 20a, 20b is from the second raw video recording 10b.

[0106] The second feature points 14ba, 14bb, 14bc respectively match the first feature points 14aa, 14ab, 14ac. That is, a first feature point 14aa of the first feature points matches a first feature point 14ba of the second feature points, a second feature point 14ab of the first feature points matches a second feature point 14bb of the second feature points, etc. The identification and matching of feature points can be carried out using, e.g., a scale-invariant feature transform algorithm.

[0107] Based on the first feature points 14aa, 14ab, 14ac and the camera calibration parameters 11, first directions respectively associated with each of the first feature points 14aa, 14ab, 14ac are determined. Correspondingly, second directions respectively associated with each of the second feature points 14ba, 14bb, 14bc are determined based on the second feature points 14ba, 14bb, 14bc and the camera calibration parameters 11.

[0108] Each of these directions are quantified by a polar angle and an azimuthal angle relative to a common viewpoint of the camera arrangement 1 and relative to a reference direction of that common viewpoint.

[0109] If the camera arrangement has a constant position, the first directions will correspond to the second directions. However, due to motion, the second directions can be different from the first directions.

[0110] This is illustrated in the middle right-hand section of the present illustration which illustrated a coordinate system having the common viewpoint as origin, and in which a first direction 13a towards a first feature point 14aa and a second direction 13b towards a second feature point 14ba are illustrated. The first direction 13a is quantified by a first azimuthal angle cpaaand a first polar angle 0aa, and the second direction 13b is quantified by a second azimuthal angle cpba and a second polar angle 0ba. From these directions 13a, 13b, an angular displacement 15 is calculated, for example based on a difference between the azimuthal angles and the polar angles.

[0111] Even though only a single example of a first direction 13a and a second direction 13b are illustrated, the directions of all of the first feature points and the matching second feature points are preferably taken into account to determine the angular displacement 15.

[0112] The determined angular displacement is associated with the first and second raw images 20a, 20b. In case more than two raw images are considered, an angular displacement can be calculated for each of the raw images, for example relative to the first raw image or another reference entity, such as a predetermined reference direction, or a mean direction derived from multiple images.

[0113] Once the angular displacement 15 has been determined, this parameter can be applied to the relevant images from the raw video recordings 10a, 10b to provide an output 21 which takes into account this angular displacement. As a simple example, the output 21 comprises an output video 18, and the angular displacement has been determined to be 3° in the azimuthal direction and 2° in the polar direction. The output video 18 comprises at least a first processed image based on the first raw image 20a, and a second processed image based on the second raw image 20b. The first processed image is established based on directional tracing of the contents in the first raw image 20a of the second video recording 10b and of the contents in a corresponding raw image from the first video recording 10a. For this first processed image, the angular displacement is not taken into account. The second processed image is established based on directional tracing of the contents in the second raw image 20b of the second video recording 10b and of the contents in a corresponding raw image from the first video recording 10a. In this directional tracing for the second processed image, the angular displacement is taken into account. That is, the directional tracing is adjusted by 3° in the azimuthal direction and 2° in the polar direction to match the determined angular displacement. Thereby, the second processed image is processed relative to the first processed image.

[0114] As indicated the figure, the output 21 can alternatively or additionally comprise tracking data 19 derived from the raw video recordings 10a, 10b.

[0115] Fig. 4 illustrates a computer program 22 according to an example of the present disclosure. The computer program 22 is configured to be executed by a computer processing system.

[0116] Upon execution, the computer program 22 receives two inputs. Firstly, it receives raw video recordings 10 of a sport event. Secondly, it receives camera calibration parameters 11. Alternatively, the camera calibration parameters 11 are partially of fully coded into the computer program 22, for example such that the computer program 22 does not need to receive these parameters from an external source to process the raw video recordings 10.

[0117] The raw video recordings 10 and the camera calibration parameters 11 may be provided from the same source, or from separate sources.

[0118] The raw video recordings 10 comprise at least a first video recording 10a and a second video recording 10b from respective recording units.

[0119] The video recordings 10 are provided to an image selector 23 of the computer program 22. This module selects at least two raw images 20 from at least one of the raw video recordings 10. The selected images are recorded at different times.

[0120] The raw images 20 are provided to a feature point determiner 24, which is a module of the computer program 22 which determines matching feature points 14 in each of the raw images 20. For example, for a first raw image 20a, at least three respective first feature points 14aa, 14ab, 14ac ... are determined, and for a second raw image 20b, at least three respective second feature points 14ba, 14bb, 14bc ... are determined, with each of the second feature points 14ba, 14bb, 14bc ... respectively matching the first feature points 14aa, 14ab, 14ac .... These feature points 14 are provided to a direction calculator 25, which also receives the camera calibration parameters 11. This direction calculator 25 is a module of the computer program 22 which determines first directions 13aa, 13ab, 13ac ... respectively associated with the first feature points 14aa, 14ab, 14ac ... and second directions 13ba, 13bb, 13bc ... respectively associated with the second feature points 14ba, 14bb, 14bc ....

[0121] The directions 13 determined by the direction calculator 25 are provided to an angular displacement calculator 26, which is a module of the computer program 22 which calculates an angular displacement 15 based on differences between the second directions 13ba, 13bb, 13bc ... and the first directions 13aa, 13ab, 13ac ....

[0122] Based on this angular displacement 15, the raw video recordings 10, and the camera calibration parameters 11, an output establisher 27 establishes an output 21 of the computer program 22. The output 21 is established by the output establisher 27 by performing directional tracing of contents in the raw video recordings 10 using the camera calibration parameters 11 with the directional tracing being adjusted according to the angular displacement 15.

[0123] In case angular displacement is only calculated for a subset of raw images from the video recordings, this angular displacement can be interpolated and / or extrapolated to provide an estimated angular displacement for raw images for which an actual angular displacement has not been calculated.

[0124] Figs. 5a-b illustrate examples of raw images 10a, 10b from a raw video recording.

[0125] The two images 10a, 10b are recorded at different times of a sport event.

[0126] In the first raw image 10a, illustrated in Fig. 5a, a first set of feature points 14aa, 14ab, 14ac, ... have been determined. Correspondingly, in the second raw image 10b, illustrated in Fig. 5b, a second set of feature points 14ba, 14bb, 14bc, ... have been determined, the second feature points respectively matching the first feature points.

[0127] Figs. 6a-b illustrate zooms of raw images from a raw video recording. The video recording is the same as the one from which raw images are illustrated in Figs. 5a-b. Once again, feature points are indicated in both images 10a, 10b. In addition, three crosses are shown in each figure. These crosses correspond to approximate specific pixel coordinates. Hence, in contrast to the feature points, these crosses do not move with the contents of the raw images upon rotation of the recording units. As evident from a comparison between the crosses and the contents of the two raw images 10a, 10b, the contents of the second raw image 10b is shifted relative to the contents of the first raw image 10a. In this particular example, this shift has occurred due to wind displacing the recording units between the times at which first raw image 10a and the second raw image 10b have been recorded.

[0128] Fig. 7 illustrates a representation of an output video without stabilization. The illustrated image has been generated as a median of a plurality of raw images. Evidently, the displacement also evident from a comparison between Fig. 6a and Fig. 6b results in a blurry image. This indicates that there is a need for stabilization of outputs from videos of sport events acquired by particular types of camera arrangements.

[0129] Fig. 8 illustrates a representation of an output video in which stabilization according to an example of the present disclosure has been applied. The illustrated image has been generated as a median of a plurality of raw images where each image (except a first raw image used for reference) has been adjusted according to a calculated angular displacement. By comparison with Fig. 7, it is evident that significantly less blur is present, indicating that aspects and examples according to the present disclosure are capable of providing stabilization of an output from videos of sport events.

[0130] Preferably, the first raw image used for reference is an image which has an angular displacement is small relative to neighbouring images, indicating that the recording units are relatively stable at the point in time at which this image has been acquired.

[0131] List of figure references:

[0132] 1 camera arrangement

[0133] 2 mount

[0134] 3 stand

[0135] 4 field of view

[0136] 5 recording unit

[0137] 6 sensing chip

[0138] 7 lens arrangement

[0139] 8 common viewpoint

[0140] 9 incoming light

[0141] 10 raw video recording

[0142] 11 camera calibration parameters

[0143] 12 feature points

[0144] 13 direction

[0145] 14 feature point

[0146] 15 angular displacement

[0147] 16 intrinsic camera parameters 17 distortion parameters

[0148] 18 output video

[0149] 19 tracking data

[0150] 20 raw image 21 output

[0151] 22 computer program

[0152] 23 image selector

[0153] 24 feature point determiner

[0154] 25 direction calculator 26 angular displacement calculator

[0155] 27 output establisher

Claims

24CLAIMS1. A computer-implemented method of stabilizing an output from a video of a sport event, the method comprising the steps of: providing raw video recordings of the sport event at a sport area, the raw video recordings comprising a first video recording from a first recording unit and a second video recording from a second recording unit, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; providing camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determining first feature points in a first raw image of the raw video recordings; determining second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determining first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determining second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, wherein the common viewpoint is associated with a reference direction, relative to which the first directions and the second directions are provided; calculating an angular displacement between the second directions and the first directions; andestablishing the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

2. A method according to claim 1, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement by changing an orientation of the common viewpoint according to the angular displacement and / or by changing an orientation of directions of the directional tracing from the common viewpoint according to the angular displacement.

3. A method according to any of the preceding claims, wherein the output comprises an output video established based on performing ray tracing from the common view point using the raw video recordings in combination with the camera calibration parameters, for example wherein the output video comprises at least a first processed image based on the first raw image and a second processed image based on the second raw image, wherein the ray tracing from the common viewpoint is adjusted according to the angular displacement between the second processed image and the first processed image.

4. A method according to claim 3, wherein the method comprises a step of determining a region of interest in relation to the sport event, wherein the output video corresponds to the region of interest, wherein the directional tracing from the common viewpoint is adjusted at least partly by shifting the region of interest to match the angular displacement.

5. A method according to any of the preceding claims, wherein the output comprises tracking data, wherein the tracking data is established by tracking one or more sport elements present in the raw video recordings, wherein the one or more sport elements comprise any of: sport players of the sport event, sport objects of the sport event, and game events of the sport event.

6. A method according to any of claims, wherein the steps of determining the second feature points, determining the second directions, calculating the angular displacement, and establishing the output video are performed for different instances of second raw images of the raw video recordings such that consecutive sections of the output are stabilized relative to the raw video recordings.

7. A method according to any of the preceding claims, wherein the first recording unit and the second recording unit are mounted in a common mount which provide the fixed positionand angle that the first recording unit and the second recording unit have relative to each other.

8. A method according to any of the preceding claims, wherein the common viewpoint is determined based on the fixed position and angle that the first recording unit and the second recording unit have relative to each other.

9. A method according to any of the preceding claims, wherein the method comprises a step of: determining a three-dimensional world space of the sport area with respect to the first recording unit and the second recording unit based on the raw video recordings, wherein the step of establishing the output based on performing directional tracing from the common viewpoint is further based on the three-dimensional world space of the sport area.

10. A method according to any of the preceding claims, wherein the camera calibration parameters are based on intrinsic camera parameters and distortion parameters of the first recording unit and of the second recording unit.

11. A method according to any of the preceding claims, wherein the raw video recordings collectively cover the entire sport area, and / or collectively cover a horizontal angle of at least 140°, for example a horizontal angle of at least 150°, for example a horizontal angle of at least 160 °, such as a horizontal angle of at least 170°.

12. A method according to any of the preceding claims, wherein the first feature points and / or the second feature points are determined based on determining a preliminary set of feature points and selecting a subset of the preliminary set of feature points to provide the first feature points and / or the second feature points, wherein feature points of the subset are selected from the preliminary set of feature points based on angular distance among these feature points and / or wherein feature points of the subset are selected from the preliminary set of feature points by excluding feature points close to an edge of the first raw image and / or the second raw image.2713. A method according to any of the preceding claims, wherein the first feature points and the second feature points are determined such that these feature points are primarily located outside the sport area, preferably primarily at a perimeter of the sport area and / or further away from the first recording unit and the second recording unit than the perimeter of the sport area.

14. A computer program for stabilizing an output from a video of a sport event at a sport area, the computer program comprising instruction which, when the program is executed by a computer processing system, cause the computer processing system to carry out the steps of: receive raw video recordings of the sport event, the raw video recordings comprising a first video recording from a first recording unit and a second video recording from a second recording unit, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; receive camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determine first feature points in a first raw image of the raw video recordings; determine second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determine first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determine second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, wherein the common viewpoint is associated with a reference28 direction, relative to which the first directions and the second directions are provided; calculate an angular displacement between the second directions and the first directions; and establish the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

15. A system for stabilizing an output from a video of a sport event at a sport area, the system comprising : a first recording unit for providing a first video recording of the sport event at the sport area; a second recording unit for providing a second video recording of the sport event at the sport area, wherein the first recording unit and the second recording unit have a fixed position and angle relative to each other; and a computer processing system communicatively linked to the first recording unit and the second recording unit, wherein the computer processing system is configured to: receive raw video recordings from the first recording unit and the second recording unit, the raw video recordings comprising the first video recording from the first recording unit and the second video recording from the second recording unit; receive camera calibration parameters for the first recording unit and the second recording unit, the camera calibration parameters being indicative of a relation between directions of incoming light and pixels of the first recording unit and / or the second recording unit which receive said incoming light; determine first feature points in a first raw image of the raw video recordings;29 determine second feature points in a second raw image of the raw video recordings, the first image and the second image being recorded at different times of the sport event, wherein the second feature points respectively match the first feature points; determine first directions respectively associated with the first feature points based on the camera calibration parameters, wherein the first directions each indicates a respective direction towards a respectively associated feature point of the first feature points relative to a common viewpoint of the first recording unit and the second recording unit; determine second directions respectively associated with the second feature points based on the camera calibration parameters, wherein the second directions each indicates a respective direction towards a respectively associated feature point of the second feature points relative to the common viewpoint, wherein the common viewpoint is associated with a reference direction, relative to which the first directions and the second directions are provided; calculate an angular displacement between the second directions and the first directions; and establish the output based on directional tracing of contents in the raw video recordings relative to the common viewpoint based on the raw video recordings in combination with the camera calibration parameters, wherein the directional tracing from the common viewpoint is adjusted according to the angular displacement.

Images