Automatic alignment of video streams

Abstract

Methods and systems for determining a time offset between a first video stream and a second video stream depicting a sporting event. A depiction of a first type of visually unique activity is identified in a first video stream and a second video stream. A temporal offset between the two video streams is determined at least in part by comparing the depiction of the first type of visually unique activity in the first video stream with the depiction of the first type of visually unique activity in the second video stream.

Description

Technical Field

[0001] The present invention relates to methods and systems for processing videos, in particular for temporal alignment of video streams, ie for determining time offsets between video streams. In embodiments disclosed herein, the video streams depict a live event, such as in particular a sporting event. Background Art

[0002] Live events, such as sporting events, particularly at the collegiate and professional levels, continue to grow in popularity and revenue, with individual colleges and franchises raking in billions of dollars in revenue each year. It will be appreciated that the temporal offset between video streams depicting live events may be important (or even necessary) to perform various kinds of processing of such video streams, such as processing the video streams to generate analytics about the event (e.g., in the case of sporting events, analytics about the game, team, and / or player) or processing the video streams to generate enhanced video content of the event, such as a single video showing highlights of the game from multiple angles. Summary of the invention

[0003] According to a first aspect of the present disclosure, a video processing method is provided, comprising: identifying one or more depictions of a first type of visually unique activity in a first video stream depicting a sporting event; identifying one or more depictions of the first type of visually unique activity in a second video stream depicting the sporting event; and determining a time offset between the first video stream and the second video stream, wherein the determination of the time offset comprises comparing one or more depictions of the first type of visually unique activity in the first video stream with one or more depictions of the first type of visually unique activity in the second video stream.

[0004] In an embodiment, the method includes processing at least one of the first video stream and the second video stream based on a time offset. In an example, the processing includes performing spatiotemporal pattern recognition based on the first video stream, the second video stream, and the time offset between them. Additionally or alternatively, the processing includes generating video content (e.g., enhanced video content) based on the first video stream, the second video stream, and the time offset between them.

[0005] In an embodiment, the method includes receiving a first video stream from a first portable device including a camera. For example, the first portable device may be a smartphone. Optionally, the method includes receiving a second video stream from a second portable device including a camera. For example, the second portable device may be a smartphone.

[0006] According to a second aspect of the present disclosure, a video processing system is provided, comprising: a memory storing a plurality of computer-executable instructions; one or more processors executing the computer-executable instructions, the computer-executable instructions causing the one or more processors to: identify one or more depictions of a first type of visually unique activity in a first video stream depicting a sporting event; identify one or more depictions of the first type of visually unique activity in a second video stream depicting the sporting event; and determine a time offset between the first video stream and the second video stream, wherein the system determines the time offset at least by comparing the one or more depictions of the first type of visually unique activity in the first video stream with one or more depictions of the first type of visually unique activity in the second video stream.

[0007] In an embodiment, the system is configured to receive a first video stream from a first portable device including a camera. For example, the first portable device may be a smartphone. Optionally, the system is configured to receive a second video stream from a second portable device including a camera. For example, the second portable device may be a smartphone.

[0008] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising a set of computer-readable instructions that, when executed by one or more processors, cause the one or more processors to: identify one or more depictions of a first type of visually unique activity in a first video stream depicting a sporting event; identify one or more depictions of the first type of visually unique activity in a second video stream depicting the sporting event; and determine a time offset between the first video stream and the second video stream, wherein the system determines the time offset at least by comparing the one or more depictions of the first type of visually unique activity in the first video stream with one or more depictions of the first type of visually unique activity in the second video stream.

[0009] In an embodiment of the second aspect and the third aspect of the present disclosure, the computer executable instructions further cause one or more processors to: process at least one of the first video stream and the second video stream based on a time offset. In an example, the processing includes performing spatiotemporal pattern recognition based on the first video stream, the second video stream, and the time offset between them. Additionally or alternatively, the processing includes generating video content (e.g., enhanced video content) based on the first video stream, the second video stream, and the time offset between them.

[0010] In an embodiment of the second and third aspects of the present disclosure, computer executable instructions cause one or more processors to: identify one or more depictions of a second type of visually unique activity in a first video stream; and identify one or more depictions of a second type of visually unique activity in a second video stream, wherein the system determines a temporal offset at least by comparing the one or more depictions of the second type of visually unique activity in the first video stream with the one or more depictions of the second type of visually unique activity in the second video stream.

[0011] While in the above aspects and embodiments, the video stream depicts a sporting event, in methods and systems according to further aspects, the event may be a non-sporting live event, such as a concert, comedy show, or play.

[0012] Further features and advantages will become apparent from the following description, made with reference to the accompanying drawings, which are given by way of example only. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart illustrating a video processing method according to an illustrative embodiment.

[0014] Figure 2A and Figure 2B Video frames from two different video streams of a sporting event are shown, and illustrate how the two video streams may be aligned using a change in the game clock for the sporting event.

[0015] FIG. 3A to FIG. 3D Two video frames from each of two video streams of a sporting event are shown, and illustrate how camera flashes occurring during the sporting event may be used to align the two video streams.

[0016] Figure 4A and Figure 4B They are respectively for Figure 3A and Figure 3B The video stream whose frames are displayed in Figure 3C and Figure 3D A graph of pixel intensity versus time for a video stream whose frames are displayed in .

[0017] FIG. 5A to FIG. 5D Two video frames from each of two video streams of a sporting event are shown, and it is illustrated how the two video streams may be aligned using billboards for the sporting event.

[0018] Figure 6 Is displayed for Figure 5A and Figure 5B A line graph showing the pixel intensity of a video stream whose frames vary over time, and a ... Figure 5C and Figure 5DA line graph showing the pixel intensity of a video stream whose frames vary over time.

[0019] Fig. 7A and Figure 7B Two video frames from a video stream of a sporting event are shown, illustrating how the movement of a ball in the sporting event can be used to align the two video streams.

[0020] Fig. 8A and Figure 8B Use Figure 3A and Figure 3B A graph of changes in the horizontal positioning of a ball determined from a video stream of its frames is displayed, and a graph of the three-dimensional positioning of the ball derived from one or more other video streams of the sporting event.

[0021] Fig. 9A and Fig. 9B The same frame of the video stream from the sporting event is shown with the head position for each player projected based on the head position determined using a corresponding frame of a second video stream from the sporting event.

[0022] Fig. 10A and Fig. 10B Body poses of several participants in a sporting event are shown as determined from a first frame and a second frame, respectively, of a video stream of the sporting event.

[0023] Fig.11 is a schematic diagram of a video processing system according to another illustrative embodiment. DETAILED DESCRIPTION

[0024] Embodiments of the present application relate to automatic alignment of video streams, particularly video streams depicting live events such as sporting events.

[0025] First of all Figure 1 Reference is made to a flowchart illustrating a video processing method 100 according to an illustrative embodiment. As shown, the method 100 includes a step 101 of identifying one or more depictions of a first category of visually distinct activities in a first video stream. In many of the examples illustrated herein, the first video stream depicts a sporting event, such as a football game, a basketball game, a tennis game, etc.; however, it is contemplated that the methods described herein may be applied to video streams depicting non-sporting events as well.

[0026] Also like Figure 1 As shown in , method 100 also includes a step 102 of identifying one or more depictions of the same type of visually unique activity in a second video stream depicting the same sporting event.

[0027] As will be described below, reference will be made to FIGS. 2 to Fig. 10B The example shown in the explanation is Figure 1 The method 100 can utilize various types of visually unique activities to determine the time offset between two video streams. For example, referring to FIG. 2A to FIG. 10B Visually unique activities of the kind described are characterized by being: visible from multiple locations and directions of the sporting event; having visible changes that can be localized in time, for example because they change abruptly in appearance from one frame to the next in a video stream (e.g. due to changes in position, motion, and / or pattern); and / or changing in appearance regularly during the sporting event. Therefore (or in addition), depictions of such visually unique activities can be easily and / or robustly identified within two (or more) video streams of the same sporting event.

[0028] Video streams of a sporting event may be received from a variety of sources. Specifically, it is envisaged that in the case where the first video stream and / or the second video stream is generated by a portable device including a camera (such as a smartphone or a tablet device), a video stream of a sporting event may be received from a variety of sources. Figure 1 Compared to video streams generated by conventional broadcast cameras, it is believed that there is a lack of alignment solutions for video streams generated by such devices. Nevertheless, in the case where the first video stream and / or the second video stream are generated by conventional broadcast cameras, an alignment solution may be employed. Figure 1 Method 100.

[0029] Back to Figure 1 It will be appreciated that method 100 also includes a step 106 of determining a time offset between the first video stream and the second video stream. As indicated at block 108, this step 106 includes comparing the depiction of the first category of visually unique activity in the first video stream with one or more depictions of the first category of visually unique activity in the second video stream.

[0030] In some examples, such as reference Figures 3A to 6 In those examples described, such comparison can include comparing the intensity (e.g., average or median intensity) of some or all of the pixels corresponding to the depiction of the first kind of visually distinctive activity in the first video stream with the intensity (e.g., average or median intensity) of some or all of the pixels corresponding to the depiction of the first kind of visually distinctive activity in the second video stream. Alternatively (or additionally), the movement and / or position of the depiction of the first kind of visually distinctive activity in the first video stream can be compared with the movement and / or position of the depiction of the first kind of visually distinctive activity in the second video stream, as described with reference to 7A to 10B The situation in the example described.

[0031] like Figure 1As further shown in , method 100 may optionally include an additional video processing step 110 after step 106, which determines the time offset between the first video stream and the second video stream. In the additional video processing step 110, the time offset thus determined is utilized when processing one or both of the first video stream and the second video stream. For example, the additional processing step 110 may include: performing spatiotemporal pattern recognition based on the first video stream, the second video stream, and the time offset between them. Additionally or alternatively, step 110 may include: generating video content (e.g., enhanced video content) based on the first video stream, the second video stream, and the time offset between them. For example, video content that simultaneously displays two views of the same highlight clip in a sporting event (the two views corresponding to the first video stream and the second video stream), wherein the videos of the two views of the highlight clip are synchronized.

[0032] Now please note FIG. 2A to FIG. 10B , which shows that it can be Figure 1 Various kinds of visually unique activities are identified in steps 101 and 102 of the video processing method 100 in order to determine a temporal offset between two video streams.

[0033] Alignment using the race clock

[0034] refer to Figure 2A and Figure 2B A first illustrative example of a suitable type of visually unique activity is changing the time displayed on a clock in a sporting event. In a specific example, the clock may be a game clock, i.e., a clock that displays the elapsed time or the remaining time in a game or its playing period.

[0035] Figure 2A and Figure 2B Video frames from two different video streams of a sporting event are shown, which in the particular example shown is a basketball game. As can be appreciated, game clock 200 is visible in both video streams, even though the video streams correspond to significantly different viewpoints of the game. The game clock is designed to be visible from multiple locations and directions of the sporting event - the function of the game clock is to allow viewers to know the current time in the game. Additionally, the game clock may be characterized by an abrupt change in appearance from one frame of the video stream to the next, so that changes in time on the game clock indicate a very narrow window of time. Furthermore, the game clock inherently changes in appearance regularly over the course of the sporting event. For these reasons (and / or other reasons), using changes in the time displayed on a clock at a sporting event as a proxy for the current time of the game clock may be a useful tool for spectating viewers. Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0036] exist Figure 1In steps 101 and 102 of method 100, for example, depictions of changing time on a clock may be identified by using an optical character recognition algorithm on each frame of each video stream. In a particular example, the optical character recognition algorithm may be applied to a particular portion of each frame that is expected to contain a clock. For example, knowledge of the actual shape, position, and orientation of the clock in the venue and knowledge of the positioning and orientation of the camera providing the video stream (e.g., by calibrating the camera) may be used to determine the portion of the frame that contains the clock. Alternatively (or additionally), a segmentation algorithm utilizing a suitably trained neural network (such as Mask R-CNN) may be used to determine the portion of the frame that contains the clock.

[0037] In step 106, for example, comparing the depiction of the changing time on the clock in the first video stream with the depiction of the changing time on the clock in the second video stream may include comparing the frame numbers for each video stream (or the times within each video stream) where one or more game clock time transitions occurred (e.g., from 12:00 to 12:01, or from 1:41 to 1:42). In general, performing such a comparison for multiple game clock transitions may more accurately estimate the time offset between the video streams.

[0038] Use camera flash for aiming

[0039] Another example of a suitable kind of visually distinctive activity is camera flashes occurring during a sporting event. Given that the intensity of the light emitted by a camera flash unit is very high, camera flashes are characterized by being visible from multiple positions and directions during a sporting event. Furthermore, due to the short duration of a camera flash (typically less than the time between two frames of the video stream) and the rapid increase in brightness produced by the camera flash unit, the appearance of the camera flash changes abruptly from one frame to the next in the video stream. Furthermore, camera flashes are characterized by changing appearance regularly over the course of a sporting event. For these reasons (and / or other reasons), the use of camera flashes as a means of visually distinctive activity is not an option. Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0040] Reference now FIG. 3A to FIG. 3D , which shows two video frames from each of the two video streams of a sporting event. Obviously, in the particular example shown, the sporting event is a basketball game. It can be noted that camera flash is particularly noticeable at indoor sporting events such as basketball or ice hockey games. Therefore, when using Figure 1 When the method 100 is used to temporally align video streams of indoor sporting events, camera flashes may be a particularly suitable choice for visually distinctive activities.

[0041] Figure 3A and Figure 3BShown are a first frame, in which no camera flashes occur, and a second frame, in which one or more camera flashes occur; both frames are from the same video stream and are therefore taken from the same viewpoint. Figure 3C and Figure 3D Similarly, the first and second frames from the second video stream are shown, where no camera flash occurs in the first frame ( Figure 3C ), but one or more camera flashes occurred in the second frame ( Figure 3D Obviously, the second video stream (such as Figure 3C and Figure 3D ) is from the first video stream (such as Figure 3A and Figure 3B The images were taken from different viewpoints (as shown in the figure).

[0042] exist Figure 1 In steps 101 and 102 of method 100 of , for example, depictions of camera flashes in a given video stream may be identified by analyzing the pixel intensities (e.g., total pixel intensities or average pixel intensities) of some or all frames in the video stream. The inventors have determined that frames depicting camera flashes will be significantly overexposed. Therefore, frames with particularly high pixel intensities are likely to depict camera flashes. This is particularly useful in Figure 4A and Figure 4B are shown in Figure 3A and Figure 3B The video stream of each frame shown in Figure 3C and Figure 3D Graphs of pixel intensity versus time for the frames of the video stream shown in . Clearly, each graph includes a very short-lived peak with a significantly higher pixel intensity than the immediately preceding or following frame. This peak in pixel intensity corresponds to Figure 3B and Figure 3D The camera flash described in .

[0043] In step 106, for example, comparing the depiction of camera flashes in the first video stream with the depiction of camera flashes in the second video stream can include comparing the frame numbers of each video stream at which particularly short duration and large amplitude peaks in pixel intensity occur. A simple approach can be used in some examples, which is to assume that the flashes are very short compared to the frame rate, so that each flash falls on only one frame (or zero frames if the flash occurs during the "dead time" of the camera sensor, such as when the shutter is closed), and each flash only significantly increases the average image intensity of that frame.

[0044] Other examples may use a more complex approach where the flash has some unknown shorter duration, and thus has the potential to span two or more consecutive frames. This approach may also take into account whether the camera has a global or rolling shutter, and may use the camera specifications / information to determine when the sensor is open or closed. From this and from the average frame intensity (in the case of a global shutter) or the intensity per row (in the case of a rolling shutter), the model can estimate the most likely flash start / end times.

[0045] Whether a simple or more complex method is used to determine the timing of each flash, performing a comparison across multiple camera flashes can provide a more accurate estimate of the time offset between video streams.

[0046] Alignment using an electronic display

[0047] Another example of a class of visually distinct activities suitable for aligning video streams is the changing of images and / or patterns displayed on one or more electronic displays at a sporting event. In a particular example, the electronic display is a billboard, but it may also be a "big screen" that displays highlights or other video content to spectators at a sporting event.

[0048] Electronic displays at sporting events are designed to be visible from multiple locations and directions during the sporting event - they are intended to be visible to most, if not all, spectators. Furthermore, electronic displays are characterized by abrupt changes in appearance from one frame of a video stream to the next. Furthermore, electronic displays inherently change appearance regularly during the course of a sporting event. For these reasons (and / or other reasons), using changes in images and / or patterns displayed on one or more electronic displays at a sporting event as a means of visualizing the sporting event is not an option. Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0049] exist Figure 1 In steps 101 and 102 of method 100, knowledge of the true shape, position, and orientation of the electronic display and knowledge of the true position and orientation of the camera providing the video stream can be used to identify the depiction of the electronic display in the video stream. Electronic displays at sporting events rarely move because they are large in size and typically have simple, regular shapes (e.g., they are rectangular). Therefore (or otherwise), knowledge of the shape, position, and orientation of the electronic display can be relatively straightforward to obtain and maintain. Knowledge of the position and orientation of the camera providing the video stream can be obtained by calibrating the intrinsic and extrinsic parameters of each camera. There are a variety of camera calibration methods to choose from, including the method disclosed in commonly assigned U.S. Patent No. 10,600,210B1 (the disclosure of which is incorporated herein by reference).

[0050] In other examples, alternative methods may be used to identify depictions of electronic displays in video streams, such as using a segmentation algorithm utilizing a suitably trained neural network, such as Mask R-CNN.

[0051] In step 106 of method 100, for example, comparing the depiction of an electronic display in a first video stream with the depiction of an electronic display in a second video stream may include comparing the change over time in pixel intensity of some or all pixels identified in step 101 as depicting (multiple) electronic displays in the first video stream with the change over time in pixel intensity of some / all pixels identified in step 102 as depicting (multiple) electronic displays in the second video stream.

[0052] FIG. 5A to FIG. 5D and Figure 6 An example of this approach is shown in . FIG. 5A to FIG. 5D Two video frames from each of two video streams from a sporting event are shown. In the particular example shown, the sporting event is a football match. There are several billboards 150 at the edge of the field. Obviously, the billboards 150 are in the first video stream (e.g., Figure 5A and Figure 5B ) and a second video stream (as shown in Figure 5C and Figure 5D ), even though the video streams correspond to significantly different viewpoints of the match. Figure 6 is a line graph depicting the pixel intensity of a subset 510 of pixels of a billboard as a function of time.

[0053] exist Figure 5A and Figure 5B In the example shown in , the first video stream (such as Figure 5A and Figure 5B The pixel subset 510 of the second video stream (shown in Figure 5C and Figure 5D 150). This can be accomplished by using knowledge of the real-world shape, position, and orientation of billboard 150 (as discussed above) and / or using image feature matching. This approach may provide a more robust and / or more accurate determination of temporal offset because the same portion of the display will be sampled from each video stream. However, the two pixel subsets 510 do not necessarily correspond to the same real-world location, for example because even if the sampled pixel subsets 510 correspond to different portions of the display, large-scale transitions (e.g., the display going black, or changing from one advertisement to the next) will still be identifiable in the pixel intensity time series of the two video streams. Such large-scale transitions are not necessarily identifiable in the real-world offsets of the two video streams. FIG. 5A to FIG. 5D Through Figure 5B and Figure 5DThe hatching of the display panel 150 in FIG. 1 shows the display, which is Figure 5A and Figure 5C Not visible in.

[0054] Back to Figure 6 , it can be seen that the first line 610 shows Figure 5A and Figure 5B The pixel intensity of each frame of the video stream is shown in FIG, while the second line 620 shows Figure 5C and Figure 5D The pixel intensity of the video stream over time for each frame is shown in . The two time series can be compared, for example using a cross-correlation function, in order to find a possible time shift between the two series.

[0055] Use ball movement to align

[0056] Another example of a type of visually distinctive activity suitable for aligning video streams is the movement of a ball, puck, or similar contested object in play at a sporting event.

[0057] As with the other kinds of visually distinctive activities discussed above, a ball (or other contested object) is characterized by being: visible from multiple locations and directions in the sporting event; changing appearance abruptly from one frame of the video stream to the next (particularly when the ball is kicked, thrown, caught, etc. by a player); and changing appearance regularly over the course of the sporting event. For these reasons (and / or others), using the movement of a ball (or other contested object) as a Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0058] In steps 101 and 102 of method 100, various methods may be used to identify depictions of a ball in a video stream. For example, various object detection algorithms may be utilized, including neural network algorithms (such as Faster R-CNN or YOLO) and non-neural network algorithms (such as SIFT).

[0059] In step 106, for example, comparing the depiction of the movement of the ball in the first video stream with the depiction of the movement of the ball in the second video stream may include comparing changes in the movement of the ball in the first video stream with changes in the movement of the ball in the second video stream. The movement of the ball may change abruptly from frame to frame in the video streams, such as Fig. 7A and Figure 7B As seen in Fig. 7A and Figure 7B Two video frames from a video stream of a sporting event are shown. Obviously, the sporting event is a football match. Fig. 7A In the example above, the ball is moving toward the player; then the player kicks the ball, which causes it to suddenly move in the opposite direction, e.g. Figure 7BSuch abrupt changes in movement are helpful for temporal alignment because they are associated with very narrow time windows and tend to be visually distinctive.

[0060] Additionally or alternatively, for example, comparing the depiction of movement of a ball in a first video stream with the depiction of movement of a ball in a second video stream may include comparing horizontal or vertical movement of the ball in the first video stream with corresponding movement of the ball in the second video stream. In this context, horizontal movement of the ball in the video streams means movement in a left-right direction within a frame of the video stream, while vertical movement means movement in an up-down direction within a frame of the video stream.

[0061] When the cameras producing the first and second video streams have been calibrated (so that their extrinsic and intrinsic parameters are known), it is possible to determine how different the viewpoints of the two video streams are. (As noted above, a variety of camera calibration methods are available, including the method disclosed in commonly assigned U.S. Patent No. 10,600,210B1.) When the viewpoints are relatively similar, suitable performance can be achieved simply by comparing the 2D horizontal motion in the first video stream (i.e., the movement of the ball in the left-right direction within the frames of the first video stream) with the 2D horizontal motion in the second video stream (i.e., the movement of the ball in the left-right direction within the frames of the second video stream).

[0062] A more general approach is to convert the movement of the ball depicted in the second video stream into 3D movement and then determine the component of this 3D movement in the horizontal (or vertical) direction of the first video stream without requiring the video streams to have similar viewpoints. In this way, a like-for-like comparison of the movements can be made. Converting the 2D movement depicted in the video streams to 3D movement can be achieved in various ways. In one example, the 3D movement can be determined by triangulating the ball using the second video stream in combination with a third video stream, where the third video stream is time-aligned with the second video stream (i.e., the time offset between the second and third video streams is known). In addition, to help perform triangulation, the cameras that produce the second and third video streams can be calibrated (so that their external and internal parameters are known).

[0063] Fig. 8A and Figure 8B The application of this method to Fig. 7A and Figure 7B An example of a video stream with frames shown in . Fig. 8A Shows the Fig. 7A and Figure 7BThe change in horizontal motion from one frame to the next of the video streams shown in FIG. 2D motion in the second video stream is converted to 3D motion using triangulation using a third synchronized video stream as described above. The horizontal component of this 3D motion in the first video stream (e.g., Fig. 7A and Figure 7B ). Figure 8B The frame-to-frame variation of the 3D motion component determined in this way is shown in . It can be seen that Fig. 8A and Figure 8B Both show peaks of very short duration. Using, for example, a cross-correlation function, the two time series can be compared in order to find a possible time shift between the two series.

[0064] Note that while the above method converts 2D motion in the second video stream into 3D motion, 2D motion in the first video stream can be converted into 3D motion. Figure 1 The method 100 processes the first and second video streams symmetrically, and the designation of a given one of the two video streams as "first" or "second" is essentially arbitrary.

[0065] Note also that, although horizontal and vertical movements are depicted above, this is for simplicity only, and 2D movements in other directions may also be compared in step 106 .

[0066] Use head position for alignment

[0067] Another example of a class of visually distinct activities suitable for aligning video streams is the change in head position of a participant in a sporting event (e.g., the change in head position of an athlete and / or referee at a sporting event). The head of a participant is characterized by being visible from multiple positions and directions during the sporting event and by changing appearance regularly during the sporting event. For these reasons (and / or other reasons), using the change in head position of a participant as a Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0068] In steps 101 and 102 of method 100, various methods may be used to identify the depiction of the head of the participant in the video stream. For example, various computer vision algorithms may be used, such as using a neural network (such as FasterR-CNN, YOLO, or OpenPose) or using a feature detection algorithm (such as SIFT or SURF).

[0069] In step 106 of method 100, for example, comparing the changes in the depiction of the head position of the participant in the first video stream with the changes in the depiction of the head position of the participant in the second video stream may include: converting the (2D) head positions depicted in several frames of the second video stream to 3D head positions and then reprojecting these 3D positions into one of the frames of the first video stream. The method of converting the 2D head positions to 3D head positions may be similar to the method of converting the 2D ball movement to 3D ball movement in the "Alignment Using Ball Movement" section above.

[0070] Re-projecting the head position from the second video stream allows for an apples-to-apples comparison of the head positions. The closest matching frame between the identified head position 910 and the re-projected head position 920 indicates that there may be a temporal offset between the two video streams. This Fig. 9A and Fig. 9B is shown in Fig. 9A and Fig. 9B The same frame from the first video stream of the football game is shown, and therefore the same identified head position 910 is shown. However, Fig. 9A and Fig. 9B Each of the 920 images shows a re-projected head position 920 corresponding to a different frame in the second video stream. Fig. 9A The re-projected head position shown in 920 is Fig. 9B The re-projected head position 920 shown in matches better.

[0071] The process of converting the head position to 3D and reprojecting to find the closest matching frame to the first video stream can be repeated for additional frames from the first video stream to provide a further estimate of the temporal offset and therefore more accurately determine the temporal offset between the video streams.

[0072] Using body posture for alignment

[0073] Another example of a class of visually distinctive activities suitable for aligning video streams is changes in the body pose of participants in a sporting event (e.g., changes in the position and orientation of the heads, limbs, and torsos of players and / or referees at a sporting event). The body pose of a participant is characterized by being visible from multiple positions and directions during the sporting event and by changing appearance regularly during the sporting event. For these reasons (and / or other reasons), using changes in the body pose of a participant as a proxy for the presence of an event in the sporting event is useful. Figure 1 The first category of visually distinct activities in method 100 may provide robust and / or accurate video stream alignment.

[0074] In steps 101 and 102 of method 100, various methods may be used to identify depictions of body poses of participants in the video stream. For example, various computer vision algorithms may be used, such as using neural networks (such as MaskR-CNN, OpenPose, AlphaPose). For example, a computer vision algorithm may be used to identify key points 1010 on the body of each participant, such as the head, shoulders, elbows, wrists, hips, knees, and ankles of the participant 1020, such as Fig. 10A and Fig. 10B The set of locations of these key points characterizes the body posture of the participant.

[0075] For example, step 106 of method 100 may include sub-steps similar to those described in the "Using Head Position for Alignment" section above. Specifically, the 2D positions of the body key points in several frames of the second video stream may be converted to 3D positions, which are then reprojected onto the frames from the first video stream to find the closest matching frames.

[0076] Compared to the position of the participant's head, the position of the body posture will tend to change more rapidly, e.g. Fig. 10B and Fig. 10A It is obvious from the comparison that although the head positions are Fig. 10A and Fig. 10B There is only a small amount of movement between the two frames shown in , but the positions of other key points (such as the elbow and wrist) move a lot. Therefore, using body pose to align the video streams can provide relatively accurate results. In addition, relatively accurate results can be achieved even if only one participant is visible in the video frame, because this still provides multiple key points that can be used for comparison.

[0077] combination

[0078] It is envisaged that in order to provide a more robust temporal alignment of video streams, two or more of the above methods may optionally be combined. In this regard, reference is again made to Figure 1 As shown, method 100 includes optional step 103 of identifying one or more depictions of a second category of visually unique activity of a different category in the first video stream, and optional step 104 of identifying one or more depictions of the second category of visually unique activity in the second video stream. Accordingly, and as Figure 1 As shown in , determining 106 the temporal offset between the first video stream and the second video stream optionally includes comparing 109 the depiction of the second type of visually unique activity in the first video stream with the depiction of the second type of visually unique activity in the second video stream. For example, the second type of visually unique activity may be the same as that described above with reference to FIG. 2A to FIG. 10B Any of the examples shown.

[0079] In an example, step 109 of comparing the depictions of the second category of visually unique activity in the first video stream with the depictions of the second category of visually unique activity in the second video stream can produce one estimate for the temporal offset, while step 108 of comparing the depictions of the first category of visually unique activity in the first video stream with one or more depictions of the first category of visually unique activity in the second video stream can produce another estimate for the temporal offset. In such an example, in step 106, the temporal offset can be determined based on both of these estimates.

[0080] It should be noted that although Figure 1 Steps 101 and 102 are shown to occur before optional steps 103 and 104, but in other embodiments, steps 101 and 102 may occur after optional steps 103 and 104, or may occur simultaneously with steps 103 and 104. In practice, steps 101 to 104 may be performed in any suitable order.

[0081] Reference now Fig.11 , which is a schematic diagram of a video processing system according to another illustrative embodiment. As shown, the video processing system 1100 includes a memory 1110 and a processor 1120. The memory 1110 stores a plurality of computer executable instructions 1112 to 1118, which when executed by the processor 1120 cause the processor to:

[0082] identifying 1112 one or more depictions of a first category of visually distinctive activity in a first video stream depicting a sporting event;

[0083] identifying 1114 one or more depictions of a first category of visually distinctive activity in a second video stream depicting the sporting event; and

[0084] A time offset between the first video stream and the second video stream is determined 1116.

[0085] Determination of the temporal offset includes comparing 1118 one or more depictions of the first category of visually unique activity in the first video stream to one or more depictions of the first category of visually unique activity in the second video stream.

[0086] In the video processing system 1100, the above reference FIG. 2A to FIG. 10B Any type of visually unique activity described in the visually unique activity. In addition, the above reference FIG. 2A to FIG. 10B The depicted method for identifying and comparing depictions of visually unique activities may be implemented in the video processing system 1100 .

[0087] Memory 1110 may be of any suitable type, such as including one or more of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), and flash memory.

[0088] although Fig.11 Only one processor 1120 is shown in FIG. 1 , but it will be understood that the system 1100 may include multiple processors. Specifically (but not exclusively), in the case where the system 1100 includes multiple processors, the system 1100 may include a GPU and / or a neural network accelerator.

[0089] The video streams of the sporting event may be received from various sources. Specifically, it is envisioned that in the case where the first video stream and / or the second video stream are generated by a portable device 1140, 1150 (such as a smartphone or tablet device) including a camera, the video stream may be generated by a portable device 1140, 1150 (such as a smartphone or tablet device). Fig.11 The inventors have recognized that there is a lack of alignment solutions for video streams generated by such devices compared to video streams generated by conventional broadcast cameras. Nevertheless, in the case where the first video stream and / or the second video stream are generated by conventional broadcast cameras, the same alignment solutions can be used. Figure 1 Method 100.

[0090] The video stream may be received via any suitable communication system. For example, the video stream may be transmitted via the Internet, via an intranet (e.g., which includes the video processing system 1100), via a cellular network (particularly but not exclusively, where the video stream is generated by the portable device 1140, 1150), or via a bus of the video processing system 1100.

[0091] It should be noted that although the above reference Figures 1 to 11 The video stream in the depicted embodiment depicts a sporting event, but it is contemplated that many embodiments, particularly those described above with reference to Figure 1 The method 100 described above with reference to Fig.11 The described system 1100 may be adapted to process video streams depicting other types of live events, such as, for example, concerts, comedy shows, or plays.

[0092] It will be understood that any feature described with respect to any one embodiment may be used alone, or in combination with other features described, or in combination with one or more features of any other embodiment in the embodiment or any combination of any other embodiment in the embodiment. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. A video processing method, comprising: identifying one or more depictions of a first category of visually distinctive activity in a first video stream depicting a sporting event; identifying one or more depictions of the first category of visually distinctive activity in a second video stream depicting the sporting event; as well as determining a time offset between the first video stream and the second video stream, Wherein the determination of the time offset comprises comparing the one or more depictions of the first kind of visually unique activity in the first video stream with the one or more depictions of the first kind of visually unique activity in the second video stream.

2. The method of claim 1, wherein the first category of visually unique activity corresponds to a change in time displayed on a clock in the sporting event. 3 . The method of claim 1 , wherein the first category of visually distinctive activity corresponds to camera flashes occurring during the sporting event.

4. The method of claim 1, wherein the first category of visually unique activity corresponds to a change in images displayed on one or more electronic displays at the sporting event.

5. The method of claim 4, wherein the one or more electronic displays comprise a billboard.

6. The method of claim 1, wherein the first category of visually unique activity corresponds to movement of a ball or puck.

7. The method of claim 1, wherein the first category of visually unique activity corresponds to movement of one or more participants in the sporting event.

8. The method of claim 7, wherein the first category of visually unique activity corresponds to movement of the heads of the one or more participants in the sporting event.

9. The method of claim 7, wherein the first category of visually unique activity corresponds to changes in respective postures of the one or more participants in the sporting event.

10. The method according to any one of claims 1 to 9, comprising: The first video stream is received from a first portable device including a camera. The method of claim 10 , wherein the first portable device is a smartphone.

12. The method according to any one of claims 1 to 11, comprising: identifying one or more depictions of a second category of visually distinctive activity in the first video stream; as well as identifying one or more depictions of the second category of visually distinctive activity in the second video stream, Wherein the determination of the time offset comprises comparing the one or more depictions of the second kind of visually unique activity in the first video stream with the one or more depictions of the second kind of visually unique activity in the second video stream.

13. A video processing system, comprising: a memory storing a plurality of computer executable instructions; one or more processors executing the computer executable instructions, the computer executable instructions causing the one or more processors to: identifying one or more depictions of a first category of visually distinctive activity in a first video stream depicting a sporting event; identifying one or more depictions of the first category of visually distinctive activity in a second video stream depicting the sporting event; as well as Determining a time offset between the first video stream and the second video stream, wherein the system determines the time offset at least by comparing the one or more depictions of the first kind of visually unique activity in the first video stream with the one or more depictions of the first kind of visually unique activity in the second video stream.

14. The system of claim 13, wherein the first category of visually unique activity corresponds to a change in time displayed on a clock in the sporting event.

15. The system of claim 13, wherein the first category of visually distinctive activity corresponds to camera flashes occurring during the sporting event.

16. The system of claim 13, wherein the first category of visually unique activity corresponds to a change in images displayed on one or more electronic displays at the sporting event.

17. The system of claim 16, wherein the one or more electronic displays comprise a billboard.

18. The system of claim 13, wherein the first category of visually unique activity corresponds to movement of a ball or puck being played in the sporting event.

19. The system of claim 13, wherein the first category of visually unique activity corresponds to movement of one or more participants in the sporting event.

20. The system of claim 19, wherein the first category of visually unique activity corresponds to movement of the heads of the one or more participants in the sporting event.

21. The system of claim 19, wherein the first category of visually unique activity corresponds to changes in respective postures of the one or more participants in the sporting event.

22. The system of any one of claims 13 to 21, wherein the system is configured to receive the first video stream from a first portable device comprising a camera.

23. The system of claim 22, wherein the first portable device is a smartphone.

24. The system of any one of claims 13 to 23, wherein the computer executable instructions cause the one or more processors to: identifying one or more depictions of a second category of visually distinctive activity in the first video stream; and identifying one or more depictions of the second category of visually distinctive activity in the second video stream, Wherein the system determines the time offset by at least comparing the one or more depictions of the second kind of visually unique activity in the first video stream with the one or more depictions of the second kind of visually unique activity in the second video stream.

25. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions that, when executed by one or more processors, cause the one or more processors to: identifying one or more depictions of a first category of visually distinctive activity in a first video stream depicting a sporting event; identifying one or more depictions of the first category of visually distinctive activity in a second video stream depicting the sporting event; as well as determining a time offset between the first video stream and the second video stream, Wherein the system determines the temporal offset by at least comparing the one or more depictions of the first kind of visually unique activity in the first video stream with the one or more depictions of the first kind of visually unique activity in the second video stream.

26. The non-transitory computer-readable storage medium of claim 25, wherein the first category of visually unique activity corresponds to a change in time displayed on a clock in the sporting event.

27. The non-transitory computer-readable storage medium of claim 25, wherein the first category of visually distinctive activity corresponds to camera flashes occurring during the sporting event.

28. The non-transitory computer-readable storage medium of claim 25, wherein the first category of visually unique activity corresponds to a change in an image displayed on one or more electronic displays at the sporting event.

29. The non-transitory computer-readable storage medium of claim 28, wherein the one or more electronic displays comprise a billboard.

30. The non-transitory computer-readable storage medium of claim 25, wherein the first category of visually unique activity corresponds to movement of a ball or puck being played at the sporting event.

31. The non-transitory computer-readable storage medium of claim 25, wherein the first category of visually unique activity corresponds to movement of one or more participants in the sporting event.

32. The non-transitory computer-readable storage medium of claim 31, wherein the first category of visually distinctive activity corresponds to movement of a head of the one or more participants in the sporting event.

33. The non-transitory computer-readable storage medium of claim 31, wherein the first category of visually unique activity corresponds to changes in respective postures of the one or more participants in the sporting event.

34. The non-transitory computer-readable storage medium of any one of claims 25 to 33, wherein the system is configured to receive the first video stream from a first portable device comprising a camera.

35. The non-transitory computer-readable storage medium of claim 34, wherein the first portable device is a smartphone.

36. The non-transitory computer-readable storage medium of any one of claims 25 to 35, wherein the computer-executable instructions cause the one or more processors to: identifying one or more depictions of a second category of visually distinctive activity in the first video stream; and identifying one or more depictions of the second category of visually distinctive activity in the second video stream, Wherein the one or more processors determine the time offset by at least comparing the one or more depictions of the second kind of visually unique activity in the first video stream with the one or more depictions of the second kind of visually unique activity in the second video stream.

Images