Method for tracking a camera bar and related methods, systems and computer program products

Abstract

A method is disclosed for setting up non-randomly positioned markers within a field of view of a camera bar and tracking a pose of the camera bar, the method comprising the steps of: (i) viewing a field of view with the camera bar, the camera bar being fixedly attached to a video camera; (ii) displaying the camera bar's view of the field of view on a display of a computer system; (iii) the computer system generating a pattern of non-random marker positions including positions within the field of view and displaying the pattern of non-random marker positions within the field of view together with the camera bar's view of the field of view on the display of the computer system; and (iv) the computer system detecting the markers in the field of view of the camera bar by displaying the non-random markers generated in step (iii). The method includes detecting when a marker is located at a marker position in the field of view of the camera bar, (v) in response to the computer system detecting in step (iv) that a marker is located at a non-random marker position, the computer system recording the markers and their respective positions in the field of view of the camera bar, (vi) repeating steps (iv) and (v) until a predetermined number of different markers and their respective marker positions have been recorded, (vii) the computer system matching the markers detected in the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar, and (viii) repeating step (vii) to track the pose of the camera bar. Related methods, systems, and computer program products are disclosed.

Description

[Technical field]

[0001] 1. Field of the invention The field of the invention relates to tracking a camera bar or a video camera that includes a camera bar, and related methods, systems, computer program products, video files, and video streams. [Background technology]

[0002] 2.Technical background When using a system for creating virtual scenes that combine live video augmented by other images, such as computer-generated images, users would like to be able to set up the system quickly, since studio time is expensive. It is also desirable to reduce the energy consumption of such systems, since video processing tends to use a significant amount of energy. Users would also like to be able to view the virtual scenes, which combine live video augmented by other images, such as computer-generated images, in real time.

[0003] 3. Consideration of Related Technologies US2007 / 0248283(A1) discloses a system for creating virtual scenes combining live video augmented with other images, including computer-generated images. In one embodiment, it includes a scene camera with an attached tracking camera that views a tracking marker pattern having a plurality of tracking markers with identifying indicia. The tracking marker patterns are closely positioned such that, when viewed by the tracking camera, the coordinate position of the scene camera can be determined in real time. The disclosure also includes a filtering algorithm that changes based on camera motion and maintains accurate positioning. It is disclosed that alternative embodiments can include an array of randomly distributed markers, as well as markers of various sizes and orientations. Summary of the Invention

[0004] According to a first aspect of the invention, there is provided a method of setting up non-randomly positioned markers within the field of view of a camera bar and tracking a pose of the camera bar, the method comprising: (i) viewing the field of view with a camera bar, the camera bar being fixedly attached to a video camera; (ii) displaying the camera bar's view of the field of view on a display of a computer system; (iii) the computer system generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view, together with a view of the camera bar of the field of view, on a display of the computer system; (iv) detecting by the computer system that a marker is located in the field of view of the camera bar at the non-random marker position generated in step (iii); (v) in response to the computer system detecting in step (iv) that a marker has been placed in a non-random marker location within the field of view of the camera bar, the computer system recording the markers and their respective locations within the field of view; (vi) repeating steps (iv) and (v) until a predetermined number of different markers and their respective marker positions have been recorded; (vii) the computer system matching the markers detected within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (viii) repeating step (vii) to track the pose of the camera bar.

[0005] The generated pattern of non-random marker positions may include positions outside the initial field of view of the camera bar.

[0006] An advantage is that the non-randomly positioned markers in the field of view of the camera bar can be set up quickly by just one person who can place the markers while looking at the display, while the computer system can detect when the person places a marker in a non-random marker position. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked using less energy, which reduces the energy requirements for the computation. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked in real time, which reduces the time requirements for the computation.

[0007] The camera bar and the computer system may be connected by a data cable, or they may be connected by a wireless data connection.

[0008] The method may include storing a predetermined number of different markers recorded and their respective marker positions.

[0009] The method may be one in which the generated pattern of non-random marker positions within the field of view starts from an initial indicator detected in the field of view by a computer system. An advantage is that the set-up process may be speeded up because it starts from a pre-selected point that may be selected to speed up the set-up.

[0010] The method may be one in which the generated pattern of non-random marker positions within the field of view is generated using a path that is represented by a continuously differentiable (C1) continuous path or function. The advantage is that since the non-randomly positioned marker positions are very well defined, the pose of the camera bar can be tracked using less energy, which reduces the energy requirements for the tracking computation.

[0011] The method may be one in which the generated pattern of non-random marker positions within the field of view is generated using a smooth continuous path or a path represented by a function. The advantage is that the pose of the camera bar can be tracked using less energy since the non-randomly positioned marker positions are very well defined, which reduces the energy requirements for the tracking computation.

[0012] The method may be one in which the marker locations are equally spaced along the path.

[0013] The method may be one in which a generated pattern of non-random marker positions within the field of view is generated from a set of parameters that are passed to one or more well-defined, deterministic equations.

[0014] The method can be one where a formula generates a spiral pattern and the parameters include the location of the initial marker, the height of the camera bar above the floor, and the orientation angle of the camera bar. An advantage is that a spiral pattern tailored to a particular environment is provided, which can speed up setup time.

[0015] The method may be one in which the generated pattern of non-random marker positions within the field of view is a spiral pattern. The advantage is that the pose of the camera bar can be tracked using less energy since the non-randomly positioned marker positions are very distinct, which reduces the energy requirements for the tracking computation.

[0016] The method can be one in which the generated pattern of non-random marker positions within the field of view is a quasi-crystalline pattern. The advantage is that the pose of the camera bar can be tracked using less energy because the non-randomly positioned marker positions are very well defined, which reduces the energy requirements for the tracking computation.

[0017] The method may be such that the generated pattern of non-random marker positions within the field of view is on a non-periodic grid, or on a periodic grid, or on a non-periodic lattice, or on a periodic lattice. The advantage is that since the non-randomly positioned marker positions are very well defined, the pose of the camera bar can be tracked using less energy, which reduces the energy requirements for the tracking computation.

[0018] The method may be one in which the generated pattern of non-random marker positions within the field of view lies on a set of non-intersecting closed curves, e.g., a set of concentric circles. The advantage is that the pose of the camera bar can be tracked using less energy since the non-randomly positioned marker positions are very well defined, which reduces the energy requirements for the tracking computation.

[0019] The method may be one in which the generated pattern of non-random marker positions within the field of view lies on a set of non-intersecting open curves, e.g., on a set of concentric semicircles. The advantage is that the pose of the camera bar can be tracked using less energy since the non-randomly positioned marker positions are very well defined, which reduces the energy requirements for the tracking computation.

[0020] The method may be such that in step (iii) a pattern of non-random marker positions within the field of view is overlaid onto the field of view of the camera bar.

[0021] The method may be such that in step (iv) the computer system indicates within the display that it has detected that a marker has been placed within a non-random marker position in the field of view of the camera bar, for example by changing the displayed colour of the non-random marker position, for example from red to green.

[0022] The method may be one in which the camera bar is a monoscopic camera bar.

[0023] The method further comprises the step (vii): (a) using a marker extraction algorithm to detect 2D positions of markers in a sequence of images viewed by a camera bar, where the first and last images in the sequence are taken from different camera bar positions, and labeling the markers throughout the sequence using nearest neighbor matching; (b) extracting a matrix using a robust extractor to extract rotations and translations with unknown scale factors from the first frame of the sequence to the last frame of the sequence; (c) using the camera bar calibration data to calculate 3D marker positions using the detected 2D marker positions, where the scale factor is unknown; (d1) matching the calculated scaled positions of the 3D markers to the 3D path of the markers, the 3D path in the world coordinate system being known and including fitting the set of calculated 3D marker positions to the 3D path of the markers, including deriving a scale factor; (d2) matching the calculated scaled positions of the 3D markers to the 3D positions of the markers, where the 3D positions of the markers in the world coordinate system are known and includes fitting the set of calculated 3D marker positions to the 3D positions of the markers, including deriving a scale factor; (e) modeling the matching solution as a SIM(3) Lie group to represent the camera bar pose, where the matching solution includes scale; and (f) extracting a camera bar pose from the matching solution in the world coordinate system. An advantage is that a simple camera bar, a monoscopic camera bar, can be used.

[0024] The method may be one in which step (d1) or (d2) is carried out by using iterative closest point (ICP) fitting or curve fitting, or by using a non-linear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), or by using a Gauss-Newton algorithm.

[0025] The method may be a method in which in step (b) a random sample consensus, RANSAC, is used.

[0026] The method may be one in which the camera bar is a stereoscopic camera bar including two cameras mounted at a fixed distance apart in a stereoscopic configuration. An advantage is that the pose of the camera bar can be tracked using less energy because the non-randomly positioned marker positions are determined quickly and accurately, which reduces the energy requirements for the tracking computation.

[0027] The method may be one in which the camera bar includes three cameras arranged in a triangle (e.g., an equilateral triangle) at a fixed distance from each other. The advantage is that the pose of the camera bar can be tracked using less energy because the non-randomly positioned marker positions are quickly and accurately determined, which reduces the energy requirements for the tracking computation.

[0028] The method may be one in which the camera bar includes multiple cameras. An advantage is that the pose of the camera bar can be tracked using less energy because the non-randomly positioned marker positions are quickly and accurately determined, which reduces the energy requirements for the tracking computation.

[0029] The method further comprises the step (vii): (a) using a marker extraction algorithm to detect the 2D positions of markers on the images seen by the camera bar; (b) calculating 3D marker positions using the detected 2D marker positions, using the camera bar calibration data, and using the camera bar coordinate system; (c1) matching the calculated 3D marker positions to a 3D path of the marker, where the 3D path in the world coordinate system is known and includes fitting a set of calculated 3D marker positions to the 3D path of the marker; (c2) matching the calculated 3D marker positions to 3D positions of the markers, where the 3D positions of the markers in the world coordinate system are known and the matching includes fitting the set of calculated 3D marker positions to the 3D positions of the markers; (d) Modeling the matching solution as an SE(3) Lie group to represent the camera bar pose; (e) extracting, in the world coordinate system, a camera bar pose from the matching solution. An advantage is that because the non-randomly positioned marker positions are quickly and accurately determined, the camera bar pose can be tracked using less energy, which reduces the energy requirements for the tracking computation.

[0030] The method may be one in which step (c1) or (c2) is carried out by using iterative closest point (ICP) fitting or curve fitting, or by using a non-linear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), or by using a Gauss-Newton algorithm.

[0031] The method may be one in which the camera bar pose is provided instantaneously.

[0032] A method could be such that the field of view of the camera bar does not coincide at all with the field of view of the video camera, the advantage being that the markers are not visible in the video recorded by the video camera.

[0033] A method may be such that the field of view of the camera bar partially overlaps with the field of view of the video camera, the advantage being that some markers are not visible in the video recorded by the video camera.

[0034] The method may be such that the field of view of the camera bar includes the entire field of view of the video camera, the advantage being that the tracking accuracy of the camera bar is improved.

[0035] The method may be such that the marker has a particular shape and / or color and / or is made of a particular material.

[0036] The method may be one in which the markers used do not differ significantly from each other in appearance. The advantage is that there is no accidental mixing of different markers, allowing for a quick setup of non-randomly positioned markers within the field of view of the camera bar.

[0037] The method may be one in which the markers used are approximately spherical or approximately circular. The advantage is that there is no orientation error of the markers, so that non-randomly positioned markers within the field of view of the camera bar can be quickly set up.

[0038] The method may be one in which the camera bar pose includes the camera bar rotation and the camera bar translation, i.e. 6 degrees of freedom. The advantage is accurate tracking of the camera bar.

[0039] The method may be one in which the computer system is a laptop computer, a desktop computer, a tablet computer, or a smartphone.

[0040] The method may be such that, after the camera pose has been tracked for the first time, the camera bar is moved to display a new field of view that overlaps the previous field of view, then a pattern of non-random marker positions in the new field of view is displayed on the display of the computer system along with a view of the camera bar in the new field of view, and then steps (iv) and (v) are performed. An advantage is that camera pose tracking can be provided for fields outside the previous field of view.

[0041] The method may comprise repeating steps (iv) and (v) together with performing step (viii). An advantage is that further non-randomly positioned markers can be added to improve tracking of the camera bar.

[0042] The method may include, when performing step (viii), the number of different markers and respective marker positions recorded is at least 10, or is in the range of 10 to 1000, or is in the range of 20 to 500. An advantage is that it improves tracking of the camera bar without requiring excessively long set-up times.

[0043] According to a second aspect of the present invention there is provided a system including a video camera, a camera bar fixedly attached to the video camera, and a computer system including a display, the computer system comprising: (i) displaying a field of view of the camera bar on a display of a computer system; (ii) generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view on a display of the computer system together with a view of the camera bar of the field of view; (iii) detecting, within the field of view of the camera bar, that a marker is located within the location of the non-random marker location generated in (ii); and (iv) in response to detecting that a marker is located within a non-random marker location within the field of view of the camera bar in (iii), recording the markers and their respective locations within the field of view; (v) repeating (iii) and (iv) until a predetermined number of different markers and their respective marker positions have been recorded; (vi) matching the detected markers within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (vii) repeating (vi) to track the pose of the camera bar.

[0044] An advantage is that the non-randomly positioned markers in the field of view of the camera bar can be set up quickly by just one person who can place the markers while looking at the display, while the computer system can detect when the person places a marker in a non-random marker position. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked using less energy, which reduces the energy requirements for the computation. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked in real time, which reduces the time requirements for the computation.

[0045] The system may be configured to carry out the method of any of the first aspects of the invention.

[0046] According to a third aspect of the present invention there is provided a computer program product, the computer program product comprising: (i) displaying a field of view of the camera bar on a display of a computer system; (ii) generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view on a display of the computer system together with a view of the camera bar of the field of view; (iii) detecting, within the field of view of the camera bar, that a marker is located within the location of the non-random marker location generated in (ii); and (iv) in response to detecting that a marker is located within a non-random marker location within the field of view of the camera bar in (iii), recording the markers and their respective locations within the field of view; (v) repeating (iii) and (iv) until a predetermined number of different markers and their respective marker positions have been recorded; (vi) matching the detected markers within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (vii) Repeat (vi) to track the pose of the camera bar.

[0047] An advantage is that the non-randomly positioned markers in the field of view of the camera bar can be set up quickly by just one person who can place the markers while looking at the display, while the computer system can detect when the person places a marker in a non-random marker position. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked using less energy, which reduces the energy requirements for the computation. An advantage is that since it is computationally efficient to match the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very specific, so the pose of the camera bar can be tracked in real time, which reduces the time requirements for the computation.

[0048] The computer program product may be executable on a computer system to carry out the method of any of the first aspects of the invention.

[0049] According to a fourth aspect of the present invention, there is provided a method for generating augmented reality video in real time by mixing or compositing computer generated 3D objects with a video feed from a video camera, the method comprising: (a) the body of the video camera can be moved in 3D and sensors including accelerometers and gyros sensing across six degrees of freedom within the video camera or attached directly or indirectly to the video camera provide real-time positioning data that allows the 3D position and 3D orientation of the video camera to be calculated; (b) for example, a camera bar including two cameras forming a stereoscopic system is fixed directly or indirectly to a video camera; (c) that real-time positioning data is then automatically used to create, invoke, render, or modify a computer-generated 3D object; (d) the resulting computer-generated 3D object is then mixed or composited in real time with a video feed from a video camera to provide augmented reality video, for example for a television broadcast, a movie, or a video game; (e) The 3D position and orientation of the video camera is determined with reference to a 3D map of the real world, for example by using video flow in which a camera bar including two cameras forming a stereoscopic vision system surveys the field of view and software running on a processor is used to detect non-randomly positioned markers in the field of view, in addition to real-time 3D positioning data from the sensor, the markers and their non-random positions being pre-recorded in the processor and included in the 3D map of the real world, the markers having been previously added to the field of view manually or artificially.

[0050] An advantage is that since it is computationally efficient to match markers detected within the field of view of the camera bar with recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very well defined, so the pose of the video camera can be tracked using less energy, which reduces the computational energy requirements.An advantage is that since it is computationally efficient to match markers detected within the field of view of the camera bar with recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very well defined, so the pose of the video camera can be tracked in real time, which reduces the computational time requirements.

[0051] The method may comprise the method of any of the aspects of the first aspect of the invention.

[0052] According to a fifth aspect of the present invention there is provided a mixing or blending system, the mixing or blending system comprising: (i) a video camera; and (ii) sensors including accelerometers and gyros sensing across six degrees of freedom; (iii) a camera bar including, for example, two cameras forming a stereoscopic system; (iv) a processor; Mixing or compositing computer-generated 3D objects with real-time video feeds from video cameras to generate augmented reality video in real time, for example for television broadcasts, movies, or video games; (a) the body of the video camera can be moved in 3D and sensors within, or attached directly or indirectly to, the video camera provide real-time positioning data that enables the 3D position and 3D orientation of the video camera to be calculated; (b) for example, a camera bar including two cameras forming a stereoscopic system is fixed directly or indirectly to a video camera; (c) the system is configured to automatically use the real-time positioning data to create, invoke, render, or modify a computer-generated 3D object; (d) the system is configured to mix or composite the resulting computer-generated 3D objects in real time with a video feed from a video camera to provide real-time augmented reality video, e.g., for a television broadcast, a movie, or a video game; (e) the system is configured to determine the 3D position and orientation of the video camera with reference to a real-world 3D map, for example by using a video flow in which a camera bar including two cameras forming a stereoscopic vision system surveys a field of view and software running on a processor is configured to detect non-randomly positioned markers within the field of view, in addition to real-time 3D positioning data from the sensor, the markers and their non-random positions being pre-recorded in the processor and included in the real-world 3D map, and the markers have been previously added to the field of view manually or artificially.

[0053] An advantage is that since it is computationally efficient to match markers detected within the field of view of the camera bar with recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very well defined, so the pose of the video camera can be tracked using less energy, which reduces the computational energy requirements.An advantage is that since it is computationally efficient to match markers detected within the field of view of the camera bar with recorded markers and their respective non-randomly positioned marker positions, the non-randomly positioned marker positions are very well defined, so the pose of the video camera can be tracked in real time, which reduces the computational time requirements.

[0054] The system may include a system according to any of the aspects of the second aspect of the invention.

[0055] According to a sixth aspect of the present invention there is provided a video file of a video created using a method of any of the aspects of the fourth aspect of the present invention.

[0056] According to a seventh aspect of the present invention there is provided a video stream of a video produced using a method according to any of the fourth aspect of the present invention.

[0057] Aspects of the invention may be combined. [Brief description of the drawings]

[0058] Aspects of the present invention will now be described, by way of example only, with reference to the following figures:

[0059] [Figure 1] 1 shows an example of a camera bar looking into a field of view, which includes an indicator. [Diagram 2] 1 illustrates an example of a camera bar looking into a field of view that includes an indicator, and in which a set of non-random marker positions are shown on a spiral starting from the indicator. [Diagram 3] 1 illustrates an exemplary user interface presented on a display of a computer system when a non-random spiral pattern is initially generated, with a camera bar pointed at an element of the non-random pattern and the display showing a live video stream including the element of the non-random pattern, the element of the non-random pattern being overlaid on top of the video stream to guide the user. [Figure 4] 4 shows an exemplary user interface presented on a display of a computer system in which several markers (solid black circles) are arranged in the non-random spiral pattern of FIG. [Diagram 5] 1 illustrates an exemplary user interface presented on a display of a computer system when a non-random quasicrystalline pattern is initially generated, with a camera bar pointed at elements of the non-random pattern and the display showing a live video stream including the elements of the non-random pattern, the elements of the non-random pattern being overlaid on top of the video stream to guide the user. [Figure 6] 6 shows an exemplary user interface presented on a display of a computer system in which several markers (solid black circles) are arranged in the non-random quasicrystalline pattern of FIG. 5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0060] Setting up and tracking non-randomly positioned markers overview When creating video media (e.g., video files, or real-time video) where real-world video is mixed with virtual objects, e.g., creating virtual 3D objects (e.g., props), a news presenter can walk around in a news show or news segment, or for example, computer-generated monsters can be added to a video of a movie set, the position of the real-world camera needs to be tracked as the real-world camera moves, so that graphics can be added realistically. In one example, a stereo vision camera bar is used that is fixedly attached to a real-world camera, where the camera bar includes two cameras mounted a fixed distance apart in a stereoscopic configuration to track the position of the real-world camera to which the camera bar is attached, and the camera bar can be used to detect previously recorded non-randomly positioned markers in the camera bar field of view to infer the position of the camera bar. The field of view of the camera bar may not coincide with the field of view of the real-world camera at all. The field of view of the camera bar may overlap partially with the field of view of the real-world camera. The field of view of the camera bar may include the entire field of view of the real-world camera. In one example, synthetic markers are used that are purposefully added to the 3D space visible from the camera bar, so that the synthetic markers are visible within the working space of the camera bar (e.g., within the maximum field of view of the camera bar). The synthetic markers may have a particular shape and / or color and / or be made of a particular material to aid in their detection in the image. They are referred to herein as markers. In one example, the markers used do not differ significantly from each other in appearance. In one example, the markers used are approximately spherical. In one example, the markers are distributed on a spiral. In one example, the markers are distributed on a quasi-crystalline lattice. In one example, the markers are distributed on an aperiodic grid, or a periodic grid, or an aperiodic lattice, or a periodic lattice. In one example, the markers are distributed on a set of non-intersecting closed curves, e.g., a set of concentric circles. A closed curve is a curve that has the same start and end points. In one example, the markers are distributed on a set of non-intersecting open curves, e.g., a set of concentric semicircles.An open curve is a curve that has a distinct start and end point.

[0061] Described herein are methods and systems for tracking camera pose (camera pose typically includes camera rotation and camera translation, i.e., six degrees of freedom) using non-random placement of markers, where the markers are visible through a camera bar. The markers are purposefully placed in non-random, predefined positions. In one example, such non-random, predefined positions are dictated by software running on a computer (e.g., Ncam), which may provide clarity and simplicity for user setup, and may also provide predictability of software calculations resulting in improved tracking quality.

[0062] set up In one example setup, a user using a computer system attempts to track a single monocular camera, where the camera includes an image sensor array and a lens, where the lens is arranged to focus image light onto the image sensor array. However, the tracking setup can also be used with any number of cameras at the same time, where the cameras are in fixed relative positions to each other. This arrangement of cameras where the cameras are in fixed relative positions to each other is called a camera bar, where the cameras of the camera bar are arranged to image the same or very similar fields of view. In one example of a camera bar, two cameras are arranged on the bar at a fixed distance from each other, where the cameras are arranged to image the same or very similar fields of view. In one example of a camera bar, three cameras are arranged in a triangle (e.g., an equilateral triangle) on the support at a fixed distance from each other, where the cameras are arranged to image the same or very similar fields of view. In the simplest case, the camera bar comprises a single camera. In less simple cases, the camera bar comprises multiple cameras at fixed relative positions to each other, where the cameras are arranged to image the same or very similar fields of view.

[0063] Marker setup may include three steps:

[0064] Initial marker placement In one example, the first step for a user is to aim the camera of the camera bar at the area they wish to decorate with markers. A portion of this area should always (or nearly always) be clearly visible from the moving area of ​​the camera bar during capture. The user then places an indicator (e.g., a fiducial marker or a single marker) to indicate the start of placement of a non-random placement of markers. An example is shown in FIG. 1. A fiducial marker or fiducial is an object placed in the field of view of the imaging system that appears in the images created to be used as a reference point or measure.

[0065] Software running on the computer (e.g., Ncam) then uses the initial indicator to generate allowable positions for the marker and displays the allowable positions for the marker to the user in the camera bar field of view in a user interface displayed by a computer system including a display (e.g., a laptop computer, desktop computer, tablet computer, or smartphone), so that the user knows where to place the marker within the camera bar field of view.

[0066] Depending on the size of the area the user is photographing, different numbers of markers are typically used. For example, the smallest space (roughly 5m 2 ) to 20-50 markers in a much larger space (approximately 80m 2 ) with 200 to 500 markers.

[0067] Typically, the camera bar only sees a subset of these markers, since the camera bar's field of view rarely, if ever, sees all markers simultaneously in most scenarios. A well-defined pattern of markers improves the reliability of determining the pose of the camera bar, since a particular marker pattern is typically only visible from a unique pose or from a limited set of poses. This reduces the range of possible ambiguities in the pose of the camera bar when inferred from non-randomly placed marker positions.

[0068] Pattern Generation A non-random arrangement of markers can be generated from a set of parameters that can be passed to one or more well-defined deterministic formulas. For example, one such formula can generate a spiral pattern, where the parameters used can include the initial marker location, the height of the camera bar above the floor in meters, and the orientation of the camera bar in degrees (e.g., tilt angle and azimuth angle in a (e.g., spherical) coordinate system). An example is shown in FIG. 2.

[0069] In one example, the markers are on a path that is represented by a continuous path or function that is continuously differentiable (C1). In one example, the placement of the markers is not random if the 3D path that intersects the marker locations is a continuous, continuously differentiable (C1) path or function. In one example, the markers are on a path that is represented by a smooth, continuous path or function. In one example, the placement of the markers is not random if the 3D path that intersects the marker locations is a smooth, continuous path or function. In one example, the marker locations are equally spaced along the path.

[0070] Now, many formulations of patterns of non-randomly positioned markers are possible, but some patterns work better than others. What defines an effective pattern comes down primarily to the following factors: ● The uniformity of the distribution of markers that can be observed from the area where the camera is moved. This is to ensure that the camera bar never looks into an area without markers and therefore does not lose its position. ● Uniqueness of the pattern within the field of view of the camera bar, which allows the tracking software running on the computer to quickly determine its position from a position where the marker is newly identified (a "cold start"), since the pattern of markers within the field of view is only possible from one viewpoint, or from a limited number of viewpoints. ● The number of markers in view at any one time. The more markers in view, the more robust the tracking will be. However, having too many markers in view increases the setup time. Therefore, a pattern that uses a reasonable number and density of markers is desirable.

[0071] Another pattern of non-randomly positioned markers that may well meet the requirements is a quasi-crystalline pattern, which is an ordered but non-periodic pattern.

[0072] A user interface presented on the display of a computer system may look like the image of FIG. 3 when the non-random spiral pattern is first generated, with a camera bar pointed at an element of the non-random pattern, and a user viewing a live video stream including the element of the non-random pattern. The element of the non-random pattern (e.g., a red circle, etc.) is overlaid on top of the video stream to guide the user. (Note: the camera bar stereo camera is referred to as a "witness" camera in the user interface (UI) of FIG. 3.) In the example of FIG. 3, an initial marker is shown in the center of the image.

[0073] A user interface presented on the display of a computer system may look like the image of FIG. 5 when the non-random quasicrystalline pattern is first generated, with the camera bar pointed at the elements of the non-random pattern, and the user viewing a live video stream including the elements of the non-random pattern. The elements of the non-random pattern (e.g., red circles, etc.) are overlaid on top of the video stream to guide the user. (Note: the camera bar stereo camera is referred to as the "witness" camera in the user interface (UI) of FIG. 5). In the example of FIG. 5, an initial marker is shown in the center of the image.

[0074] Covering the Remaining Tracking Space Once a certain number of markers are placed in a non-random pattern, the movement of the camera bar can be tracked as long as some markers are still visible. In one example, the user interface notifies the user that the markers have been placed in the correct position and whether the minimum amount of markers has been achieved. An example of several markers (black filled circles) placed in a non-random spiral pattern is shown in FIG. 4. In the example of FIG. 4, the initial marker is shown in the center of the image. An example of several markers (black filled circles) placed in a non-random quasi-crystalline pattern is shown in FIG. 6. In the example of FIG. 6, the initial marker is shown in the center of the image. In one example, elements of the pattern may change color (e.g., from red to green) when a marker is detected within them.

[0075] Once the camera bar pose tracking is working, a pattern can be locked to the scene in 3D space, allowing the user to gradually move the camera bar around the moving area and receive guidance in placing additional markers. Once the moving area is sufficiently covered with markers, the user can finish the setup process and track the camera bar pose with confidence.

[0076] The following describes an exemplary algorithm used to robustly extract the pose of the camera bar when it looks at markers set up on a known 3D path, e.g., C1 (smoothness coefficient), or at known 3D positions (e.g., quasicrystalline lattice positions).

[0077] Camera bar pose calculation The following exemplary pseudo algorithms or methods vary depending on the number of cameras mounted on the camera bar.

[0078] Pseudo-algorithms or methods: the 2-n (multiple) camera case This is an instantaneous tracking / relocalization algorithm that requires only a subset of the markers visible by the camera bar (e.g., 8 or more). It does not require any specific movement from the camera bar. It includes the following steps: 1. Detecting the 2D location of the markers on the image using a marker extraction algorithm (e.g., blob extraction, etc.) 2. Calculating the 3D positions of the detected 2D marker positions, where the camera bar is fully calibrated (e.g., using its intrinsic (e.g., the distance between its two cameras) and extrinsic (e.g., its tilt and azimuth) parameters) using the camera bar coordinate system. 3. Matching the 3D constructed marker to the 3D path of a marker whose 3D path in the world coordinate system is known, or to the 3D position of a marker whose 3D position in the world coordinate system is known. a. Fit the set of 3D points to a noise-robust 3D path of the marker or 3D position of the marker (e.g., using iterative closest point (ICP) fitting or similar or curve fitting, or even a nonlinear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), also known as the damped least squares (DLS) method, or Gauss-Newton (the Gauss-Newton algorithm is used to solve nonlinear least squares problems), etc.). The solution is then modeled as a Lie group SE(3) that represents the camera bar pose (e.g., 6 degrees of freedom: position, orientation). Physically, SE(3) (Special Euclidean Group in Three Dimensions) is a group of simultaneous rotations and translations of vectors. It is used in robotics and general kinematics. SE(3) moves vectors from one frame to another. b. Extract the pose of the camera bar in the world coordinate system from the fit.

[0079] Pseudo-algorithms or methods: the case of monocular camera bars This technique requires motion to be applied to the camera bar. It is used to calculate the relative 3D positions of the markers from the camera bar position at the end of the movement and then calculate the absolute pose of the camera bar using the perfectly known 3D path of the markers or using the perfectly known 3D positions of the markers. 1. Use a marker extraction algorithm (e.g., blob extraction) on successive frames to detect the 2D location of the markers on the images and use nearest neighbor matching to label the markers throughout the sequence. 2. For example, using the opencv library, extract the essential matrices using a robust extractor (e.g., Random Sample Consensus, or RANSAC, which is an iterative method for estimating mathematical models from datasets that contain outliers) to extract the rotation and translation (up to scale) from the first frame to the last frame of the sequence. 3. With the intrinsic parameters of the camera bar fully calibrated (e.g. focal length is calibrated), calculate the 3D positions (up to a scale factor S) of the detected 2D marker positions. 4. Matching a 3D constructed scaled marker to a 3D path, where the 3D path in the world coordinate system is known, or matching a 3D constructed scaled marker to a 3D marker position, where the 3D marker position in the world coordinate system is known. a. Fit the set of 3D points to a noise-robust 3D path or 3D position (e.g., using iterative closest point (ICP) fitting or similar or curve fitting, or even a nonlinear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), also known as the damped least squares (DLS) method, or Gauss-Newton (the Gauss-Newton algorithm is used to solve nonlinear least squares problems), etc.). The scale factor is one more variable to the problem solved by the minimization technique. The solution is then modeled as a Lie group SIM(3) that represents similarity (7 degrees of freedom: scale, position, orientation). SIM(3) is the group of similarity transformations in 3D space, the semidirect product SE(3)XR*. It has 7 degrees of freedom, 3 translations, 3 rotations, and 1 scale. b. Extract the pose of the camera bar in the world coordinate system from the fit.

[0080] application 1. A method for generating augmented reality video in real time by mixing or compositing computer-generated 3D objects with a video feed from a video camera, comprising: (a) the body of the video camera can be moved in 3D and sensors including accelerometers and gyros sensing across six degrees of freedom within the video camera or attached directly or indirectly to the video camera provide real-time positioning data that allows the 3D position and 3D orientation of the video camera to be calculated; (b) for example, a camera bar including two cameras forming a stereoscopic system is fixed directly or indirectly to a video camera; (c) that real-time positioning data is then automatically used to create, invoke, render, or modify a computer-generated 3D object; (d) the resulting computer-generated 3D object is then mixed or composited in real time with a video feed from a video camera to provide augmented reality video, for example for a television broadcast, a movie, or a video game; (e) A method in which the 3D position and orientation of the video camera is determined with reference to a 3D map of the real world, for example by using a video flow in which a camera bar including two cameras forming a stereoscopic vision system surveys the field of view and software running on a processor is used to detect non-randomly positioned markers in the field of view, the markers and their non-random positions being pre-recorded in the processor and included in the 3D map of the real world, the markers having been previously added to the field of view manually or artificially. The non-randomly positioned markers may be non-randomly positioned markers as described elsewhere herein. The field of view of the camera bar may not coincide at all with the field of view of the video camera. The field of view of the camera bar may partially overlap with the field of view of the video camera. The field of view of the camera bar may include the entire field of view of the video camera.

[0081] A mixed or composite system comprising: (i) a video camera; and (ii) sensors including accelerometers and gyros sensing across six degrees of freedom; (iii) a camera bar including, for example, two cameras forming a stereoscopic system; (iv) a processor; Mixing or compositing computer-generated 3D objects with real-time video feeds from video cameras to generate augmented reality video in real time, for example for television broadcasts, movies, or video games; (a) the body of the video camera can be moved in 3D and sensors within, or attached directly or indirectly to, the video camera provide real-time positioning data that enables the 3D position and 3D orientation of the video camera to be calculated; (b) for example, a camera bar including two cameras forming a stereoscopic system is fixed directly or indirectly to a video camera; (c) the system is configured to automatically use the real-time positioning data to create, invoke, render, or modify a computer-generated 3D object; (d) the system is configured to mix or composite the resulting computer-generated 3D objects in real time with a video feed from a video camera to provide real-time augmented reality video, e.g., for a television broadcast, a movie, or a video game; (e) A mixed or synthetic system, in which the system is configured to determine the 3D position and orientation of the video camera with reference to a real-world 3D map, for example by using a video flow in which a camera bar including two cameras forming a stereoscopic vision system surveys the field of view and software running on a processor is configured to detect non-randomly positioned markers in the field of view, in addition to real-time 3D positioning data from the sensor, and the markers and their non-random positions are pre-recorded and included in the real-world 3D map in the processor, and the markers have been added to the field of view manually or artificially in advance. The non-randomly positioned markers may be non-randomly positioned markers as described elsewhere herein. The field of view of the camera bar may not coincide at all with the field of view of the video camera. The field of view of the camera bar may partially overlap with the field of view of the video camera. The field of view of the camera bar may include the entire field of view of the video camera.

[0082] In one example, accelerometers and gyros sensing over six degrees of freedom are used to predict the position of the video camera to make the matching process between what the camera bar sees and a 3D map of the real world faster and / or more robust. An added benefit of using accelerometers and gyros sensing over six degrees of freedom and a 3D map of the real world is the possibility of frame interpolation between frames and / or frame extrapolation from one or more frames.

[0083] Note It is to be understood that the above-referenced arrangements are merely illustrative of the application of the principles of the present invention. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the invention. While the invention has been fully described above with specificity and detail in connection with what are presently considered to be the most practical and preferred embodiments of the invention as illustrated in the drawings, it will be apparent to those skilled in the art that numerous modifications can be made without departing from the principles and concepts of the invention as set forth herein.

Claims

1. 1. A method for setting up non-randomly positioned markers within a field of view of a camera bar and tracking a pose of the camera bar, the method comprising: (i) viewing a field of view with a camera bar, the camera bar being fixedly attached to a video camera; (ii) displaying a view of the field of view of the camera bar on a display of a computer system; (iii) the computer system generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view, along with a view of the camera bar of the field of view, on the display of the computer system; (iv) the computer system detecting that a marker is located in the field of view of the camera bar at a non-random marker position generated in step (iii); (v) in response to the computer system detecting that markers have been placed at non-random marker locations within the field of view of the camera bar in step (iv), the computer system records the markers and their respective locations within the field of view; (vi) repeating steps (iv) and (v) until a predetermined number of different markers and their respective marker positions have been recorded; (vii) the computer system matching markers detected within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (viii) repeating step (vii) to track the pose of the camera bar.

2. The method of claim 1 , further comprising the step of storing the recorded predetermined number of different markers and their respective marker positions.

3. The method of claim 1 or 2, wherein the generated pattern of non-random marker positions within the field of view begins with an initial indicator detected within the field of view by the computer system.

4. the generated pattern of non-random marker positions within the field of view is generated using a path that is represented by a continuously differentiable (C1) continuous path or function; or The method of any one of claims 1 to 3, wherein the generated pattern of non-random marker positions within the field of view is generated using a smooth continuous path or a path represented by a function.

5. The method of claim 4 , wherein the marker locations are equally spaced along the path.

6. 6. The method of claim 1, wherein the generated pattern of non-random marker positions within the field of view is generated from a set of parameters passed to one or more well-defined deterministic equations.

7. The method of claim 6 , wherein a formula generates a spiral pattern and the parameters include a position of the initial marker, a height of the camera bar above a floor, and an orientation angle of the camera bar.

8. The method of any one of claims 1 to 7, wherein the generated pattern of non-random marker positions within the field of view is a spiral pattern.

9. the generated pattern of non-random marker positions within the field of view is a quasicrystalline pattern; or the generated pattern of non-random marker positions within the field of view is on a non-periodic grid, or on a periodic grid, or on a non-periodic grating, or on a periodic grating; or the generated pattern of non-random marker positions within the field of view lies on a set of non-intersecting closed curves, e.g., on a set of concentric circles; or the generated pattern of non-random marker positions within the field of view lies on a set of non-intersecting open curves, e.g., on a set of concentric semicircles; The method according to any one of claims 1 to 3.

10. A method according to any preceding claim, wherein in step (iii) the pattern of non-random marker positions within the field of view is overlaid onto the field of view of the camera bar.

11. A method according to any one of claims 1 to 10, wherein in step (iv) the computer system indicates within the display that it has detected that a marker has been placed within a non-random marker position in the field of view of the camera bar, for example by changing the displayed colour of the non-random marker position, for example from red to green.

12. The method of any one of claims 1 to 11, wherein the camera bar is a monoscopic camera bar.

13. Step (vii) (a) using a marker extraction algorithm to detect 2D positions of markers in a sequence of images viewed by the camera bar, where a first and a last image in the sequence are taken from different camera bar positions, and labeling markers throughout the sequence using nearest neighbor matching; (b) extracting matrices using a robust extractor to extract rotations and translations with unknown scale factors from a first frame of the sequence to a last frame of the sequence; (c) calculating 3D marker positions with unknown scale factors using the detected 2D marker positions using camera bar calibration data; (d1) matching the calculated scaled positions of the 3D markers to a 3D path of the markers, the 3D path in a world coordinate system being known, comprising fitting the set of calculated 3D marker positions to the 3D path of the markers, the 3D path being known, comprising deriving the scale factor; (d2) matching the calculated scaled positions of the 3D markers to the 3D positions of the markers, the 3D positions of the markers in a world coordinate system being known, comprising fitting the set of calculated 3D marker positions to the 3D positions of the markers, comprising deriving the scale factor; (e) modeling a matching solution as a SIM(3) on a Lie group to represent the camera bar pose, where the matching solution includes scale; (f) extracting the camera bar pose from the matching solution in the world coordinate system.

14. Step (d1) or (d2) is performed by using iterative closest point (ICP) fitting or curve fitting, or by using a non-linear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), or by using a Gauss-Newton algorithm, or In step (b), Random Sample Consensus (RANSAC) is used; The method of claim 13.

15. the camera bar is a stereoscopic camera bar comprising two cameras mounted a fixed distance apart in a stereoscopic configuration; or the camera bar includes three cameras arranged in a triangle (e.g., an equilateral triangle) at a fixed distance from each other; The method according to any one of claims 1 to 11.

16. Step (vii) (a) detecting 2D positions of markers on images seen by the camera bar using a marker extraction algorithm; (b) calculating 3D marker positions using the detected 2D marker positions, using camera bar calibration data, and using a camera bar coordinate system; (c1) matching the calculated 3D marker positions to a 3D path of the marker, where the 3D path in a world coordinate system is known, and includes fitting the set of calculated 3D marker positions to the 3D path of the marker; (c2) matching the calculated 3D marker positions to the 3D positions of the markers, where the 3D positions of the markers in a world coordinate system are known and includes fitting the set of calculated 3D marker positions to the 3D positions of the markers; (d) modeling the matching solution as an SE(3) on the Lie group to represent the camera bar pose; (e) extracting the camera bar pose from the matching solution in the world coordinate system.

17. 17. The method of claim 16, wherein step (c1) or (c2) is performed by using iterative closest point (ICP) fitting or curve fitting, or by using a non-linear minimization technique such as the Levenberg-Marquardt algorithm (LMA or simply LM), or by using a Gauss-Newton algorithm.

18. The method of any one of claims 1 to 17, wherein the field of view of the camera bar does not exactly coincide with the field of view of the video camera.

19. the field of view of the camera bar partially overlaps with the field of view of the video camera, or the field of view of the camera bar includes the entire field of view of the video camera; The method according to any one of claims 1 to 17.

20. 1. A system comprising a video camera, a camera bar fixedly attached to the video camera, and a computer system including a display, the computer system comprising: (i) displaying a field of view of the camera bar on the display of the computer system; (ii) generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view on the display of the computer system together with a view of the camera bar of the field of view; (iii) detecting within the field of view of the camera bar that a marker is located within a non-random marker location generated in (ii); and (iv) in response to detecting that markers are located within a non-random marker location within the field of view of the camera bar in (iii), recording the markers and their respective locations within the field of view; (v) repeating (iii) and (iv) until a predetermined number of different markers and their respective marker positions have been recorded; (vi) matching markers detected within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (vii) repeating (vi) to track the pose of the camera bar.

21. A computer program product, the computer program product executing, on a computer system including a display, (i) displaying a field of view of a camera bar on the display of the computer system; (ii) generating a pattern of non-random marker positions that includes positions within the field of view and displaying the pattern of non-random marker positions within the field of view on the display of the computer system together with a view of the camera bar of the field of view; (iii) detecting within the field of view of the camera bar that a marker is located within a non-random marker location generated in (ii); and (iv) in response to detecting that markers are located within a non-random marker location within the field of view of the camera bar in (iii), recording the markers and their respective locations within the field of view; (v) repeating (iii) and (iv) until a predetermined number of different markers and their respective marker positions have been recorded; (vi) matching markers detected within the field of view of the camera bar with the recorded markers and their respective marker positions to obtain a pose of the camera bar; (vii) repeating (vi) to track the pose of the camera bar.

Images