Computer-Aided Camera and Control System

Abstract

The lightweight and portable microprocessor-assisted remotely controlled camera system provides precise, repeatable camera movements through the improved stability of a 3-axis gimbal design, allowing a lightweight, large sensor camera to be placed almost anywhere and smoothly controlled remotely via a simple touchscreen interface. The computer control module provides a wide range of control options (including stabilization and movement smoothing via the touchscreen interface), including remote adjustment of all camera, lens, and gimbal parameters (which can be programmed, set, and recalled).

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 001,028, filed March 27, 2020, which is incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION The present invention relates to a remotely controlled camera system, and more particularly to a camera platform with three-axis stabilization and control via a touch screen interface and precision movement control. [Background technology]

[0003] background Video recording of public activities and sporting events is common for entertainment and educational / training purposes. Entire industries are devoted to filming professional and amateur athletic competitions, performances, and other events using sophisticated camera equipment operated by professionals. Such camera equipment tends to be expensive and requires specialized training to achieve the quality results required for televised sporting events or concerts. For video recording of activities that do not typically generate income (e.g., youth athletic leagues and tournaments, surfing competitions, and closed-door athletic events), there is interest in the ability to produce high-quality video of the activity for personal entertainment and for use in training and coaching. Other activities where affordable video capabilities are often desirable include corporate events and conferences, theatrical or musical performances, weddings and parties, and public festivals. Unfortunately, without the significant financial resources of a commercial entertainment business, the video capabilities available for small events are often less than ideal due to limitations in both equipment quality and location.

[0004] Significant efforts to capture images (including video) from the air have been poured into the development of unmanned aerial vehicles (UAVs), or drones. Many UAV systems are currently available that provide image and video capture and remote control from ground-based devices. However, the use of drones may be restricted in many situations pursuant to FAA regulations and / or local ordinances. Furthermore, currently available systems require pilots to use direct control of the UAV, similar to other fixed-wing or rotary-wing aircraft. Control of these drones, for example, to adjust the UAV's pitch, roll, yaw, and power, relies on common control inputs, such as joystick and throttle controls. While somewhat effective, such control systems require specialized knowledge on the part of the remote pilot and are vulnerable to collisions due to piloting errors. The risk of collision is further exacerbated by pilots' tendency to focus on what they are seeing on a monitor screen to follow the action in the field. Popular commercial UAVs are typically designed to communicate remotely with a smartphone or tablet to allow the pilot to see what the camera sees, but wireless communication systems can be interrupted and / or subject to delays, with potentially catastrophic consequences for the UAV.

[0005] Solutions to some of the problems encountered in video capture systems have been described in the prior art. For example, U.S. Patent No. 9,900,551 to Rekimto discloses a communication terminal for videoconferencing with camera control and image processing. While this system is suitable for mounting on a stable base such as a table, it does not provide control at a distance and therefore requires time-consuming adjustments to ensure level adjustment. U.S. Patent Application No. 2019 / 0137597 to Glover et al. describes a system and method for automatically orienting a camera that relies on a transceiver worn by the subject to communicate with the camera and cause the camera's processor to track the subject and adjust the focal length. While this approach may be useful for a single subject or a small number of subjects, tracking an entire team with multiple transceivers can be challenging from many perspectives, not to mention the computational demands involved in managing real-time interactions with a large number of targets. U.S. Patent No. 5,959,667 to Maeng discloses a voice-activated camera system including software that automatically tracks a speaking person. The disclosures of each of the above-identified patent documents are incorporated herein by reference.

[0006] While the described approaches may address some of the difficulties encountered in producing high-quality video, none provide a system with the desired performance capabilities in an affordable, user-friendly package for general-purpose video recording of events. Thus, there remains a need for such a system. Summary of the Invention [Means for solving the problem]

[0007] overview According to embodiments of the present invention, a lightweight, portable, microprocessor-assisted, remotely controlled camera system provides precise, repeatable camera movements that were previously difficult or impossible to achieve in any other way. The improved stability of the system's gimbal design allows the lightweight, large sensor camera to be placed almost anywhere and smoothly controlled remotely via a simple touchscreen interface. All camera and lens parameters can be set and recalled, and movements can be smoothed / improved via electronic microprocessor control. This ingenious robotic camera effectively enables amateurs to shoot like professionals.

[0008] In one exemplary embodiment, the inventive system allows for remote mounting, stabilization, and control of image capture technology. The system includes a high-resolution camera with a lens, a 3-axis gimbal, a control module, a camera mount, a power supply, a wireless or wired connection path (either direct point-to-point or connected via a local or wide area network), optional lens control servos, and a touchscreen device or other user interface. For wireless connections, WiFi, cellular network connections (e.g., 5G) can be used. For wired connections, alternatives include serial or parallel control (digital or analog) and / or high-speed mixed signals. In some embodiments, a web-based interface can be used for control and monitoring via any web-connected device.

[0009] The lens is attached to the camera with an optional lens-mounted zoom / focus servo attachment. The camera is supported by a 3-axis gimbal via a novel, adjustable-balance mounting device. The gimbal and camera are connected to a control module (also known as the "AIRstation"). The AIRstation functions as an Ethernet distribution point, an Ethernet-to-serial control bridge, and power distribution for all system components. The gimbal and AIRstation are supported on a camera platform, which can be a pole, tripod, monopod, robotic arm, slider, jib arm, fixed mount, ground-based vehicle, or other device. The AIRstation is hardwired to a power source (either a battery pack or an AC converter) that provides power for all system components. The system combines camera control, lens control (zoom, focus, aperture), camera angle (pan, tilt, and roll), and camera position (X, Y, Z axes in 3D space) into a touchscreen interface. The unit combines Ethernet routing / bridging, video and audio input / output, serial control, and power distribution into one device. The device can be controlled by a tablet / computer via either a wireless (e.g., cellular or WiFi) connection and / or a wired connection, which can be configured as an override feature.

[0010] The tablet / computer may connect wirelessly into the system via any hardwired Ethernet point or may control individual system components via a direct point-to-point connection. The tablet / computer provides a user interface that may be operated via a touch screen, user voice, and / or external attached control devices. The tablet / computer control connection may either be a direct point-to-point connection or may be networked via a local area network (LAN), wide area network (WAN), or wireless (e.g., WiFi or cellular) connection. For wired connections, alternatives include serial or parallel control (digital or analog) and / or high-speed mixed signal.

[0011] The lens can be controlled through the camera lens mount connection or via an external servo motor fixed to the lens. Control of the external servo can be either through the camera, through an AIRstation serial connection, or through a direct wireless connection to a tablet / computer or attached device.

[0012] Additional movement of the camera stage (eg, a robotic arm or mobile camera stage) can also be controlled via the tablet / computer user interface.

[0013] Included is a software implementation for integrated visual (in addition to numerical) programming of camera, lens zoom, focus, and aperture control, pan / tilt positioning, and camera X / Y / Z position in 3D space. All camera and lens settings, as well as camera position and orientation in 3D space, are set through a simple What-You-See-Is-What-You-Get ("WYSIWYG") interface. Settings can be recalled and smoothly transitioned between with a single tap on the touchscreen or using other user interfaces. Multiple settings can be transitioned between simultaneously; for example, pan and tilt direction and camera zoom and focus can all be adjusted simultaneously, as can the position of the robotic platform on which the camera and gimbal are supported.

[0014] A key improvement offered by this inventive camera system is a three-axis gimbal that allows for rapid and precise rebalancing with lens changes. The gimbal design allows the camera / lens assembly to be pre-balanced and centered so that lens changes and other modifications require minimal rebalancing effort. Center-of-gravity balance between the camera and lens is maintained through the use of two sliding mechanisms. The three-axis gimbal can be used to adjust both the vertical and horizontal axes as well as provide self-leveling so that the entire setup remains level if moved or placed on a moving platform. The gimbal can also be servo-controlled and connected to an inertial measurement unit (IMU) within the camera assembly for automatic balancing. The IMU can incorporate a gyroscope, accelerometer, magnetometer, or any combination thereof.

[0015] An optional feature may include using an Apple® pencil-like device (see, e.g., U.S. Pat. No. 9,329,703, incorporated herein by reference) or similar stylus for enhanced control, with an optional zoom rocker switch or slide incorporated within the pencil, allowing for variable speed dynamic control of touchscreen functions. This feature offers the significant advantage of allowing for sole manual control of pan / tilt as well as zoom functions. By comparison, alternatives such as joysticks incorporating twist-to-zoom functionality tend to be difficult to operate smoothly.

[0016] Additional features may include distance transmitters for focus assistance and pan / tilt control. In one implementation, a dog tag, clip-on device, or other transmitter may be worn by a subject / participant during an activity to transmit distance and / or direction data to a receiver calibrated with the camera focal plane. The camera may then use the transmitted data to precisely adjust the focal length and follow the tracked subject. Alternatively, the transmitter / receiver may be incorporated into a Fitbit® or Apple® watch-style personal smart device, or an application for communicating with the built-in tracking capabilities of such devices may be included in the system software, allowing the camera system to follow the subject and automatically adjust camera settings as needed. Similarly, a smartphone may be used as a transmitter / receiver tracking device to provide feedback for controlling focus, pan, and / or tilt. To add interest, a tracker or sensor may also be incorporated into a game ball to automatically follow the ball's movement with the camera.

[0017] In one aspect of the present invention, a remotely controlled camera system includes: a high-resolution camera configured to generate digital images and further configured to generate data corresponding to image-related parameters detected by a sensor associated with the camera; a lens detachably and interchangeably attached to the camera; a three-axis gimbal configured to assist or correct one or more of pan, tilt, and roll of the camera; a camera platform configured to support the gimbal; a control module configured to receive digital images and image-related parameters generated by the camera, process the image-related parameters to generate image information associated with the digital images, and generate control signals for each of the camera and the gimbal; a communication path; and a user interface in communication with the control module via the communication path, configured to display the digital images and the image information associated with the digital images and to input user instructions to the control module for activating control signals to perform operations on one or both of the camera and the gimbal. In some embodiments, the camera platform can be one or more of a tripod, a monopod, a slider, a jib arm, a robotic arm, a ground-based vehicle, and a wall mount. If the camera platform is a robotic arm, the control module is further configured to generate control signals for operating the robotic arm. If the camera platform is a ground-based vehicle, the control module may be further configured to generate control signals for operation of the ground-based vehicle. The communication path may be a wireless or wired path.

[0018] The gimbal may include z-axis motion assemblies symmetrically positioned on either side of the camera such that activation of the z-axis motion assemblies provides forward / backward adjustment of the camera and lens combination. The z-axis motion assemblies may be dual rack-and-pinion assemblies, where a pinion of each rack-and-pinion assembly is connected to the left and right sides of the camera, respectively, and each pinion is connected to an axis for synchronous movement. The racks of each rack-and-pinion assembly are supported on opposite ends of a U-arm, and the camera is positioned symmetrically about the center between the opposite ends of the U-arm. A first end of the U-arm supports a tilt motor, and a second end of the U-arm supports a corresponding bearing, such that activation of the tilt motor tilts the camera relative to the U-arm. The base of the U-arm includes a roll motor.

[0019] The image information associated with the digital image includes one or a combination of lens data, focal length, exposure, aperture, color tone, flicker reduction, shutter angle / speed, image profile, sharpness, brightness, saturation, contrast, luminance, LUT, white balance, ISO, and audio level, and the control module is configured to generate an interactive display of the image information on a user interface; and in response to input of user commands via the user interface, activate control signals to perform operations on one or both of the camera and gimbal to adjust one or more of the image-related parameters. The control module is further configured for a network for communicating the digital image and the image information to a storage medium.

[0020] The gimbal may be configured to generate a signal corresponding to the gimbal-related parameter, and the control module is further configured to receive the gimbal-related parameter and associate the gimbal-related parameter with the image information.

[0021] The system may further include one or more depth sensors for determining a distance between the object and a focal plane of the camera and generating a depth signal, the control module receiving and processing the depth signal to generate depth information associated with the image information. The depth information is processed by the control module to generate tracking instructions to one or both of the camera, lens, and gimbal to track the movement of the object. The depth information may also be used to generate a depth map from the digital image.

[0022] The control module may be configured to control the movement of the camera and gimbal to smooth the transition between two different points in the field of view, and may further be configured to obtain lens focal length information from the camera to dynamically change the degree of pan / tilt / roll movement based on the focal length.

[0023] In some embodiments, the user interface is a touch screen device, and the user controls the camera pan and tilt by touching the part of the image where the object of interest is found and following the movement of the object of interest.

[0024] The system may further include a transmitter / receiver system, where a transmitter located on the object generates a signal to a control module to provide depth information to assist in focus and pan / tilt control. [Brief explanation of the drawings]

[0025] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1 is a diagram of the main components of an embodiment of the inventive camera system. [Figure 2A] 10A-10C are diagrams of alternative mounting options for use with the inventive camera system. [Figure 2B] 10A-10C are diagrams of alternative mounting options for use with the inventive camera system. [Figure 2C] 10A-10C are diagrams of alternative mounting options for use with the inventive camera system. [Figure 2D] 10A-10C are diagrams of alternative mounting options for use with the inventive camera system. [Figure 3A] FIG. 1 is a front view of a three-axis gimbal according to an embodiment of the inventive camera system. [Figure 3B] FIG. 1 is a side view of a three-axis gimbal according to an embodiment of the inventive camera system. [Figure 3C] FIG. 1 is a top view of a three-axis gimbal according to an embodiment of the inventive camera system. [Figure 4] FIG. 2 is a rear perspective view of a camera and rack and pinion mechanism according to an embodiment of the present camera system. [Figure 5] 1 is a diagram of integrated communication between various components of a camera system according to one embodiment of the present invention. [Figure 6] 1 is a platform stack illustrating an exemplary architecture of an AIRstation control module. [Figure 7] 1 is a simulated screen display of a user interface according to one embodiment of the inventive camera system. [Figure 8] 10 is a simulated screen display of a user interface showing smoothing features for tracking motion. [Figure 9] FIG. 1 is a block diagram of one embodiment of the inventive camera system incorporating one or more depth sensors. [Figure 10] FIG. 1 is a block diagram of one embodiment of the inventive system configured for metadata recording. DETAILED DESCRIPTION OF THE INVENTION

[0026] Detailed Description of the Embodiments Referring to FIG. 1, the inventive camera system 100 includes five main components: a high-resolution camera 10, a three-axis gimbal 20, a control module 30, a camera stand 40, and a user interface 50. The camera stand 40 can take many different forms as known in the art. In FIG. 1, the camera stand 40 shown is a conventional tripod. A single-leg or similar conference camera support can be used. Briefly referring to FIGS. 2A-2C, some examples of options that can be used as the camera stand include a slider or jib arm 40a (FIG. 2A), a robotic arm 40b (FIG. 2B), a ground-based vehicle 40c (FIG. 2C), or a combination of multiple supports (e.g., a robotic arm on a robotic vehicle). Additional options include wall mounts or other camera supports. In one embodiment, a tripod such as that shown in FIG. 1 can be attached to and extend upward from a chair, allowing the chair occupant to act as a "human counterweight." Alternatively, a folding chair can extend from the base of the tripod.

[0027] Returning again to FIG. 1 , located adjacent to, incorporated into, or attached to the camera platform 40 is a power source 60, such as a battery pack, AC / DC converter module, generator, or other power source, suitably positioned to provide a low center of gravity and counterbalance for the camera and gimbal. In some embodiments, the power source 60 may be held within a base 62 that acts as a camera platform in combination with an axle / pole 48 (which may include telescoping capabilities) as shown in FIG. 2D . In another implementation, the power source may be incorporated into a single leg at its bottom for a low center of gravity. This configuration may also be combined with the “human counterbalance” approach described above, a chair attached to the bottom of the axle or single leg, or a chair having a receptacle for receiving the bottom of the axle or single leg.

[0028] The connector cable 42 between the power supply 40 and the control module may be enclosed within the hollow tripod as shown, or may be off-axis (preferably with appropriate stabilization via cable ties or other cable control to ensure centering of any objects that may affect the balance of the assembly or present a risk of accidental contact). Bundled with the connector cable 42 may be a wired connection path providing either direct point-to-point or connection via a local area network, wide area network, or other hardwired connection to an external controller (e.g., user interface 50). Alternatively, a wireless connection (e.g., WiFi, 5G mobile) may be used. In some embodiments, a Gigabit Ethernet (GigE) cable is used.

[0029] In the exemplary embodiment, camera 20 is a commercially available high-resolution camera such as one in the Z CAM product line of compact, high-resolution cinema cameras (Z Cam, Shenzhen, China). As will be apparent to one skilled in the art, other cameras can be used in the inventive system, including commercially available cameras such as the Blackmagic Pocket Cinema Camera (BMPCC), Komodo, GoPro®, Panasonic BGH1, and others. The camera includes a mount for an interchangeable lens 12, as well as electronics and motors for controlling lens focus and zoom. The exemplary camera includes ports for interfacing via multiple connection protocols, including HDMI (video output), Gigabit Ethernet (camera control, settings, and live streaming), USB (camera control and data conversion), XLR audio input / output, 3.5mm audio input / output, COM, RS232, remote, WiFi antenna, I / O, and others. These examples are not intended to be limiting. It will be apparent to one skilled in the art that other connector types and protocols can be selected to match various camera manufacturer specifications.

[0030] Traditional gimbals require adjustments across four to five different axes to achieve proper payload balance. Precise balancing must be achieved to minimize stress on the motors and battery drain. Payloads not only need to be balanced when the camera is initially attached, but also whenever a lens is changed or a peripheral that alters the payload's center of gravity is attached. When modifications are made to a traditional gimbal, it can take 5 to 30 minutes to adjust the various sliders and locks to bring the gimbal to proper balance. Most gimbals place the camera mounting plate below the camera. This inventive design utilizes a unique side-mounting approach that ensures the camera payload (camera / lens combination) is always naturally centered vertically and laterally. No vertical (up / down in Figure 3A) or lateral (left to right in Figure 3A) adjustments are necessary when modifications (e.g., lens changes) are made to the payload. Once modifications are made, only horizontal adjustments—i.e., a shift forward or backward along line 110 (Figures 3B and 3C)—are required to compensate for the payload center due to the balance change.

[0031] The use of a side mount as opposed to a traditional undermount eliminates the need for vertical adjustment (up / down in FIG. 3A). The evenly weighted U-arm 22 eliminates the need for lateral adjustment (i.e., left or right in FIG. 3). Because a traditional undermount, by its nature, positions the camera / lens asymmetrically relative to the horizontal axis, rebalancing is always required when even small changes are made.

[0032] The three-axis gimbal used in the inventive system offers a significant improvement over existing technology. The camera 10 is mounted to a three-axis gimbal 20 to provide enhanced control, balance, and response. The inventive three-axis gimbal allows adjustment across an entire axis in seconds. This is achieved through a mounting system that places the camera and lens at their natural center of gravity with many separate elements.

[0033] The first element of the improved gimbal involves the use of a side mount on the camera body. Commercial cameras of the type contemplated for use in the inventive system include standard-sized threaded holes in multiple locations on the camera body. Typically, a hole on the bottom of the camera screws onto a threaded mount on a tripod head or other camera support. Referring briefly to FIG. 1 , a standard threaded mounting hole 14 is shown on the side of the camera, but can also be located on the bottom and back as well. In the inventive system, a side mount is used in most embodiments. For cameras without inherent side mount holes, the camera cage or bracket enclosing the camera body can provide suitable mounting holes.

[0034] 3A-3C, the camera is centered between the ends of the U-arm 22. Looking at FIG. 3C, the U-arm 22 has a tilt motor 24 on its left side that provides power to tilt adjustment of the camera. The motor 24 is activated to rotate the camera and lens about axis 116 relative to the U-arm 22. The motor provides a feedback signal that is provided to the control module to indicate its position. A corresponding bearing 25 on the right side of the U-arm ensures smooth rotation. Disposed between the motor 24 and bearing 25 and the side of the camera 10 is a z-axis motion assembly that balances the camera / lens combination about centerline 110. In one embodiment, these assemblies include a dual rack and pinion combination. Each rack and pinion assembly includes a mounting plate 210 that attaches to the side of the camera 10 and connects perpendicularly to a rack guide housing 212. (Note that the mounting plate 210 and housing 212 may be integrated into a single component.) A pinion gear (not shown in FIG. 3C) attached to the back plate 29 engages with teeth on the rack 21 to move the camera forward and backward relative to the axis 110, allowing for precise balancing of the camera 10 / lens 12 within the entire gimbal assembly.

[0035] Figure 4 shows further details of the rack assembly. Pinion gears 216 rotatably mounted to the back plate 29 engage their corresponding racks 21, which are mounted via the housing 28 to the motor 24 on the right side of the camera and the bearing 25 on the left side. The side plates that attach to each side of the camera 10 are not shown in this view. Rather, the back plate 29 attaches to the camera back using standard built-in screw holes, if available. If no screw holes are provided in the camera back, a U-shaped bracket that wraps around the rear and sides of the camera can be used to attach the z-axis motion assembly through the camera's side mount holes.

[0036] The dual rack and pinion assembly allows for quick and precise z-axis adjustment. The pinions 216 are synchronized via the shaft 218, ensuring that the camera remains perfectly aligned across the dual rack but does not distort during adjustment. For adjustment, the shaft 218 and pinion 216 can be rotated manually, or some embodiments may include a pinion motor that would be located on the backplate 28 to drive the rotation of the shaft 218 and / or pinion. In motorized implementations, the motor can be driven by a controller tied to the camera inertial measurement unit (IMU), which enables automatic balancing. The IMU may incorporate a gyroscope, accelerometer, magnetometer, or any combination thereof.

[0037] An IMU may be mounted parallel to the camera to measure camera tilt, roll, yaw, and translation rate, and provide this information to a system controller for processing. The controller may automatically determine adjustments as needed, or may display parameters that allow the system operator to make adjustments either manually or by sending commands via the controller to motors to make the desired adjustments.

[0038] In an alternative embodiment, each rack and pinion assembly may be replaced by a screwdriver assembly attached to each side of the camera via a side bracket similar to that described for the rack and pinion assembly, and the drive motors of the screwdrivers would be synchronized for simultaneous activation for z-axis adjustment.

[0039] A key feature of the improved gimbal is to distribute the weight of the gimbal components evenly across the entire vertical plane. This starts with the symmetrical configuration of the U-arm 22, ensuring that the camera payload is always evenly spaced between the two arms and always centered on the roll motor 220. The heaviest component, the gimbal tilt motor 24, is counterbalanced on the other side of the arm with ball bearings 25, the IMU, wiring, lens servos, and other gimbal components. The bearing side of the U-arm or z-axis adjustment assembly can be fine-tuned for balance with a small counterweight if needed.

[0040] 3B and 3C, the base of the roll motor 220 is attached to a block 222 so that activation of the motor 220 rotates the U-arm 22 relative to the block 222. The symmetrical positioning of the motors 220 between the blocks further contributes to centering the gimbal about axis 112. Each block 222 is connected to the upper end of a corresponding elbow 224 by a rod 223. A rod 228 connects the lower end of the elbow 224 to a block that is attached to a yaw motor 230, which provides camera panning. Although not shown in these figures, the yaw motor 230 would be attached to the base or other structure, for example, via a set screw-type or other conventional connector. The base may house, or be attached to a housing surrounding, a gimbal control card containing circuitry for connection to the various electrical components of the three-axis gimbal. Alternatively, the gimbal control card (for balance) could be placed within the housing on the yaw motor housing 230 between the rods 228. This same structure may serve to house the system controller 30, or in some embodiments, the system controller may be held in a housing attached to a block within the gimbal assembly. For example, the system controller 30 may be supported on the rear of the yaw / roll arm of the gimbal assembly (i.e., at blocks 222-224). The symmetrical and balanced gimbal design means that when a camera is initially mounted and lenses are changed or other components are added or subtracted, optimal balance can be achieved within seconds rather than minutes. Each motor within the gimbal provides a feedback signal that is provided to the control module to indicate its position.

[0041] FIG. 5 illustrates components of an embodiment of the inventive system and their interconnections. The system can be constructed using a variety of different components and sources. The legend in the lower right corner of the figure indicates sample sources of components, including commercial off-the-shelf ("COTS"), custom, and combinations thereof. As noted above, the main components of the inventive system are a high-resolution camera 10, a three-axis gimbal 20, a control module 30, a camera platform 40, and a user interface 50. The camera 10 connects to the control module 30 (also called an "AIRstation") via an electrical interface cable 42b. In a preferred embodiment, the cable 42b is a Gigabit Ethernet (GigE) cable (a digital interface for digital cameras widely used in the field of industrial image processing). Cameras contemplated for use with the inventive system typically include a standard industry-standard GigE port, as previously described. For cameras that do not have a built-in (intrinsic) lens servo, additional lens servo capability 15 may be added and connected to the camera 10 via the camera's built-in 4-pin RS232 port, or may be controlled directly from the AIRstation. As will be apparent to those skilled in the art, additional interface capabilities may be provided as new industry standards are adopted. Thus, the description of specific industry standards within the exemplary embodiments is not intended to be limiting.

[0042] The camera 10 is mounted on a gimbal 20 as previously described. The gimbal 20 is electronically (and in most cases physically) attached to a gimbal control module 32, which is located in a common housing with the AIRstation 30 and can be connected via a serial connector 42a. The AIRstation 30 functions as an Ethernet distribution point, an Ethernet-to-serial control bridge, and power distribution to all system components. As shown, the power connections are shown as dashed lines with connections to a power supply 60 that provides power to all system components. In the exemplary embodiment, the power supply 60 is a 14.8V / 10A power supply connected via a cable (e.g., cable 42 in FIG. 1) to a power hub 62, which can be housed within the AIRstation housing for connection to the gimbal module 32 (14.8V / 3A) and the camera 10 (12V / 5A) and a voltage converter 64 (which provides 5V / 3A to the AIRstation 30). In some embodiments, the AIRstation 30 is preferably contained within a waterproof enclosure that can serve as a physical base for the gimbal and is attached to a camera mount, for example, via a fixed screw-type connection. The AIRstation 30 is wired to an Ethernet LAN switch 74 via a GigE cable 70, which provides connection to a network or storage device or database. As shown, an Internet connection 76 to the cloud 77 provides access for video editing, video switching, multicasting, and live streaming to other devices, as well as large-capacity cloud-based data storage. For purposes of this specification, storage devices, databases, and cloud storage all fall within the meaning of "storage media." The user interface 50 can be a tablet / computer that can be operated via a touchscreen, user voice, and / or external accessory control devices. The tablet / computer control connection can be either a direct point-to-point connection or can be networked via a local area network (LAN), wide area network (WAN), or wireless (e.g., WiFi or cellular) connection. With regard to wired connections, alternatives include serial or parallel control (digital or analog) and / or high-speed mixed signal.In an exemplary embodiment, an Ethernet switch 74 provides a connection to the user interface 50. For example, if the user interface is an Apple® iPad®, the cable 75 can be a Lightning®-to-GigE cable. For other tablet brands (e.g., Samsung®, Microsoft®), a USB-C-to-GigE cable can be used for LAN connection to the switch 74. If wireless communication (e.g., 5G or other cellular networks) is used, the AIRstation 30 can be configured via a SIM card or other add-on module 47. This configuration would provide direct connectivity to the cloud 77. In a preferred embodiment, the AIRstation is configured for both local (e.g., wired) and remote connectivity. For local (in-premise) connectivity, the application can connect to the AIRstation on a LAN via Wi-Fi or other mobile camera platform to control the camera, gimbal, and, if applicable, robotic arm. For remote connectivity, the application can connect to the AIRstation via a VPN into the local network or directly to the AIRstation via a carrier's IP address. In this way, all connections will be secure and no unauthorized access will be allowed. To further reduce the latency inherent in all VPN implementations, the Websockets-based cloud gateway can also be used as a central aggregation point for transmitting real-time control and telemetry data from the AIRstation to the operator and vice versa.

[0043] As will be readily apparent to one skilled in the art, user interface 50 is not limited to tablets or other touchscreen devices. In some implementations, a tablet, laptop, or desktop computer may provide a visual display whose control is achieved by a keyboard, joystick, game controller, trackball, mouse, touchpad, or some combination thereof. In other embodiments, an augmented reality (AR) or virtual reality (VR) headset may be implemented for gyroscope and / or accelerometer control of the pan / tilt / roll functions of the gimbal.

[0044] In an exemplary embodiment, the AIRstation 30 is implemented using a Raspberry Pi 4 computer. As will be apparent to those skilled in the art, other CPUs may be used, including, but not limited to, an NVIDIA® Jetson™ or a Qualcomm® Snapdragon™ 865 or 888. The AIRstation API is built on secure and lightweight IoT protocols such as Websockets, MQTT (Message Queuing Telemetry Transport), and REST (representational state transfer). FIG. 6 shows a platform stack 300 of an exemplary architecture for supporting AIRstation applications. The firmware level 310 includes interfaces for communication with external networks, such as IO drivers 311, file system 312, memory management 313, process management 314, scheduler 315, network protocols 316, and network drivers 317, as well as interfaces for connectivity between the AIRstation and other system hardware components. The service level 320 includes an operating system, which in the exemplary embodiment is a Linux operating system. There are several Linux operating systems available for Raspberry Pi computers (including Raspberry Pi OS, Ubuntu, and others). Choosing an appropriate OS for a selected CPU is within the skill of one in the art. This level provides programs operating at the application level with basic interaction with the hardware. The application level 330 captures the logic and interfaces needed to coordinate operations between system components with two API blocks: control module API 331 and component API 332. As shown, component APIs are provided for operation of the camera 10, gimbal 20, and optionally, robotic arm 40b. The communications level 340 includes applications for interaction with external networks (i.e., the Internet and / or the cloud).

[0045] The AIRstation control module software implementation allows for integrated visual (in addition to numerical) programming of camera, lens zoom, focus, and aperture control, pan / tilt positioning, and camera X / Y / Z position in 3D space. All camera and lens settings, as well as camera position and orientation in 3D space, are set through a single What-You-See-Is-What-You-Get (WYSIWYG) interface. Settings can be recalled with a simple tap and smoothly transitioned between. Multiple settings can be transitioned between simultaneously (e.g., pan and tilt direction and camera zoom and focus can all occur simultaneously). The touchscreen of the user interface 50 provides user controls for pan, tilt, zoom, focus, aperture, audio levels, camera settings, streaming destinations, etc.

[0046] 7 shows an example of a display on user interface 50 according to one embodiment of the inventive system. The display includes user-selectable "buttons" for initiating many different actions. Table 1 below provides a list of features that can be controlled via user interface 50.

[0047] [Table 1]

[0048] Additional control features can be enabled by touching an area of ​​the image 510 and dragging a finger (or stylus) to allow the user to control camera pan and tilt. For example, to follow a player with a ball in a playing field, the user touches the image where the player / ball is located and follows the player / ball's movement with a finger or stylus. Auto-exposure control can be activated via multiple taps on the image to display a temporary icon for changing the aperture setting 504 by using the camera's automatic settings. Similarly, auto-focus can be selected by multiple taps on a selected spot on the image 510.

[0049] Existing methods for displaying video exposure involve specific values ​​(e.g., above 90 IRE) or false-color banding, where the specific IRE value is displayed in various colors or in a waveform / vectorscope that displays the brightness / color values ​​as a curve. These methods are difficult to correlate with a portion of the video picture area (waveform). These methods can also be highly distracting (banding, false-color overlay) when the user is concentrating on the subject or activity being recorded. These features also tend to involve intensive computer and video display resources.

[0050] The present inventive technique avoids such problems by providing a temporary display of spot exposure. To activate this feature, the camera operator simply touches an area of ​​the screen to display an IRE value from 0-100 to help determine the appropriate exposure level. The present inventive technique provides a fast and non-intrusive way to display exposure to help the camera operator capture a correctly exposed image.

[0051] Existing methods for magnifying a video image to check focus accuracy involve either physical zooming at the lens or digital zooming at the picture area. Physical zoom methods have the disadvantage of altering the video image being recorded, so they cannot be used while recording is in progress. As a result, focus checks must be performed beforehand. Other approaches that employ digital magnification methods increase the size of the entire picture frame while leaving the recorded output unchanged and cropping out areas of potential interest.

[0052] The inventive system temporarily magnifies only a specific portion of the image, leaving the recorded image unchanged and not cropping out other possible areas of interest.

[0053] Some embodiments may include a smart focal length feature for pan / tilt / rotate control. The longer the focal length of a camera lens, the narrower the viewing angle. At narrower viewing angles, camera movement is visually amplified; for example, when shooting with a 300 mm lens (8° viewing angle), a 1° movement command will produce more apparent visual movement than if the same command were sent to a camera with a 12 mm lens (122° viewing angle). Because existing control devices (flight surfaces, joysticks, or other devices) use static control signals for pan / tilt, smooth control of longer focal length lenses can be challenging. This is further complicated when a camera with a variable zoom lens is performing live-recording zoom. Apparent movement is amplified as the lens zoom makes precise control more difficult.

[0054] The inventive system addresses these challenges by using a proportional control approach. The greater the zoom in a lens, the greater the controller movement required for the same degree of movement. For example, a lens at 25mm requires a touchscreen movement of 5mm from the origin to move the camera 10° per second, while for a lens at 50mm, the same 5mm movement will cause the camera to pan at 5° per second.

[0055] The inventive approach dynamically varies the degree of camera movement to compensate for the "long lens effect" regardless of the control device (touchscreen, joystick, AR headset, etc.). This implementation involves control module 30 obtaining lens focal length information from the lens and then interpolating the data to dynamically vary the degree of pan / tilt / roll movement appropriate for the focal length. The amount of camera pan / tilt / roll movement is dynamically adjusted downward for a given control input as the lens focal length increases.

[0056] In some embodiments, a software-based "bumper" may be included to cushion extreme camera and lens movements. This software implementation provides camera lens zoom, focus, and aperture control as well as camera movement. A "delta" that controls the amount and speed of ease-in and ease-out can be set point-by-point, allowing for a smooth, dynamic, ramping transition between two different settings, effectively providing "more-human, less-robotic" camera movement control. Setting these features may be accomplished by programming the call button 520 and selecting the corresponding call button when the feature is activated. The software implementation also provides microprocessor-assisted dampening to prevent runaway or excessively fast manual camera / lens movements. The degree of damping is variable and can be controlled by user settings.

[0057] Manual control of the transition between two preset positions may be included (in this context, "position" means the location of any movable system part (i.e., camera, lens, gimbal, or mount)). The software implementation of AIRstation allows for on-the-fly, variable speed, manually controlled transition between multiple preset points. For example, a start point "A" and an end point "B" may be set. Various "way points" may also be added between the start and end points. The start / end / way points may include any or all of the camera, lens, gimbal, or camera mount settings.

[0058] FIG. 8 provides an example of the system's WYSIWYG smoothing of programmed movements for trajectory planning. The system software can generate an overlay 610 with a visual motion path on the video display. The camera movement is mapped onto the picture area as a tracking line 612. The endpoints of the tracking line ("A" and "B") are marked on a slider bar 620. By sliding a manual slider button 622 along the slider bar 620 between these points, the user can manually control the overall speed of the movement between these fixed points, rather than relying on a pre-programmed speed. Keyframe points can be set and / or modified to smooth simulated or preset movement points. Similar procedures exist for post-production motion graphics but have not been commonly available for live recording.

[0059] Adjustment points can be added to the path that allow the movement curve (e.g., a quadratic Bezier curve, familiar from the field of computer graphics) to be modified and smoothed. Recording and camera movement require that the path in Cartesian space be smooth and flowing rather than angular when changing movement direction. This approach is beneficial for control systems for both positioning and velocity.

[0060] Sharp focus (especially when fast-moving or dynamic objects are captured on video) is generally difficult to achieve with any camera lens. Manual focusing can be fast but is prone to human error. Automatic focusing systems (also known as "autofocus") primarily use a contrast-based detection method that calculates the intensity between adjacent pixels on the image sensor to determine focus. While this method is useful, it is severely limited by the image sensor's capabilities. This method can be slow and inaccurate, and requires the lens to "hunt" to find the proper focal length. This in-and-out "hunting" movement of the focus element causes distracting and highly undesirable softening / sharpening of the image. Contrast-based autofocus is also particularly unreliable in environments where hazy or low light levels obscure the subject.

[0061] Focusing difficulties are exacerbated when using some lens types. Manufacturers have produced "varifocal" lenses that are smaller, lighter, and less expensive, but have the drawback of being unable to maintain focus as the camera is zoomed. "Parfocal" lenses, on the other hand, maintain focus while zooming, but are generally larger, heavier, and more expensive. Varifocal lenses can produce optical properties that are equal to or better than parfocal lenses. If the varifocal focusing limitations could be overcome, varifocal lenses could be deployed in a much wider range of use cases.

[0062] In some embodiments, the inventive system employs a depth sensor, such as a LIDAR, radar, sonar, laser, or other distance sensor, to determine the distance of an object from the camera and generate a signal for use by the AIRstation control module to use the measured distance and achieve precision focus for all lens types. This approach allows a variable-focus lens to behave as if it were parfocal and maintain focus while zooming. The AIRstation control module also has the advantage of being controllable over the entire IP, allowing for safe lens focus and tracking control at long distances.

[0063] 9, a user identifies an area of ​​interest ("AoI") on the touchscreen interface 50. The AoI 700 is brought to the AIRstation 30 and then correlated with depth information provided by one or more depth sensor modules 702 to determine the precise distance of the AoI-subject 700 to the camera sensor focal plane. When using a parfocal lens, the AIRstation 30 transmits this distance information directly to the lens focus control motor to quickly and precisely establish the desired focal length setting.

[0064] For a variable focus lens, the lens focus element position varies with the zoom focal length, so a pre-calibrated "lens profile" is required to adjust the lens focus element to the correct position for a given zoom focal length. The AIRstation 30 dynamically adjusts the lens focus element to the correct position for a given zoom focal length, allowing the variable focus lens to function parfocally and creating a closed-loop communication loop 710 with the lens and camera to maintain focus while zooming.

[0065] It should be noted that some manufacturers (e.g., Sony) have incorporated lens profiles into their camera bodies. These profiles are proprietary systems that do not allow for use on other camera systems. The present inventive system offers the advantage of not only allowing programmable lens profiles to be generated for any variable focus lens from any manufacturer, but also allowing precise calibration of each individual lens regardless of manufacturer, thus allowing any variable focus lens to be used on any camera body capable of system integration.

[0066] Incorporating distance measurement capabilities into the inventive system via LIDAR, radar, sonar, laser, or other depth sensors can also assist in tracking. In the same way, the AIRstation control module can utilize focal point AoI information, and the integration of depth information with image recognition allows for more precise object tracking to be achieved. This can be replaced or augmented by the use of radar for moving objects or ultra-wideband for positioning via radio beacons. The benefit and utility of using multiple modalities for location information lies in the reduction of single-point-of-failure scenarios. For example, because LIDAR is based on the return time of a scanning laser, anything that obscures or blocks the path of the emitted or returning laser scan can cause the LIDAR to produce inaccurate or incomplete information. Ultra-wideband, while not subject to visibility challenges, has a weakness in that a clear line of sight must exist between the radio tag and at least three beacons. Radar does not have any of the weaknesses mentioned above, but it does require complex algorithms (such as fast Fourier transforms) to process the signals reflected back to the emitter, so a combination of various sensors will produce the most accurate and reliable results.

[0067] Most commercial video cameras generate a large amount of metadata, only a portion of which is typically used. Apart from the most basic information, this metadata is discarded or ignored in post-video production applications, primarily due to its static nature. Video is a dynamic medium with information that changes every few milliseconds. Therefore, having static data for a dynamic medium is only marginally useful.

[0068] The inventive system leverages the camera's standard metadata and generates additional metadata using an added distance sensor, significantly expanding capabilities for video post-production and file indexing. Figure 10 is a block diagram illustrating various system components where metadata can be collected and compiled within the system's memory (within control module 30) and transmitted to the cloud (see, e.g., Figure 5) or other database for storage. Specifically, the system's time-stamped, frame-accurate dynamic-metadata recording adds to traditional static metadata. Recorded information can include lens data, focal length, focal length, and aperture. Static camera data can also include color tone, flicker reduction, shutter angle / speed, image profile, sharpness, brightness, saturation, contrast, luminance, LUT, and audio input source. Dynamic camera data can include white balance, ISO, and audio levels. Gimbal data can include IMU information, gyro heading, movement rate, pan degree, tilt degree, roll degree, motor encoder, etc. The position / depth data includes information about the position of the camera in 3D space and a depth map corresponding to the RGB information.

[0069] Data from one or more depth sensors can be utilized to construct a 3D depth map. In one specific example, millimeter-wave radar has a sufficiently high level of resolution to allow the assignment of a depth value to each pixel in an image to generate a depth map. This facilitates separation of foreground from background to significantly expand the ability to modify footage through post-production editing (e.g., adding video effects, background replacement, removing undesirable objects, adding additional objects, etc.).

[0070] An optional additional feature of the inventive system may include a transceiver system with separate distance transmitters to provide focus assistance and pan / tilt control. In one example, a dog tag-like necklace or other transmitter may be worn by the subject to transmit distance and / or direction data to a receiver calibrated with the camera focal plane. The camera can then use that data to precisely adjust the focal length. In another example, the transceiver may be incorporated into a Fitbit® or Apple® watch-type personal smart device, or an application may be included in the system software to communicate with the built-in tracking features (e.g., GPS capabilities) of such devices. Similarly, a smartphone may be used as a transceiver tracking device to provide feedback for controlling focus, zoom, pan, and / or tilt. To add interest, a tracker or sensor may be incorporated into the game ball to enable the system to automatically follow the ball's movement with the camera and adjust focus / zoom / pan / tilt as needed based on detected signals. Additional focus assistance and zoom / pan / tilt control may be implemented via various voice-activated techniques. Tracking provides a 3D Cartesian coordinate position (x, y, depth) that relays the tracked position back to the AIRStation, enabling a "follow-me" capability. The AIRStation uses the x,y coordinate information to control the gimbal motors (right, left, up, down). The AIRStation also uses the tracked object information to send focus and zoom information to a connected camera. The combination of gimbal and camera allows for fully autonomous camera operation.

[0071] The inventive camera system disclosed herein provides microprocessor-assisted remote control for precise, repeatable camera movements previously difficult or impossible to achieve using existing techniques. The significantly improved stability of the gimbal design allows lightweight, large sensor cameras to be placed almost anywhere and smoothly controlled remotely via a simple touchscreen interface. All camera and lens parameters can be set and recalled, and movements can be smoothed / improved via electronic microprocessor control. The data generated by the system provides expanded post-production capabilities. The inventive robotic camera effectively enables amateurs to shoot like experts and experts to increase the versatility of footage during capture and in post-production editing.

Claims

1. a high-resolution camera configured to generate digital images, the camera further configured to generate data corresponding to image-related parameters detected by a sensor associated with the camera; a lens detachably and interchangeably attached to said camera; a three-axis gimbal configured to assist and correct one or more of pan, tilt, and roll of the camera, the gimbal including a U-arm assembly having arm ends configured to symmetrically support and center the camera therebetween, and a z-axis motion assembly disposed opposite the camera, the z-axis motion assembly configured to provide forward / aft adjustment of the camera relative to the U-arm; a camera mount configured to support the gimbal; a control module configured to receive digital images and image-related parameters generated by the camera, process the image-related parameters to generate image information associated with the digital images, and generate control signals for each of the camera and gimbal; communication paths; and a user interface in communication with the control module via the communication path, the user interface configured to display the digital images and the image information associated with the digital images and to input user commands to the control module for activating control signals to perform operations on one or both of the camera and the gimbal.

2. The system of claim 1 , wherein the camera platform comprises one or more of a tripod, a monopod, a slider, a jib arm, a robotic arm, a ground-based vehicle, and a wall mount.

3. The system of claim 2 , wherein the camera platform includes a robotic arm, and the control module is further configured to generate control signals for operation of the robotic arm.

4. The system of claim 2 , wherein the camera platform comprises a ground-based vehicle, and the control module is further configured to generate control signals for operation of the ground-based vehicle.

5. The system of claim 1 , wherein the communication path is a wireless or wired connection path.

6. 2. The system of claim 1, wherein the z-axis motion assembly includes a dual rack and gear assembly, with a gear of each rack and gear assembly disposed to one of the left and right sides of the camera.

7. The system of claim 6 , wherein the rack of each rack and gear assembly is supported on opposite ends of an arm of the U-arm assembly.

8. 2. The system of claim 1, wherein a first end of the U-arm assembly includes a tilt motor and a second end of the U-arm assembly includes a bearing, and activation of the tilt motor tilts the camera relative to the U-arm assembly.

9. The system of claim 1 , wherein the base of the U-arm assembly includes a roll motor.

10. the image information associated with the digital image includes one or a combination of lens data, focal length, focal distance, exposure, aperture, color tone, flicker reduction, shutter angle / speed, image profile, sharpness, brightness, saturation, contrast, luminance, LUT, white balance, ISO, and audio level; 2. The system of claim 1, wherein the control module is configured to generate an interactive display of the image information on the user interface; and the control module is further configured, in response to input of user commands via the user interface, to activate control signals to perform operations on one or both of the camera and the gimbal to adjust one or more of the image-related parameters.

11. 11. The system of claim 10, wherein the control module is further configured for a network for communicating the digital images and image information to a storage medium.

12. 2. The system of claim 1, wherein the gimbal is further configured to generate a signal corresponding to a gimbal-related parameter, and the control module is further configured to receive the gimbal-related parameter and associate the gimbal-related parameter with the image information.

13. The system of claim 1, further comprising one or more depth sensors configured to determine the distance between a subject and a focal plane of the camera and generate a depth signal, wherein the control module receives and processes the depth signal to generate depth information associated with the image information.

14. The system of claim 13 , wherein the depth information is processed by the control module to generate tracking instructions to one or both of a camera, the lens, and a gimbal to track the movement of the subject.

15. The system of claim 13 , wherein the depth information is processed by the control module to generate a depth map from the digital image.

16. The system of claim 1 , wherein the control module is further configured to control movement of the camera and gimbal to smooth a transition between two different points in a field of view.

17. The system of claim 1 , wherein the control module is further configured to obtain lens focal length information from the camera to dynamically change the degree of pan / tilt / roll movement based on lens focal length.

18. 10. The system of claim 1, wherein the user interface includes a touch screen device, and the user controls camera pan and tilt by touching a portion of the image where an object is found and by following the movement of the object.

19. 10. The system of claim 1, further comprising a transmitter / receiver system, wherein a transmitter positioned on the subject generates a signal to the control module to provide depth information to assist in focus and pan / tilt control.

Images