Method and system for correcting distortion caused by camera tilt

Abstract

One disclosed example provides a video conferencing system including a processor and a storage device storing instructions executable by the processor to obtain an image of a scene captured via a camera, the image of the scene including an image distortion caused by a camera tilt angle at which the image of the scene is captured. The instructions are further executable to apply a projection mapping to the image of the scene to map the image of the scene to a projection including a tilt parameter based on the camera tilt angle at which the image of the scene is captured, thereby obtaining a corrected image; and output the corrected image.

Description

Background Technology

[0001] Cameras are used for communication in many different environments. For example, cameras are often used in conference rooms to allow video image data of a meeting to be sent to a remote location for others to participate in the meeting. In this way, people at a remote location can enjoy a more collaborative experience compared to using only audio. Summary of the Invention

[0002] This disclosure is provided to introduce, in a simplified form, a selection of concepts also described in the detailed description. This disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure.

[0003] One disclosed example provides a video conferencing system including a processor and a storage device storing instructions executable by the processor to obtain an image of a scene captured via a camera, the image of the scene including image distortion caused by the camera pitch angle used to acquire the image of the scene. The instructions are also executable to apply a projection mapping to the image of the scene to map the image of the scene to a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; and to output the corrected image. Attached Figure Description

[0004] Figure 1 An example camera that can be used in a video conferencing system is shown schematically.

[0005] Figure 2 An example camera controller is schematically shown, configured to generate a distortion-corrected image by applying a projection mapping that includes tilt parameters.

[0006] Figure 3 An example conference room is shown, including cameras arranged at a downward tilt angle.

[0007] Figure 4 An example uncorrected image is shown, captured by a wide-angle camera with a downward tilt angle.

[0008] Figure 5 This shows the result after applying a cylindrical projection mapping without a tilt parameter. Figure 4 The image.

[0009] Figure 6 The geometric representation of an example untilted cylindrical projection is shown.

[0010] Figure 7The geometric representation of an example tilted cylindrical projection is shown.

[0011] Figure 8 An example mapping of pixel locations in the original image to pixel locations in the corrected image is illustrated schematically.

[0012] Figure 9 This shows the result after applying a cylindrical projection mapping that includes tilt parameters. Figure 4 The image.

[0013] Figure 10 The geometric representation of an example tilted line projection is shown.

[0014] Figure 11 The geometric representation of an example tilted spherical distortion projection is shown.

[0015] Figure 12 A schematic representation of the field of view of a cropped, undistorted image is shown.

[0016] Figure 13 An example of cropping the original image before applying a tilted cylindrical distortion projection is illustrated.

[0017] Figure 14 This is a flowchart describing an example method for correcting tilt distortion in an image.

[0018] Figure 15 An example computing system is illustrated schematically. Detailed Implementation

[0019] A camera directs light from a scene onto a planar image sensor to capture an image of the scene in an image frame. In many environments, especially indoors such as conference rooms, wide-angle cameras can be used to image multiple dispersed objects of interest. Cameras employing wide-angle lenses capture light from a wide field of view (FOV) but suffer from various types of optical distortion, such as trapezoidal distortion, barrel distortion, pincushion distortion, and / or TV distortion. The level of distortion can be more pronounced at higher field angles from the camera's optical axis. Standard low geometry distortion (or low TV distortion) camera lenses (with a horizontal field of view between 40° and 70°) may have slight distortion, while wide-angle (70°–90° FOV) and ultra-wide-angle (>90° FOV) lenses may have greater distortion at the edges and corners of the image frame. Low TV distortion requires increased complexity, such as aspherical profiles, element counts, and tighter tolerances in higher FOV lens designs; for example, a reasonably mass-produced 120° horizontal FOV lens design may suffer from 6% TV distortion. Furthermore, because there is often a difference between the angular orientation on the HFOV captured by the camera and the effective angular orientation seen by the viewer when viewing the scene captured by the camera on a computer and / or interactive monitor, viewers often cannot fully understand low TV distortion, as wide FOV frame capture will require a shorter viewing distance to optimally view the displayed scene content. Therefore, wide FOV scenes shot using low TV distortion lenses may exhibit trapezoidal stretching in the corners of the field of view. Since low TV distortion lenses are not readily available above 120 degrees and are theoretically limited to 180 degrees, fisheye lenses can be used to capture wide FOV scenes. Therefore, for example, when using an ultra-wide-angle lens in a conference room environment, people standing near the sides of the FOV in the image may appear distorted and standing at non-perpendicular angles, along with other distortions. The unnatural appearance of people and other objects in the image can lead to an unsatisfactory experience for participants in remote video conferences.

[0020] In some distortion correction methods, the raw image from the camera is mapped onto an analytical geometry projection that approximates the arrangement of people and / or other objects of interest in the imaging space. In the example of a conference room, a semi-circular arrangement of people standing or seated around the camera can approximate a cylinder. Thus, a cylindrical projection can be used to correct image distortion by mapping the raw image onto a cylindrical projection. In some examples, such mapping can be based on the optical center of the camera's lens, such that the optical center and the center of the projection are aligned. Similarly, the projection can also be based on a known or approximate distortion function of the lens. In the corrected image, vertically oriented objects in the scene, such as people standing near the sides of the camera's field of view (FOV), may appear straight and vertically oriented, rather than distorted and / or tilted, and therefore have a more natural appearance. As used herein, the term "raw image" refers to an image generated without any distortion correction and can include monochrome images, color images, and at least partially processed images.

[0021] However, some camera placements can pose challenges to distortion correction. For example, a video conferencing camera might be mounted on a wall above a video conferencing display or attached to the top of the display. When the area of ​​interest includes the participants in the room, a camera positioned horizontally outward from that location might waste most of its image frames imaging the ceiling. Offset imaging, where the optical center of the image sensor is offset relative to the optical axis of the lens, can be used as a method to achieve proper correction in such cases. However, this solution involves using a dedicated offset image sensor or an oversized image sensor. While offset imaging allows the camera to capture the field of view downward at an offset angle, it may not adapt well to different camera heights and mounting angles unless the image sensor includes a sufficient pixel area to accommodate the offset caused by the offset angle.

[0022] Another way to avoid over-imagening the ceiling is to tilt the camera downwards. However, cylindrical projection may not adequately correct distortions in images acquired at a tilted pitch angle. Cylindrical projection distortion correction may presuppose a horizontally oriented camera optical axis. With the optical axis tilted downwards, the projection can straighten vertically oriented objects, but these objects may appear tilted to one side of the image.

[0023] Therefore, examples involving the correction of distortion in images acquired using a camera with a tilt angle are disclosed. In short, the disclosed example application involves projection based on a tilt parameter of the camera's tilt angle. Using such tilt projection to correct distortion in images from a standard wide-angle camera avoids the angular constraints of using oversized image sensors and offset image sensors. Furthermore, different projections can be determined and / or stored for different tilt angles, allowing appropriate correction to be applied based on the current camera tilt angle. With this correction, people and other vertically oriented objects near the sides of the image may appear to be standing upright rather than tilted outwards. A more natural appearance of people in the image compared to an uncorrected image or an image not corrected using the tilt parameter can help remote video conferencing participants feel more "present" in the video conference. Although described in the context of a conference room environment, the disclosed examples can be used with any suitable camera in any suitable internal or external environment.

[0024] Before discussing the use of tilt parameters in projection models, refer to Figure 1 Example camera 100 is described. Camera 100 is configured to image scene 102. Camera 100 includes image sensor 104 and lens 106, which is positioned to guide object light 107 from scene 102 onto image sensor 104. Lens 106 can take any suitable shape and can comprise any suitable optical material with any suitable optical properties. In some implementations, lens 106 can be a system of two or more lenses or other optical elements. As a more specific example, lens 106 can be an f-θ lens, which can comprise a known linear or near-linear lens distortion function. Using an f-θ lens in a camera helps ensure an approximately linear relationship between the image sensor space and the angle of incidence. Lens 106 (with lens elements held in a lens barrel 108) can be held in a fixed position relative to the center of image sensor 104 via a bracket mounting structure 110. In some examples, the mount structure 110 is a rigid mount structure that fixes the lens barrel 108 along each of the six degrees of freedom (e.g., x, y, z, tip, tilt, azimuth rotation) and thus fixes all elements of the lens 106 relative to the image sensor 104. For example, a fixed-focus camera may have such an arrangement. In some examples, the mount structure 110 may allow the lens barrel 108 to move relative to the image sensor 104 along at least one axis (e.g., for image stabilization and / or focusing, such as by placing an autofocus voice coil actuator between the lens barrel 108 and the mount structure 110). In such examples, even though the position of the lens 106 relative to the image sensor 104 can move along the z-axis, the lens 106 remains fixed relative to the center of the image sensor 104.

[0025] In the non-limiting example camera shown, the lens barrel 108 is operatively coupled to a mount structure 110. The mount structure 110 is mounted to a printed circuit board (PCB) 112. In one example, the mount structure 110 is bonded to the PCB 112 via adhesive. An image sensor 104 is mounted on the PCB 112 such that the optical axis 114 of the lens 106 is substantially aligned with the center of the image sensor 104. Specifically, the lens barrel 108, the mount structure 110, and the PCB 112 collectively maintain the optical alignment of the lens 106 with the image sensor 104 (e.g., in the case of using a threaded lens barrel 108 and mount structure 110, the mount structure 110 can engage relative to the PCB 112 to fix the x, y, z positions and tip / tilt angle, while the thread can be substantially used to set the focus). Alternatively, as in the case of using active alignment (AA), the pre-focus position can be set by optically or mechanically fixing the focus position between the lens barrel 108 and the mount structure 110. Once fixed in this manner, the lens and support assembly can be actively adjusted in all degrees of freedom and engaged via a gap engagement between the support mounting structure 110 and the PCB 112 to fix the x, y, final z focal point, tip, tilt, and azimuth rotation. Alternatively, the threadless lens support can be first engaged to the PCB, and then the lens with the threadless lens barrel can be positioned and engaged using AA.

[0026] Camera 100 also includes a controller 116 configured to control image sensor 104 to acquire images of scene 102, and to perform other control operations of camera 100 discussed herein. Controller 116 may include a logic subsystem and a storage subsystem. The logic subsystem includes one or more physical devices configured to execute instructions held in the storage subsystem to perform any operations, algorithms, calculations, or transformations disclosed herein. In some implementations, the logic subsystem may take the form of an application-specific integrated circuit (ASIC) or a system-on-a-chip (SoC), wherein some or all of the instructions are hardware-coded or firmware-coded. References to the logic subsystem and storage subsystem of controller 116 are provided. Figure 15 Further detailed discussion is needed.

[0027] During the manufacturing process of camera 100, manufacturing tolerances can cause camera-to-camera variations in the optical alignment of image sensor 104 and lens 106, potentially leading to a deviation of the image sensor's position from its ideal alignment with lens 106. Image sensor 104' is shown with a position offset relative to its ideal alignment with lens 106. Furthermore, manufacturing tolerances in lens 106 itself can cause variations in the optical alignment of image sensor 104 and lens 106. As shown in sidebar 122, when image sensor 104 is ideally aligned with lens 106, the optical axis 114 of lens 106 is centered relative to image sensor 104 at the actual optical center 118 of lens 106. However, when the position of image sensor 104' is offset relative to the ideal position of image sensor 104, the actual optical center 118 of lens 106 deviates from the center 120 of image sensor 104'. The difference between the center of image sensor 104' and the actual optical center 118 of lens 106 can affect image distortion that contributes to the lens.

[0028] The actual optical center 118 can vary from camera to camera, resulting in different cameras generating original images with different distortions based on different camera-dependent optical centers. Thus, in some examples, applying tilt projection distortion correction to the image can be based on the determined optical center and the lens's nominal distortion data. The optical center can be determined in any suitable manner. As an example, during manufacturing, the optical center can be determined camera-by-camera by projecting a plane illumination field through lens 106 onto image sensor 104 and measuring the centroid of the image intensity distribution. Note that if the camera includes a lens with finite relative illumination attenuation, pointing measurements based on live images, including a calibrated test setup (which has an optical target with a positional reference relative to the lens's optical axis, which can be repeatedly mounted using a kinematic mechanical mounting of the lens barrel), can be expected to provide higher accuracy for the optical center measurement. As another example, a laser beam aligned with optical axis 114 can be used to measure the optical center 118. In other examples, tilt projection distortion correction can be based on any other suitable calibration.

[0029] Figure 2A block diagram of controller 116 is shown in more detail. Controller 116 is configured to acquire a raw image 204 of a scene via image sensor 104. Controller 116 can be configured to load the raw image 204 into memory 202 of camera 100. Controller 116 is also configured to generate a distortion-corrected image 214 based on the raw image 204, which may be based on camera-specific optical center 118 or other suitable calibration of camera 100. The measured camera-specific optical center 118 or other calibration may be stored in memory 202 (e.g., electrically erasable programmable read-only memory (EEPROM) or other non-volatile memory) so that the camera-specific optical center 118 or other calibration can be used to perform distortion correction operations. The measured camera-specific optical center 118 may be represented as (x,y) coordinates in the coordinate space of image sensor 104. Due to lens distortion and / or alignment differences, the measured camera-specific optical center 118 can deviate from the actual center position of the image sensor 104. While optical center (OC) position data is not required to perform distortion correction on the image, the magnitude of the effective distortion of the corrected image can increase as the OC deviates from the origin position used in the projection mapping. For example, in some applications, small OC errors (i.e., on the order of <5-10 pixels) can provide sufficient accuracy, while other applications may utilize higher-precision OC data. This means that the accuracy of the OC error achieved through lens alignment and integration into the image sensor PCB can help inform the level of requirements and results. Note that including accurate OC position data enables and maintains high-quality correction for a wide range of OC errors because the distortion correction mesh file (described below) is generated to compensate for the OC.

[0030] Controller 116 may include a distortion correction machine 206 configured to translate the pixel positions of pixels in the original image 204 according to a distortion correction projection 212 including tilt parameters to generate a distortion-corrected image 214. In other examples, distortion correction may be performed on another computing device, such as a computer receiving image data from camera 100 (e.g., a computing device to which camera 100 is integrated), rather than on controller 116. Note that, for example, pixel positions of different pixels in the original image can be translated and / or interpolated on a basis of individual pixels based on the distortion correction projection by applying a mesh (e.g., the mesh file described above) that indicates a floating-point position within the original input image (x', y') for each integer (x, y) pixel of the distortion-corrected image. Thus, in different instances, the pixel positions of different pixels can be translated differently (e.g., different translation directions and / or distances for different pixels), the pixel positions of different pixels can be translated identically (e.g., the same translation direction and / or distance for different pixels), and / or the pixel positions of some pixels can remain the same between the original image 204 and the distortion-corrected image 214. Furthermore, distortion correction may include stretching and / or compressing portions of the image. Additionally, in cases where a tilt sensor is not provided, the distortion-corrected image 214 can be analyzed via software (host) or firmware (embedded) to detect and locate line objects representing vertical objects in the scene, i.e., the corners of room walls, which are expected to be vertical after distortion correction, and the adjustment of the tilt angle is fed back to the tilt sensor as input. Such an iterative process thus utilizes scene content as feedback to determine the correct tilt parameters without the need for a tilt sensor.

[0031] More specifically, distortion correction machine 206 can be configured to perform distortion correction mapping based on distortion correction projection 212, which includes pitch tilt parameters. Distortion correction machine 206 may optionally utilize measured camera-specific optical center 118, image sensor parameters 208, and / or lens distortion parameters 210 as input. In one example, image sensor parameters 208 may include the resolution of image sensor 104 (e.g., the number of pixels included in the image sensor in both the x and y dimensions) and the pixel size of the pixels of image sensor 104 (e.g., the size of the pixels in both the x and y dimensions). In other examples, other image sensor parameters may be considered for distortion correction projection 212. In one example, lens distortion parameters 210 may include distortion data, such as the field angle of the actual image height relative to lens 106. In other examples, other lens distortion parameters may be considered for distortion correction projection 212. In some examples, multiple distortion-corrected projections 212 can be stored, for example, corresponding to different tilt angles and / or different types of projections (e.g., cylindrical, spherical, and / or linear). Furthermore, in some examples, tilt angle data from tilt sensor 213 can be used to select the distortion-corrected projection 212 to be applied, such that the tilt angle of the projection corresponds to the camera tilt angle determined according to the tilt sensor data. As an example, tilt sensor 213 can be integrated into a camera or a computing device including a camera. Additionally, tilt parameters can be provided using a dedicated mechanical camera mount with a known tilt angle relative to the lens optical axis. In other examples, the tilt angle can be provided as user input rather than determined based on tilt sensor data. It should be noted that other geometric distortion-corrected projections not shown herein are envisioned to be compatible with tilt distortion correction, such as toroidal or hybrid projections, such as an upper cylindrical and lower toroidal or spherical projection with a transition at the equator. Furthermore, geometric distortion-corrected projections compatible with tilt distortion correction can include a parabolic profile along the vertical FOV that sweeps or curves radially along the horizontal FOV.

[0032] For a specific type of camera configuration using lens 106 and image sensor 104, sensor parameters 208 and lens distortion parameters 210 may be known a priori. For example, sensor parameters 208 and lens distortion parameters 210 may be the same for each camera in a particular manufacturing batch, while each camera in a particular manufacturing batch may have different measured camera-specific optical centers due to manufacturing variations. In some implementations, sensor parameters 208 and lens distortion parameters 210 may be stored in memory 202, while in other implementations, sensor parameters 208 and lens distortion parameters may be hardcoded into the distortion correction algorithm. Furthermore, in some examples, the mesh used for interpolation of the distortion-corrected image may be implemented too large due to the expected pixel tolerance (e.g., + / - 50 pixels) of the optical center variation in that manufacturing batch. The mesh can be trimmed in firmware and / or software to match the output size based on a given offset from the stored optical center data for each camera case. In other examples, the mesh used for interpolation can have the same size as the distortion-corrected output image, where the mesh is calculated once based on stored optical center data during camera initialization.

[0033] The distortion-corrected projection 212 defines the relationship between the pixel positions of the original image 204 and the translated pixel positions of the distortion-corrected image 214 as an inverse function, where sensor coordinates are mapped to the projection plane and / or surface coordinates of the distortion-corrected projection 212. As described above, the distortion-corrected projection 212 includes tilt parameters based on the camera's pitch tilt angle.

[0034] Figure 3 An example meeting room 300 is shown, including a video conferencing display and a camera, wherein the camera 100 is positioned on one side of the display 302 at a downward tilt angle. Figure 3 In this example, camera 100 is located above display 302, but in other examples, camera 100 may be located on different sides of display 302, including the left, right or bottom in some examples. Figure 4 The example raw image 400, acquired by a camera equipped with a wide-angle f-θ, is shown, having the characteristics of... Figure 1 A similar arrangement in the room depicted. The original image 400 includes a horizontal FOV (HFOV) of 145.5° and was acquired with a downward tilt angle of 17 degrees to the horizontal plane. As can be seen, objects on each side of the image (e.g., object 402) appear curved and distorted. Furthermore, objects closer to the center of the image appear unnaturally distorted or compressed (e.g., object 404 is compressed at the waist). Due to the unnatural appearance of attendees in the meeting room image, the original image 400 may not provide a satisfactory meeting room experience for remote participants.

[0035] Figure 5 Image 500 is shown, resulting from applying tilt-free cylindrical projection distortion correction to image 400. Cylindrical projection maps image sensor coordinates in image sensor space to cylindrical coordinates. Prior to this mapping, the pixel grid of the original image is parameterized to the corresponding physical dimensions and can be offset based on a measured, camera-varying optical center to align the projection with the optical axis of the lens. In other examples, this shift can be omitted depending on the required level of accuracy for the application.

[0036] Figure 6 The geometric representation of the untilted cylindrical projection 600 used to generate image 500 is shown, where the image sensor coordinates are represented by (x, y) for the original image. The projection radius R... c It can be used as a scaler to set the field of view captured within the corrected image pixel location. For example, for the actual height H representing the horizontal edge of the image sensor. Re The field angle θ of (θ) is obtained by using R c Set to The corrected image can be set to face and include the target HFOV. Untilted cylindrical projection combines the image sensor coordinates and cylindrical coordinates (from azimuth arc length x...) c and height length y c The relationship between the coordinates on the cylinder (and therefore their positions in the corrected image) is defined as follows: The inverse function of the form, where

[0037] The radius r0 can be determined as follows:

[0038] Therefore, for :

[0039] For a unit radius, the factor c can be defined as follows: The ratio to "sin": ,in The distortion lookup table is defined as follows: For the corner The percentage distortion p0 at 0 (radians).

[0040] In image 500, object 402, while no longer curved and distorted, appears tilted to the right. Similar distortions can be seen in other people and objects in the image. For example, the top edge of monitor 504 is curved, giving monitor 504 a distorted appearance. Similarly, the whiteboards on either side of the conference room (e.g., whiteboard 506) also appear tilted outwards. Therefore, although participants and other objects appear more natural than in the uncorrected image, the remaining distortions can lead to an unsatisfactory experience for remote participants.

[0041] Therefore, as mentioned above, in order to avoid objects appearing tilted outwards in the image, the tilt parameter can be incorporated into the projection map for image distortion correction. Figure 7 This diagram illustrates the geometry of an example tilted cylindrical projection 700. The tilted cylindrical projection 700 maps image sensor coordinates in image sensor space to cylindrical coordinates of a tilted cylinder to generate a corrected image from the original image. For the original image, the image sensor coordinates are represented by (x, y). The projection radius R... c It can be used as a scaler to set the field of view captured within the corrected image pixel location. For example, for the actual height H representing the horizontal edge of the image sensor. Re The field angle θ of (θ) is obtained by using R c Set to The corrected image can be set to face and include the target HFOV. Cylindrical projection combines the image sensor coordinates and cylindrical coordinates (from azimuth arc length x...) c and height length y c The coordinates on the cylinder represent the tilt angle of all points rotated about point P in the YZ plane. The relationship between x(f, R) and its corresponding position in the corrected image is defined as x(f, R). c , x c , y c , ) & y(f, R c , x c ,y c , The inverse function of the form ).

[0042] For a given tilt angle Horizontal position x o For a given (x) c , y c ) remains constant, however y o and z o Depends on the distance from point C=(l,m,n) to point =( , , The radial offset in the yz plane. Here, the coordinate (x... o , y o , z o ) is the offset point Cartesian coordinates relative to the origin (i.e., in the reference frame of the camera located at the center of the untilted cylinder). Coordinates (x... c ,y c , z c () represents the cylindrical coordinates of point C relative to the origin, equivalent to the offset point. Cylindrical coordinates relative to the center of the tilted cylinder. The coordinates (l, m, n) are Cartesian coordinates relative to the pivot point P of the cylinder, located at a distance R from the origin of the camera reference frame. c First, determine the radial offset corresponding to point C. y o (x c , y c , )and z o (x c , y c , ):

[0043] For (x) o , y o , z o ):

[0044] For the corner , with radius r o The relationship is:

[0045] Therefore, for the field angle ,because ,so

[0046] Now, the x and y positions on the sensor plane (from the projection origin at the optical axis, representing the rotation tilt angle) The xc and yc positions on a cylinder can be defined as: The distortion lookup table is defined as follows: For the corner The percentage distortion p0 at 0 (radians).

[0047] Distortion-corrected projection can be pre-computed and stored as a pixel mapping function (mesh file), which is applied to pixels in the original image to produce a distortion-corrected image. Figure 8 An example original image 800 and an example distortion-corrected image 802 are shown. The original image 800 includes contour lines in the form of concentric circles, representing the field angle of the lens (in degrees). In the example shown, the contour lines represent 10-degree increments from 0 to 90 degrees. The contour 804 in the original image 800 represents the frame edges of the distortion-corrected image 802. In the example shown, the original image 800 was generated by a camera supporting a ~137-degree HFOV using spherical or cylindrical projection. In other examples, the distortion-corrected projection can be calculated during use, rather than pre-calculated.

[0048] Figure 9 A corrected image 900 is shown, obtained by applying tilted cylindrical projection correction to image 400. In the corrected image 900, object 402 is no longer distorted as in the original image 400, or tilted as in image 500 corrected by untilted cylindrical projection, but appears neither distorted nor vertically oriented. Furthermore, whiteboards along the sides of the room (e.g., whiteboard 506) also appear vertically oriented, and the top edge of display 504 appears straight rather than curved. Additionally, object 404 no longer appears narrower at the bottom of the image. Thus, the corrected image 900 provides a more satisfying experience for participants in remote meetings, giving them a greater sense of "being there" due to the more natural appearance and orientation of the user and other objects in the room.

[0049] As described above, in some examples, the camera can be configured to be used with multiple different pitch angles. For example, the camera may have a pivot joint at which the camera body is attached to a mounting bracket. For cameras integrated into the device body (e.g., cameras positioned in the bezel of a display), the device or the device's mounting bracket may include similar joints. Similarly, a device including a camera (such as a display) may have different pitch angles when mounted on a bracket compared to a wall. Therefore, to allow the application of appropriate tilt projection, the camera or the device including the camera may also include a tilt sensor (e.g., an inertial motion sensor, a rotary encoder integrated into the pivot joint, etc.) to allow sensing of the pitch tilt angle. In some examples, the desired tilt angle parameter may be associated one-to-one with a specific device's mounting bracket, allowing the camera to detect the tilt angle based on communication with that device or a specific mounting bracket on that device, via a communication connection with the device at the mounting bracket. Based on the sensed pitch tilt angle, a corresponding projection mapping can be applied to correct the image acquired at the sensed tilt angle.

[0050] In some examples, tilt distortion corrected projections can be pre-computed for each of multiple pitch angles. For example, a tilt distortion corrected pixel map can be computed for each integer degree between 0° and 30°. In other examples, tilt distortion correction can be computed for every half degree. In still other examples, pixel maps can be computed for any other suitable angular range and / or increment, and the angular range can include positive or negative tilt angles. In further examples, tilt distortion maps can be computed, for example, in real time during use.

[0051] In the example above, a tilted cylindrical projection was used for distortion correction. In other examples, other projections that include a tilt parameter may be used. Figure 10 The pitch angle is shown. The example is a geometric representation of a tilted linear distortion correction projection 1000. The tilted linear projection maps the image sensor coordinates in image sensor space to the projection plane coordinates of the tilted linear projection plane. Prior to this mapping, the pixel grid of the original image is parameterized to the corresponding physical dimensions and may optionally be offset based on a measured, camera-dependent optical center.

[0052] For the original image, the image sensor coordinates are determined by (x... s ,y s () represents the projected distance z. p It can be used as a scaler to set the field of view captured within the corrected image pixel location. For example, for the actual height H of the horizontal edge of the image sensor. Re By z p Set to The corrected image can be set to face and include the target horizontal field of view (HFOV). Tilt linear projection combines the image sensor coordinates and the projection plane coordinates (from position (x...)) p , y p The coordinates on the plane are represented by ), and all points are located about a distance z along the z-axis. p The point at that location was rotated by the tilt angle. Therefore, the relationship between the positions in the corrected image is defined as... The inverse function of the form.

[0053] For a point P=(l,m,n) on a plane without inclination, with respect to the pivot point along the z-axis, l=x p m=y p And n=0. For a given tilt angle Horizontal position x o Maintain a constant. This includes the tilt of the plane. Point P moves to =( , , ),in =x p , =n sin +m cos =y p cos ,and =n cos -m sin =-y p sin Therefore, the coordinates relative to the XYZ origin become

[0054] Given the above (x) o , y o , z o The field angle is expressed as: ,and , and z p = Distance scaler to the plane .

[0055] For the projection plane coordinates (x) p ,y p ),

[0056] The radius r of the image sensor from the angle to the image height s It can be represented as a function: r s ( For example, this relationship can be determined using a distortion lookup table, which can be used to interpolate the radial image height for any given field angle, fit an equation, or estimate it using the parabolic percentage (p) distortion from f-θ at the field angle. Radius r s This can be determined as follows:

[0057] For the field angle in the radius

[0058] A distortion lookup table can be defined as follows:

[0059] For sensor coordinates (x) s , y s ), including those attributed to tilt angle Rotation:

[0060] Figure 11An example tilted spherical distortion-corrected projection 1100 is schematically illustrated. The tilted spherical projection maps image sensor coordinates in image sensor space to spherical coordinates. The projection radius R... s It can be used as a scaler to set the field of view captured within the corrected image pixel location. For example, for the actual height H representing the horizontal edge of the image sensor. Re The field angle θ of (θ) is obtained by using R s Set to The corrected image can be set to face and include the target HFOV. Spherical projection combines the image sensor coordinates and spherical coordinates (from azimuth arc length x...) s And elevation arc length y s The coordinates on the sphere represent the coordinates of all points rotated about point P in the YZ plane by an angle of inclination. The relationship between x(f, x) and its corresponding position in the corrected image is defined as x(f, x) s , y s , R s , ) & y(f, x s , y s , R s , The inverse function of the form ).

[0061] For a given tilt angle Horizontal position x o For a given (x) s , y s ) remains constant, however y o and z o Depends on the distance from point S=(l,m,n) to point S =( , , The radial offset in the yz plane. Here, the coordinates S=(l,m,n) are the Cartesian coordinates in the reference frame of the pivot point P, and the distance R from the origin of the camera's reference frame. s First, determine :

[0062] For (x) o , y o , z o ):

[0063] For the corner Having the same radius r o The relationship, and because :

[0064] Therefore, for the field angle :

[0065] Now, the x and y positions on the sensor plane (from the projection origin at the optical axis, representing the rotation tilt angle) x on the sphere s and y s Location can be defined as: The distortion lookup table is defined as follows: For the corner The percentage distortion p0 at 0 (radians).

[0066] Applying tilt distortion correction projection as disclosed herein allows images to be cropped to remove any empty areas on each side resulting from correction, while maintaining a significant horizontal field of view (HFOV) in the corrected image. Figure 12 An example projection mapping is shown, projecting the field angle of an f-θ lens onto a corrected image using a tilted cylindrical distortion-corrected projection. In this example, the camera pitch angle is 12°, as shown by the position of the horizontal H-plane 1204 relative to the optical center of the f-θ lens. In this example, the initial raw image includes an HFOV of 145.5°, while the corrected cropped portion 1206 includes an HFOV of 136° and a vertical FOV (VFOV) of 66.5°. In some examples, the image can be further cropped to an HFOV of 133° and a VFOV of 65.0°, as shown in the corrected cropped portion 1208. In some examples, the raw image can be captured using a camera including a 2560 × 1920 pixel image sensor (pixel size approximately 2.2µm). In other examples, the raw image can be captured using a high-resolution 4K camera including a 3840 × 2160 pixel image sensor (pixel size approximately 1.85µm). In some examples, the raw image can be captured as a 4K (16:9) ROI on a 12MP (4:3) image sensor. In some examples, the final corrected image may include a 16:9 aspect ratio. In other examples, the corrected image may include a 4:3 aspect ratio. In other examples, HFOV and VFOV may have any other suitable values ​​after correction is applied.

[0067] In some examples, the original image can be cropped before applying tilt projection mapping, such as when using oversized sensors to help explain OC errors while maintaining consistency in camera FOV pointing per camera in a group of devices. Figure 13An exemplary oversized original image 1300 of a conference room scene captured using a camera tilted downwards at a 17° pitch angle is shown. As an example, the oversized original image 1300 can be cropped at the center using cropping region 1302. As another example, the oversized original image 1300 can be cropped using a cropping region vertically offset from the center, thus cropping the image into a vertically offset region of interest. In this example, cropping region 1304 is vertically offset downwards by 340 pixels to capture the original region of interest 1306. Corrected image 1308 represents untilted cylindrical distortion correction applied to the cropped region of interest. Corrected image 1310 represents tilted cylindrical distortion correction applied to the cropped region of interest 1306 using a 17° tilt angle. Corrected image 1310 can be further cropped via cropping region 1312 to produce a corrected image for presentation to a user. The combination of camera pointing angle (or camera pitch) and vertical offset can be used to achieve a greater vertical offset in the captured image than the camera pitch angle or the vertical offset caused by the ROI, so that the camera can appear to be pointing at a downward tilt angle that is more subtle in appearance than the angle it can capture. This provides industrial design with some freedom regarding the physical appearance of the camera in the application, while also enabling similar scene capture functionality for a variety of pitch angles.

[0068] Figure 14 A flowchart illustrating an example method 1400 for correcting tilt distortion in images acquired from a camera with a pitch angle is shown. For example, method 1400 could be derived from... Figure 1 The method is executed by the controller 116 of the camera 100 shown, or on a computing device that receives image data from the camera 100. At 1402, method 1400 includes acquiring an image of the scene acquired via the camera. The image of the scene includes image distortion caused by the camera's pitch angle at which the scene image is acquired. In some implementations, at 1404, method 1400 may optionally include cropping the image of the scene before applying projection mapping. For example, the image of the scene may be cropped with a vertical offset similar to that of cropping region 1304. In some implementations, at 1406, method 1400 may optionally include acquiring a scene image comprising a horizontal field of view between 70° and 145°. In other implementations, method 1400 may optionally include acquiring a scene image comprising a horizontal field of view up to 180° or even greater. For example, linear projection can be used for corrections up to 180 degrees (although <120 degrees may be more practical for many biometric applications when considering the significant drop in on-axis pixel density at higher FOVs using linear projection), while cylindrical and spherical projections can be applied to angles greater than 180 degrees, such as using fisheye lenses. As mentioned above, image distortion can be more severe in such wide and ultra-wide angle ranges than in lower angle ranges.

[0069] At 1408, method 1400 includes applying a projection map to an image of the scene to map the image of the scene onto a projection including a tilt parameter based on the camera pitch angle at which the image of the scene was acquired, thereby obtaining a corrected image. In some implementations, at 1410, method 1400 may optionally include applying one of a tilted cylindrical projection 700, a tilted linear projection 1000, or a tilted spherical projection 1100. In some implementations, at 1412, method 1400 may optionally include selecting the projection map from a plurality of projection maps, each corresponding to a different camera pitch angle. For example, the plurality of projection maps may be stored in a table indexed by the tilt parameter of the projection. In some implementations, at 1414, method 1400 may optionally include applying the projection map based on the position of the optical center of the camera lens relative to the image sensor of the camera. For example, the camera acquiring the scene image at 1402 may include an image sensor and a camera lens having an optical center offset relative to the image sensor. In this example, method 1400 may apply the projection based on this offset.

[0070] In 1416, method 1400 includes outputting an image. In some implementations, in 1418, method 1400 may optionally include outputting the image to a video conferencing application or any other suitable application.

[0071] In some embodiments, the methods and processes described herein may be associated with a computing system including one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0072] Figure 15 A non-limiting embodiment of a computing system 1500 capable of performing one or more of the methods and processes described above is illustrated schematically. The computing system 1500 is shown in a simplified form. The computing system 1500 may take the form of one or more of the following: a personal computer, a server computer, a tablet computer, a home entertainment computer, a network computing device, a gaming device, a mobile computing device, a mobile communication device (e.g., a smartphone), and / or other computing devices. For example, the computing device 1500 may represent a controller 116, a camera 100, a computing device including the camera 100 (e.g., a video conferencing system with an integrated camera), or any other suitable computing system.

[0073] The computing system 1500 includes a logic subsystem 1502 and a storage subsystem 1504. The computing system 1500 may optionally include a display subsystem 1506, an input subsystem 1508, a communication subsystem 1510, and / or... Figure 15 Other components not shown.

[0074] The logic subsystem 1502 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise obtain desired results.

[0075] A logical subsystem may include one or more processors configured to execute software instructions. Additionally or alternatively, a logical subsystem may include one or more hardware or firmware logical subsystems configured to execute hardware or firmware instructions. The processor of the logical subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel, and / or distributed processing. The individual components of the logical subsystem may optionally be distributed across two or more separate devices, which may be located remotely and / or configured for collaborative processing. Aspects of the logical subsystem may be virtualized and executed by remotely accessible, networked computing devices configured with cloud computing capabilities.

[0076] Storage subsystem 1504 includes one or more physical storage devices configured to hold instructions executable by a logical subsystem to implement the methods and processes described herein. In implementing such methods and processes, the state of storage subsystem 1504 can be transformed—for example, to maintain different data. For example, storage subsystem 1504 may store instructions executable to apply distortion correction as described above. Furthermore, storage subsystem 1504 may store applications executable by logical subsystem 1502, such as video conferencing application 1512 for receiving distortion-corrected images for transmission and / or display.

[0077] Storage subsystem 1504 may include removable and / or built-in devices. Storage subsystem 1504 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. Storage subsystem 1504 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices.

[0078] It is understood that the storage subsystem 1504 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated via a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not held by the physical device for a finite duration.

[0079] Various aspects of the logic subsystem 1502 and the storage subsystem 1504 can be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), systems-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0080] The terms "module," "program," and "engine" can be used to describe aspects of the computing system 1500 implemented to perform specific functions. In some cases, a module, program, or engine can be instantiated by executing instructions held by the storage subsystem 1504 within the logical subsystem 1502. It should be understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass single or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0081] It should be understood that, as used herein, a "service" is an application that can execute across multiple user sessions. A service can be used for one or more system components, programs, and / or other services. In some implementations, a service can run on one or more server computing devices.

[0082] When included, display subsystem 1506 can be used to present a visual representation of the data held by storage subsystem 1504. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data held by the storage subsystem, and thereby transform the state of the storage subsystem, the state of display subsystem 1506 can also be transformed to visually represent the changes in the underlying data. Display subsystem 1506 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined with logic subsystem 1502 and / or storage subsystem 1504 in a shared package, or such display devices may be peripheral display devices.

[0083] When the input subsystem 1508 is included, it may include one or more user input devices such as a keyboard, mouse, touchscreen, or game controller, or interface with such user input devices. In some embodiments, the input subsystem may include or interface with selected Natural User Input (NUI) components. Such components may be integrated or peripheral, and the transduction and / or processing of input actions may be handled on-board or off-board. Example NUI components may include a microphone for speech and / or voice recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity.

[0084] When a communication subsystem 1510 is included, the communication subsystem 1510 may be configured to communicatively couple the computing system 1500 to one or more other computing devices. The communication subsystem 1510 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem may allow the computing system 1500 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0085] Another example provides a video conferencing system including: a processor and a storage device storing instructions executable by the processor to: acquire an image of a scene captured via a camera, the image of the scene including image distortion caused by a camera pitch angle used to acquire the image of the scene; apply a projection map to the image of the scene to map the image of the scene to a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; and output the corrected image. In some such examples, the video conferencing system includes a camera and a display, wherein the camera is positioned to the side of the display. In some such examples, the camera includes a tilt sensor configured to provide an output based on the camera pitch angle, and the instructions are also executable to select a projection map from a plurality of projection maps based on the output from the tilt sensor, each projection map being based on a different camera pitch angle. In some such examples, the camera includes an f-θ lens. In some such examples, the instructions are also executable to select a projection map based on the rotational position of the camera. In some such examples, the camera includes an excessively large image sensor, and instructions are executable to crop the image to form a vertically offset region of interest. In some such examples, the scene image includes 7 and 14 The horizontal field of view between. In some such examples, the projection mapping includes tilted cylindrical projection, tilted spherical projection, or tilted linear projection. In some such examples, the instructions may also execute to apply the projection mapping based on the camera-specific optical center of the camera lens relative to the image sensor. In some such examples, the instructions may execute to crop the scene image before applying the projection mapping.

[0086] Another example provides a method for operating a video conferencing system, the method comprising: acquiring an image of a scene captured by a camera, the image of the scene including image distortion caused by a camera pitch angle used to acquire the scene image; applying a projection map to the scene image to map the scene image to a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; and outputting the corrected image. In some such examples, applying a projection map to the scene image includes applying a tilted cylindrical projection, a tilted spherical projection, or a tilted linear projection. In some such examples, the method further includes selecting the projection map from a plurality of projection maps, each of the plurality of projection maps corresponding to a different camera pitch angle. In some such examples, applying the projection map is based on the position of the optical center of the camera lens relative to the image sensor of the camera. In some such examples, the method further includes cropping the scene image before applying the projection map. In some such examples, the scene image includes 7 and 14 The horizontal field of view between them.

[0087] Another example provides a computer-readable storage device including instructions executable by a video conferencing system to correct image distortion by: acquiring an image of a scene captured by a camera; applying a projection map to the scene image to map the scene image to a projection including tilt parameters based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; and outputting the corrected image. In some such examples, the instructions can be executed to apply a projection map to the scene image of the scene by applying a tilted cylindrical projection, a tilted spherical projection, or a tilted linear projection. In some such examples, the instructions can also be executed to receive input about the camera pitch angle from a tilt sensor and select a projection map from a plurality of projection maps, each of the plurality of projection maps corresponding to a different camera pitch angle. In some such examples, the instructions can also be executed to select a projection map based on the rotational position of the camera.

[0088] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions explained and / or described may be performed in the explained and / or described order, in a different order, in parallel, or omitted. Similarly, the order of the processes described above may be changed.

[0089] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as other features, functions, actions and / or attributes, and any and all equivalents thereof.

Claims

1. A video conferencing system, comprising: processor; as well as A storage device for storing instructions, which can be executed by the processor to: Obtain an image of a scene captured by a camera, the image of the scene including image distortion caused by the camera pitch angle used to capture the image of the scene; The projection mapping is applied to the image of the scene to map the image of the scene onto a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; as well as Output the corrected image; The projection mappings mentioned above include tilted cylindrical projection, tilted spherical projection, or tilted linear projection; The tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is tilted relative to the optical axis of the camera based on the tilt parameter. and The tilt of the tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is performed around a pivot point located on the optical axis of the camera and spaced a distance from the camera; The camera includes a tilt sensor configured to provide an output based on the camera's pitch angle, and the instructions are also executable to select a projection map from a plurality of projection maps based on the output from the tilt sensor, each projection map being based on a different camera pitch angle.

2. The video conferencing system according to claim 1, characterized in that, The video conferencing system includes the camera and the display, wherein the camera is positioned on one side of the display.

3. The video conferencing system according to claim 2, characterized in that, The camera includes an f-θ lens.

4. The video conferencing system according to any one of claims 2, characterized in that, The instructions can also be executed to select the projection map based on the rotational position of the camera.

5. The video conferencing system according to any one of claims 2, characterized in that, The camera includes a large-size image sensor, and the instructions are executable to crop the image using a cropping region vertically offset from the image center to form a vertically offset region of interest.

6. The video conferencing system according to any one of claims 1 to 2, characterized in that, The image of the scene includes a horizontal field of view between 70 and 145 degrees.

7. The video conferencing system according to any one of claims 1 to 2, characterized in that, The instructions can also execute the application of the projection mapping based on the position of the camera lens's varying optical center relative to the image sensor.

8. The video conferencing system according to any one of claims 1 to 2, characterized in that, The instructions can be executed to crop the image of the scene before applying the projection mapping.

9. A method for operating a video conferencing system, the method comprising: Obtain an image of a scene captured by a camera, the image of the scene including image distortion caused by the camera pitch angle used to capture the image of the scene; The projection mapping is applied to the image of the scene to map the image of the scene onto a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; as well as Output the corrected image; The image to which the projection mapping is applied to the scene includes the application of tilted cylindrical projection, tilted spherical projection, or tilted linear projection; The tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is tilted relative to the optical axis of the camera based on the tilt parameter. and The tilt of the tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is performed around a pivot point located on the optical axis of the camera and spaced a distance from the camera; The camera includes a tilt sensor configured to provide an output based on the camera's pitch angle, and to select a projection map from a plurality of projection maps based on the output from the tilt sensor, each projection map being based on a different camera pitch angle.

10. The method according to claim 9, characterized in that, It also includes selecting the projection map from a plurality of projection maps, each of which corresponds to a different camera pitch angle.

11. The method according to any one of claims 9 or 10, characterized in that, The projection mapping is applied based on the position of the optical center of the camera lens relative to the image sensor of the camera.

12. The method according to any one of claims 9 or 10, characterized in that, It also includes cropping the image of the scene before applying the projection mapping.

13. The method according to any one of claims 9 or 10, characterized in that, The image of the scene includes a horizontal field of view between 70 and 145 degrees.

14. A computer-readable storage device including instructions executable by a video conferencing system to correct image distortion by: Obtain images of the scene captured by the camera; The projection mapping is applied to the image of the scene to map the image of the scene onto a projection including a tilt parameter based on the camera pitch angle used to acquire the image of the scene, thereby obtaining a corrected image; as well as Output the corrected image; wherein the instructions are executable to apply the projection mapping to a scene image of the scene by applying a tilted cylindrical projection, a tilted spherical projection, or a tilted linear projection; The tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is tilted relative to the optical axis of the camera based on the tilt parameter. and The tilt of the tilted cylindrical projection, the tilted spherical projection, or the tilted linear projection is performed around a pivot point located on the optical axis of the camera and spaced a distance from the camera; The camera includes a tilt sensor configured to provide an output based on the camera's pitch angle, and the instructions are also executable to select a projection map from a plurality of projection maps based on the output from the tilt sensor, each projection map being based on a different camera pitch angle.

15. The computer-readable storage device according to claim 14, characterized in that, The instructions can also be executed to select the projection map based on the rotational position of the camera.

Images