How to control the camera

Abstract

The method includes displaying a first image acquired from a camera having an input camera projection model including a first focal length and optical axis parameter value. A portion of the first image is selected as a second image associated with an output camera projection model in which any of the focal length and / or optical axis parameter value differs from parameters of the input camera projection model. The method includes iteratively adjusting any of the focal length and / or optical axis parameter value of the camera lens to approach a corresponding value of the output camera projection model, acquiring a subsequent image using the adjusted focal length or optical axis parameter value, mapping pixel coordinates in the second image to respective positions in the subsequent image via a normalized three-dimensional coordinate system to determine values ​​for each of the pixel coordinates, and displaying the second image.

Description

[Technical field]

[0001] The present invention relates to a method for controlling a camera. [Background technology]

[0002] 1, in a typical electronically controlled zoom camera 10, a processor 16 running a camera application (not shown) can respond to user interaction by causing an actuator 18 for the zoom lens 12 to change the focal length, thereby varying the magnification and field of view of the camera 10. Typically, the actuator 18 comprises a stepper motor (not shown) or equivalent mechanism, and the speed of the optical zoom transition from one magnification to another is limited by the speed at which the lens assembly can be adjusted.

[0003] In this case, the zoom lens 12 is shown as a single module, but it will be understood that it may comprise at least an optical lens assembly coupled to an image sensor which provides any acquired image to an image processing pipeline which performs basic image processing of the acquired image before it is written to memory 14 via system bus 30, each of these components being positioned as needed within a given camera.

[0004] In this regard, it will be understood that the term camera includes any image capture device with zoom capabilities, including dedicated still cameras, video cameras, smart phones, tablet computers, and security cameras. In the last case in particular, the camera application typically runs on a remote computing device with a network connection to the non-interactive application running on the camera processor 16.

[0005] Any such camera may also have a digital zoom module 20 that extends the zoom range beyond the optical zoom capability of the lens 12; thus, referring to FIG. 2, once the optical zoom capability of the lens 12 reaches its maximum value (in this case 30x magnification), the digital zoom module 20 is activated to provide additional image magnification (in this case up to 40x magnification).

[0006] 1, for purposes of illustration, the zoom module 20 is shown separate from the lens 12, but it may be integrated into the image processing pipeline firmware so that images written to memory are immediately provided at the required magnification. Alternatively, the functionality of the zoom module 20 may be implemented in a camera application, whether local or remote to the camera 10, so that images acquired from the camera 10 can be magnified before being displayed.

[0007] In FIG. 1, at startup, the digital zoom module 20 takes a required portion 14″ of a relatively low magnification image 14′ that has been acquired or acquired from the lens 12 and written (written) to the memory 14′, and interpolates this image to provide a display image that is typically the same size and resolution as the original low magnification image 14′. Such interpolation can utilize bilinear or bicubic interpolation, but the zoom module 20 can also utilize neural network type processing to provide improved quality enlarged images, such as deep backpropagation networks (DBPNs), as disclosed in, for example, “Deep Back-Projection Networks for Single Image Super-resolution” by Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita, arXiv:1904.05677v1, [cs.CV], 4 Apr 2019, but it will be understood that any super-resolution network can be utilized.

[0008] The camera or camera application may have a dedicated zoom button or screen icon that allows the user to select to increase or decrease magnification; in such a case, the portion of the display field of view that is selected to be magnified is typically on the center of the display field of view.

[0009] However, in some cases, for example when a camera application provides an interactive display, the portion of the display field of view that is selected to be enlarged may not be centered within the display.

[0010] In other cases, the camera may also have controls that allow a user to request that the camera pan (P) and / or tilt (T) simultaneously with or after a zoom (Z) request. An example of such a PTZ camera is a fixed-mount security camera that is remotely controlled from a user application running on a network-connected device. Again, such a pan or tilt request is accompanied by a delay as the pan / tilt actuator physically moves the camera so that the optical axis of the camera is centered in the field of view.

[0011] If the camera 10 is equipped with an interactive display, such as a touchscreen display (not shown), whether directly on the camera or on a network-connected device running a camera application, this interaction can include the time it takes for the user to perform interactive gestures when selecting the portion of the display field of view that they wish to magnify, either using a pointer device or touchscreen gestures.

[0012] As shown in FIG. 2, the time taken to complete an interactive gesture (e.g., t0-t1 and t2-t3) or the time taken to select a button / icon, t4, can be extremely short, while optical zoom, as well as pan or tilt, are limited by the speed of the respective actuators, and user interaction with the camera 10 may be delayed in reaching the requested zoom level or position.

[0013] This can be particularly noticeable when a user, such as at time t4, can instantly specify an increase in zoom level, for example, by selecting a touch screen icon or camera button to request maximum optical zoom. In the example of FIG. 2, the optical zoom only reaches this zoom level after time t5, when the user requests a further increase in zoom level, which requires digital zoom. From this point on, image interpolation can typically be performed at least as fast as the camera's image frame acquisition rate, so that the camera can respond almost instantly to a user request to change the level of digital zoom in the region beyond maximum optical magnification and maximum optical magnification plus digital magnification (40x in this case).

[0014] US 10469759 discloses a device including a lens assembly for focusing or optically zooming an image, a sensor array for capturing an image, a memory for storing instructions, and a processor for executing the instructions to cause the sensor array to capture an image, determine a field of view in the captured image, obtain parameter values ​​based on sensor measurements associated with the captured image, obtain a digital zoom profile and an optical zoom profile based on the parameter values, the captured image, and the field of view, apply the digital zoom profile and the optical zoom profile together, and configure the camera to the received zoom request for the field of view.

[0015] The approach of U.S. Patent No. 10,469,759 is deficient in that it does not take into account the changing relationship between a potential off-center selected portion of the display field of view and the optical axis of the camera as the camera transitions from a given zoom level to a requested zoom level. This means that as the optical image magnification increases during a zoom transition, the field of view is continually distorted. Thus, as the optical portion of the increased magnification increases as the lens actuator 18 drives the lens to the requested magnification, the advantage of instantly switching to the increased magnification is lost due to the continuously changing distortion of the field of view. [Prior art documents] [Patent documents]

[0016] [Patent Document 1] U.S. Pat. No. 10,469,759 [Patent Document 2] U.S. Patent Application Serial No. 16 / 575,748 [Patent Document 3] International Publication No. 2014 / 146983 [Patent Document 4] International Publication No. 2017 / 1404382003 [Non-patent literature]

[0017] [Non-Patent Document 1] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita, "Deep Back-Projection Networks for Single Image Super-resolution", arXiv:1904.05677v1, [cs.CV], 4 Apr 2019 Summary of the Invention

[0018] According to a first aspect of the present invention, there is provided a method for controlling a camera as claimed in claim 1. An embodiment of the present invention is based on image reprojection using the following two camera models: 1. An input camera model that represents the physical lens properties calibrated to acquire the image and 2. An output camera model that represents the desired projection of the output image. Using two separate camera models for reprojection allows: Supplementing the optical zoom with the digital zoom when there is a difference in focal length between the two camera models, without introducing distortion into the field of view as the optical zoom changes towards the desired zoom level. Distortion correction - for example, if the output camera has a rectilinear projection model and the input camera model is based on a different type of projection. Virtual camera rotation - if the camera model is rotated relative to the projection, perform electronic image stabilization to account for camera movement, especially during zoom transitions.

[0019] This two-stage re-projection is not the same operation as simply cropping and enlarging the input image: cropping does not preserve realistic perspective, but our re-projection uses two camera models to allow for distortion correction and allows for virtual camera rotation to enable image stabilization while still maintaining realistic perspective.

[0020] Thus, embodiments of the present method improve the user experience by appropriately combining digital and optical zoom for zoom levels below the maximum range of the lens without introducing field of view distortion during zoom transitions.

[0021] Response to zooming, as well as any panning or tilting required to enlarge or view selections off-center of the displayed field of view, is essentially instantaneous.

[0022] The input camera model is calibrated for all zoom levels and changes with changes in optical zoom level during zoom transitions. Thus, for each input image captured while the camera is transitioning from one zoom level to another, the changing zoom levels need to be taken into account and the method receives information about the current optical zoom from the camera before processing the captured image.

[0023] When a user wishes to select a particular zoom level, it can be set virtually instantaneously by adjusting the focal length of the output camera model. This results in instantaneous digital zoom. At the same time, the new zoom setting is sent to the optical zoom actuator, which starts moving the zoom position to the selected level. While the zoom actuator adjusts the lens, the effective focal length of the display image does not change, as it iteratively recalculates the lens projection from the current zoom level to the desired zoom level. The only effect the user sees is a gradual improvement in image sharpness as digital zoom replaces optical zoom.

[0024] In a second aspect, there is provided a handheld camera comprising a lens, a lens actuator configured to change a focal length of the lens, and a processor configured to run a camera application and control the camera to perform the steps of claim 1.

[0025] In a third aspect, there is provided a computer program product comprising computer readable instructions stored on a computer readable medium and configured to perform the steps of claim 1 when executed on a handheld camera.

[0026] In a fourth aspect, there is provided a system comprising a fixed mounted camera and a computer program product comprising computer readable instructions stored on a computer readable medium and configured to perform the steps of claim 1 when executed on a computing device operatively connected to the camera.

[0027] Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 illustrates a conventional camera having optical and digital zoom capabilities. [Diagram 2] FIG. 2 illustrates a transition from optical zoom to digital zoom in the camera of FIG. [Diagram 3] FIG. 2 illustrates an exemplary input / output camera projection utilized in one embodiment of the present invention. [Figure 4] FIG. 2 illustrates exemplary input and rotated output camera projections utilized in one embodiment of the present invention. [Diagram 5] FIG. 2 illustrates another exemplary input and output camera and rotating output projection utilized in one embodiment of the present invention. [Figure 6] FIG. 2 illustrates another exemplary input and output camera and rotating output projection utilized in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] 3, as will be appreciated, the field of view of a camera for acquiring an image can be modeled using a projection model, and for simplicity, the projection model of an input camera for acquiring an image can be represented as a 3D view frustum 30 with the base of the view frustum corresponding to the acquired image. Similarly, the portion of the displayed version of the acquired image that is selected to be magnified or the orientation to which the camera is panned / tilted can be represented as a view frustum 40 with an associated projection model, with each view frustum 30 and 40 having a common viewpoint P (also known as the center of projection).

[0030] In this example, camera 10 provides a level of optical magnification that produces a captured image having a field of view that corresponds to the rectangular base of view frustum 30. In this example, an output rectangular image from within the field of view of camera 10 is selected by a user either by interacting with a touch screen display or by actuation of a zoom button detected by a camera application. The base of view frustum 40 is shown elevated above the base of view frustum 30 for ease of illustration and to indicate that the projection module for view frustum 40 includes a longer focal length than for view frustum 30.

[0031] In the embodiment of FIG. 3, the output image corresponding to the base of the viewing frustum 40 is at the center of the acquired input image and therefore their respective optical axes O0 / O are coincident, although as will be understood from the detailed description below, this is not required.

[0032] An embodiment of the present application transforms points from the acquired input image into a normalized 3D space using an input camera model, and projects these points back into a 2D output image space using an output camera model.

[0033] The output camera model that transforms the acquired image through the input camera model to generate the output image is called a forward projection. In practice, however, any pixel location for which a value in the output image is determined is transformed to a location in normalized three-dimensional space using the output camera model, and then transformed to a location in the acquired image space using the input camera model. The acquired image data for pixel locations surrounding or adjacent to the projected location in the acquired image space is then used, typically interpolated, to generate a pixel value for the output image pixel location.

[0034] For monochrome images, only one value needs to be calculated per output image location, but for other image formats, including color formats such as RGB, YCC, LAB, etc., the above process is repeated for each color plane in the output image.

[0035] In either case, if both cameras have the parameters, the projection will result in the same transformation (the image is unchanged). However, if the focal length of the output camera is longer than the input camera, digital zoom is provided. Alternatively, or in addition, if the optical axis of the output camera is offset from that of the input camera, digital pan / tilt is provided.

[0036] F O is the focal length, O O Define the output camera projection that represents the desired magnification level as the optical axis position in the output coordinate space.

[0037] If digital enlargement is required, O is different from the calibration value F of the input camera projection. If the part (sub-region) of the display field of view selected to be magnified is off-center, O O is different from the optical axis O of the input camera projection.

[0038] As explained in the introduction, this can occur when the user freely selects a sub-region of the touch panel display to be magnified, or it can occur when the user requests to pan or tilt the camera.

[0039] The output camera model can be of any type, including fisheye or rectilinear, but for simplicity, pinhole projection is used in this example.

[0040] When using backprojection, the first step is to project the output image coordinates, typically pixel locations (X,Y), into a normalized 3D space, i.e. TIFF2024513783000002.tif32150

[0041] The resulting vectors [x,y,z] may be normalized to unit length, but this step is optional and does not affect the final result. In this specification, a normalized coordinate system is one in which all linear dimensions are divided by the effective focal length (as described above), and a normalized vector is one with a length of 1 (this is not important to this application).

[0042] Optionally, the points of the 3D coordinate system can be additionally rotated about the origin of the 3D coordinate system to counteract movement of the camera itself while the optical magnification and / or optical axis of the input camera moves towards the desired magnification and optical axis of the output camera. Obviously, this is typically only a necessary requirement for handheld cameras, which are subject to camera (and not lens) movement.

[0043] One technique that can determine and limit such rotations to avoid black boundary conditions resulting from the output image extending beyond the boundaries of the input image is described in detail in U.S. Application No. 16 / 575,748, filed September 19, 2019 (Reference: FN-648-US).

[0044] Alternatively, other processes for determining inter-frame motion can be used, such as those disclosed in WO 2014 / 146983 (Document: FN-389-PCT), as well as other forms of optical or electronic image stabilization, such as those disclosed in WO 2017 / 140438 (Document: FN-495-PCT).

[0045] In either case, whether rotated for image stabilization or not, the next step is to project the coordinates in 3D space onto the source image using a calibrated model of the input lens. Again, any projection model (rectilinear, fisheye, etc.) can be used, including any type of lens distortion modeling (radial, tangential, etc.).

[0046] In the following example, pinhole projection is extended with a radial distortion model for added realism. The first step in pinhole (perspective) projection is to divide the x,y coordinates by the z coordinate value. This projects the point from the normalized 3D coordinate system onto the X,Y plane (2D). TIFF2024513783000003.tif26150

[0047] Next, a distortion model is applied. TIFF2024513783000004.tif13150 Here TIFF2024513783000005.tif13150 and kn are the distortion coefficients. The final step is the transformation into the source image pixel space. TIFF2024513783000006.tif20150

[0048] In this final transformation, the optical axis coordinates O of the output camera model are Ox ,O Oy The optical axis coordinates of the input image, O, which may not be the same as x ,O y Note that is utilized.

[0049] Typically, O x and O y are determined through a camera calibration process and typically do not coincide with the center of the input image. They represent the position of the input camera's optical axis in the input image coordinate system (pixels). The output optical axis position O Ox ,O Oy can be set to the center of the output image. In this case, if there is no associated pan / tilt, the input image will be cropped symmetrically around the optical axis position. The crop area is (O x and O y If a crop is required to be centered in the input image, O Ox , O OyThe value of can be set such that when projected back onto the input image, the center of the output image coincides with the center of the input image. If pan / tilt of the output camera is applied, the center of the output image projected onto the input image will change position depending on the position and shape of the crop window. The nature of these changes will depend on the projection parameters of both cameras.

[0050] Note that if the camera does not have a pan / tilt mechanism and an off-center portion of the display image is selected for magnification, there is a limit to how far that portion can be magnified and still be fully visible. In this case, the camera application must choose between stopping the zoom transition at a point where all of the selected portion remains visible or continuing to zoom to the requested level and either cropping out part of the originally selected portion of the display image or displaying an empty area outside the input image.

[0051] If the camera motion is not being countered, the output view frustum can remain constant during zoom, pan and / or tilt transitions of the camera. However, if image stabilization is utilized, the view frustum of the output camera can move as the desired portion of the field of view to be displayed moves to counteract the motion of the camera itself while the optical magnification shifts towards the desired magnification and optical axis of the output camera.

[0052] Input camera (O x ,O y ) may be necessary if the camera is equipped with an optical image stabilization system that moves either the lens barrel or the sensor. In such cases, this movement is measured in real time by the camera system and passed on to the digital zoom module.

[0053] It will be appreciated that any difference between the view frustum sizes corresponding to the input and output images allows some freedom of rotation of the output view frustum, which can be used to change the orientation of a virtual camera to facilitate digital pan or tilt and / or to provide video stabilization.

[0054] 4 shows the output camera frustum 50 rotated relative to the input camera frustum 30. At any given time during the zoom transition, the greater the difference between the selected magnification, i.e., the output camera frustum 50, and the currently acquired input image frustum 30, the greater the range of stabilization (or rotational freedom) will be, since the angle of view difference between the two camera models will be greater. This can be understood by the relationship of the rotational range of the input camera's frustum 30 and output camera frustum 70 in FIG. 6 to the rotational range available to the unrotated frustum 60 in FIG. 5 and the frustum 50 in FIG. 4.

[0055] As will be appreciated, if the camera has a variable focus (zoom) lens, during the zoom transition the input camera model is updated, including any changes in focal length and any changes to account for the pan and / or tilt of the camera, and the input camera frustum 30 is updated accordingly. If the output camera model does not need to be changed, for image stabilization reasons, the effective output field of view remains the same. As the focal length of the lens increases, the only effect the user sees is improved image resolution, as digital zoom increasingly replaces optical zoom.

[0056] If image stabilization is utilized, the expanded field of view can remain constant and distortion-free even as the camera moves during the transition (possibly caused by the user interacting with the camera controls to select the desired zoom level).

[0057] As explained, a view frustum having a rectangular base is shown here for purposes of illustration and clarity, and is not limiting. The shape of the input camera's view frustum is typically determined by the type of lens projection and lens distortion used. The shape of the output camera's view frustum represents the desired final projection. This may be the same projection as the input camera, or any other projection if a model change of the projection is desired. For example, lens distortion can be corrected by creating the input view frustum 30 using a real lens projection model and creating the output camera view frustums 50, 60, 70 using a pinhole camera model. Similarly, the optical axis O of the output camera is determined by the type of lens projection and the lens distortion. O It should be appreciated that if x deviates from the optical axis O of the input camera, then the reprojection means that the flattum of the output camera will have a trapezoidal base rather than a rectangular base.

[0058] Although the above examples are based on zoom transitions, the above concepts can be applied when the selected portion of the displayed field of view is off-center, i.e., camera rotations including pan or tilt, in which case the selected portion of the displayed field of view can be instantly brought to the center of the display through rotation of the virtual camera and as the pan / tilt actuator rotates the physical camera, the virtual camera is rotated by the difference between the desired orientation and the actual orientation of the pan / tilt mechanism of the physical camera, and can also take into account the movement of the camera during the transition.

[0059] Note that in some cases, virtual pan / tilt can be applied without zooming.

[0060] In this case, the user may request to pan the camera to capture a field of view that overlaps with the current field of view (a typical use case would be when controlling a fixed camera).

[0061] Using the above method, the output camera's frustum initially extends outside the input camera's frustum, and a period of time is required for the input camera to pan / tilt so that its field of view covers the entirety of the output camera's frustum. Nevertheless, for those parts of the output camera's frustum that overlap the input camera's frustum at any given moment during the pan / tilt transition, the above-mentioned reprojection can be performed so that the visible parts of the output image can be displayed immediately at a fixed location, and as the input camera pans / tilts to the required position, the non-visible (available) parts of the output image are gradually reduced, i.e., the display jumps immediately to the required pan / tilt position, without distortion as the camera pans / tilts, and the missing parts of the display are filled in as they become available. This is an example where the black border is intentionally provided to maintain a stable field of view during the camera transition.

[0062] It should be understood that any sub-region of the display field of view selected to be magnified need not have the same aspect ratio of the selected image. In this case, the camera application can decide whether to crop or distort the selected sub-region when reprojected for display. Similarly, in many cases, the reprojected image will be the same size and resolution as the image from which the sub-region was selected, but this is not necessarily the case. [Explanation of symbols]

[0063] 12 Lenses 14. Memory 16 processors 18 Actuator 20 Digital Zoom 30 Input View Frustum 50, 60, 70 Output Camera Frustum

Claims

1. 1. A method of controlling a camera, comprising: acquiring a first image having a first field of view using a camera (10) having a lens with a first focal length, the camera having an associated input camera projection model (30) representing calibrated physical lens characteristics for acquiring the image, including focal length and optical axis parameter values; displaying the first image at a current optical zoom in the first field of view; receiving from a user a selection of at least a portion of the first field of view to be displayed as a second image associated with an output camera projection model (40, 50, 60, 70) having a common viewpoint with the input camera projection model and having at least one of optical axis parameter values ​​including focal length and one or both of pan and tilt different from corresponding parameters of the input camera projection model, and immediately displaying the second image with the digital zoom corresponding to the portion of the first field of view; Iteratively, (a) adjusting at least one of the focal length and optical axis parameter values ​​of the camera lens to approximate corresponding focal length and optical axis parameters of the output camera projection model; (b) acquiring a subsequent image with the camera using the adjusted focal length or optical axis parameter value; (c) mapping pixel coordinates in the second image to respective positions in the subsequent image using the adjusted focal length or optical axis parameter values ​​of the input camera projection model via a normalized three-dimensional coordinate system using the output camera projection model to determine respective values ​​of pixel coordinates, thereby providing a progressive improvement in sharpness of the image as the digital zoom is replaced by the optical zoom; (d) re-displaying the second image at the determined pixel coordinate values; Including, The steps (a) to (d) include until the focal length and optical axis parameter values ​​of the camera lens match the corresponding values ​​of the output camera projection model, or until such time that all of the selected portion of the first field of view can remain visible in the display image. repeated in either method.

2. The mapping step further comprises: obtaining an indication of movement of the camera in time between acquiring the subsequent image and acquiring a previous image; rotating the mapped coordinates in the normalized 3D coordinate system to account for the translation of the camera; The method of claim 1 , comprising:

3. The method of claim 1 , wherein the input camera projection model comprises one of a pinhole, a straight line, or a fisheye projection model, and the output camera projection model comprises one of a pinhole, a straight line, or a fisheye projection model.

4. The method of claim 1 , wherein the output camera projection model comprises a rectilinear projection model and the input camera projection model comprises a different projection model.

5. The method of claim 1 , wherein the selecting step comprises selecting a sub-region of the displayed first image, and the adjusting step comprises adjusting the focal length of the lens.

6. 2. The method of claim 1 , wherein the selecting step comprises selecting an out-of-center region of the displayed first image, and the adjusting step comprises adjusting a pan or tilt angle of the lens such that the input camera projection model optical axis parameter value approaches the output camera projection model optical axis parameter value.

7. The method of claim 1 , wherein the selecting step comprises selecting an out-of-center sub-region of the displayed first image, and the adjusting step comprises adjusting the focal length of the lens.

8. 2. The method of claim 1 , wherein the selecting step comprises selecting an out-of-center sub-region of the displayed first image, and the adjusting step comprises adjusting the focal length of the lens and adjusting a pan or tilt angle of the lens such that the input camera projection model optical axis parameter value approaches the output camera projection model optical axis parameter value.

9. 13. A handheld camera comprising: a lens; a lens actuator configured to change a focal length of the lens; and a processor configured to execute a camera application to control the camera to perform the steps recited in claim 1.

10. The steps (b) and (c) in camera firmware prior to writing the pixel coordinate values ​​of the second image to memory; or after writing the pixel coordinate values ​​of the subsequent image to memory, in camera firmware or by the camera application, 10. The portable camera according to claim 9, wherein the method is carried out in any one of the following manners.

11. A computer program product comprising computer readable instructions stored on a computer readable medium and configured to perform the steps of claim 1 when executed on a handheld camera.

12. 13. A system comprising: a fixed-mount camera comprising a lens and a lens actuator configured to change the focal length and pan and tilt angles of the lens; and a computer program product comprising computer readable instructions stored on a computer readable medium and configured to perform the steps recited in claim 1 when executed on a computing device operatively connected to the camera.

13. The system of claim 12 , wherein steps (b) and (c) are performed in camera firmware prior to writing the pixel coordinate values ​​of the second image to memory.

14. 13. The system of claim 12, wherein the camera application is non-interactive, and the computer readable instructions are configured to display the first image, allow a user of the computing device to select the portion of the first field of view to be displayed as the second image, and cause the computing device to repeatedly display the second image.

Images