Asymmetric camera sensor positioning for improved package detection

Abstract

To relate asymmetric camera sensor positioning for improved package detection.SOLUTION: An electronic doorbell includes an image sensor that is rotated to a portrait orientation and vertically shifted relative to a lens of a camera, resulting in asymmetric positioning of the image sensor relative to the lens. The lens projects an image circle onto the image sensor and the image sensor has a sensor detection area having upper corners within the image circle and lower corners outside of the image circle to enable capture of an object located in a lower portion of the image circle and proximate to an edge of the image circle. Then, lens distortion correction is performed on a captured image to provide a final image usable to detect the package, which may be located within the image circle but outside of a conventional sensor detection area.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] background With the advancement of electronic doorbells to capture images and / or video, many users have begun to rely on electronic doorbell data to determine whether a package has been delivered. However, many existing electronic doorbells are equipped with cameras with a limited field of view (FOV). Typically, the doorbell is oriented to allow image capture of a person's face, but image capture of a person's feet is not necessary, as a user may be more interested in seeing the person's face (for safety reasons). In many cases, if the package is delivered too close to the doorbell (e.g., underneath the doorbell and on the ground adjacent to the wall on which the doorbell is mounted), the package may be outside the camera's FOV. Summary of the Invention [Problem to be solved by the invention]

[0002] Although a user may receive delivery notifications and obtain image data of arriving and departing delivery drivers, if a user wants to check the status of a delivered package, the user may not be able to see the package in the doorbell data if the package may be outside the camera's FOV (e.g., if the package is too close to the doorbell). Additionally, package detection algorithms may be applied to the doorbell data, but if the package is outside the camera's FOV, the package may not be detected in the captured image and the user may not be notified of the package. If the package is not in the camera's FOV and a person approaches to take or steal the package, the user may not be notified that the package has been taken.

[0003] One solution to increasing the camera's FOV is to use a doorbell camera with a 180° angle of view (AOV). However, such a camera increases manufacturing costs and may require additional structural features to prevent infrared flare. Using such a camera may also result in additional image data (e.g., pixel data) that may not be important being displayed on the user device, including the area above a person's head. Because the number of pixels that an application can display on a screen (e.g., a smartphone display) when a user is viewing doorbell data is finite, the additional image data that is not important may consume a practical area of ​​the screen, resulting in a smaller display image, which may degrade the image quality and degrade the user experience.

[0004] summary This specification describes asymmetric camera sensor positioning for enhanced package detection. In one aspect, an electronic doorbell has an image sensor that is rotated vertically and shifted vertically relative to the camera lens, resulting in asymmetric positioning of the image sensor relative to the lens. The lens projects an image circle onto the image sensor, which has a sensor detection area located at the bottom of the image circle and with an upper corner within the image circle and a lower corner outside the image circle to enable capturing of objects close to the edge of the image circle. Lens distortion correction is then performed on the captured image to provide a final image that can be used to detect packages that may be located within the image circle but outside the conventional sensor detection area. [Means for solving the problem]

[0005] In some aspects, an electronic doorbell is disclosed. The electronic doorbell includes a lens optical axis. The electronic doorbell includes a lens having an image circle that represents a scene captured by the lens. The image circle has an upper portion and a lower portion separated by a middle portion and arranged vertically stacked. The upper portion is located proximate to the upper edge of the image circle, and the lower portion is located proximate to the lower edge of the image circle. The electronic doorbell also includes an image sensor having a sensor detection area. The image sensor is oriented vertically relative to the vertical stack of the image circle portions, with the vertical orientation of the sensor detection area having a longer vertical dimension than its horizontal dimension. The image sensor is also vertically offset from the optical axis of the lens by an offset distance so as to enable the image sensor to capture an image of an object located in the lower portion of the image circle near the lower edge of the image circle.

[0006] This Summary is provided to introduce simplified concepts involving asymmetric camera sensor positioning for enhanced package detection, which are further described in the Detailed Description and Figures below. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

[0007] Details of one or more aspects of asymmetric camera sensor positioning for improved package detection are described herein with reference to the following figures: The use of the same reference numbers in different examples of the disclosure and figures represents similar elements. [Brief explanation of the drawings]

[0008] [Figure 1] 10A-10C illustrate examples of asymmetric camera sensor positioning for improved package detection compared to conventional camera doorbells. [Figure 2] FIG. 2 is an isometric view of the electronic doorbell embodiment of FIG. 1. [Figure 3] 3 is a cross-sectional view of the electronic doorbell of FIG. 2 taken along line 3-3, and an enlarged view of the end of the electronic doorbell on the camera side in the cross-sectional view. [Figure 4]10 is an example of a sensor detection area relative to an image circle based on asymmetric positioning of the image sensor relative to the lens. [Figure 5] 10A-10C illustrate examples of sensor detection areas offset relative to the image circle for improved package detection. [Figure 6] FIG. 10 illustrates an example of lens distortion correction in response to asymmetric camera sensor positioning in an electronic doorbell. [Figure 7] 1A-1C illustrate an example method for correcting lens distortion in images captured by an asymmetrically positioned camera sensor in accordance with the techniques described herein. [Figure 8] FIG. 8 is a block diagram illustrating an example system including an example device that can be implemented as any electronic device (e.g., an electronic doorbell) that implements aspects of asymmetric camera sensor positioning as described with reference to FIGS. 1-7. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description overview This specification describes asymmetric camera sensor positioning for improved package detection. The technology described herein provides an electronic doorbell camera with a wider vertical FOV compared to conventional doorbell cameras. For example, the electronic doorbell camera includes an image sensor that is rotated vertically and vertically offset from the camera's lens. This asymmetric positioning allows the image sensor to capture images of objects (e.g., packages) on the ground generally below the electronic doorbell camera, which may be captured by a conventional lens but not adequately projected onto the conventional image sensor by the lens. Lens distortion in the captured image is corrected in post-processing to provide a clean final image to the user, which may include the package on the ground as well as the face of a standing person.

[0010] Not only is the vertical FOV increased, but the number of pixels available to the user is also increased, resulting in improved image quality, efficiency, and a better user experience. Lens distortion associated with objects near the corners of the image is reduced. Additionally, the overall efficiency of using a standard 160° lens is improved. The described features and concepts of asymmetric camera sensor positioning for improved package detection can be implemented in any number of different environments, and some aspects are described in the context of the following examples.

[0011] Device Examples Figure 1 shows an example of asymmetric camera sensor positioning for improved package detection compared to conventional doorbell cameras. Many conventional doorbell cameras have a wide horizontal field of view that allows for imaging of people rather than objects (e.g., packages) on the ground. The conventional doorbell camera 102 captures images with a vertical field of view (hFOV) and a short vertical field of view (vFOV). Example 100 shows a conventional doorbell camera 102 mounted on a wall 104 (represented by the vertical dashed line), with a vertical AOV 106 of approximately 104 degrees, and oriented to capture the head and face 108 of a person 110 who is approximately 6 feet (ft) 2 inches (in) (1.88 meters (m)) tall and standing approximately 2 ft (0.6 m) horizontally 112 from the conventional doorbell camera. In this example, the conventional doorbell camera 102 is unable to capture an image of a package 114 located in an area (e.g., area 116) substantially below the conventional doorbell camera 102 and its vertical AOV 106 (e.g., the volume between the edge of the vertical AOV 106, the conventional doorbell camera 102, the ground, and the wall 104). In the example, the package 114 is located at a distance 118 of approximately 6 inches (0.15 meters) from the wall 104 and is “hidden” from the conventional doorbell camera 102 .

[0012] Example 120 illustrates an electronic doorbell 122 with asymmetric camera sensor positioning, as described herein. The electronic doorbell 122 includes a vertical AOV 124 of approximately 130 degrees, providing a corresponding vFOV that enables image capture of the head and face 108 of the person 110 described above, as well as a package 114 on the ground. The use of a larger vertical AOV (e.g., vertical AOV 124) reduces the size of the area 126 and increases the corresponding vFOV for image capture, as compared to a conventional vertical AOV 106, preventing the package 114 from being “hidden.” To improve the vertical AOV 124 of the electronic doorbell 122 over the vertical AOV 106 of the conventional doorbell 102, the electronic doorbell 122 includes a camera sensor (e.g., image sensor 128) that is vertically oriented and asymmetrically aligned with the camera lens (e.g., lens 130) of the electronic doorbell 122, such that the sensor detection area 132 is offset (e.g., not optically centered) from the image circle 134. Notably, the image sensor 128 is not optically aligned coaxially with the lens 130, but is vertically offset. In this manner, the electronic doorbell 122 can achieve a larger vertical AOV (and a larger vFOV) than a conventional doorbell camera 102 by using similar components and structures, without implementing a more expensive image sensor and / or lens. The image circle 134 is a cross-section of the cone of light transmitted by the lens 130 onto the image sensor 128. The sensor detection area 132 of the image sensor 128 represents the area of ​​light sensed by the image sensor 128. Generally, the image circle 134 is projected onto the sensor detection area 132 in a manner that allows the image sensor 128 to detect at least a portion of the image circle 134. Depending on certain factors (e.g., focal length, distance between the lens 130 and the image sensor 128, placement, size of the image sensor 128), the sensor detection area 132 may capture some or all of the image circle 134 projected onto the image sensor 128. Further details of these and other features are described below.

[0013] 2 shows an isometric view 200 of an embodiment of the electronic doorbell 122 of FIG. 1. The electronic doorbell 122 has a housing 202 that has a rounded edge that intersects a longitudinal axis 204 of the housing 202. The device includes a housing 202 having an elongated shape (e.g., a generally oval shape from the front) with two elongated ends. A camera module (e.g., camera module 206 having image sensor 128 and lens 130 in FIG. 1) is disposed within housing 202 adjacent to one end (camera end 208) of housing 202. A push button 210 is disposed adjacent to the other end (button end 212) of housing 202.

[0014] Housing 202 may include a plastic material and may be formed, for example, using plastic injection molding techniques. Housing 202 may include any suitable shape, including the example shape shown in FIG. 2. For example, housing 202 may include multiple components that fit together (close together) to form an outer frame (e.g., a hollow, substantially oval outer frame) to form a cavity for housing various components of electronic doorbell 122. Housing 202 may also include an opening or transparent area aligned with camera module 206 to allow camera module 206 to see through the opening or transparent area and capture images or video of a scene.

[0015] Button 210 may include any suitable button usable to initiate a function (e.g., a mechanical button for opening and closing a switch, a capacitive sensor for detecting a user's touch). For example, activation of button 210 may initiate a function including an audible doorbell ring, sending an electronic notification to the doorbell owner's smartphone, initiating camera module 206, etc. Any suitable function may be initiated by activating button 210.

[0016] FIG. 3 shows a cross-sectional view 300 of the electronic doorbell 122 in FIG. 2 taken along line 3-3, and a close-up view 302 of the camera end 208 of the electronic doorbell in cross-sectional view 300. Within housing 202, electronic doorbell 122 includes multiple printed circuit boards (PCB(s)), including at least a main logic board 304 and a camera board 306. Additional PCB(s) may also be used. The PCB(s) may include various integrated circuit (IC) components, such as a system-on-chip (SoC) IC device, a processor, and IC components for sensors for detecting inputs such as light-emitting diode(s) (LED(s)), microphone(s), touch input, button press, voice command, or motion. Electronic doorbell 122 also includes a camera module 206 (e.g., camera), a battery 308, a button 210, and a speaker module 310. Battery 308 may be located between camera end 208 and button end 212.

[0017] The speaker module 310 may output audio toward the front and / or side of the electronic doorbell 122 (e.g., a side perpendicular to the front surface 312 of the housing 202). The speaker module 310 may allow a visitor (e.g., a user pressing the button 210) to hear an audible message, including a recorded voice message or a real-time voice transmission from the doorbell owner.

[0018] The battery 308 provides power to the electronic doorbell 122 and enables the electronic doorbell 122 to be wireless. Because the electronic doorbell 122 is battery powered, the electronic doorbell 122 can be mounted in any suitable location without hardwiring the electronic doorbell 122 to a power source. For example, the electronic doorbell 122 can be mounted in a user's home adjacent to the front door of the user's home without drilling holes in the home to connect wiring to a power source inside the home.

[0019] The PCBs (e.g., main logic board 304, camera board 306) may be formed from a glass-reinforced epoxy material such as FR4. In some cases, the PCBs may include a single layer of conductive traces and may be single-layer boards. In other cases, the PCBs may be multi-layer boards that include multiple layers of conductive traces separated by layers of dielectric material. stomach.

[0020] The electronic doorbell 122 also includes a passive infrared (PIR) sensor 314 mounted within the housing 202 proximate the camera end 208. The PIR sensor 314 is configured to detect the movement of an object (e.g., a person or an animal) within the FOV of the PIR sensor 314.

[0021] The camera module 206 includes various components, including an image sensor 128 and a lens 130. In certain aspects, the lens 130 has an optical center (e.g., a lens optical axis 316 that represents a line that passes through the geometric center of the lens 130 and joins the centers of curvature of the lens surfaces). The image sensor 128 also has an optical center (e.g., an image sensor optical axis 318), and the optical center in a conventional camera system is typically aligned with the lens optical axis 316 of the lens 130. However, in the electronic doorbell 122 described herein, the image sensor optical axis 318 is vertically offset from the lens optical axis 316. Further details are described in connection with FIG. 4.

[0022] FIG. 4 illustrates an example embodiment 400 of the sensor detection area 132 with respect to the image circle 134 based on asymmetric positioning of the image sensor 128 relative to the lens 130 (shown in FIG. 1 ). In one aspect, the image sensor 128 is shifted relative to the lens 130 to shift the sensor detection area 132 relative to the image circle 134. In addition, the image sensor 128 is rotated 90 degrees to place the sensor detection area 132 in a portrait orientation (e.g., a 3:4 portrait orientation). This portrait orientation provides the image sensor 128 with a larger vertical plane (e.g., height 402) than the horizontal plane (e.g., width 404). By both rotating and shifting the image sensor 128 vertically, the vertical AOV (and corresponding vFOV) is increased (see, e.g., the example 120 in FIG. 1 , which shows a vertical AOV 124 of approximately 130°). Thus, height 402 of image sensor 128 allows image sensor 128 to image an area of ​​image circle 134 corresponding to a vertical AOV of lens 130 of approximately 130°.

[0023] The image circle 134 may include multiple sections, including an upper section 406 and a lower section 408 arranged vertically stacked and separated by a middle section 410. The upper section is positioned proximate an upper edge 412 of the image circle. The lower section is positioned proximate a lower edge 414 of the image circle. In some examples, the middle section may be further divided into a left section 416 and a right section 418 separated by a middle section 420 and stacked horizontally. Note that the terms “upper” and “lower” are described with reference to the illustrated examples and are not intended to be limiting with respect to a particular orientation of the components relative to external factors (e.g., Earth, gravity). The described techniques may also be implemented by interchanging the terms “upper” and “lower” herein and applying the techniques to a lens arrangement with mirror inversion, in which the image of the projected scene is inverted when detected by the image sensor 128.

[0024] In some aspects, the image circle 134 may be associated with a lens 130 having an AOV range of approximately 160° (referred to as a 160° lens), which may be substantially less expensive than a 180° lens. In conventional camera systems using a 160° lens, the sensor detection area 132 (e.g., having a rectangular shape) may not capture the top and bottom (e.g., top 406 and bottom 408, respectively) of an image captured in the image circle 134 (e.g., having an elliptical shape) when all of the corners of the sensor detection area 132 are positioned within the image circle 134.

[0025] However, in the illustrated example, the upper two corners of the sensor detection area 132 (e.g. For example, the upper corner 422) is positioned within the oval (e.g., circular) shape of the image circle 134, and the lower two corners (e.g., lower corner 424) are positioned outside the boundary of the image circle 134. The image sensor optical axis 318 is horizontally aligned with the lens optical axis 316 (e.g., aligned along a horizontal axis 426) but is vertically offset (along a vertical axis 428) so as to be vertically asymmetric with the lens optical axis 316. In an example using a mirrored inverted lens arrangement, the lower corner 424 may be positioned inside the image circle 134, while the upper corner 422 may be positioned outside the image circle 134.

[0026] The image sensor optical axis 318 may be offset relative to the lens optical axis 316 by any suitable distance (e.g., offset distance 430) to enable the image sensor 128 to capture a lower portion 408 of the image circle 134 that is proximate a boundary (e.g., edge) of the image circle 134. In one example, the offset distance 430 may be substantially within a range of 0.15 millimeters (mm) to 0.35 mm. In some aspects, the upper portion 406 (e.g., the region between the upper edge 412 of the image circle 134 and the upper edge 432 of the sensor detection area 132) may not contain useful pixels because it includes the upper region of a person's head, which generally has uninteresting or unimportant image data. Therefore, locating the upper corners 422 of the sensor detection area 132 within the image circle prevents the upper two corner regions of the captured image from having black pixels. However, the lower portion 408 may include useful pixels because it may include the package 114 (shown in FIG. 1 ) positioned on the ground proximate the wall to which the electronic doorbell 122 is mounted. Therefore, the bottom corner 424 is positioned outside the image circle 134, allowing the bottom 408 to be imaged.

[0027] Because a portion of the sensor detection area 132 extends beyond the bottom edge 414 of the image circle 134 projected onto the image sensor 128, the sensor detection area 132 may include two bottom corner regions 434 that result in black pixels (e.g., vignetting) in the corresponding bottom corner regions of the captured image. The resulting black bottom corner regions of the image captured by the image sensor 128 can be corrected using distortion correction techniques. However, offsetting the image sensor 128 reduces distortion and image artifacts that may be introduced by distortion correction techniques applied to the image, particularly for objects near the corners of the image. In one aspect, the center (e.g., midpoint 436) of the bottom edge (bottom edge 438 between corners (e.g., bottom corners 424) outside the image circle 134) of the sensor detection area 132 is located within the image circle 134, rather than tangent to or outside the edge of the image circle 134. This positioning of the bottom edge 438 of the sensor detection area 132 relative to the image circle 134 reduces the number of black pixels contained near the bottom of the captured image and improves the efficiency of the lens distortion correction techniques described herein. However, in other examples, the center of the bottom edge 438 of the sensor detection area 132 may be tangent to an edge (e.g., the lower edge 414) of the image circle 134. In yet other examples, the midpoint 436 of the bottom edge 438 of the sensor detection area 132 may be located beyond the lower edge 414 of the image circle 134, which may result in additional black pixels that require some additional post-processing steps to remove.

[0028] Additionally, shifting the sensor detection area relative to the image circle 134 improves image quality by increasing (e.g., maximizing) the number of pixels that are useful to the user. For example, capturing the top portion 406 and subsequently cropping the corresponding data reduces image quality because pixels are "thrown away." Capturing a finite number of pixels, removing some (e.g., top portion 406), and enlarging the resulting image for display on a user screen may cause the remaining portions of the image to be blurred or pixelated. Thus, by shifting the sensor detection area 132, the image sensor 128 is capturing those portions of the image that have useful image data and / or are important to the user (e.g., , a larger vFOV for capturing images of a person's face as well as a package placed on the ground below the electronic doorbell 122. As a result, shifting the image sensor 128 and performing post-processing on the captured images as described herein not only increases the vFOV for capturing images, but also reduces (e.g., minimizes) the number of pixels used in the final result, thereby increasing (e.g., maximizing) image quality.

[0029] FIG. 5 shows an example embodiment 500 of a sensor detection area 134 that is offset relative to the image circle for improved package detection. In the illustrated example, the image circle 134 is shown as a substantially ellipse (e.g., a circle). However, the image circle 134 may be any suitable shape based on the shape and curvature of the lens 130. The sensor detection area 132 captures a portion of the image circle 134 projected onto the image sensor 128. In the illustrated example, the image circle 134 includes an image of a person (e.g., person 110) standing in front of the electronic doorbell 122. A package (e.g., package 114) is shown in the lower portion 408 of the image circle 134. The upper portion 406 of the image circle 134 captures the space above the head and face 108 of the person 110 in the image and does not include any data related to the person 110. After the image sensor 128 captures a portion of the image bounded within the sensor detection area 132, distortion correction may be performed on the captured image, an example of which is described with respect to FIG. 6.

[0030] FIG. 6 shows an example embodiment 600 of lens distortion correction according to asymmetric camera sensor positioning in an electronic doorbell. First, image sensor 128 (offset from FIG. 1 and relative to lens 130) captures an image (e.g., image 602-1) that includes a portion of image circle 134 and a portion outside image circle 134 near the bottom of sensor detection area 132 (e.g., the two lower corners of area 434). As shown, captured image 602-1 includes bottom 408 of image circle 134 and black corner areas 604-1 corresponding to the two lower corner areas 434 of sensor detection area 132. However, captured area 602-1 does not include top 406 of image circle 134 above upper edge 432 of sensor detection area 132. Captured image 602-1 also includes package 114-1. In some aspects, captured image 602-1 may also include dark pixels (e.g., region 606-1) or other artifacts near the edge of image circle 134 adjacent to black corner region 604-1.

[0031] Keystone correction 608 is applied to captured image 602-1 to provide a trapezoidal image 602-2 for asymmetry correction. Because captured image 602-1 contains vertical asymmetry due to asymmetric positioning of image sensor 128 relative to lens 130, keystone correction 608 is applied to remove some of the lens distortion.

[0032] Trapezoidal image 602-2 is de-distorted to provide de-distorted image 602-3, which corrects some of the lens distortion (e.g., vertical asymmetry) in original captured image 602-1. Note that person 110-2 in de-distorted image 602-3 is thinner (e.g., less distorted) than person 110-1 in original captured image 602-1. Furthermore, black corner region 604-1 has shrunk to the size of black corner region 604-2. Region 606-1 has also shrunk to the size of region 606-2. In some aspects, package 114-1 may be altered to de-distort trapezoidal image 602-2. For example, package 114-2 in de-distorted image 602-3 is smaller than package 114-1 in original captured image 602-1. However, package 114-2 is large enough to be identifiable as a package at a user's doorstep. In one aspect, keystone correction 608 involves applying a trapezoidal shape to the image, where the trapezoidal shape is narrower at the top of the image and wider at the bottom of the image, which reduces the risk of artifacts toward the top of the image during subsequent distortion correction of the image.

[0033] The edges of the dewarped image 602-3 are dewarped to provide a dewarped image 602-4. Dewarping the edges of the dewarped image 602-3 removes the dark corner region 604-2 from the bottom of the image. In some aspects, dewarping may cause some distortion in the package 114-3, but the lens distortion of the person 110-3 is significantly reduced. Furthermore, some distortion may be acceptable because completely correcting the distortion may introduce artifacts into the image, which degrade the final image and reduce the user experience. In one example, the dewarped image 602-4 may include a region 606-3 of darkened pixels that was near the edge of the image circle 134.

[0034] The distortion-corrected image 602-4 may be brightened to remove darkened pixels in region 606-3 and provide a final image 602-5. The final image 602-5 includes a distortion-corrected image including the person 110-3 (including the person's head and face) and package 114-3 without black or darkened pixels in the bottom corners of the final image 602-5 (e.g., bottom corner region 610). The final image 602-5 may then be provided to a user device (e.g., a smartphone) for display.

[0035] Example method 7 shows an example method 700 for correcting lens distortion caused by an electronic doorbell with asymmetric image sensor positioning, according to the techniques described herein. In one aspect, method 700 may be performed by one or more processors of electronic doorbell 122.

[0036] At 702, an image having at least a lens distortion is captured and received based on an image sensor positioned asymmetrically relative to the lens. In one aspect, the image sensor 128 is positioned asymmetrically relative to the lens 130, resulting in a vertical offset of the image sensor optical axis 318 (e.g., the optical center of the image sensor 128) from the lens optical axis 316 (e.g., the optical center of the lens 130). Additionally, the image sensor 128 is oriented vertically. As a result of the orientation and asymmetric positioning of the image sensor 128 relative to the lens 130, the sensor detection area 132 is vertically offset from the center of the image circle 134 to include the bottom of the image circle 134 (e.g., bottom 408) and areas beyond the boundaries of the image circle 134 (e.g., bottom corner areas 434), resulting in the captured image having black corner areas 604-1.

[0037] At 704, keystone correction is applied to the captured image to provide a trapezoidal image. For example, keystone correction 608 may be applied to the captured image 602-1 to provide a trapezoidal image 602-2 that can be used to correct vertical asymmetry resulting from asymmetric positioning of the image sensor 128 relative to the lens 130.

[0038] At 706, the trapezoidal image is de-warped to provide a de-warped image. For example, trapezoidal image 602-2 may be de-warped to correct vertical asymmetry and provide de-warped image 602-3. For example, straightening trapezoidal image 602-2 essentially stretches trapezoid 602-2 more at the top than at the bottom of the image.

[0039] At 708, the dewarped image is dewarped to provide a dewarped image. For example, the edges of dewarped image 602-3 are dewarped to provide dewarped image 602-4. Dewarping removes black areas (e.g., black corner areas 604) corresponding to the two lower corner areas 434 of the sensor sensing area 132.

[0040] At 710, the distortion-corrected image is brightened to provide a final image. For example, distortion-corrected image 602-4 is brightened to provide final image 602-5, which is lens distortion-corrected and has no black corners.

[0041] At 712, the final image 602-5 is provided to a computer-readable storage memory. For example, the final image 602-5 may be stored in local storage or remote storage (e.g., online storage). Because the electronic doorbell 122 is battery-powered, the final image 602-5 may be wirelessly communicated to remote data storage.

[0042] At 714, the final image is output to a user device (e.g., a smartphone). In one example, the final image 602-5 may be provided to the user device based on a request to access local or remote storage. If necessary, the final image may be output to the user device at 714 before or simultaneously with providing the final image to memory storage at 712. Accordingly, the techniques described herein include keystone correction of a 3:4 portrait-oriented image sensor to provide the final image 602-5 with an improved vFOV without black corners. These techniques enable improved package detection of packages or other objects located in the lower portion 408 of the image circle 134, which may typically be outside the FOV of a conventional image sensor. Accordingly, such packages may be detected based on the final image 602-5, and a notification may be provided to a user of another electronic device associated with the electronic doorbell 122 to notify the user that the package is present or, in some cases, that the package has been removed.

[0043] Example of a Computing System FIG. 8 is a block diagram illustrating an example system 800 including an example device 802 that can be implemented as any electronic device (e.g., electronic doorbell 122) implementing aspects of asymmetric camera sensor positioning, as described with reference to FIGS. 1-7 . The example device 802 can be any type of computing device, client device, mobile phone, tablet, communication, entertainment, gaming, media playback, and / or other type of device. Additionally, the example device 802 can be implemented as any other type of electronic device configured to communicate over a network, such as a thermostat, doorbell, hazard detector, camera, lighting unit, commissioning device, router, border router, joiner router, joiner device, end device, reader, access point, hub, and / or other electronic device. The example device 802 can be integrated with electronic circuits, microprocessors, memory, input / output (I / O) logic controls, communication interfaces and components, and other hardware, firmware, and / or software for communicating over a network. Additionally, the device 802 can be implemented with a variety of components, such as any number of different components and combinations thereof, as further described below.

[0044] The device 802 includes a communication device 804 that enables wired and / or wireless communication of device data 806, such as data communicated between devices in a network, data being received, data scheduled for broadcast, data packets, and data synchronized between devices. The device data can include any type of communication data, as well as audio, video, and / or image data generated by applications executing on the device. The communication device 804 can also include a transceiver for cellular communication and / or network data communication. The communication device 804 can include a wireless radio system for multiple different wireless communication systems. The wireless radio system can include Wi-Fi, Bluetooth, mobile broadband, Bluetooth Low Energy (BLE), and / or point-to-point IEEE 802.11 standard. 802.15.4 Each of the different wireless systems may include radio devices, antennas, and chipsets implemented for a particular wireless communication technology.

[0045] The device 802 also includes an input / output (I / O) interface 808, such as a data network interface, that provides a connection and / or communication link between the device, a data network (e.g., an internal network, an external network, etc.), and other devices. The I / O interface may be used to couple the device to any type of component, peripheral, and / or accessory device. The I / O interface also includes a data input port through which any type of data, media content, and / or input, such as user input to the device, may be received, as well as any type of communication data, such as audio, video, and / or image data received from any content and / or data source.

[0046] The device 802 includes a processing system 810, which may be implemented at least partially in hardware with any type of microprocessor, controller, or the like, that processes executable instructions. The processing system may include integrated circuits, such as processor and memory systems implemented as a system-on-chip (SoC), programmable logic devices, logic devices formed using one or more semiconductors, and other implementations in silicon and / or hardware. Alternatively, or in addition, the device may be implemented in any one or combination of software, hardware, firmware, or fixed logic circuits that may be implemented with processing and control circuitry. The device 802 may further include any type of system bus or other data and command transfer system that couples various components within the device. The system bus may include any one or combination of different bus configurations and structures, as well as control and data lines.

[0047] The device 802 also includes computer-readable storage memory 812, such as a data storage device that can be accessed by a computing device and provides persistent storage of data and executable instructions (e.g., software applications, modules, programs, functions, etc.). Computer-readable storage memory as described herein excludes propagating signals. Examples of computer-readable storage devices include volatile and non-volatile memory, fixed and removable media devices, and any suitable memory device or electronic data storage that holds data for computing device access. Computer-readable storage memory includes various implementations of random access memory (RAM), read-only memory (ROM), flash memory, and other types of storage memory in various memory device configurations.

[0048] The computer-readable storage memory 812 provides storage 806 for device data and various device applications 814, such as an operating system maintained by the computer-readable storage memory as software applications and executed by the processing system 810. The device applications may also include any type of control application, software application, signal processing and control module, device manager, such as code native to a particular device, a hardware abstraction layer for a particular device, etc. In this example, the device applications also include a smart home application 816 that implements aspects of asymmetric camera sensor positioning for improved package detection, such as when the example device 802 is implemented as any electronic device described herein. The device 802 also includes a power source 818, such as the battery 308. An alternating current (AC) power source may also be used to charge the device's battery.

[0049] In some aspects, at least a portion of the technology described for electronic doorbell 122 may be implemented in a distributed system, such as on a "cloud" 820 within a platform 822. Cloud 820 includes and / or represents platform 822 for services 824 and / or resources 826.

[0050] Platform 822 abstracts underlying hardware functionality, such as server devices (e.g., included in services 824) and / or software resources (e.g., included as resources 826), and communicatively connects example device 802 to other devices, servers, etc. Resources 826 may include applications and / or data that may be utilized while computer processing is performed on a server remote from example device 802. Additionally, services 824 and / or resources 826 may facilitate subscriber network services, such as over the Internet, a cellular network, or a Wi-Fi network. Platform 822 can act as a resource abstraction and scaling mechanism to service demand for resources 826 implemented via the platform, such as in interconnected device implementations with functionality distributed throughout system 800. For example, functionality may be implemented partially in example device 802 as well as via platform 822, which abstracts functionality from cloud 820.

[0051] Some examples are described below. The electronic doorbell comprises a lens having a lens optical axis, the lens providing an image circle representing a scene photographed by the lens, the image circle having an upper portion and a lower portion separated at a middle portion and arranged vertically stacked, the upper portion being arranged adjacent to an upper edge of the image circle and the lower portion being arranged adjacent to a lower edge of the image circle, the electronic doorbell further comprises an image sensor having a sensor detection area, the image sensor being arranged vertically with respect to the vertically stacked portions of the image circle, the vertical orientation of the sensor detection area having a vertical FOV that is larger than the horizontal FOV, and the image sensor being vertically offset relative to the lens toward the lower portion of the image circle by an offset distance from the lens optical axis so as to enable the image sensor to photograph an image of an object arranged at the lower portion of the image circle adjacent to the lower edge of the image circle.

[0052] The image sensor may be configured in a 3:4 portrait orientation. The sensor detection area may have a rectangular shape, the image circle may have an elliptical shape, and the sensor detection area may include two upper corners located inside the image circle and two lower corners located outside the image circle.

[0053] The image sensor may have an image sensor optical axis, and the image sensor may be vertically displaced relative to the lens such that the image sensor optical axis is offset from the lens optical axis by an offset distance along the normal axis of the lens.

[0054] The image sensor optical axis may be parallel to the lens optical axis. The offset distance may be substantially within the range of 0.15 millimeters to 0.35 millimeters.

[0055] The image circle represents the scene based on a lens with a field of view of approximately 160 degrees. Portrait orientation and vertical offset of the image sensor relative to the lens may allow the image sensor to capture a region of the image circle corresponding to the vertical field of view of the lens of approximately 130 degrees.

[0056] The electronic doorbell may further comprise a processor configured to perform lens distortion correction on the image captured by the image sensor to remove black areas in the bottom corner regions of the captured image.

[0057] The processor may be configured to perform lens distortion correction by applying keystone correction to the captured image to provide a keystone image that can be used to correct for vertical asymmetry, and undistorting the keystone image to provide a undistorted image to correct for vertical asymmetry.

[0058] The processor may be configured to perform lens distortion correction by further rectifying the de-warped image to remove black areas in the bottom corner regions and provide a rectified image.

[0059] The processor may be configured to perform lens distortion correction by further brightening the distortion-corrected image to provide a final image for display.

[0060] The image sensor may be positioned to have the midpoint of the bottom edge of the sensor detection area located within the image circle.

[0061] 1. A method for correcting lens distortion caused by an electronic doorbell having asymmetric image sensor positioning, the method comprising: capturing an image using an image sensor of the electronic doorbell, the electronic doorbell having a portrait-oriented sensor detection area and an image sensor optical axis that is vertically offset from a lens optical axis of the lens of the electronic doorbell, the sensor detection area having an upper corner positioned inside an image circle projected by the lens onto the image sensor and a lower corner positioned outside the image circle; applying keystone correction to the captured image to provide a trapezoidal image that can be used to correct vertical asymmetry resulting from the vertical offset of the image sensor optical axis; undistorting the trapezoidal image to correct the vertical asymmetry and provide a distortion-free image; distortion-correcting the distortion-free image to remove black areas corresponding to the lower corners of the sensor detection area and provide the distortion-corrected image; and brightening the distortion-corrected image to provide a final image for display on an electronic device.

[0062] The method may further comprise detecting, based on the final image, a package located at a bottom of the image circle proximate an edge of the image circle, and providing an indication to a user of the electronic device that a package is present.

[0063] conclusion Furthermore, aspects of the asymmetric camera sensor positioned for improved package detection have been described in feature and / or method-specific language, and the subject matter of the appended claims is not limited to the particular features or methods described. Rather, the specific features and methods are disclosed as examples of the claimed asymmetric camera sensor positioned for improved package detection, and other equivalent features and methods are intended to be within the scope of the appended claims. Moreover, various different aspects have been described, and it is understood that each described aspect can be implemented independently or in conjunction with one or more other described aspects.

Claims

1. An electronic doorbell, a lens having a lens optical axis, the lens providing an image circle representing a scene photographed by the lens, the image circle having an upper portion and a lower portion separated at a middle portion and arranged vertically stacked, the upper portion being positioned adjacent to an upper edge of the image circle and the lower portion being positioned adjacent to a lower edge of the image circle; The electronic doorbell further comprises an image sensor having a sensor detection area, the image sensor comprising: a vertical orientation of the sensor detection area relative to the vertically stacked portions of the image circle, the vertical orientation of the sensor detection area having a vertical dimension greater than a horizontal dimension; and an image sensor that is vertically displaced relative to the lens toward the lower portion of the image circle by an offset distance from the lens optical axis to enable the image sensor to capture an image of an object positioned at the lower portion of the image circle proximate the lower edge of the image circle;

2. The electronic doorbell of claim 1 , wherein the image sensor is configured in a 3:4 portrait orientation.

3. the sensor detection area has a rectangular shape, and the image circle has an elliptical shape; 3. The electronic doorbell of claim 1, wherein the sensor detection area includes two upper corners positioned inside the image circle and two lower corners positioned outside the image circle.

4. the image sensor has an image sensor optical axis; 10. An electronic doorbell as claimed in any one of the preceding claims, wherein the image sensor is vertically displaced relative to the lens such that the image sensor optical axis is offset from the lens optical axis by the offset distance along the normal axis of the lens.

5. The electronic doorbell of claim 4 , wherein the image sensor optical axis is parallel to the lens optical axis.

6. 10. An electronic doorbell according to any one of the preceding claims, wherein the offset distance is substantially within the range of 0.15 mm to 0.35 mm.

7. 10. An electronic doorbell according to any one of the preceding claims, wherein the image circle represents the scene based on the lens having an angle of view of approximately 160 degrees.

8. 10. The electronic doorbell of claim 1, wherein the portrait orientation and vertical offset of the image sensor relative to the lens enables the image sensor to capture an area of ​​the image circle corresponding to a vertical angle of view of the lens of approximately 130 degrees.

9. 10. The electronic doorbell of claim 1, further comprising a processor configured to perform lens distortion correction on images captured by the image sensor so as to remove black areas in lower corner regions of the captured images.

10. The processor provides a trapezoidal image that can be used to correct for vertical asymmetry. applying keystone correction to the captured image so as to 10. The electronic doorbell of claim 9, configured to perform lens distortion correction by de-distorting the trapezoidal image to correct for the asymmetry in the vertical direction and providing a de-distorted image.

11. 11. The electronic doorbell of claim 10, wherein the processor is configured to perform the lens distortion correction by further distortion correcting the de-warped image to remove the black areas in the bottom corner regions and provide a distortion-corrected image.

12. The electronic doorbell of claim 11 , wherein the processor is configured to perform the lens distortion correction by further brightening the distortion-corrected image to provide a final image for display.

13. 10. An electronic doorbell according to any one of the preceding claims, wherein the image sensor is positioned to have the midpoint of a bottom edge of the sensor detection area located within the image circle.

14. 1. A method for correcting lens distortion caused by an electronic doorbell having asymmetric image sensor positioning, the method comprising: and capturing an image using an image sensor of the electronic doorbell, the electronic doorbell comprising: a vertically oriented sensor detection area; and The electronic doorbell includes an image sensor optical axis that is vertically offset from a lens optical axis of a lens of the electronic doorbell, and the sensor detection area has an upper corner that is positioned inside an image circle projected onto the image sensor by the lens and a lower corner that is positioned outside the image circle, and the method includes: applying keystone correction to the captured image to provide a keystone image that can be used to correct vertical asymmetry resulting from vertical displacement of the image sensor optical axis; undistorting the trapezoidal image to correct the asymmetry in the vertical direction and provide a undistorted image; rectifying the rectified image to remove dark areas corresponding to lower corners of the sensor detection area to provide a rectified image; and brightening the distortion-corrected image to provide a final image for display on an electronic device.

15. The method further comprises: Detecting a package located at a lower portion of the image circle near an edge of the image circle based on the final image; and providing an indication to a user of the electronic device that the package is present.

Images