System and method for privacy protected mobile camera

Abstract

A mobile device includes a privacy camera configured and operative to capture privacy-protected image data of a scene, where the privacy camera is a pop-up camera or a regular camera covered with an optical element or a coded aperture mask, where the mobile device is configured to communicate the captured privacy-protected image data to a communication network, and where the communication of the privacy-protected image data to the communication network does not infringe on the privacy of a person appearing in the scene. In some examples, the communication network is the Internet.

Description

Cross-reference to related applications

[0001] This application claims the benefits of U.S. Provisional Patent Application No. 63 / 495139, filed April 10, 2023, and U.S. Provisional Patent Application No. 63 / 582247, filed September 13, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The examples disclosed herein generally relate to privacy-preserving photography using mobile devices, and more specifically to systems and methods for generating and sharing privacy-preserving image data using mobile devices. Background Technology

[0003] Mobile handheld electronic devices (hereinafter “mobile devices”), such as smartphones, tablets, headphones, and smartwatches, are known to be configured to capture images (or video streams of images) and share these images via communication networks (such as, but not limited to, the Internet). In particular, video streaming of images between such mobile devices is becoming increasingly popular.

[0004] Figure 1A A known method 100 for generating and transmitting images or video streams of images by means of a mobile device is shown. For simplicity, in the following text we refer only to (single) images, but unless otherwise explicitly stated, it should also include (or refer to) video streams of images.

[0005] In step 102, the user or a program activates the camera included in the mobile device. Typically, the program for activating the camera is instructed to do so by the user. In other scenarios, and sometimes undesirable ones, the program for activating the camera is instructed by a third party or third party, without (or even against) the user's intent.

[0006] In step 104, the camera captures the scene at high resolution, i.e., the camera generates high-resolution image data. "High resolution" here means allowing for more detailed capture of the scene's spatial (or pixel) resolution. Specifically, "high resolution" here means a spatial resolution high enough to analyze details of the captured scene that might infringe on a person's privacy. High resolution may correspond to a pixel resolution of approximately 0.5 megapixels ("MP") or greater. In some examples, the person may be the owner of a mobile device (hereinafter referred to as the "owner") intentionally capturing an image of themselves or someone else. In other examples, the person may not be the owner of the mobile device, but rather someone unintentionally captured by the mobile device, for example, by simultaneously entering the background captured by the camera. Such a person will subsequently be referred to as a "bystander."

[0007] In optional step 106, high-resolution image data can be processed.

[0008] In step 108, the high-resolution image data may be transmitted to a communication network such as (but not limited to) the Internet (e.g., to a cloud server). For clarity, this document will refer to any technology that (1) allows information to be shared between electronic devices such as mobile devices and (2) does not require physical proximity of the mobile devices involved as performing “transmission to the Internet”. In particular, transmission does not necessarily have to be performed via a specific protocol or standard commonly used in the “Internet”. For example, sharing an image via a messenger app or social media app is referred to as transmission to the Internet, while sharing an image via Bluetooth is not referred to as transmission to the Internet. Typically, high-resolution image data is shared with a third party or a group of third parties, hereinafter referred to as “high-resolution image data sharing”. Transmission of high-resolution image data to the Internet may be performed intentionally by the owner, or may be performed unintentionally, for example by mistake, or may be performed against the owner’s intent (or will), for example in the case of Internet fraud or hacking. Therefore and unintentionally, a person’s privacy may be violated. Privacy violations may be caused, for example, by the fact that a third party learns about the appearance or actions of the owner or bystander by analyzing the transmitted high-resolution image data.

[0009] Figure 1B illustrates a known mobile device 150 having a rear surface 152 that generally points toward the scene. The mobile device 150 may also include a front surface (not shown) oriented relative to the rear surface 152, which generally points toward the user and includes a screen (not shown). The rear surface 152 includes a known pop-up (PO) camera 154 having a pop-up camera field of view (FOV). Pop-up cameras are disclosed, for example, in international patent applications PCT / IB2020 / 058697 and PCT / IB2022 / 050575, the entire contents of which are incorporated herein by reference. The PO camera 154 is in a “PO state” or “active state”, in which the PO camera 154 is operated to image (or capture) the scene at high resolution. We note that in the PO state, the PO camera 154 protrudes (or “pops out”) from the rear surface 152, as indicated by arrow 156. The mobile device 150 also includes another camera 158, which may be a wide-angle camera or an ultra-wide-angle (UW) camera with a field of view (FOV) greater than that of a PO camera, or a telephoto camera with a telephoto camera with a field of view (FOV less than that of a PO camera).

[0010] Figure 1C exemplarily illustrates the function of a PO-type camera 154 in PO state. Dots (or "dots") 160 indicate a diameter ("D") as seen in the scene. REAL A circular object ("). Point 162 represents a high-resolution image of point 160 captured by PO-type camera 154 in PO state. Point 162 has a diameter ("D") P-OIt can be seen that the characteristics of (real) point 160, such as size and shape, are reproduced in great detail by (imaging) point 162, especially D. REAL =D P-O .

[0011] Figure 1D shows the known moving device 150 of Figure 1B, in which the PO camera 154 is in a "retracted state" or "inactive state," in which the PO camera 154 is not operable to image the scene at high image resolution. We note that, as shown at position 164, in the retracted state, the PO camera 154 does not protrude from the rear surface 152 (or protrudes significantly less).

[0012] Figure 1E exemplarily illustrates the function of the PO camera 154 in its retracted state. Point 166 represents a high-resolution image of point 160 captured by the PO camera 154 in its retracted state. Point 166 has a diameter (“D”). C “C” indicates “contraction”. It can be seen that features such as size and shape of the (real) point 160 are only partially reproduced by the (imaging) point 166. Specifically, D… REAL <D C That is, point 166 is blurry.

[0013] What is needed and beneficial is a mobile camera that prevents invasion of the privacy of people (whether owners or bystanders). Summary of the Invention

[0014] In various examples, a mobile device including a privacy camera is provided, the privacy camera being configured and operated to capture privacy-preserving image data of a scene, wherein the privacy camera is a pop-up camera, wherein the mobile device is configured to transmit the captured privacy-preserving image data to a communication network, and wherein transmitting the privacy-preserving image data to the communication network does not infringe on the privacy of people present in the scene.

[0015] In some examples, the communication network is the Internet.

[0016] In some examples, privacy cameras are configured and operated to capture privacy-preserving image data of a scene using a pop-up camera in a retracted state.

[0017] In some examples, the mobile device has a rear surface and a front surface positioned relative to the rear surface, wherein the front surface includes a screen, and a privacy camera is included in the rear surface and has a privacy-protected camera state, in which the pop-up camera protrudes from the rear surface to a less extent than it does in the non-privacy-protected camera state.

[0018] In some examples, the privacy camera includes a privacy actuator that is operated to switch between a privacy-protected camera state and one or more additional non-privacy-protected camera states.

[0019] In some examples, the pop-up camera includes optical components, one or more additional camera states including a retracted state and a pop-up state, and a privacy actuator is configured to move the optical components to switch the privacy camera between the privacy-preserving camera state, the retracted state, and the pop-up state.

[0020] In some examples, the first movement of the optics of the privacy camera switching between the retracted state and the privacy-preserving camera state is shorter than the second movement of the optics of the privacy camera switching between the retracted state and the pop-up state.

[0021] In some examples, the mobile device has a rear surface and a front surface positioned relative to the rear surface, the front surface including a screen, a privacy camera being included in the rear surface, and in the privacy-protected camera state, the pop-up camera protrudes from the rear surface to a lesser extent than in the pop-up state.

[0022] In some examples, the privacy camera is a multi-state zoom camera with a zoom lens having two or more zoom states, each zoom state corresponding to a corresponding zoom state lens position, and wherein the zoom lens operates as a privacy camera when it is not operating at the corresponding zoom state lens position.

[0023] In some examples, the mobile device is configured to extract context-specific information from privacy-preserving image data. In some examples, the context-specific information is a person's actions, gestures, or postures. In some examples, the mobile device is further configured to create an avatar, and the avatar is displayed performing a person's actions, gestures, or postures.

[0024] In some examples, the privacy camera includes an adaptive aperture (AA) to define the level of privacy. In some examples, the pop-up camera has an f-number f / #, where f / # depends on the AA, and has a minimum value of f / #. Min The range is from f / 1.0 to f / 5. In some examples, the pop-up camera has an f-number f / #, which depends on AA, and has a maximum value of f / #. Max Within the range of f / 1.2 to f / 50. In some examples, the pop-up camera is a wide-angle camera with a minimum f / # in the range of f / 1.5 to f / 3. Min And the maximum value f / # in the range of f / 2 to f / 16. Max In some examples, the pop-up camera is a telephoto camera with a minimum f / # in the range of f / 1.8 to f / 5. Min And the maximum value f / # in the range of f / 2.5 to f / 16.Max .

[0025] In some examples, the mobile device includes at least one additional camera, and image data captured by the at least one additional camera is used to define the AA.

[0026] In various examples, a mobile device is provided, comprising: a privacy camera configured and operated to capture privacy-preserving image data of a scene, wherein the mobile device is configured to transmit the captured privacy-preserving image data to a communication network, wherein the privacy camera is a conventional camera covered with optical elements or an coded aperture mask, and wherein transmitting the privacy-preserving image data to the communication network does not infringe on the privacy of people present in the scene.

[0027] In some examples, the privacy camera includes a privacy actuator that is operated to switch between a privacy-protected state and a non-privacy-protected camera state.

[0028] In some examples, the mobile device includes a manual physical shutter that is operated to switch between a privacy camera state and a non-privacy camera state.

[0029] In some examples, the mobile device is configured to extract scene-specific information from privacy-preserving image data. In some examples, the scene-specific information is a person's actions, gestures, or postures.

[0030] In some examples, the mobile device is further configured to create an avatar, and the avatar is displayed as performing human actions, gestures, or postures.

[0031] In some examples, the privacy camera includes an adaptive aperture (AA) for defining the level of privacy. In some examples, the pop-up camera has a variable f-number (f / #), which depends on the AA, and a minimum value (f / #). Min Within the range of f / 1.0 to f / 5 and / or the maximum value f / # Max Within the range of f / 1.2 to f / 50.

[0032] In some examples, AA is controlled by the user. In other examples, AA is controlled by an algorithm.

[0033] In some examples, the privacy camera has an effective focal length ranging from 2.5mm to 50mm. In other examples, the privacy camera has an effective focal length ranging from 2.5mm to 15mm.

[0034] In some examples, the mobile device is a smartphone. In some examples, the mobile device is a tablet. In some examples, the mobile device is a laptop. Attached Figure Description

[0035] The following description, with reference to the accompanying drawings listed after this paragraph, illustrates non-limiting examples of the embodiments disclosed herein. The drawings and descriptions are intended to illustrate and clarify the embodiments disclosed herein and should not be construed as limiting in any way. Identical elements in different drawings may be indicated by the same numerals. Elements in the drawings are not necessarily drawn to scale. In the drawings: Figure 1A Methods known in the art for generating and transmitting image data are illustrated; Figure 1B illustrates a mobile device known in the art that includes a PO-type camera in a PO state; Figure 1C shows a real object and an object captured by the moving device shown in Figure 1B; Figure 1D shows the moving device of Figure 1B, including the PO-type camera in a retracted state; Figure 1E shows a real object and an object captured by the moving device shown in Figure 1D; Figure 2A Examples of mobile devices including privacy cameras as disclosed herein are shown; Figure 2B Show real objects and those made of Figure 2A The objects shown are captured by a mobile device, including a privacy camera. Figure 2C Another example of a mobile device including a privacy camera as disclosed herein is shown; Figure 2D Show real objects and those made of Figure 2C The object shown is captured by another mobile device, including a privacy camera. Figure 3A Examples of methods for generating and transmitting privacy-preserving image data, as disclosed herein, are shown. Figure 3B This document illustrates another example of a method for generating and transmitting privacy-preserving image data as disclosed herein; Figure 4 An example of a mobile device disclosed herein, configured to generate and transmit privacy-preserving image data, is illustrated schematically. Figure 5A A camera is shown that can operate as a privacy camera as disclosed herein in the first camera state; Figure 5B Showing the second camera state Figure 5A The camera; Figure 5C This illustrates another camera that can operate as a privacy camera as disclosed herein in the first camera state; Figure 5D Showing the second camera state Figure 5C The camera. Detailed Implementation

[0036] Figure 2A An example of a mobile device designated 200 is shown, which includes a privacy-preserving camera (or simply "privacy camera") designated 202 disclosed herein. To protect privacy, the aperture of the privacy camera 202 may be covered by a suitable optical element 204. "Suitable" here means that the optical element 204 is configured to interfere with (or "encrypt") the image information before it reaches the privacy camera 202. Encryption prevents the capture of high-resolution image data that may infringe on human privacy. The optical element 204 may be, for example, an optical diffuser known in the art, a grating such as a phase grating known in the art, etc. Due to the optical element 204, the privacy camera 202 is operated to capture privacy-preserving image data (or simply "privacy-preserving data"), i.e., image data that does not infringe on human privacy. The privacy camera 202 is included in the rear surface 206 of the mobile device 200. The mobile device 200 may also include a second camera 208. The privacy camera configured to perform the methods disclosed herein may have an effective focal length ("EFL") in the range of 2.5 mm to 50 mm. Preferably, the EFL is in the range of 2.5 mm to 15 mm.

[0037] In some examples, optical elements such as optical element 204 can be used in laptop computers (or “notebook computers”). We note that laptop computers typically already include a manual physical shutter for closing the aperture of the camera included in the laptop computer. Adding an optical element such as optical element 204 to such a manual physical shutter could be relatively simple, allowing the user to choose to close the aperture of the camera included in the laptop computer, or to cover the aperture of the camera included in the laptop computer with an optical element that is operated to turn the camera into a privacy camera. Similarly, in mobile devices, optical elements can be implemented as manual physical shutters.

[0038] Figure 2B The functionality of privacy camera 202 is illustrated exemplarily. Point 210 represents an image of point 160 captured by privacy camera 202. Point 210 has a diameter (“D”). P "P" stands for "privacy". It is evident that characteristics such as size and shape of the (real) point 160 are only partially reproduced by the (imaging) point 210. Specifically, D... REAL <D P That is, point 210 is blurry.

[0039] Figure 2CAnother example of a mobile device numbered 250 is shown, which includes another privacy camera numbered 252 disclosed herein. To protect privacy, the aperture of the privacy camera 252 can be covered with a "coded aperture" mask 254 as known in the art. A coded aperture mask is a grid, grating, etc. of materials with different opacities. Generally, some parts of the coded aperture block light (i.e., are opaque), while other parts do not block light (i.e., are transparent). In other examples, light can be only partially blocked. In still other examples, light can be not blocked, but rather manipulated in other ways, such as by a phase grating, as known in the art. The coded aperture mask 254 is configured to interfere with image information before the image information reaches the privacy camera 202, such that the coded aperture mask 254 prevents the capture of high-resolution image data that may violate a person's privacy. Due to the coded aperture mask 254, the privacy camera 252 is operated to capture privacy-protected image data. The privacy camera 252 is included in the rear surface 256 of the mobile device 250. The mobile device 250 may also include a second camera 258.

[0040] Figure 2D Exemplarily shows the function of the privacy camera 252. The point 260 represents the image of the point 160 captured by the privacy camera 202. The point 260 has a diameter (“D P ”). It can be seen that the characteristics such as the size, shape, etc. of the (real) point 160 are only partially reproduced by the point 260. In particular, D REAL <DP, that is, the point 260 is blurred. In addition, the coded aperture mask 254 applies a coded pattern known in the art.

[0041] In other examples, privacy according to the method disclosed herein can be provided by using a P-O type camera in a retracted state, as shown in FIG. 1D for example. The retracted state "blurs" some parts of the image in the same or similar manner as the above-described optical elements. As for the prominent differences of the P-O type camera in the popped state, retracted state, or intermediate state, the P-O type camera provides a visual indication of the "privacy mode" via protrusion. Any image captured with the P-O type camera in this state will provide privacy.

[0042] In some examples, the optical elements of the privacy camera can be designed such that they meet the following two conditions: 1. Sufficiently interfere with image information, for example, by blurring, before the image information reaches the image sensor. "Sufficiently" here means that the image information is interfered such that it is impossible to capture high-resolution image data (which may violate privacy).

[0043] 2. Retain sufficient image information to be able to interpret a specific situation or action occurring in the scene.

[0044] In other words, the optics of a privacy camera can be designed to preserve human privacy while still allowing the extraction of specific scene information that is considered (or classified) as not infringing on human privacy. In the following text, we will refer to these two conditions as the "privacy camera conditions." Specific scene information can include actions and gestures performed by a person. The relevant optics can be, for example, lenses included in PO-type camera 154, or optics such as optics 204 included in privacy camera 202, or coded aperture masks such as coded aperture masks 254 included in privacy camera 252.

[0045] In some examples, the optics of a privacy camera can be co-designed with an algorithm (or program) configured to analyze privacy-preserving data. This is beneficial for achieving a privacy camera that, on the one hand, does not infringe on human privacy, but on the other hand, still allows for the extraction of scene-specific information from the privacy-preserving data captured by the privacy camera.

[0046] For example, privacy-preserving data captured by a privacy camera may not be sufficiently detailed for tasks such as human identification or for extracting detailed information about a person's body. The definition of "privacy-preserving data" depends on the specific use case (or scenario) and the relevant contextual information, and therefore may vary from situation to situation.

[0047] Figure 3A An example of method number 300 for generating and transmitting privacy-preserving image data is shown. This includes a mobile device 400, such as a privacy camera. Figure 4 The mobile device can be configured to execute method 300.

[0048] In step 302, the user or program activates the privacy camera included in the mobile device.

[0049] In step 304, the camera captures privacy-preserving image data of the scene.

[0050] In optional step 306, the privacy-preserving image data can be processed. As an example, the privacy-preserving image data can be analyzed to extract information about gestures or actions performed in the scene.

[0051] In step 308, the privacy-preserving image data is transmitted to the Internet (e.g., to a cloud server), where it is typically shared with a third party or a group of third parties. Note that transmitting the privacy-preserving image data to the Internet does not transmit potentially privacy-infringing data. Therefore, and expectably, human privacy is protected.

[0052] Figure 3B Another example of a method numbered 350 for generating and transmitting privacy-preserving image data is shown. This includes privacy cameras with adaptive aperture (“AA”) (such as Camera 500). Figure 5A , Figure 5B The moving device can be configured to perform method 350. AA changes the camera's f-number (f / #) by opening or closing the camera's aperture ("DA"). By changing the camera's f / #, the camera's depth of focus ("DoFo") changes according to DoFo = c * f / #, where c is a "circle of confusion" known in the art. c does not change when the camera's f / # is changed, making DoFo dependent on f / # (DoFo ​​~ f / #). Regarding a defocused (or "out-of-focus") camera, we note that by increasing the camera's DoFo, the camera can be brought into focus. This does not involve any further movement for focusing, such as movement through the camera's lens or through the camera's image sensor. In the following example, we refer to a PO-type camera 154 in a retracted state, as shown in Figure 1D. In the retracted state, the PO-type camera 154 is inoperable for imaging the scene at high image resolution, as shown in Figure 1E: the imaged point 166 is significantly blurred compared to the (real) point 160. However, the amount of blur can be adjusted by changing the DoFo. In other words, when using AA, the amount of blur can be defined via the f / # of the PO-type camera 154. Specifically, this means that AA can be used to meet specific privacy camera conditions. For example, the level of privacy can be increased by turning on AA (lowering f / #), and vice versa. AA can be operated to, for example, at f / # Min =1 to f / # Min The minimum value f / # (or simply "f / #") within the range of 5. Min =f / 1 to f / 5”) and approximately f / # Max =The f / # of the PO-type camera 154 is changed discretely or continuously between the maximum value f / # in the range of f / 1.2 to f / 50. In the wide-angle camera example, f / # Min It can be in the range of f / 1.5 to f / 3, and f / # Max It can be in the range of f / 2 to f / 16. In the telephoto camera example, f / # Min It can be in the range of f / 1.8 to f / 5, and f / # Max It can be in the range of f / 2.5 to f / 16.

[0053] As shown in the figure, method 350 includes all steps 302, 304, 306, and 308 of method 300. Furthermore, method 350 includes two additional steps 352 and 354. In these two additional steps, specific privacy camera conditions for the camera are implemented.

[0054] In step 352, the desired settings for AA and (optionally) the image sensor are determined. To do this, image data of the scene is captured and analyzed. In the case of using a privacy camera such as a PO-type camera 154, the analysis can, for example, assess the degree of blur in the image data. In step 352, the privacy camera may be used with a relatively large f / # corresponding to a relatively large DoFo. In the case of using a conventional camera (i.e., a camera that is not a privacy camera), the analysis can, for example, estimate (or “predict”) the degree of blur in the image data captured by the privacy camera. Based on the analysis of the image data, the desired AA settings are determined.

[0055] Note that by changing f / #, the amount of light reaching the image sensor is related to f / #. 2 Change proportionally. When the camera's f / # is changed from its minimum value "f / # Min "Add to the new greater than f / #" Min "f / #" New "At that time, the amount of light entering the camera and the factor (f / #)" New / f / # Min ) 2 The exposure time is reduced proportionally. Therefore, in order to capture a comparable amount of light in both scenarios, the exposure time of the image sensor can be modified accordingly. For example, when the camera is at the position corresponding to f / # Min Under AA conditions, the exposure time T Exp It can be marked as "T" Exp-Min "In order to position the camera at the corresponding f / # " New A considerable amount of light can be collected in the AA state, which can be achieved through T Exp-New =T Exp-Min (f / # New / f / # Min ) 2 To calculate the new exposure time "T" Exp-New Note that when capturing a video stream of images (“video stream”), the maximum exposure time is limited by the frame rate used to capture the video stream. For example, when using a frame rate of 30 frames per second (“fps”), T must be satisfied. Exp-New <33ms. When using a frame rate of 60 frames per second (“fps”), T must be satisfied. Exp-New<16ms, etc. In some examples, image sensors known in the art that are operated to perform "pixel binning" can be used. Pixel binning is a technique in which multiple adjacent or "neighboring" (smaller) pixels on an image sensor are combined (or "merged," "grouped," or "fused") to work together as a single (larger) pixel. By using pixel binning, an image sensor can switch between at least two different pixel (or "spatial") resolutions: a first higher pixel resolution, in which small pixels are captured and read out individually; and a second lower pixel resolution, in which multiple small pixels are combined into a larger pixel that is captured and read out. It should be noted that in the first pixel resolution, the area size (or "area") of a single pixel is smaller than the area size in the second pixel resolution. The reduced area results in a reduction in the amount of light entering (or being "captured" by) a single pixel, the amount of light being given, for example, by the number of photons entering the pixel per unit time. In some examples, and in order to compensate for (or "mitigate") the effect of the reduced amount of light entering the camera when the camera's f / # is increased, the image sensor can switch from the first higher pixel resolution to the second lower pixel resolution.

[0056] In step 354, the desired AA and (optionally) image sensor settings are applied. For example, a specific f / # for the privacy camera is achieved by changing the AA. After applying the desired AA settings, specific privacy camera conditions are met. In other examples, the user can manually control the AA to define the desired level of obscurity corresponding to the desired level of privacy.

[0057] Figure 4A mobile device, numbered 400, is shown configured to perform a method for generating and transmitting privacy-preserving image data. The mobile device 400 may be configured to perform method 300 or method 350 disclosed herein. The mobile device 400 includes a privacy camera 410 configured to capture privacy-preserving image data. The privacy camera 410 includes an image sensor 412, a lens 414 having a lens optical axis, and (optionally) a privacy actuator 416. In some examples, the privacy camera 410 may be a PO-type camera in a retracted state (Figures 1D-E). Note that a PO-type camera can be a privacy camera 410 in a retracted state, and it can be a regular camera (operated to capture high-resolution image data) in a PO state. The privacy actuator 416 may be configured to switch the PO-type camera from a retracted (privacy-preserving) state to a PO (regular) state. In some examples, the privacy actuator 416 may operate to cause the privacy camera 410 to enter a specific privacy state, such as an intermediate state. In physical terms, the intermediate state may be located between the retracted state and the PO state. "Middle" here means that the first movement of the privacy camera, actuated by the privacy actuator 416, to switch between the retracted and privacy-protected camera states is shorter than the second movement to switch between the retracted and pop-up camera states. Therefore, in the middle state, the pop-up camera protrudes less from the surface of the mobile device than in the second camera state.

[0058] In some PO-type cameras, there may be a large (or even an infinite) number of such intermediate states. In other examples and such... Figures 5A to 5D As shown, privacy actuator 416 may include AA to change the f / # of privacy camera 410 such that a specific privacy camera condition is met. In other examples, privacy camera 410 may be a privacy camera, such as privacy camera 202. Figures 2A to 2B ) or privacy camera 252 ( Figures 2C to 2D In other examples, the privacy camera 410 may be represented by a two-state (or more generally, multi-state) zoom camera, for example, as disclosed in PCT / IB2020 / 051405. A multi-state zoom camera has a zoom lens with two or more zoom states and is configured to capture privacy-preserving image data by operating the zoom lens at zoom lens positions between two or more zoom states.

[0059] Mobile device 400 also includes an application processor (AP) 420. AP 420 includes: a scene estimator 422, configured to perform basic scene understanding such as human or animal detection; an aperture controller 424, for example, to derive AA settings at step 352; a sensor controller 426, for example, to derive sensor settings at step 352; a motion recognizer 428, configured to recognize actions, postures, or gestures of people appearing in a privacy-preserving image; and (optionally) a scene designer 430. Scene designer 430 can operate to process any privacy-preserving image data to design (or generate) images that include necessary but privacy-infringing information about the scene. For example, and based on results or outputs from scene estimator 422 and motion recognizer 426, scene designer 430 can design images that do not show the actual person performing a particular action but instead show an avatar chosen by the actual person performing the particular action. The avatar may or may not resemble an actual person. The mobile device 400 further includes a screen 440 (e.g., for displaying high-resolution and / or privacy-preserving image data) and a memory 450 (e.g., for storing an avatar of an actual person).

[0060] We note that the methods disclosed herein can also be performed by other electronic devices, including cameras. These other electronic devices can be, for example, laptop computers or monitors with cameras connected to computers. In other examples, electronic devices connected to external cameras can be configured to perform the methods disclosed herein.

[0061] Figure 5A A camera, designated 500, is schematically shown, including an adaptive aperture (“AA”) mechanism for performing method 350 as disclosed herein. Camera 500 includes a lens 502 comprising an exemplary plurality of lens elements (five in total) labeled L1 to L5 and having an EFL, an image sensor 504 having a full image sensor diagonal (“SD”), and an AA 510. As can be seen, the AA 510 is located between two lens elements of lens 502. The AA 510 changes the f / # of camera 500 to define a level of privacy by varying the aperture diameter (“DA”). The AA 510 can operate as a privacy actuator, such as privacy actuator 416. Camera 500 is shown in a first camera state where the AA 510 is maximally closed, achieving a minimum aperture diameter DA. Min In the first camera mode, f / # is f / # Max , which is made by f / # Max =EFL / DA Min Given. In the first camera state, f / # can be satisfied. Max=f / 1.2 to f / 50. Image data captured in the first camera state provides a low (or small) level of blur corresponding to a low level of privacy.

[0062] Figure 5B This schematically illustrates the second camera configuration. Figure 5A The camera 500. AA 510 is opened to the maximum extent, enabling a result greater than DA. Min Maximum aperture DA Max In the first camera mode, f / # is f / # Min , it is made by f / # Min =EFL / DA Max Given. In the second camera configuration, f / # can be satisfied. Min =f / 1.0 to f / 5. Image data captured in the second camera state provides a high (or large) degree of blur, corresponding to a high degree of privacy. AA 510 is operated to discretely or continuously change the f / # of camera 500 between the first and second states. The second state provides a higher degree of privacy, which is desirable. On the other hand, the first state enables the execution of methods such as method 350 in more detail, which may or may not be desirable. It can switch between the first and second states to satisfy specific privacy camera conditions.

[0063] Figure 5C A camera 520, including an AA mechanism with AA 522, is schematically shown for performing the method 350 disclosed herein. As can be seen, AA 522 is located on the object side of lens 502. Camera 520 is shown positioned with f / # Max The first camera state.

[0064] Figure 5D Schematic illustration of the second camera state Figure 5C The camera. In the second camera mode, camera 520 has a smaller f / # Max f / # Min .

[0065] While this disclosure has been described with reference to certain examples and generally related methods, changes and substitutions to these examples and methods will be apparent to those skilled in the art. This disclosure should be understood not to be limited to the specific examples described herein, but only to the scope of the appended claims.

[0066] All references mentioned in this specification are incorporated herein by reference in their entirety, as if each individual reference were specifically and individually indicated to be incorporated herein by reference. Furthermore, any reference or designation of any reference in this application should not be construed as an admission that such reference is available as prior art relative to this application.

Claims

1. A mobile device, comprising: Privacy cameras are configured and operated to capture privacy-preserving image data of a scene. The privacy camera is a pop-up camera. The mobile device is configured to transmit captured privacy-preserving image data to a communication network. Furthermore, transmitting the privacy-protected image data to the communication network does not infringe upon the privacy of people present in the scene.

2. The mobile device according to claim 1, wherein, The communication network is the Internet.

3. The mobile device according to claim 1, wherein, The privacy camera is configured and operated to capture privacy-preserving image data of the scene using the pop-up camera in its retracted state.

4. The mobile device according to claim 3, wherein, The mobile device has a rear surface and a front surface positioned opposite the rear surface, wherein the front surface includes a screen, and wherein the privacy camera is included in the rear surface and has a privacy-protected camera state, wherein in the privacy-protected camera state, the pop-up camera protrudes from the rear surface to a less extent than in the non-privacy-protected camera state.

5. The mobile device according to claim 1, wherein, The privacy camera includes a privacy actuator, and the privacy actuator is operated to switch the privacy camera between a privacy-protected camera state and one or more additional non-privacy-protected camera states.

6. The mobile device according to claim 5, wherein, The pop-up camera includes an optical assembly, wherein the one or more additional camera states include a retracted state and a pop-up state, and wherein the privacy actuator is configured to move the optical assembly to switch the privacy camera between the privacy-protecting camera state, the retracted state, and the pop-up state.

7. The mobile device according to claim 6, wherein, The first movement of the optical component used to switch the privacy camera between the retracted state and the privacy protection camera state is shorter than the second movement of the optical component used to switch the privacy camera between the retracted state and the pop-up state.

8. The mobile device according to claim 7, wherein, The mobile device has a rear surface and a front surface positioned opposite the rear surface, wherein the front surface includes a screen, wherein the privacy camera is included in the rear surface, and wherein, in the privacy-protected camera state, the pop-up camera protrudes from the rear surface to a lesser extent than it protrudes from the rear surface in the pop-up state.

9. The mobile device according to claim 5, wherein, The privacy camera is a multi-state zoom camera with a zoom lens having two or more zoom states, wherein each zoom state corresponds to a corresponding zoom state lens position, and wherein the zoom lens operates as a privacy camera when it is not operating at the corresponding zoom state lens position.

10. The mobile device according to claim 1, wherein, The mobile device is configured to extract specific scene information from the privacy-preserving image data.

11. The mobile device according to claim 10, wherein, The specific scene information refers to a person's actions, gestures, or postures.

12. The mobile device according to claim 11, wherein, The mobile device is also configured to create an avatar, wherein the avatar is displayed as performing the person's actions, gestures, or postures.

13. The mobile device according to claim 1, wherein, The privacy camera includes an adaptive aperture (AA), wherein the AA is used to limit the level of privacy.

14. The mobile device according to claim 13, wherein, The pop-up camera has an f-number f / #, wherein f / # depends on the AA, and wherein the minimum value of f / # is... Min Within the range of f / 1.0 to f / 5.

15. The mobile device according to claim 13, wherein, The pop-up camera has an f-number f / #, wherein f / # depends on the AA, and wherein the maximum value of f / # is... Max Within the range of f / 1.2 to f / 50.

16. The mobile device according to claim 13, wherein, The pop-up camera is a wide-angle camera with an f-number f / #, wherein f / # depends on the AA, and wherein the minimum value f / # is... Min It can be in the range of f / 1.5 to f / 3, with a maximum value of f / #. Max It can be in the range of f / 2 to f / 16.

17. The mobile device according to claim 13, wherein, The pop-up camera is a telephoto camera with an f-number f / #, wherein f / # depends on the AA, and wherein f / # Min It can be in the range of f / 1.8 to f / 5 and f / # Max It can be in the range of f / 2.5 to f / 16.

18. The mobile device according to claim 13, wherein, The AA is controlled by the user.

19. The mobile device according to claim 13, wherein, The AA is controlled by an algorithm.

20. The mobile device according to claim 13, wherein, The mobile device includes at least one additional camera, and wherein image data captured by the at least one additional camera is used to define the AA.

21. The mobile device according to claim 1, wherein, The privacy camera has an effective focal length ranging from 2.5mm to 50mm.

22. The mobile device according to claim 21, wherein, The privacy camera has an effective focal length in the range of 2.5mm to 15mm.

23. The mobile device according to any one of claims 1-22, wherein, The mobile device is a smartphone.

24. The mobile device according to any one of claims 1-22, wherein, The mobile device is a tablet computer.

25. A mobile device, comprising: Privacy cameras are configured and operated to capture privacy-preserving image data of a scene. The mobile device is configured to transmit captured privacy-preserving image data to a communication network. The privacy camera is a conventional camera covered with optical elements or a coded aperture mask, and In this process, transmitting the privacy-protected image data to the communication network does not infringe on the privacy of people appearing in the scene.

26. The mobile device according to claim 25, wherein, The communication network is the Internet.

27. The mobile device according to claim 25, wherein, The privacy camera includes a privacy actuator, and the privacy actuator is operated to switch the privacy camera between a privacy-protected state and a non-privacy-protected camera state.

28. The mobile device according to claim 25, wherein, The mobile device includes a manual physical shutter, wherein the manual physical shutter is operated to switch the privacy camera between a privacy-protected state and a non-privacy-protected camera state.

29. The mobile device according to claim 25, wherein, The mobile device is configured to extract specific scene information from the privacy-preserving image data.

30. The mobile device according to claim 29, wherein, The specific scene information refers to a person's actions, gestures, or postures.

31. The mobile device according to claim 30, wherein, The mobile device is also configured to create an avatar, wherein the avatar is displayed as performing the person's actions, gestures, or postures.

32. The mobile device according to claim 25, wherein, The privacy camera includes an adaptive aperture (AA), wherein the AA is used to limit the level of privacy.

33. The mobile device according to claim 25, wherein, The privacy camera has a variable f-number f / #, where the variation of f / # depends on the AA, and where the minimum value of f / # is... Min Within the range of f / 1.0 to f / 5.

34. The mobile device according to claim 25, wherein, The privacy camera has a variable f-number f / #, where the change of f / # depends on the AA, and where the maximum value of f / # is... Max Within the range of f / 1.2 to f / 50.

35. The mobile device according to claim 25, wherein, The AA is controlled by the user.

36. The mobile device according to claim 25, wherein, The AA is controlled by an algorithm.

37. The mobile device according to claim 25, wherein, The privacy camera has an effective focal length ranging from 2.5mm to 50mm.

38. The mobile device according to claim 25, wherein, The privacy camera has an effective focal length in the range of 2.5mm to 15mm.

39. The mobile device according to any one of claims 25-38, wherein, The mobile device is a smartphone.

40. The mobile device according to any one of claims 25-38, wherein, The mobile device is a tablet computer.

41. The mobile device according to any one of claims 25-38, wherein, The mobile device is a laptop computer.

Images