Using image sensors for always-on applications in mobile devices

Abstract

A mobile device includes an application processor and an image sensor. The application processor includes: an imaging subsystem configured to process high-resolution image data through a first interface; and a sensor hub configured to process sensor data through a second interface. The image sensor operates in one of a first mode and a second mode. During the first mode, the image sensor is configured to capture high-resolution image data in response to a request from the imaging subsystem, and the imaging subsystem is configured to use the first interface to access the high-resolution image data to perform a first operation. During the second mode, the image sensor is configured to capture low-resolution image data, and the sensor hub is configured to use a second bus to access the low-resolution image data to perform a second operation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 996,079, filed on August 18, 2020, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] Exemplary embodiments of the inventive concept relate to an image sensor of a mobile device. Background Art

[0004] Mobile devices such as smart phones include ultra-low power (ULP) sensors such as accelerometers, magnetometers, and gyroscopes that can continue to operate during low-power states. The mobile device can exit the normal power state and enter the low power state to save battery power. Various components of the mobile device can be powered off. When the mobile device determines that a certain condition has been met and switches from the low power state to the normal power state, these components can be repowered. For example, based on an analysis of sensor data received from the sensor, the condition can be met when a certain threshold is reached. However, when there is not enough sensor data, the condition may not be met. Smart phones typically include one or more cameras, each camera having one or more image sensors. However, these image sensors do not operate in ULP mode and do not provide image data in the low power state. Summary of the invention

[0005] Embodiments of the present disclosure allow the image sensor of a mobile device to be used as a ULP sensor.

[0006] According to an exemplary embodiment of the present invention, a mobile device includes an application processor and an image sensor. The application processor includes: an imaging subsystem configured to process high-resolution image data through a first interface; and a sensor hub configured to process sensor data through a second interface. The image sensor operates in one of a first mode and a second mode. During the first mode, the image sensor is configured to capture high-resolution image data in response to a request from the imaging subsystem, and the imaging subsystem is configured to use the first interface to access the high-resolution image data to perform a first operation. During the second mode, the image sensor is configured to capture low-resolution image data, and the sensor hub is configured to use the second interface to access the low-resolution image data to perform a second operation. In an exemplary embodiment, the first operation is a default operation of the image sensor. For example, before being specifically configured to perform the second operation, the image sensor will only perform the first operation.

[0007] According to an exemplary embodiment of the inventive concept, an application processor includes a sensor hub and an imaging subsystem. The imaging subsystem is configured to process a high-resolution image from an image sensor through a first interface during a first mode. The sensor hub is configured to process sensor data from at least one non-image sensor and low-resolution image data from an image sensor through a second interface. The sensor hub is configured to determine whether to exit a sleep state by analyzing the sensor data and the low-resolution image data during another second mode.

[0008] According to an exemplary embodiment of the present invention, an image sensing device includes an image sensor. The image sensor includes a pixel array and a correlated double sampler (CDS). The CDS is configured to sample an image signal provided from the pixel array at a first frequency to generate high-resolution image data in response to receiving a control signal indicating a first mode from an application processor, and to generate low-resolution image data at a second frequency lower than the first frequency when the control signal indicates a second mode different from the first mode. During the first mode, the image sensor provides high-resolution image data to a first bus connected to the application processor. During the second mode, the image sensor provides low-resolution image data to a second bus connected to the application processor.

[0009] According to an exemplary embodiment of the present invention, a method of operating an image sensor of a mobile device includes: an application processor of the mobile device transmits a control signal to the image sensor indicating whether the mobile device is in a low power state or in a normal power state; when the control signal indicates that the mobile device is in a low power state, the image sensor generates low-resolution image data; and when the image sensor receives a request from the application processor and the control signal indicates that the mobile device is in a normal power mode, the image sensor generates high-resolution image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features of the present inventive concept will become more apparent by describing in detail exemplary embodiments of the present inventive concept with reference to the accompanying drawings, in which:

[0011] Figure 1 is a block diagram illustrating a portion of a mobile device according to an exemplary embodiment of the inventive concept.

[0012] Figure 2 is a block diagram illustrating an image sensor of a mobile device according to an exemplary embodiment of the inventive concept.

[0013] Figure 3 is a block diagram of a portion of a mobile device according to an exemplary embodiment of the inventive concept.

[0014] Figure 4An operation of a mobile device when an image sensor of the mobile device is in an AON mode according to an exemplary embodiment of the inventive concept is illustrated.

[0015] Figure 5A A method for synchronizing an image sensor in an AON mode with an application processor of a mobile device according to an exemplary embodiment of the inventive concept is illustrated.

[0016] Figure 5B Included for Figure 5A An exemplary timing diagram of the method.

[0017] Fig. 6A A method for synchronizing an image sensor in an AON mode with an application processor of a mobile device according to an exemplary embodiment of the inventive concept is illustrated.

[0018] Figure 6B Included for Fig. 6A An exemplary timing diagram of the method.

[0019] Fig. 7A A method of operating a mobile device according to an exemplary embodiment of the inventive concept is illustrated.

[0020] Figure 7B A method of operating a mobile device according to an exemplary embodiment of the inventive concept is illustrated.

[0021] Figure 8 is a block diagram of a mobile device according to an exemplary embodiment of the inventive concept.

[0022] Fig. 9 An example of a mobile device is shown. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the inventive concept will be described more fully hereinafter with reference to the accompanying drawings.Throughout the drawings, like reference numerals may refer to like elements.

[0024] It should be understood that, unless the context clearly indicates otherwise, the description of features or aspects within each exemplary embodiment should generally be considered as applicable to other similar features or aspects in other exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well.

[0025] Figure 1 is a block diagram illustrating a portion of a mobile device according to an exemplary embodiment of the inventive concept. For example, the mobile device may be a smart phone or a tablet computer, but is not limited thereto.

[0026] refer to Figure 1, which includes an application processor 110, an image sensor 150, and a non-image sensor (e.g., 140-1, 140-2, ..., 140-N). In an exemplary embodiment of the inventive concept, the image sensor 150 is a complementary metal oxide semiconductor (CMOS) image sensor (IS), and may also be referred to as a CIS, where C is an abbreviation of CMOS. The image sensor 150 is not limited to being a CMOS image sensor, and may be another type of image sensor, such as a charge coupled device (CCD) image sensor, in an alternative embodiment.

[0027] The imaging subsystem 130 of the application processor 110 can control the image sensor 150. After the user triggers the camera function or activates the camera (e.g., the rear camera) of the mobile device, the imaging subsystem 130 can send a request to the image sensor 150, wherein the request causes the image sensor 150 to prepare high-resolution image data. The imaging subsystem 130 includes a camera control interface (CCI) host 132, an image data interface 133-1 (ImDatal / F), and a processor 134-1 (e.g., an image signal processing unit). The imaging subsystem 130 may additionally include a memory 136 for storing setting files, each setting file defining a different operating mode (resolution, frame rate, internal configuration, etc.). The imaging subsystem 130 may store one or more drivers (e.g., CIS drivers) for each available CIS. The processor 134-1 may drive the image sensor 150 using one of the CIS drivers. The CCI host 132 of the imaging subsystem 130 may control the image sensor 150 using the camera control interface (CCI). The CCI may use an I2C protocol, a serial peripheral interface (SPI) protocol, or an I3C protocol. The CCI host 132 may use one of the above protocols to transmit a control signal (e.g., CCldata) to the image sensor 150. The ImData I / F 133-1 of the imaging subsystem 130 may receive raw image data (e.g., high-resolution image data HiReslmData1) from the CCI host 156 of the image sensor 150. The processor 134-1 of the imaging subsystem 130 may then process the raw image data. For example, the raw image data may be in a MIPI format.

[0028] The non-image sensors communicate with the sensor hub 120 of the application processor 110 on the sensor bus 160. In an exemplary embodiment, one of the non-image sensors is an ambient light sensor. For example, the ambient light sensor may output first sensor data SData1 indicating the amount of ambient light present on the sensor bus 160. In an exemplary embodiment, one of the non-image sensors is an inertial motion unit (IMU) that measures at least one of the applied force, angular rate, or orientation. The IMU may include a combination of an accelerometer, a gyroscope, and a magnetometer. For example, the IMU may output second sensor data sData2 indicating the amount of force applied to the mobile device on the sensor bus 160. Additional non-image sensors may be present to output additional sensor data sDataN. For example, the additional non-image sensors may include a motion sensor, a temperature sensor, an infrared sensor, or a barometer.

[0029] The sensor hub 120 may be referred to as an always-on sensor hub. The sensor hub 120 may be a low-power island on the application processor 110. The sensor hub 120 may be optimized to operate in an always-on mode. The sensor hub 120 collects data from sensors and analyzes the collected data to determine whether a change of state is required on the application processor 110. Examples of state changes include waking up from a sleep state or a low-power state (i.e., waking up or entering a normal power state), partially waking up from a sleep or low-power state (i.e., partially waking up), or entering a sleep or low-power state. For example, all components of the mobile device become enabled or receive power during wake-up, and only some components become enabled or receive power during partial wake-up. For example, the sensor hub 120 may determine that the user has picked up, touched, or shaken the mobile device based on the analyzed data, and then notify the application processor 110 that the mobile device may need to exit a low-power state.

[0030] When the mobile device is in a low power state, various components of the mobile device may be disabled so that the components do not consume power. However, the sensor hub 120 and the non-image sensors are still enabled during the low power state and thus may exchange sensor data (e.g., sdataN) and control signals (e.g., Ctrl1, Ctrl2, ..., CtrlN) with each other.

[0031] The sensor hub 120 includes a sensor interface 122 (e.g., an interface circuit), a processor 124, and a memory 126. The processor 124 may run a driver to drive a corresponding one of the non-image sensors. Driving the non-image sensor may include: the sensor hub 120 sending one or more control signals (e.g., Ctrl1, Ctrl2, ..., CtrlN) to the non-image sensor on the sensor bus 160. The sensor hub 120 receives non-image sensor data (e.g., sdataN) through its sensor interface 122.

[0032] Unlike conventional image sensors, non-image sensors are executed using ultra-low power (ULP) consumption. According to an exemplary embodiment of the inventive concept, the image sensor is modified to include a ULP always-on (AON) capability (e.g., AON mode), thereby generating an image sensor 150. The image sensor 150 is capable of operating in one of the CIS mode and the AON mode. The sensor hub 120 or the application processor 110 may set a register indicating whether the image sensor 150 is operating in the CIS mode or in the AON mode. The image sensor 150 may then periodically or as needed check the register to determine which mode to operate in the CIS mode or the AON mode. For example, the value of the register may include a first value indicating the CIS mode or another second value indicating the AON mode. In an alternative embodiment, there is a first setting file for the AON mode and a second setting file for the CIS mode, and the sensor hub 120 loads the first setting file so that the image sensor 150 operates in the AON mode and loads the second setting file so that the image sensor 150 operates in the CIS mode. The setting file may be stored in a memory such as the memory 126.

[0033] In an exemplary embodiment, when the mobile device is in a normal power state and the user has selected an application that triggers a camera function or a camera of the mobile device, the register indicates a CIS mode. In an exemplary embodiment, when the mobile device is in a low power state, the register indicates an AON mode.

[0034] When the image sensor 150 operates in the CIS mode, the image sensor 150 may generate high-resolution raw image data (e.g., HiReslmData1) and send it to the imaging subsystem 130 through the camera serial interface (CSI) host 156. The imaging subsystem 130 may receive HiReslmData1 through its ImData I / F 133-1. In an exemplary embodiment, when the image sensor 150 operates in the AON mode, the image sensor 150 (e.g., the control I / F 155) sends low-resolution image data (e.g., LoResImData) to the sensor bus 160, and the sensor hub 120 receives the data through its sensor I / F 122.

[0035] When the application processor 110 acts as a master device and the image sensor 150 acts as a slave device, the sensor hub 120 retrieves LoResImData from the image sensor 150. When the image sensor 150 acts as a sub-master device, the image sensor 150 transmits LoResImData to the sensor hub 120. The application processor 110 may send a first signal to the image sensor 150 notifying the image sensor 150 that it is to act as a sub-master device, and another second signal to the image sensor 150 notifying the image sensor 150 that it is no longer acting as a sub-master device or that it has become a slave device again. Alternatively, the application processor 110 may set a register with information indicating whether the image sensor 150 is to act as a sub-master device or a slave device, and the image sensor 150 may check the register to determine how to act.

[0036] In an exemplary embodiment, a power management integrated circuit (eg, see Figure 8 580) delivers a first amount of power to the image sensor 150 during the CIS mode and delivers a second amount of power lower than the first amount of power to the image sensor 150 during the AON mode.

[0037] The sensor hub 120 may process the sensor data (e.g., data from non-image sensors and / or low-resolution image data from CIS 150) received by the sensor hub in a given time period to determine whether the mobile device should exit (e.g., wake up) from a low power or sleep state. When the image sensor 150 is not present, the sensor hub 120 receives sensor data only from one or more of the non-image sensors. When the image sensor 150 is present, the sensor hub 120 may receive only low-resolution image data or both low-resolution image data and non-image sensor data. In an exemplary embodiment, the size of the low-resolution image data is smaller than the size of the high-resolution image data. In an exemplary embodiment, if it is determined based on the low-resolution image data that the scene or background has changed, the sensor hub 120 may provide a status signal to the mobile device indicating that the mobile device is to exit from a low power state or a sleep state. The scene or background change may be determined based on comparing the previous low-resolution image data with the newly received low-resolution image data to determine a similarity value. For example, when the similarity value is less than a certain threshold (i.e., not very similar to the previous image), it may be determined that the scene or background has changed. In an exemplary embodiment, the sensor hub 120 performs a face detection algorithm on the low resolution image data, and if a face is detected, the sensor hub 120 may provide a status signal to the mobile device indicating that the mobile device is to exit from a low power state or a sleep state.

[0038] The image sensor 150 may send low-resolution image data with the application processor 110 via a dedicated CCI port for each driver of the application processor 110 or via a single physical port with different bus identifier (ID) addresses. The image sensor 150 may send low-resolution image data via a CCI bus supporting multiple masters or using a single ID with two masters and different address mapping or mode registers, or the application processor may have an internal multiplexer mechanism between the image sensor 150 and the driver of the application processor 110. In an exemplary embodiment, the image sensor 150 may operate in an AON mode or a ULP mode by reducing a sampling rate or a sampling frequency.

[0039] Figure 2 A configuration of the image sensor 150 according to an exemplary embodiment of the inventive concept is shown.

[0040] The image sensor 150 is configured to generate image data of an object 280 incident through a lens 201. The image sensor 150 includes a CIS pixel array 210, a row decoder 220, a correlated double sampler (CDS) 230, an analog-to-digital converter (ADC) 240, an output buffer 250, a timing controller 260, and a ramp generator 270.

[0041] The CIS pixel array 210 may include a plurality of CIS pixels (PX) 211 arranged in rows and columns. In an embodiment, each CIS pixel among the plurality of CIS pixels 211 may have a three-transistor (3TR) pixel structure that implements a pixel using three transistors, a four-transistor (4TR) pixel structure that implements a pixel using four transistors, or a five-transistor (5TR) pixel structure that implements a pixel using five transistors. Alternatively, at least two of the plurality of CIS pixels constituting the CIS pixel array 210 may share the same floating diffusion region FD (or floating diffusion node). However, the structure of the CIS pixel is not limited to the above configuration.

[0042] The row decoder 220 may select and drive a row of the CIS pixel array 210. In an embodiment, the row decoder 220 decodes a row address and / or a control signal output from the timing controller 260, and generates a control signal for selecting and driving a row of the CIS pixel array 210 indicated by the row address and / or the control signal. For example, the row decoder 220 may generate a selection signal, a reset signal, and a transfer signal, and may transmit the generated signals to pixels corresponding to the selected row.

[0043] The correlated double sampler 230 may sequentially sample and hold a set of reference signals and image signals provided from the CIS pixel array 210 through the column lines CL1 to CLn. In other words, the correlated double sampler 230 may sample and hold the levels of the reference signal and image signal corresponding to each of the columns. The correlated double sampler 230 may provide the set of reference signals and image signals sampled for each column to the analog-to-digital converter 240 under the control of the timing controller 260.

[0044] The analog-to-digital converter 240 may convert the correlated double sampling signal of each column output from the correlated double sampler 230 into a digital signal. In an embodiment, the analog-to-digital converter 240 may compare the correlated double sampling signal with the ramp signal output from the ramp generator 270, and may generate a digital signal corresponding to the comparison result.

[0045] The output buffer 250 may temporarily store the digital signal provided from the analog-to-digital converter 240 .

[0046] The timing controller 260 may control the operation of at least one of the CIS pixel array 210 , the row decoder 220 , the correlated double sampler 230 , the analog-to-digital converter 240 , the output buffer 250 , and the ramp generator 270 .

[0047] The ramp generator 270 may generate a ramp signal, and may provide the ramp signal to the analog-to-digital converter 240 .

[0048] In an exemplary embodiment, the correlated double sampler 230 receives the control signal CISctrl1, performs its sampling at a first frequency when the control signal CISctrl1 indicates the CIS mode and at a second frequency lower than the first frequency when the control signal CISctrl1 indicates the AON mode. Therefore, the output buffer 250 outputs high-resolution image data (e.g., HiResImData1) during the CIS mode and outputs low-resolution image data (e.g., LoResImData) during the AON mode.

[0049] Figure 3 Shows the Figure 1 A variation of the embodiment described in Figure 1 Differently, the system includes one or more additional image sensors (e.g., 351, ..., 35N). The additional image sensors differ from the image sensor 150 in that they do not provide / prepare low-resolution image data to be accessed by the sensor hub 120. The additional image sensors may prepare high-resolution image data and provide it to the imaging subsystem 130. For example, the second image sensor 351 may provide high-resolution image data HiReslmData2 to the second lmData I / F 133-2 of the imaging subsystem 130, and the Nth image sensor 35N may provide high-resolution image data HiReslmDataN to the Nth lmData I / F 133-N of the imaging subsystem 130. Figure 3 As shown, additional processors (e.g., 134-2, ..., 134-N) may be provided in the imaging subsystem 130 to process the additional high-resolution image data accordingly. In an alternative embodiment, there is only a single processor 134-1 to process all of the high-resolution image data. Therefore, the additional image sensors do not operate in the ULP or AON mode. In an alternative embodiment, one or more of the additional image sensors are configured in a manner similar to the image sensor 150 to prepare low-resolution image data to be accessed by the sensor hub 120 via the sensor bus 160 and operate in the ULP or AON mode. For example, there may be a total of N image sensors (e.g., CIS), of which only M image sensors operate similarly to the image sensor 150 to perform in one of the AON mode and the CIS mode, where M is less than N. CCI host 132 may transmit a first control signal (e.g., CCldata) to image sensor 150, a second control signal to image sensor 351, and an Nth control signal to Nth image sensor 35N using separate dedicated control lines using one of the above-described CCI protocols. Figure 3A single line connected to the CCI host 132 is shown, but in an alternative embodiment, the single line is replaced with separate lines that each reach a different one of the image sensors 150, 351, ..., 35N to provide different CCI data (e.g., control signals) to each of the image sensors.

[0050] Figure 4 An example of an image sensor 150 operating in an AON mode is shown. In this example, the image sensor 150 determines whether a scene or background change has occurred based on an analysis of low-resolution image data. For example, the image sensor 150 may include a memory that stores previous low-resolution image data, and the image sensor 150 may compare the previous low-resolution image data with the current low-resolution image data to determine whether a scene or background change has occurred. The image sensor 150 may send a scene change detection signal ScdSignal indicating whether a scene or background change has occurred to the sensor hub 120. For example, the value in the scene change detection signal ScdSignal may indicate a first value that a scene or background change has occurred and a second value different from the first value indicating that a scene or background change has not occurred. In an exemplary embodiment, the ScdSignal as the first value triggers face detection of the sensor hub 120. Therefore, the sensor hub 120 performs a face detection algorithm on the low-resolution image data to detect a face. If a face is detected, the sensor hub 120 may output a signal indicating that the mobile device should be awakened. If the sensor hub 120 cannot detect a face, the mobile device may remain in its current state.

[0051] In an alternative embodiment, both scene and face detection are performed on the image sensor 150 or both are performed on the application processor 110. In another embodiment, the image sensor 150 analyzes the low-resolution image data to determine whether the amount of ambient light exceeds a certain threshold, and then the image sensor 150 notifies the sensor hub 120 whether the amount of ambient light exceeds the certain threshold. In this embodiment, when the amount of ambient light exceeds the certain threshold, the sensor hub 120 notifies the application processor 110 of the wake-up state. When the intensity or brightness of the low-resolution image data exceeds a certain threshold, the image sensor 150 can determine that the ambient light exceeds the threshold. For example, the average brightness of the low-resolution image data can be calculated by the brightness or color of the image data of each pixel, and the average brightness can be compared with a certain threshold. In another embodiment, the sensor hub 120 analyzes the low-resolution image data to determine whether the amount of ambient light exceeds a certain threshold, and when the amount of ambient light exceeds the certain threshold, the sensor hub 120 notifies the application processor 110 of the wake-up state.

[0052] Figure 5A The image sensor 150 is shown synchronized with the sensor hub 120 of the application processor 110 during the AON mode. Since the sensor hub 120 may not be ready to process low-resolution image data, the sensor hub 120 may inform the image sensor 150 of the start time t_st and the end time t_end that the sensor hub can process low-resolution image data. For example, the sensor hub 120 may send the start time t_st and the end time t_end to the image sensor 150, and set a register indicating that the image sensor 150 should switch to the AON mode. Then, once the image sensor 150 has created the current low-resolution image data, the image sensor 150 may send an interrupt signal to the sensor hub 120 at a time between the start time t_st and the end time t_end. The image sensor 150 may send the low-resolution image data to the sensor hub 120 together with the interrupt, or the sensor hub 120 may retrieve the low-resolution image data from the memory of the image sensor 150 upon receiving the interrupt. For example, if sensor hub 120 sends start time t_st and end time t_end at time 0, start time t_st is 5 and end time t_end is 10, and image sensor 150 creates low resolution image data at time 3, image sensor 150 will not send the low resolution image data until at least time 5.

[0053] During the AON mode, the image sensor 150 may periodically perform readout to read the data of the CIS pixels, and then generate low-resolution image data of the current frame based on the read data. Figure 5B As shown, even if the low-resolution image data of frame N (e.g., AON frame) is ready before the start time T_st, the image sensor 150 does not send the Nth interrupt to the sensor hub 120 until the start time T_st or during the activation window of the sensor hub 120 between the start time T_st and the end time T_end. The sensor hub 120 can use the previously received interrupt as a zero reference. The interrupt can be shared on a separate line or as an In-Band interrupt in the case of I3C.

[0054] If the frame rate is loose enough, and the overall usage is determined in terms of processing time, an alternative approach can be used. In the alternative approach, the frame rate at which the image sensor 150 generates low-resolution image data is synchronized with the sensor hub activation frequency. The sensor hub 120 requests the image sensor 150 to prepare data at time t_valid, and assumes that the data is available by that time. The image sensor 150 then holds the data until time t_store after time t_valid. Fig. 6A As shown, the sensor hub 120 may send a signal (e.g., CSIctrl1) to the image sensor 150 to send the time t_valid and the time t_store to the image sensor 150. The image sensor 150 needs to complete the operation for generating the low-resolution image data before the time t_valid occurs, and then store the data at least until the time t_store. Figure 6B As shown, even though data is ready at time t_valid, the sensor hub 120 is still not ready to process the data (i.e., not activated). However, when the sensor hub 120 then becomes ready to process the data (i.e., activated), the sensor hub 120 reads data corresponding to the Nth frame from the image sensor 150 at a time occurring before time t_store. After time t_store, the image sensor 150 may capture a new image and generate new corresponding low-resolution image data.

[0055] Fig. 7A A method of operating an image sensor according to an exemplary embodiment of the inventive concept is illustrated. Fig. 7A The method includes determining whether the mobile device has entered a low power state (S701). In an exemplary embodiment, whenever a user presses a button of the mobile device to turn off the display, the mobile device enters a low power state. Fig. 7A The method further includes: when it is determined that the mobile device has entered the low power state, the sensor hub 120 sets the mode of the image sensor to the AON mode (step S702). Fig. 7A The method also includes generating low-resolution image data by the image sensor 150 in response to being set to the AON mode (step S703). In the AON mode, the image sensor 150 can periodically capture low-resolution images to generate low-resolution image data. In an exemplary embodiment, the front camera of the mobile device is used to capture the low-resolution image instead of the rear camera of the mobile device. Fig. 7A The method also includes the image sensor 150 transmitting the low-resolution image data to the application processor 110 (step 704). For example, the image sensor 150 may transmit the low-resolution image data to the sensor I / F 122. In an alternative embodiment, as described above with respect to Figure 5A and Figure 5B As discussed, the image sensor 150 may send an interrupt to the application processor 110, and the sensor hub 120 may retrieve the low-resolution image data from the image sensor 150 in response to the interrupt. For example, the interrupt may be sent to the sensor I / F 122. In another alternative embodiment, as described above with respect to Fig. 6A and Figure 6BAs discussed, the sensor hub 120 retrieves low resolution image data from the image sensor 150 between times t_valid and t_store. Fig. 7A The method also includes the sensor hub 120 performing an operation based on the low-resolution image data (step 705). In an exemplary embodiment, the operation includes determining whether to wake up the mobile device. For example, if the low-resolution image data depicts a scene change or includes a face, the sensor hub 120 can notify the application processor 110 that it should wake up or exit from a low-power or sleep state (e.g., enter a normal operating mode or enter a normal power state). For example, when exiting from a low-power or sleep state, the mobile device can turn on its display and present a user interface on the display.

[0056] Figure 7B A method of operating an image sensor according to an exemplary embodiment of the inventive concept is illustrated. Figure 7B The method includes determining whether the mobile device has entered a normal power state (step S706). If it is determined that the mobile device has entered a normal power state, then Figure 7B The method further includes the sensor hub 120 setting the operation mode of the image sensor 150 to the CIS mode (step S707). Figure 7B The method also includes generating high-resolution image data by the image sensor 150 in response to being set to the CIS mode (step 708). In the CIS mode, the image sensor 150 no longer periodically captures low-resolution images to be exchanged with the sensor hub 120. In an exemplary embodiment, during the CIS mode, once the user triggers a camera function of the mobile device using a rear camera of the mobile device, the image sensor 150 captures a high-resolution image or high-resolution image data that is not shared with the sensor hub 120. Figure 7B The method further includes the image sensor 150 transmitting the high-resolution image data to the application processor 110 (step 709). Figure 7B The method further includes applying the imaging subsystem 130 of the processor 110 to perform operations on the high-resolution image data (step S710).

[0057] Figure 8 is a block diagram illustrating a mobile device according to an embodiment of the inventive concept. Fig. 9 It shows Figure 8 A block diagram of an example in which a mobile device is implemented as a smart phone.

[0058] refer to Figure 8 and Fig. 9, the mobile device 500 includes: an application processor 110; a memory device 520; a storage device 530; a plurality of functional modules 540, 550, 560 and 570; and a PMIC 580, which provides operating voltages to the application processor 110, the memory device 520, the storage device 530, the functional modules 540, 550, 560 and 570 respectively; and a non-image sensor 140. For example, Fig. 9 As shown, mobile device 500 may be implemented as a smart phone.

[0059] The application processor 110 controls the overall operation of the mobile device 500. For example, the application processor 110 controls the memory device 520, the storage device 530, the non-image sensor, and the function modules 540, 550, 560, and 570.

[0060] Application processor 110 may include: a central processing unit that operates based on a clock signal; a clock generation unit that generates a clock signal to provide the clock signal to the central processing unit; and a clock management unit that predicts an operating state of the central processing unit, provides operating frequency information to PMIC 580 based on the predicted operating state of the central processing unit, and changes the operating frequency of application processor 510 based on the predicted operating state of the central processing unit. In an exemplary embodiment, PMIC 580 changes an operating voltage of application processor 510 based on the operating frequency information indicating a change in the operating frequency of application processor 110.

[0061] The memory device 520 and the storage device 530 store data for the operation of the mobile device 500. The memory device 520 may correspond to a volatile semiconductor memory device, such as a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a mobile DRAM, etc. In addition, the storage device 530 may correspond to a non-volatile semiconductor memory device, such as an erasable programmable read-only memory (EPROM) device, an electrically erasable programmable read-only memory (EEPROM) device, a flash memory device, a phase change random access memory (PRAM) device, a resistive random access memory (RRAM) device, a nano floating gate memory (NFGM) device, a polymer random access memory (PoRAM) device, a magnetic random access memory (MRAM) device, a ferroelectric random access memory (FRAM) device, etc. In some embodiments, the storage device 530 may correspond to a solid state drive (SSD) device, a hard disk drive (HDD) device, a CD-ROM device, etc.

[0062] Functional modules 540, 550, 560, and 570 perform various functions of the mobile device 500. For example, the mobile device 500 may include: a communication module 540 that performs a communication function (e.g., a code division multiple access (CDMA) module, a long term evolution (LTE) module, a radio frequency (RF) module, an ultra-wideband (UWB) module, a wireless local area network (WLAN) module, a world interoperability for microwave access (WIMAX) module, etc.); a camera module 550 that performs a camera function; a display module 560 that performs a display function; a touch panel module 570 that performs a touch sensing function, etc. In an exemplary embodiment, the camera module 550 includes the image sensor 150 and / or the image sensors 351-351N.

[0063] The inventive concept can be applied to electronic devices having an application processor. For example, the inventive concept can be applied to computers, laptop computers, digital cameras, cellular phones, smart phones, smart tablets, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, navigation systems, cameras, portable game consoles, etc.

[0064] As is conventional in the field of the present invention, exemplary embodiments are described and shown in the accompanying drawings from the aspects of functional blocks, units and / or modules. It will be appreciated by those skilled in the art that these blocks, units and / or modules are physically implemented by electronic (or optical) circuits such as logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc., wherein semiconductor-based manufacturing techniques or other manufacturing techniques can be used to form these electronic (or optical) circuits. In the case of implementing blocks, units and / or modules by microprocessors, etc., they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and can be optionally driven by firmware and / or software. Alternatively, each block, unit and / or module can be implemented by dedicated hardware or implemented as a combination of dedicated hardware that performs some functions and processors (e.g., one or more programmed microprocessors and associated circuits) that perform other functions. Moreover, without departing from the scope of the present invention, each block, unit and / or module of the exemplary embodiment can be physically separated into two or more interactive and discrete blocks, units and / or modules. Furthermore, the blocks, units and / or modules of the exemplary embodiments may be physically combined into more complex blocks, units and / or modules without departing from the scope of the inventive concept.

[0065] The exemplary embodiments of the present invention can be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. For example, the software module can be implemented on a non-transitory program storage device (e.g., in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, or any other form of storage medium as known in the art). An exemplary storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to it. In an alternative, the storage medium can be integrated into the processor. In addition, in some aspects, the processor and the storage medium can reside in an application specific integrated circuit (ASIC).

[0066] While the inventive concept has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the following claims.

Claims

1. A mobile device, comprising: a sensor bus connected to at least one non-image sensor to receive sensor data; an application processor including an imaging subsystem having a first interface and a sensor hub having a second interface, the imaging subsystem being configured to process high-resolution image data received via the first interface; an image sensor configured to operate in one of a first mode and a second mode, wherein, during the first mode, the image sensor is configured to capture the high-resolution image data in response to a request from the imaging subsystem, and the imaging subsystem is configured to perform a first operation on the high-resolution image data received through the first interface, wherein during the second mode, the image sensor is configured to capture low-resolution image data and output the low-resolution image data to the sensor bus, and Wherein, during the second mode, the sensor hub is configured to perform a second operation using the low-resolution image data and the sensor data received from the sensor bus through the second interface.

2. The mobile device according to claim 1, wherein: The image sensor operates in the second mode when the mobile device operates in a low power state and operates in the first mode when the mobile device operates in a normal power state.

3. The mobile device according to claim 2, wherein: The second operation determines whether a scene change has occurred based on an analysis of the low-resolution image data, and the application processor exits from the low-power state when it is determined that the scene change has occurred.

4. The mobile device according to claim 2, wherein: The second operation detects whether a human face exists based on an analysis of the low-resolution image data, and the application processor exits from the low-power state when the human face is detected.

5. The mobile device according to claim 1, wherein: During the second mode, when the image sensor determines that a scene change has occurred based on analysis of the low resolution image data, the sensor hub detects whether a human face is present in the low resolution image data.

6. The mobile device according to claim 1, wherein: The first operation processes the high resolution image data to generate an image.

7. The mobile device according to claim 1, wherein: The at least one non-image sensor includes at least one of an inertial motion unit IMU and an ambient light sensor ALS.

8. The mobile device according to claim 1, wherein: The sensor hub provides a time range to the image sensor indicating that the sensor hub is ready to process the low resolution image data, and the image sensor provides an interrupt to the sensor hub during a time between the time ranges after generating the low resolution image data.

9. The mobile device according to claim 1, wherein: The sensor hub provides a valid time and a storage time to the image sensor, the image sensor completes preparation of the low-resolution image data before the valid time and saves the prepared low-resolution image data until at least the storage time, and the sensor hub retrieves the prepared low-resolution image data from the image sensor between the valid time and the storage time.

10. An application processor, comprising: an imaging subsystem including a first interface, the imaging subsystem configured to process high resolution image data received from an image sensor via the first interface during a first mode; as well as a sensor hub comprising a second interface connected to a sensor bus, the sensor hub being configured to process sensor data received through the second interface from at least one non-image sensor connected to the sensor bus and low-resolution image data received through the second interface from an image sensor connected to the sensor bus, The sensor hub is configured to determine whether to exit the sleep state by analyzing the sensor data and the low-resolution image data during another second mode.

11. The application processor according to claim 10, wherein: The application processor is located in a mobile device, the first mode occurs when the mobile device is in a normal power state, and the second mode occurs when the mobile device is in a low power state.

12. The application processor according to claim 10, wherein: The sensor hub provides a time range to the image sensor indicating that the sensor hub is ready to process the low resolution image data, and the image sensor provides an interrupt to the sensor hub during a time within the time range after generating the low resolution image data.

13. The application processor according to claim 10, wherein: The sensor hub provides a valid time and a storage time to the image sensor, the image sensor completes preparation of the low-resolution image data before the valid time and saves the prepared low-resolution image data until at least the storage time, and the sensor hub retrieves the prepared low-resolution image data from the image sensor between the valid time and the storage time.

14. A method of operating a mobile device comprising an image sensor and an application processor, the method comprising: When the mobile device is in a low power state, generating low resolution image data by the image sensor; generating, by the image sensor, high resolution image data when the image sensor receives a request from the application processor and the mobile device is in a normal power state; performing, by the application processor, a first operation using sensor data received from a first bus connected to at least one non-image sensor and the low-resolution image data received via the first bus while the mobile device is in the low power state; as well as When the mobile device is in the normal power state, a second operation is performed by the application processor using the high-resolution image data received through the second bus.

15. The method according to claim 14, wherein: The first operation determines whether to switch to the normal power state.

16. The method according to claim 15, wherein: When the mobile device is in the low-power state, the sensor hub of the application processor provides an effective time and a storage time to the image sensor, the image sensor completes preparation of the low-resolution image data before the effective time and saves the prepared low-resolution image data until at least the storage time, and the sensor hub retrieves the prepared low-resolution image data from the image sensor between the effective time and the storage time.

Images