Image processing pipeline, image processing method, camera assembly, and electronic device

Abstract

An image processing pipeline (100), an image processing method, a camera assembly (500) and an electronic device (1000). An image sensor (200) comprises a pixel array (201), the pixel array (201) comprises imaging pixels (202) and identification pixels (204), the imaging pixels (202) are used to output a scene image, the scene image is used to represent scene information, and the image processing pipeline (100) is used to obtain a to-be-identified image according to an output signal of the identification pixels (204), determine whether preset information exists in the to-be-identified image, and control the electronic device (1000) to perform a corresponding operation according to the preset information when the preset information exists in the to-be-identified image.

Description

Technical Field

[0001] This application relates to the field of consumer electronics, and more specifically, to an image processing pipeline, an image processing method, a camera assembly, and an electronic device. Background Technology

[0002] In related technologies, in order to realize face recognition, gesture recognition, posture recognition, etc. in real time, electronic devices need to be equipped with a camera, which will occupy the internal space of the electronic device and increase the manufacturing cost of the electronic device. Summary of the Invention

[0003] Embodiments of this application provide an image processing pipeline, an image processing method, a camera assembly, and an electronic device.

[0004] The image processing pipeline of this application is used in an image sensor, the image sensor including a pixel array, the pixel array including imaging pixels and recognition pixels, the imaging pixels being used to output a scene image, the scene image being used to characterize scene information, and the image processing pipeline being used to obtain an image to be recognized based on the output signal of the recognition pixels, to determine whether there is preset information in the image to be recognized, and when the preset information exists in the image to be recognized, to control an electronic device to perform a corresponding operation based on the preset information.

[0005] The image processing method of this application is used in an image sensor, the image sensor including a pixel array, the pixel array including imaging pixels and recognition pixels, the imaging pixels being used to output a scene image, the scene image being used to characterize scene information, the image processing method including: obtaining an image to be recognized based on the output signal of the recognition pixels; determining whether preset information exists in the image to be recognized, and if the preset information exists in the image to be recognized, controlling an electronic device to perform a corresponding operation based on the preset information.

[0006] The camera assembly of the embodiments of this application includes an image sensor and the image processing pipeline described above, the image processing pipeline being used to process the image output by the image sensor.

[0007] The electronic device according to the embodiments of this application includes a housing and the camera assembly described above, the camera assembly being disposed on the housing.

[0008] The image processing pipeline, image processing method, camera component, and electronic device of the present application embodiment can obtain both scene images and images to be identified through an image sensor, determine whether there is preset information in the image to be identified, and control the electronic device to perform corresponding operations according to the preset information when the preset information exists, thereby realizing the AON (alwayson) function.

[0009] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0010] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0011] Figure 1 This is a schematic diagram of a camera assembly according to an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the image processing pipeline according to an embodiment of this application;

[0013] Figure 3 This is a schematic diagram of a pixel array according to an embodiment of this application;

[0014] Figure 4 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0015] Figure 5 This is a schematic diagram of a pixel array according to an embodiment of this application;

[0016] Figure 6 This is a cross-sectional schematic diagram of a photosensitive pixel according to an embodiment of this application;

[0017] Figure 7 This is a pixel circuit diagram of the photosensitive pixel according to an embodiment of this application;

[0018] Figure 8 This is a schematic diagram of an image processing method according to an embodiment of this application;

[0019] Figure 9 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0020] Figure 10 and Figure 11 This is a schematic diagram of a pixel array according to an embodiment of this application;

[0021] Figure 12 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0022] Figure 13 This is a schematic diagram of an image processing method according to an embodiment of this application;

[0023] Figure 14 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0024] Figure 15This is a schematic diagram of an image processing method according to an embodiment of this application;

[0025] Figure 16 This is a schematic flowchart of the image processing method according to an embodiment of this application;

[0026] Figure 17 and Figure 18 This is a schematic diagram of a pixel array according to an embodiment of this application;

[0027] Figure 19 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. These embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0029] Please refer to the following: Figure 1 , Figure 2 and Figure 3 The camera assembly 500 of this application embodiment includes an image sensor 200 and an image processing pipeline 100. The image sensor 200 includes a pixel array 201, which includes imaging pixels 202 and recognition pixels 204. The imaging pixels 202 are used to output a scene image, which is used to characterize scene information. The image processing pipeline 100 is used to: obtain an image to be recognized based on the output signal of the recognition pixels 204; determine whether preset information exists in the image to be recognized; and control the electronic device 1000 according to the preset information when the preset information exists in the image to be recognized (see [link to relevant documentation]). Figure 19 ) Perform the corresponding operation.

[0030] This application also discloses an image processing method for an image sensor 200. The image sensor 200 includes a pixel array 201, which includes imaging pixels 202 and recognition pixels 204. The imaging pixels 202 are used to output a scene image, which is used to characterize scene information. The image processing method can be implemented by the image processing pipeline 100 of this application's embodiments. Please refer to [link to relevant documentation]. Figure 4 Image processing methods include:

[0031] 01: Obtain the image to be recognized based on the output signal of recognition pixel 204;

[0032] 02: Determine whether there is preset information in the image to be recognized, and if there is preset information in the image to be recognized, control the electronic device 1000 to perform the corresponding operation according to the preset information.

[0033] The image processing method of this application embodiment can be implemented by the image processing pipeline 100 of this application embodiment, wherein steps 01 and 02 can both be implemented by the image processing pipeline 100. That is to say, the image processing pipeline 100 is used to: obtain an image to be recognized based on the output signal of the recognition pixel 204; determine whether there is preset information in the image to be recognized, and when there is preset information in the image to be recognized, control the electronic device 1000 to perform corresponding operations based on the preset information.

[0034] In this way, an image sensor 200 can acquire an image to be recognized, determine whether there is preset information in the image to be recognized, and control the electronic device 1000 to perform corresponding operations according to the preset information when the preset information is present, thereby realizing the AON function.

[0035] It's worth noting that the context-aware application functions achievable based on the image to be recognized obtained from the output signal of the recognition pixel 204 include at least one of the following: privacy protection, air gestures, gaze-based screen-on, and lying-down screen-off. Specifically, privacy protection: For example, if a social media app receives a new message from a girlfriend or a bank sends a text message confirming salary deposit, and the user doesn't want others to see the private information, the terminal can detect when a stranger is looking at the user's phone screen and turn it off. Air gestures: For example, if a user is cooking and has their phone placed aside to view a recipe, and an important call comes in, but their hands are oily and they can't directly operate the phone, the terminal can detect the user's air gestures and execute the corresponding operation. Gaze-based screen-on: For example, when reading a recipe or e-book, there's often a page that users read repeatedly, and the automatic screen-off time is approaching. The terminal can detect when the user is still looking at the screen and will not activate the automatic screen-off function. No rotation when lying down: For example, when the user lies down and the screen orientation of the electronic device 1000 changes, such as from vertical to horizontal, the electronic device 1000 can detect the change in the user's eye gaze direction through the recognition pixel 204, and the screen will not rotate.

[0036] In related technologies, in order to realize real-time facial recognition, gesture recognition, posture recognition and the above-mentioned scene awareness application functions, electronic devices need to be equipped with an always-on (AON) camera, which will occupy the internal space of the electronic device and increase the manufacturing cost of the electronic device.

[0037] The camera assembly 500, image processing pipeline 100, and image processing method of this application can acquire both scene images and images to be recognized using an image sensor 200, and determine whether preset information exists based on the images to be recognized. Since the images to be recognized are obtained from the output signal of the recognition pixel 204, and the recognition pixel 204 receives light at a preset period, it can periodically detect the presence of preset information to achieve the AON function. The preset period can be a default setting or can be set according to user input. In one embodiment, the preset period is short, such as 10 seconds, 1 second, 300 milliseconds, 100 milliseconds, or 10 milliseconds. This allows the recognition pixel 204 to sense light at a higher frequency, enabling it to periodically detect changes in the light signal, acquire the images to be recognized, determine whether preset information exists in the images to be recognized, and control the electronic device 1000 to perform corresponding operations based on the preset information when preset information is present in the images to be recognized.

[0038] The camera assembly 500 includes an image sensor 200. The image sensor 200 may be a complementary metal-oxide-semiconductor (CMOS) photosensitive element or a charge-coupled device (CCD) photosensitive element. The camera assembly 500 of this embodiment obtains a scene image and an image to be identified by exposing the pixel array 201.

[0039] Please see Figure 5 It is worth mentioning that the image sensor 200 also includes a vertical drive unit 22, a control unit 23, a column processing unit 24, and a horizontal drive unit 25.

[0040] For example, the image sensor 200 may employ a complementary metal-oxide-semiconductor (CMOS) photosensitive element or a charge-coupled device (CCD) photosensitive element.

[0041] For example, pixel array 201 includes a plurality of photosensitive pixels 2011 arranged in a two-dimensional array (i.e., a two-dimensional matrix). Figure 6 As shown), each photosensitive pixel 2011 includes a photoelectric conversion element 2012 (as shown). Figure 7 (As shown). Each photosensitive pixel 2011 converts light into electrical charge based on the intensity of the light incident on it.

[0042] For example, the vertical drive unit 22 includes a shift register and an address decoder. The vertical drive unit 22 includes readout scan and reset scan functions. Readout scan refers to sequentially scanning each photosensitive pixel 2011 row by row, reading signals from these photosensitive pixels 2011 one row at a time. For example, the signal output from each photosensitive pixel 2011 in the selected and scanned row of photosensitive pixels is transmitted to the column processing unit 24. Reset scan is used to reset the charge; the photocharge of the photoelectric conversion element 2012 is discarded, allowing the accumulation of new photocharge to begin.

[0043] For example, the signal processing performed by the column processing unit 24 is correlated double sampling (CDS) processing. In CDS processing, the reset level and signal level output from each photosensitive pixel 2011 in the selected row of photosensitive pixels are extracted, and the level difference is calculated. Thus, the signal of the photosensitive pixels 2011 in a row is obtained. The column processing unit 24 may have an analog-to-digital (A / D) conversion function for converting analog pixel signals into digital format.

[0044] For example, the horizontal drive unit 25 includes a shift register and an address decoder. The horizontal drive unit 25 sequentially scans the pixel array 201 column by column. Through the selection scan operation performed by the horizontal drive unit 25, each column of photosensitive pixels is sequentially processed by the column processing unit 24 and sequentially output.

[0045] For example, the control unit 23 configures timing signals according to the operating mode and uses multiple timing signals to control the vertical drive unit 22, column processing unit 24 and horizontal drive unit 25 to work together.

[0046] Please see Figure 6 The photosensitive pixel 2011 includes a pixel circuit 211, a filter 212, and a microlens 213. Along the light-receiving direction of the photosensitive pixel 2011, the microlens 213, the filter 212, and the pixel circuit 211 are arranged sequentially. The microlens 213 is used to converge light, and the filter 212 is used to allow light of a certain wavelength to pass through while filtering out light of other wavelengths. The pixel circuit 211 is used to convert the received light into an electrical signal and provide the generated electrical signal to... Figure 5 The column processing unit 24 shown.

[0047] Please see Figure 7 The pixel circuit 211 can be applied in Figure 5 Each photosensitive pixel 2011 in the pixel array 201 shown Figure 6 As shown in the image. The following is in conjunction with... Figures 2 to 4 The working principle of pixel circuit 211 is explained.

[0048] like Figure 7As shown, the pixel circuit 211 includes a photoelectric conversion element 2012 (e.g., a photodiode), an exposure control circuit (e.g., a transfer transistor 2112), a reset circuit (e.g., a reset transistor 2113), an amplification circuit (e.g., an amplification transistor 2114), and a selection circuit (e.g., a selection transistor 2115). In embodiments of this application, the transfer transistor 2112, the reset transistor 2113, the amplification transistor 2114, and the selection transistor 2115 are, for example, MOSFETs, but are not limited thereto.

[0049] For example, photoelectric conversion element 2012 includes a photodiode, the anode of which is connected to ground, for example. The photodiode converts received light into electrical charge. The cathode of the photodiode is connected to a floating diffusion unit FD via an exposure control circuit (e.g., transfer transistor 2112). The floating diffusion unit FD is connected to the gate of amplification transistor 2114 and the source of reset transistor 2113.

[0050] For example, the exposure control circuit is a transfer transistor 2112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 2112. When an effective level pulse (e.g., VPIX level) is transmitted to the gate of the transfer transistor 2112 through the exposure control line, the transfer transistor 2112 is turned on. The transfer transistor 2112 transfers the charge converted by the photodiode to the floating diffusion unit FD.

[0051] For example, the drain of reset transistor 2113 is connected to the pixel power supply VPIX. The source of reset transistor 113 is connected to the floating diffusion unit FD. Before charge is transferred from the photodiode to the floating diffusion unit FD, a pulse of the effective reset level is transmitted to the gate of reset transistor 113 via the reset line, turning on reset transistor 113. Reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

[0052] For example, the gate of amplifying transistor 2114 is connected to the floating diffusion unit FD. The drain of amplifying transistor 2114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by reset transistor 2113, amplifying transistor 2114 outputs a reset level via select transistor 2115 through output terminal OUT. After the charge of the photodiode is transferred by transfer transistor 2112, amplifying transistor 2114 outputs a signal level via select transistor 2115 through output terminal OUT.

[0053] For example, the drain of the selector transistor 2115 is connected to the source of the amplifying transistor 2114. The source of the selector transistor 2115 is connected to the output terminal OUT. Figure 5The column processing unit 24 is used in the selection transistor 2115. When an active level pulse is transmitted to the gate of the selection transistor 2115 through the selection line, the selection transistor 2115 is turned on. The signal output from the amplification transistor 2114 is transmitted to the column processing unit 24 through the selection transistor 2115.

[0054] It should be noted that the pixel structure of the pixel circuit 211 in this embodiment is not limited to... Figure 7 The structure shown is illustrated. For example, pixel circuit 211 can also have a three-transistor pixel structure, where the functions of amplification transistor 2114 and selection transistor 2115 are performed by a single transistor. For example, the exposure control circuit is not limited to a single transfer transistor 2112; other electronic devices or structures with control terminal control functions can be used as the exposure control circuit in the embodiments of this application. The implementation of the single transfer transistor 2112 in the embodiments of this application is simple, low-cost, and easy to control.

[0055] Please see Figure 8 In some embodiments, the image processing pipeline 100 includes a first image processing pipeline 10 and a second image processing pipeline 20. The first image processing pipeline 10 is used to obtain a scene image based on the output signal of the imaging pixel 202. The second image processing pipeline 20 is used to obtain an image to be recognized based on the output signal of the recognition pixel 204, determine whether there is preset information in the image to be recognized, and control the electronic device 1000 to perform corresponding operations based on the preset information when there is preset information in the image to be recognized.

[0056] Please see Figure 9 In some embodiments, the image processing method includes:

[0057] 03: Obtain the scene image based on the output signal of imaging pixel 202.

[0058] In some embodiments, the image processing pipeline 100 includes a first image processing pipeline 10 and a second image processing pipeline 20. Step 03 can be implemented by the first image processing pipeline 10. That is, the first image processing pipeline 10 is used to: obtain a scene image based on the output signal of the imaging pixel 202.

[0059] In this way, the image sensor 200 can obtain both scene images and images to be recognized, determine whether there is preset information in the images to be recognized, and control the electronic device 1000 to perform corresponding operations according to the preset information when the preset information is present, thereby realizing the AON function.

[0060] Specifically, the image processing method can be implemented through an image processing pipeline 100, which includes a first image processing pipeline 10 and a second image processing pipeline 20. The first image processing pipeline 10 can obtain a scene image based on the output signal of the imaging pixel 202. The scene image is used to represent scene information, including color information, brightness information, etc. After processing, the scene information is used to realize the imaging function of the camera component and obtain an image. The first image processing pipeline 10 may also include image processing functions such as black level correction, lens attenuation, white balance processing, image correction and adjustment, advanced noise reduction, temporal filtering, color anomaly correction, color space conversion, local tone mapping, color correction, gamma correction, color adjustment and chroma enhancement, and chroma suppression. Users can enable applications on the electronic device 1000 to obtain scene images, such as photo-taking applications and video recording applications. The second image processing pipeline 20 can obtain the image to be recognized based on the output signal of the recognition pixel 204, determine whether there is preset information in the image to be recognized, and control the electronic device 1000 to perform corresponding operations according to the preset information when the preset information is present in the image to be recognized. Users can achieve context-aware application functions such as privacy protection, air gestures, gaze-based screen keeping, and lying-down screen keeping without rotation through the second image processing pipeline 20. The processing workload of the second image processing pipeline 20 for each frame of the image to be recognized is lower than that of the first image processing pipeline 20 for the scene image. For example, the second image processing pipeline 20 does not need to perform image processing such as color anomaly correction, color space conversion, and color correction. This reduces the workload of the second image processing pipeline 20, thereby reducing its power consumption and facilitating the implementation of AON (Aspect-On-Demand) functionality.

[0061] In some embodiments, a scene image can be obtained based on the output signal of the imaging pixel 202, or it can be obtained based on both the output signal of the imaging pixel 202 and the output signal of the recognition pixel 204. The number of pixels in the recognition pixel 204 accounts for less than 5% of the pixel array 201; in one example, the number of pixels in the recognition pixel 204 accounts for 2.5% of the pixel array 201. Thus, the smaller number of pixels in the recognition pixel 204 reduces the power consumption required to obtain the image to be recognized and decreases the workload required for processing the image. Furthermore, the smaller number of pixels in the recognition pixel 204 also reduces its impact on the scene image.

[0062] It is worth mentioning that the camera assembly 500 may also include a lens, an imaging device, and a lens barrel. Specifically, the lens may include lens elements and multiple lens groups, which can achieve zoom at any focal length within any focal range to ensure the clarity of the image to be identified. The imaging device includes a voice coil motor, an infrared cutoff filter, etc. The voice coil motor can be located above the image sensor 200 and includes an anti-magnetic cover, an upper fixing ring, a pressure plate, an upper spring, a lens mount, a coil, a magnet and a magnet holder, a lower spring, and a bottom fixing base. The voice coil motor can convert electrical energy into mechanical energy, and the lens can achieve autofocus through the voice coil motor. The voice coil motor can adjust its position to focus and present a clear image to be identified. The infrared cutoff filter can be a filter that filters the infrared band, which can prevent infrared light from passing through the lens and causing image distortion. The camera assembly 500 can be applied to electronic devices with photo and video recording functions, such as smartphones and tablets.

[0063] In embodiments of this application, the image sensor 200 includes a pixel array 201, which includes imaging pixels 202 and recognition pixels 204. In one embodiment, the pixel array 201 includes imaging pixels 202 and recognition pixels 204 arranged in a two-dimensional array (i.e., a two-dimensional matrix) and in a two-dimensional matrix (e.g., a two-dimensional matrix arrangement). Figure 3 As shown, the imaging pixels 202 and the recognition pixels 204 can be arranged in a Bayer array.

[0064] In another embodiment, the image sensor 200 includes a pixel array 201, which includes a plurality of pixel units, each pixel unit including a plurality of photosensitive pixels, the plurality of photosensitive pixels of the same pixel unit covering the same color channel, and the plurality of pixel units arranged in a Bayer array.

[0065] Please see Figure 10 Specifically, pixel array 201 includes imaging pixels 202 and recognition pixels 204. Pixel array 201 also includes first-type pixel units UR, second-type pixel units UG, and third-type pixel units UB. The first-type pixel unit UA includes multiple first-color photosensitive pixels R, the second-type pixel unit UG includes multiple second-color photosensitive pixels G, and the third-type pixel unit UB includes multiple third-color photosensitive pixels B. The multiple first-type pixel units UR, multiple second-type pixel units UG, and multiple third-type pixel units UB are arranged in a Bayer array.

[0066] In yet another embodiment, the image sensor 200 includes a pixel array 201, which further includes a plurality of pixel units, each pixel unit including at least one color photosensitive pixel and at least one panchromatic photosensitive pixel W, the color photosensitive pixel having a narrower spectral response than the panchromatic photosensitive pixel W.

[0067] Please see Figure 11 Specifically, the pixel array 201 includes imaging pixels 202 and recognition pixels 204. The pixel array 201 also includes a first type pixel unit UR, a second type pixel unit UG, and a third type pixel unit UB. The first type pixel unit UR includes multiple first color photosensitive pixels R and multiple panchromatic photosensitive pixels W; the second type pixel unit UG includes multiple second color photosensitive pixels G and multiple panchromatic photosensitive pixels W; and the third type pixel unit UB includes multiple third color photosensitive pixels B and multiple panchromatic photosensitive pixels W. Because the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels W, the signal-to-noise ratio of the image can be improved, resulting in higher image sharpness.

[0068] Please refer to it again. Figure 3 In some implementations, multiple identification pixels 204 form identification pixel combinations 205, and the multiple identification pixels 204 of each identification pixel combination 205 cover multiple color channels.

[0069] Specifically, in one example, the image sensor can be an array of filters arranged in a Bayer array configuration, so that multiple imaging pixels 202 and recognition pixels 204 in the image sensor can each receive light passing through their respective filters, thereby generating pixel signals with different color channels. For example... Figure 3 As shown, the recognition pixel 204 can include pixel signals from three color channels: RGB. RGB stands for: R (red channel), G (green channel), and B (blue channel). Thus, because the recognition pixel 204 can cover multiple channels, compared to a recognition pixel 204 covering only one channel, the output signal of the recognition pixel 204 obtains an image to be recognized containing information from multiple color channels. This makes the image formed by the recognition pixel 204 more accurate, avoiding recognition errors caused by color limitations during the image recognition process, thereby resulting in a more precise recognition result.

[0070] Please see Figure 12 In some embodiments, multiple recognition pixels 204 of each recognition pixel combination 205 are arranged adjacently in the image sensor 200, and the image to be recognized is processed to determine whether preset information exists, including:

[0071] 021: Merge the output signals of multiple recognition pixels 204 of each recognition pixel combination 205 to obtain merged pixel 206;

[0072] 022: Determine whether preset information exists in the image to be identified based on multiple merged pixels 206.

[0073] In some embodiments, the plurality of recognition pixels 204 of each recognition pixel combination 205 are arranged adjacently in the image sensor 200, and steps 021 and 022 can both be implemented by the second image processing pipeline 20. That is, the second image processing pipeline 20 is used to: merge the output signals of the plurality of recognition pixels 204 of each recognition pixel combination 205 to obtain a merged pixel 206; and determine whether preset information exists in the image to be recognized based on the plurality of merged pixels 206.

[0074] Please refer to the following: Figure 3 and Figure 13 In each recognition pixel combination 205, multiple recognition pixels 204 are arranged adjacently in the image sensor 200. Adjacent recognition pixels 204 have high correlation and can be used to represent the same point in the scene, thus facilitating merging to obtain a merged pixel 206. The pixel data of the merged pixel 206 can be the sum or weighted average of the pixel data of multiple recognition pixels 204. Figure 13 For example, the identification pixel combination 205 includes four identification pixels 204. The sum or weighted average of the pixel data of the four identification pixels 204 can be used as the pixel data of the merged pixel 206. The merged pixel 206 includes various pixel information, and the pixel data of the merged pixel 206 is more accurate. In one embodiment, the merged pixel 206 can be used to characterize the brightness information and color information of an object, so that in the process of determining whether the preset information exists in the image to be identified, the brightness information and color information of the object can be more accurately identified and judged. In another embodiment, such as Figure 13 As shown, the merged pixel 206 can be used to represent the brightness information of an object, but does not include the color information of the object. In the process of determining whether the preset information exists in the image to be identified, it is not necessary to involve the identification and judgment of color information. In this way, the calculation of subsequent processing can be reduced to realize the AON function by determining whether the preset information exists in the image to be identified based on multiple merged pixels 206.

[0075] Please see Figure 14 In some implementations, processing the image to be identified to determine whether preset information exists further includes:

[0076] 023: Convert the first pixel bit depth of multiple merged pixels 206 into the second pixel bit depth, where the second pixel bit depth is less than the first pixel bit depth;

[0077] Determine whether preset information exists based on multiple merged pixels 206, including:

[0078] 0221: Determine whether there is preset information in the image to be identified based on multiple merged pixels 206 of the second pixel bit depth.

[0079] In some implementations, steps 023 and 0221 can both be implemented by the second image processing pipeline 20. That is, the second image processing pipeline 20 is used to: convert the first pixel bit depth of the multiple merged pixels 206 into a second pixel bit depth, the second pixel bit depth being less than the first pixel bit depth; and determine whether preset information exists in the image to be identified based on the multiple merged pixels 206 with the second pixel bit depth.

[0080] Please see Figure 15 In one example, the first pixel bit depth of multiple merged pixels 206 is converted to a second pixel bit depth, which is less than the first pixel bit depth. The first pixel bit depth of the merged pixels 206 can be 10 bits, and the second pixel depth can be 8 bits. The presence of preset information is determined based on the multiple merged pixels 206 at the second pixel bit depth (i.e., 8 bits). This reduces the amount of data processing and allows for faster recognition and detection.

[0081] Please see Figure 16 In some implementations, the preset information includes a preset face, preset gesture, or preset human posture, and the image processing method includes:

[0082] 024: Determine whether there is a preset face, preset gesture, or preset human posture in the image to be recognized;

[0083] 025: When a preset face, preset gesture, or preset human posture exists in the image to be recognized, control the electronic device 1000 to perform the corresponding operation.

[0084] In some implementations, steps 024 and 025 can both be implemented by the second image processing pipeline 20. That is, the second image processing pipeline 20 is used to: determine whether a preset face, preset gesture, or preset human posture exists in the image to be recognized; and control the electronic device 1000 to perform the corresponding operation when a preset face, preset gesture, or preset human posture exists in the image to be recognized.

[0085] Specifically, the second image processing pipeline 20 can obtain the image to be recognized based on the output signal of the recognition pixel 204 and determine whether preset information exists in the image to be recognized. The preset information includes a preset face, a preset gesture, or a preset human posture. In one example, the preset information can be a preset face for unlocking. After the second image processing pipeline 20 obtains the image to be recognized, it can process the image. If it is determined that a preset face exists in the image to be recognized, the corresponding preset information is determined, and thus unlocking can be performed. In another example, the preset information can be a preset gesture for operating the electronic device. For example, it can identify whether the user's gesture is a preset screen-on gesture. When the gesture is identified as a preset screen-on gesture, it controls the electronic device 1000 to turn on the screen. In yet another example, the preset information can be a preset human posture for operating the electronic device 1000. For example, it can identify whether the user's waving gesture is a preset screen-capturing gesture. When the waving gesture is identified as a preset screen-capturing gesture, it controls the electronic device 1000 to capture the screen.

[0086] In some implementations, the spectral response of the identification pixel 204 is narrower than that of the imaging pixel 202.

[0087] In some implementations, the spectral response band of the identification pixel 204 can be pre-designed for different countries of sale and user groups. A narrower spectral response than the imaging pixel 202 allows the light received by the identification pixel 204 to be more targeted, thus avoiding the influence of other light sources (stray light) and improving the signal-to-noise ratio, thereby enhancing the accuracy of identifying races, objects, etc.

[0088] In some implementations, the spectral response of the identification pixel 204 is wider than that of the imaging pixel 202.

[0089] In some implementations, the spectral response bands of the recognition pixel 204 can be pre-designed for different commonly used application scenarios. A wider spectral response for the recognition pixel 204 compared to the imaging pixel 202 can increase the amount of light received, thereby improving recognition accuracy in dark environments.

[0090] Please see Figure 17 In some implementations, the identification pixels 204 are uniformly distributed in the image sensor 200.

[0091] Specifically, the image to be identified obtained by uniformly distributing the recognition pixels 204 in the image sensor 200 can more comprehensively identify whether there are preset faces, preset gestures and preset human postures within the shooting range, thus avoiding omissions.

[0092] Please see Figure 18In some embodiments, the image sensor 200 includes a central region 207 and an edge region 208 surrounding the central region 207, and the density of recognition pixels 204 in the central region 207 is greater than the density of recognition pixels 204 in the edge region 208.

[0093] Specifically, the image sensor 200 may include a center point 209, and the central region 207 may refer to the region whose distance from the center point 209 is less than a preset distance. The central region 207 may be a circular region or a square region, etc. The edge region 208 may be any other region of the image sensor 200 besides the central region 207. Since the density of the central region 207 is greater than the density of the recognition pixels 204 in the edge region 208, the central region 207 can be considered the region of interest. This region of interest will serve as the focus area of ​​the image to be recognized, and the presence of preset information will be periodically detected within this focus area to achieve the AON (Active On-Demand) function.

[0094] It is worth mentioning that the filter array in this embodiment can use filters with other arrangements. Figure 3 In the example shown, the filter is a Bayer array filter in the form of "R, G, G, B". Other embodiments may also include filters in the form of "R, G, B, W", etc., which are not specifically limited here.

[0095] Please see Figure 19 This application also discloses an electronic device 1000, which includes a housing 600 and the aforementioned camera assembly 500.

[0096] Specifically, the electronic device 1000 can be a terminal device equipped with a camera component 500. For example, the electronic device 1000 may include a smartphone, tablet computer, or other terminal device equipped with a camera component 500. The electronic device 1000 can acquire an image to be recognized through an image sensor 200, determine whether preset information exists in the image to be recognized, and when preset information exists in the image to be recognized, control the electronic device 1000 to perform corresponding operations according to the preset information, thereby realizing the AON function.

[0097] In the description of this specification, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0098] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0099] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An image processing pipeline comprising a first image processing pipeline and a second image processing pipeline for an image sensor, characterized in that, The image sensor comprises a pixel array, the pixel array comprises imaging pixels and identification pixels, the identification pixels receive light in a preset period, the image sensor comprises a central region and an edge region surrounding the central region, the density of the identification pixels in the central region is greater than the density of the identification pixels in the edge region; the first image processing pipeline is used to obtain a scene image according to the output signal of the imaging pixels, the second image processing pipeline is used to obtain a to-be-identified image according to the output signal of the identification pixels, judge whether the preset face, the preset gesture or the preset human body posture exists in the to-be-identified image, and control the electronic device to perform corresponding operation when the preset face, the preset gesture or the preset human body posture exists in the to-be-identified image, so as to realize the AON function, and the second image processing pipeline is used to realize the application function based on the scene awareness such as privacy protection, air operation, gaze non-erasing screen and lying non-rotating.

2. The image processing pipeline of claim 1, wherein, The processing amount of the second image processing pipeline for each frame of to-be-identified image is lower than the processing amount of the first image processing pipeline for the scene image.

3. The image processing pipeline of claim 1, wherein, A plurality of identification pixels form an identification pixel combination, and a plurality of identification pixels of each identification pixel combination cover a plurality of color channels.

4. The image processing pipeline of claim 3, wherein, A plurality of identification pixels of each identification pixel combination are arranged adjacent to each other in the image sensor, and the image processing pipeline is used to: merge the output signals of a plurality of identification pixels of each identification pixel combination to obtain a merged pixel; and judge whether the preset information exists in the to-be-identified image according to a plurality of merged pixels.

5. The image processing pipeline of claim 4, wherein, The image processing pipeline is further used to: convert a first pixel bit depth of a plurality of merged pixels into a second pixel bit depth, the second pixel bit depth being smaller than the first pixel bit depth; and judge whether the preset information exists in the to-be-identified image according to a plurality of merged pixels of the second pixel bit depth.

6. The image processing pipeline of claim 1, wherein, The application function based on the scene awareness realized according to the output signal of the identification pixels includes at least one of the following: privacy protection, air operation, gaze non-erasing screen and lying non-rotating.

7. The image processing pipeline of claim 1, wherein, The spectral response of the identification pixels is narrower than the spectral response of the imaging pixels.

8. The image processing pipeline of claim 1, wherein, The spectral response of the identification pixels is wider than the spectral response of the imaging pixels.

9. The image processing pipeline of claim 1, wherein, The proportion of the number of pixels of the identification pixels in the pixel array is less than 5%.

10. The image processing pipeline of claim 1, wherein, The preset period is a default setting value, or is set according to user input.

11. The image processing pipeline of claim 1, wherein, The pixel array comprises a plurality of light-sensitive pixels, and the plurality of light-sensitive pixels are arranged in a Bayer array.

12. The image processing pipeline of claim 1, wherein, The pixel array comprises a plurality of pixel units, each pixel unit comprises a plurality of light-sensitive pixels, the plurality of light-sensitive pixels of the same pixel unit cover the same color channel, and the plurality of pixel units are arranged in a Bayer array.

13. The image processing pipeline of claim 1, wherein, The pixel array comprises a plurality of pixel units, each pixel unit comprises at least one color light-sensitive pixel and at least one panchromatic light-sensitive pixel, and the color light-sensitive pixel has a narrower spectral response than the panchromatic light-sensitive pixel.

14. An image processing method for an image sensor, the method comprising: The image sensor comprises a pixel array, the pixel array comprises imaging pixels and identification pixels, the identification pixels receive light in a preset period, the image sensor comprises a central region and an edge region surrounding the central region, the density of the identification pixels in the central region is greater than the density of the identification pixels in the edge region, the imaging pixels are used to output a scene image, the scene image is used to represent scene information, and the image processing method comprises: obtaining a to-be-identified image according to an output signal of the identification pixels; determining whether a preset face, a preset gesture or a preset human body posture exists in the to-be-identified image; and controlling an electronic device to perform a corresponding operation when the preset face, the preset gesture or the preset human body posture exists in the to-be-identified image, so as to realize an AON function.

15. The image processing method of claim 14, wherein, The image processing method comprises: obtaining the scene image according to an output signal of the imaging pixels.

16. The image processing method of claim 14, wherein, A plurality of identification pixels form an identification pixel combination, and a plurality of identification pixels of each identification pixel combination cover a plurality of color channels.

17. The image processing method of claim 16, wherein, The plurality of identification pixels of each identification pixel combination are arranged adjacent to each other in the image sensor, and the determination of whether the preset information exists in the to-be-identified image comprises: merging output signals of the plurality of identification pixels of each identification pixel combination to obtain a merged pixel; determining whether the preset information exists in the to-be-identified image according to a plurality of merged pixels.

18. The image processing method of claim 17, wherein, The determination of whether the preset information exists in the to-be-identified image comprises: converting a first pixel bit depth of a plurality of merged pixels into a second pixel bit depth, the second pixel bit depth being smaller than the first pixel bit depth; The determination of whether the preset information exists according to a plurality of merged pixels comprises: determining whether the preset information exists in the to-be-identified image according to the plurality of merged pixels with the second pixel bit depth.

19. The image processing method of claim 14, wherein, The pixel array comprises imaging pixels and identification pixels arranged in an array form in two dimensions, and the imaging pixels and the identification pixels are arranged in a Bayer array form.

20. The image processing method of claim 14, wherein, The spectral response of the identification pixels is narrower than the spectral response of the imaging pixels.

21. The image processing method of claim 14, wherein, The spectral response of the identification pixels is wider than the spectral response of the imaging pixels.

22. The image processing method of claim 14, wherein, The proportion of the number of pixels of the identification pixels in the pixel array is less than 5%.

23. The image processing method of claim 14, wherein, The preset period is a default setting value, or is set according to user input.

24. The image processing method of claim 14, wherein, The pixel array comprises a plurality of light-sensitive pixels, and the plurality of light-sensitive pixels are arranged in a Bayer array.

25. The image processing method of claim 14, wherein, The pixel array comprises a plurality of pixel units, each pixel unit comprises a plurality of light-sensitive pixels, the plurality of light-sensitive pixels of the same pixel unit cover the same color channel, and the plurality of pixel units are arranged in a Bayer array.

26. The image processing method of claim 14, wherein, The pixel array comprises a plurality of pixel units, each pixel unit comprises at least one color light-sensitive pixel and at least one panchromatic light-sensitive pixel, and the color light-sensitive pixel has a narrower spectral response than the panchromatic light-sensitive pixel.

27. A camera assembly comprising: The camera assembly comprises an image sensor and an image processing pipeline according to any one of claims 1-13, and the image processing pipeline is used to process an image output by the image sensor.

28. An electronic device, comprising: The electronic device includes a housing and the camera assembly of claim 27, the camera assembly disposed on the housing.

Images