Image signal processors and electronic devices and systems including them.
By using neural network technology to split and correct the light information sensed by the image signal processor, the problem of image quality degradation is solved, and the image quality of the image sensor and the reliability of the electronic device are improved.
Patent Information
- Application Number
- CN202111086001.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-09-17
- Filing Date
- 2021-09-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-09-16
AI Technical Summary
Because the light information sensed by the image sensor passing through the display panel differs from the actual light information, image quality deteriorates, and existing technologies struggle to effectively address this issue.
An image signal processor is used to generate feature values through a first neural network and merge global pixel values. Then, the first and second neural networks are used to split and correct sub-pixel values to improve image quality.
It effectively corrects the light information sensed by the image sensor, improves image quality, reduces flare, haze, blur and ghosting, and enhances the product reliability of electronic devices.
Smart Images

Figure CN114205544B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0119874, filed with the Korean Intellectual Property Office on September 17, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field
[0003] This disclosure relates to an image signal processor and electronic devices and electronic systems including the same. Background Technology
[0004] An image sensing device is a semiconductor device that converts optical information into electrical signals. Examples of image sensing devices may include charge-coupled device (CCD) image sensing devices and complementary metal-oxide-semiconductor (CMOS) image sensing devices.
[0005] A CMOS image sensor (CIS) may include multiple pixels arranged in a two-dimensional manner. Each pixel may include, for example, a photodiode (PD). The photodiode can be used to convert incident light into an electrical signal.
[0006] With the development of the computer and telecommunications industries, the demand for image sensors with improved performance is constantly increasing in various fields such as digital cameras, camcorders, smartphones, gaming devices, security cameras, medical miniature cameras, and robots.
[0007] Recently, image sensors have been installed beneath the display panel of electronic devices. Since image sensors sense light passing through the display panel, the information about the sensed light may differ from the information about the actual light. For example, the quality of an image sensor that senses light that has passed through the display panel may be degraded compared to the quality of an image output by an image sensor that senses light that has not passed through the display panel. Therefore, a method to address this problem is needed. Summary of the Invention
[0008] An image signal processor is provided to improve product reliability by enhancing image quality.
[0009] An electronic device is also provided to improve product reliability by enhancing image quality.
[0010] An electronic system is also provided to improve product reliability by enhancing image quality.
[0011] Additional aspects will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practice of the presented embodiments.
[0012] According to one aspect of this disclosure, an image processing apparatus includes: a memory; and at least one image signal processor configured to: use a first neural network to generate feature values indicating whether to correct global pixel values sensed during a unit frame interval, and generate a feature signal including the feature values; generate an image signal by merging the global pixel values and the feature signal; split the pixel values included in the image signal into first sub-pixel values and second sub-pixel values, split the frame feature signal included in the image signal into first sub-feature values corresponding to the first sub-pixel values and second sub-feature values corresponding to the second sub-pixel values, and generate a first sub-image signal including the first sub-pixel values and the first sub-feature values and a second sub-image signal including the second sub-pixel values and the second sub-feature values; and sequentially correct the first sub-image signal and the second sub-image signal using a second neural network.
[0013] According to one aspect of this disclosure, an electronic device includes: an image sensor configured to sense light passing through a display and output a first image signal including global pixel values sensed during a unit frame interval; and an image signal processor connected to the image sensor and configured to: receive the first image signal; generate a second image signal using a first neural network by correcting the global pixel values included in the first image signal; split the first image signal into a first sub-image signal and a second sub-image signal, and split the second image signal into a third sub-image signal and a fourth sub-image signal; and generate a first signal based on the first sub-image signal and the third sub-image signal using a second neural network, and generate a second signal based on the second sub-image signal and the fourth sub-image signal.
[0014] According to one aspect of this disclosure, an electronic system includes: an electronic device including an image sensor, an image signal processor, and a communication processor; and a data center configured to exchange data with the communication processor, wherein the image sensor is configured to output a first image signal including global pixel values sensed during a unit frame interval, and to provide the first image signal to the image signal processor, wherein the image signal processor is configured to generate a second image signal using a first neural network based on a first weight and the first image signal provided from the image sensor, and to generate a third image signal using a second neural network different from the first neural network based on a second weight, the first image signal, and the second image signal, wherein the communication processor is configured to provide the first weight, the second weight, the first image signal, and the third image signal to the data center, and the data center is configured to generate a third weight obtained by correcting the first weight and a fourth weight obtained by correcting the second weight based on the first weight, the second weight, the first image signal, and the third image signal, and to provide the third weight and the fourth weight to the electronic device. Attached Figure Description
[0015] The above and other aspects, features and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:
[0016] Figure 1 This is a perspective view showing an electronic device according to an embodiment.
[0017] Figure 2 It is according to the embodiment along Figure 1 A cross-sectional view of the electronic device taken by line AA.
[0018] Figure 3 When viewed in the first direction according to the embodiment Figure 1 A perspective view of an electronic device.
[0019] Figure 4 This is a block diagram illustrating an image sensing system according to an embodiment.
[0020] Figure 5 It is used to describe the embodiments. Figure 4 A diagram illustrating the conceptual layout of an image sensor.
[0021] Figure 6 It is a diagram used to describe a pixel array according to some embodiments.
[0022] Figure 7 It is a diagram used to describe the image signal according to the embodiment.
[0023] Figure 8 This is a block diagram illustrating a first image signal processor according to an embodiment.
[0024] Figures 9 to 10 This illustrates an embodiment. Figure 8 A block diagram illustrating the operation of the first corrector and merger.
[0025] Figure 11 It is used to illustrate the embodiments. Figure 8 The flowchart shows the operation of the first corrector and the merger.
[0026] Figure 12 This illustrates an embodiment. Figure 8 A block diagram illustrating the operation of the splitter, the second corrector, and the combiner.
[0027] Figure 13 It is used to illustrate the embodiments. Figure 8 The flowchart shows the operation of the splitter, the second corrector, and the combiner.
[0028] Figure 14 This is a block diagram illustrating a first image signal processor according to an embodiment.
[0029] Figure 15 This illustrates an embodiment. Figure 14 The flowchart shows the operation of the first image signal processor.
[0030] Figure 16 This is a block diagram illustrating a communication system according to an embodiment.
[0031] Figure 17 This illustrates an embodiment. Figure 16 A block diagram illustrating the operation of a communication system.
[0032] Figure 18 This is a block diagram illustrating an image sensing system according to an embodiment.
[0033] Figure 19 This is a block diagram illustrating an image sensing system according to an embodiment.
[0034] Figure 20 This is a block diagram used to describe an electronic device including a multi-camera module according to an embodiment.
[0035] Figure 21 According to the embodiments Figure 20 Detailed block diagram of the camera module. Detailed Implementation
[0036] Hereinafter, embodiments will be described with reference to the accompanying drawings. All embodiments described below are exemplary; therefore, the inventive concept is not limited to these disclosed embodiments and can be implemented in various other forms. Similar reference numerals always denote similar elements.
[0037] As is customary in the art, embodiments can be described and illustrated in terms of blocks that perform the described functions. These blocks (which may be referred to herein as cells or modules, or by names such as correctors, splitters, mergers, combiners, encoders, decoders, networks, etc.) can be physically implemented by analog or digital circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuitry, passive electronic components, active electronic components, optical components, hardwired circuitry, etc., and can be driven by firmware and software. For example, the circuitry can be implemented in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuitry included in a block can be implemented by dedicated hardware or by a processor (e.g., one or more programmable microprocessors and associated circuitry) or by a combination of dedicated hardware performing some functions of the block and a processor performing other functions of the block. The individual blocks of an embodiment can be physically separated into two or more interacting and discrete blocks. Similarly, the blocks of an embodiment can be physically combined into more complex blocks.
[0038] The following will refer to Figures 1 to 3 The description includes an electronic device comprising an image sensor 100.
[0039] Figure 1 This is a perspective view showing an electronic device according to some embodiments. Figure 2 It is along Figure 1 A cross-sectional view of the electronic device taken by line AA. Figure 3 When looking in the first direction Figure 1 A perspective view of an electronic device.
[0040] Reference Figures 1 to 3 The electronic device 1 may include a cover glass 10, a display 20, a rear glass 30, a rear cover 40, an image sensor 100, an image sensor 101, etc.
[0041] In some embodiments, the electronic device 1 may include a housing. For example, the housing may include a cover glass 10 facing a first direction D1 and a rear glass 30 facing a fourth direction opposite to the first direction D1. Furthermore, for example, the housing may include a connection portion for connecting the cover glass 10 to the rear glass 30. The housing protects the internal components of the electronic device 1 from external influences.
[0042] The cover glass 10 may include a transparent material, thereby allowing the electrical information signals displayed by the display 20 to be identified from the outside. For example, the cover glass 10 may include glass or plastic.
[0043] The cover glass 10 and the rear glass 30 may have a flat shape. For example, the length of each of the cover glass 10 and the rear glass 30 in the second direction D2 and the third direction D3 may be greater than its length or thickness in the first direction D1. Therefore, the electronic device 1 may have a flat shape. However, the embodiments are not limited to this.
[0044] The surface of electronic device 1 in the first direction D1 can be the front surface, and the surface of electronic device 1 in the fourth direction D4 can be the rear surface. However, this is exemplary, and the embodiments are not limited.
[0045] Reference Figure 1 and Figure 2 The cover glass 10 may cover the display 20. For example, the display 20 may be positioned on the fourth direction D4 from the cover glass 10.
[0046] Display 20 may include a plurality of pixels arranged along rows and columns. For example, display 20 may include organic light-emitting diode display (OLED), liquid crystal display (LCD), plasma display panel (PDP), electrochromic display (ECD), digital mirror device (DMD), actuator mirror device (AMD), grating light valve (GLV) device, electroluminescent display (ELD), etc.
[0047] The display 20 may include a plurality of display holes 20H. A display hole 20H may be a hole passing through the display 20 in a first direction D1. For example, pixels of the display 20 may not be disposed in the display holes 20H. However, the embodiments are not limited thereto.
[0048] Display 20 can output images provided from electronic device 1. For example, signals are sent to multiple pixels so that display 20 can output corresponding images. For example, display 20 may include a touch screen or touch panel that displays images or receives touch input from a user.
[0049] Image sensor 100 can sense light incident on display 20 from outside the electronic device 1 or a portion of light output from display 20. For example, image sensor 100 may be covered by display 20 and can sense light incident on outside the electronic device 1. For example, image sensor 100 can sense light output from display 20 and reflected by cover glass 10. Furthermore, image sensor 100 can sense light that has passed through display aperture 20H. For example, image sensor 100 can sense light passing through cover glass 10 and pixels of display 20, as well as the gaps between pixels of display 20.
[0050] Therefore, the image generated by sensing light passing through the display 20 may be distorted due to the cover glass 10, the display aperture 20H, and the pixels of the display 20. For example, light incident through the display aperture 20H may be diffracted, and the image generated by sensing the diffracted light may be distorted. For example, the distorted image may be an image with flare, haze, blur, or ghosting. However, the embodiments are not limited to this.
[0051] The image sensor 100 may be covered by the display 20 and surrounded by the back cover 40. For example, the image sensor 100 may be disposed in an opening formed in the back cover 40.
[0052] The back cover 40 prevents light generated inside the electronic device 1 from affecting the display 20. Furthermore, the back cover 40 prevents light emitted from the display 20 from entering the interior of the electronic device 1.
[0053] Reference Figure 3 The rear glass 30 may be disposed on the rear surface of the electronic device 1 (e.g., the surface of the electronic device 1 in the fourth direction D4). Furthermore, the image sensor 101 may be disposed on the rear surface of the electronic device 1 (e.g., the surface of the electronic device 1 in the fourth direction D4). However, the embodiment is not limited thereto; the image sensor 101 may be disposed on the side surface or the front surface.
[0054] Although only one image sensor 101 is shown, in embodiments, image sensor 101 may include multiple image sensors. For example, image sensor 101 may include a conventional camera, a wide-angle camera, and a telephoto camera. The rear glass 30 prevents light from entering the electronics 1 from the outside. That is, it prevents light from entering the portion other than the opening where image sensor 101 is located.
[0055] The following will refer to Figures 4 to 7 The description includes an image sensing system 2 comprising an image sensor 100, a pixel array 140, a first image signal processor 200, and an application processor 300.
[0056] Figure 4 This is a block diagram illustrating an image sensing system according to some embodiments.
[0057] Reference Figure 4 The image sensing system 2 may include an image sensor 100, a first image signal processor 200, and an application processor 300.
[0058] Image sensor 100 can generate an image signal IS by sensing an image of a target using incident light. In some embodiments, the generated image signal IS can be, for example, a digital signal, but the embodiments are not limited thereto.
[0059] The image signal IS can be provided and processed by the first image signal processor 200. For example, the first image signal processor 200 can process the image signal IS to generate the image signal IS.
[0060] Application processor 300 may receive image signal IS” from first image signal processor 200. Application processor 300 may modify or process image signal IS” to make it suitable for display. For example, second image signal processor 310 included in application processor 300 may perform image processing on image signal IS”.
[0061] Application processor 300 may include neural processor 320, graphics processor 330, etc. In this configuration, neural processor 320 may perform processing such as neural networks, machine learning, or deep learning, and graphics processor 330 may perform graphics processing. Neural processor 320 and graphics processor 330 may exchange signals with first image signal processor 200. For example, neural processor 320 and graphics processor 330 may receive data from first image signal processor 200 and may output processed data. Subsequently, neural processor 320 and graphics processor 330 may return the processed data to first image signal processor 200. However, embodiments are not limited thereto, and neural processor 320 and graphics processor 330 may be included within first image signal processor 200.
[0062] In some embodiments, the image sensor 100, the first image signal processor 200, and the application processor 300 may be disposed separately as shown. For example, the image sensor 100 and the first image signal processor 200 may be mounted on a first chip, and the application processor 300 may be mounted on a second chip so that they can communicate via an interface. However, the embodiments are not limited thereto, and the image sensor 100, the first image signal processor 200, and the application processor 300 may be implemented in a single package (e.g., a multi-chip package (MCP)).
[0063] The image sensor 100 may include a control register block 110, a timing generator 120, a line driver 130, a pixel array 140, a readout circuit 150, a ramp signal generator 160, and a buffer 170.
[0064] Control register block 110 controls the overall operation of image sensor 100. Specifically, control register block 110 can directly send operation signals to timing generator 120, ramp signal generator 160, and buffer 170.
[0065] Timing generator 120 can generate reference signals for the timing of operation of various components of image sensor 100. The reference signals for timing of operation generated by timing generator 120 can be sent to line driver 130, readout circuit 150, ramp signal generator 160, etc.
[0066] The ramp signal generator 160 can generate and transmit a ramp signal used in the readout circuit 150. For example, the readout circuit 150 may include a correlated double sampler (CDS), a comparator, etc., and the ramp signal generator 160 can generate and transmit a ramp signal used in the correlated double sampler (CDS), the comparator, etc.
[0067] The buffer 170 may include, for example, a latch. The buffer 170 may temporarily store the image signal IS to be provided to the outside and send the image signal IS to an external memory or external device.
[0068] Pixel array 140 can sense external images. Pixel array 140 may include multiple pixels (or unit pixels). Row driver 130 can selectively activate one row of pixel array 140.
[0069] The readout circuit 150 can sample the pixel signal provided from the pixel array 140, compare it with the ramp signal, and convert the analog image signal (or, for example, data of the analog image signal) into a digital image signal (or, for example, data of the digital image signal) based on the comparison result.
[0070] Figure 5 It is used to describe Figure 4 A diagram illustrating the conceptual layout of an image sensor.
[0071] Reference Figure 5 The image sensor 100 may include a first region R1 and a second region R2 stacked in a first direction D1 (e.g., the vertical direction). As shown, the first region R1 and the second region R2 may extend in a second direction D2 and a third direction D3 intersecting the first direction, and Figure 4 The blocks shown can be disposed in the first region R1 and the second region R2. In an embodiment, the first region R1 and the second region R2 can be stacked in other directions, and the first region R1 and the second region R2 can extend in other directions.
[0072] In one embodiment, the memory may be disposed in a third region, and the third region may be located below the second region R2. In this case, the memory disposed in the third region can receive image data from the first region R1 and the second region R2, store or process the image data, and resend the image data to the first region R1 and the second region R2. In this case, the memory may include memory elements such as dynamic random access memory (DRAM), static random access memory (SRAM), spin-torque magnetic random access memory (STT-MRAM), and flash memory. When the memory includes, for example, DRAM, the memory can receive and process image data at a relatively high speed. Furthermore, in some embodiments, the memory may be disposed in the second region R2.
[0073] The first region R1 may include a pixel array PA and a first peripheral region PH1, and the second region R2 may include a logic circuit region LC and a second peripheral region PH2. The first region R1 and the second region R2 may be stacked vertically in sequence.
[0074] In the first region R1, the pixel array PA can be compared with a reference. Figure 4 The pixel array 140 described is the same. The pixel array PA may include multiple unit pixels arranged in a matrix. Each pixel may include a photodiode and a transistor.
[0075] The first peripheral region PH1 may include multiple pads and may be positioned around the pixel array PA. The pads may send / receive electrical signals to / from external devices.
[0076] In the second region R2, the logic circuit region LC may include electronic components having multiple transistors. The electronic components included in the logic circuit region LC may be electrically connected to the pixel array PA to provide constant signals or control output signals to individual pixels of the pixel array PA.
[0077] In the logic circuit region LC, for example, a reference can be set. Figure 4The described components include control register block 110, timing generator 120, line driver 130, readout circuit 150, ramp signal generator 160, and buffer 170. For example, in... Figure 4 In the block, blocks other than pixel array 140 can be set in the logic circuit area LC.
[0078] In addition, in the second region R2, the second peripheral region PH2 can be set in the region corresponding to the first peripheral region PH1 of the first region R1, but the embodiments are not limited to this.
[0079] Figure 6 It is a diagram used to describe a pixel array according to some embodiments.
[0080] Reference Figure 6 A pixel array PA may include multiple unit pixels PX. The multiple unit pixels PX can be arranged in two dimensions. For example, the multiple unit pixels PX can be arranged repeatedly in a second direction and a third direction. The unit pixels PX can be arranged at regular intervals. For example, the pixel array PA can be arranged in a Bayer pattern. However, the embodiments are not limited to this; the pixel array PA can be arranged in a tetra pattern, a nona pattern, etc.
[0081] Figure 7 It is a diagram used to describe image signals according to some embodiments.
[0082] Reference Figure 7 The image signal IS can be a signal output by the image sensor 100 sensing light from the pixel array PA. For example, light can pass through the color filter of the pixel array 140 to reach the photodiode, and the image signal IS can be output from the logic circuit region LC.
[0083] For example, the image signal IS may include first green pixel values G1 to eighth green pixel values G8 output by sensing light that has passed through a green color filter. Furthermore, the image signal IS may include first blue pixel values B1 to fourth blue pixel values B4 output by sensing light that has passed through a blue color filter. Additionally, the image signal IS may include first red pixel values R1 to fourth red pixel values R4 output by sensing light that has passed through a red color filter.
[0084] Right now, Figure 7 The first green pixel values G1 to the eighth green pixel values G8, the first blue pixel values B1 to the fourth blue pixel values B4, and the first red pixel values R1 to the fourth red pixel values R4 shown can generate an image signal IS output by the image sensor 100 by sensing light that has passed through each color filter with a corresponding color.
[0085] The pixel array 140 can be arranged in a red-green-blue (RGB) Bayer pattern. That is, the color filters of the pixel array 140 can be a combination of red, green and blue color filters, and can be arranged in a Bayer pattern. However, the embodiments are not limited to this, and the pixel array 140 can be an RGB quaternary pattern, an RGB nine-element pattern, etc.
[0086] In the image signal IS, such as Figure 7 As shown, pixel values can be arranged to correspond to the colors of the color filters in pixel array 140. However, Figure 7 This is merely an example of arranging pixel values according to the position of each unit pixel PX. The actual storage location of the pixel values of the output image signal IS is not limited to the positions shown.
[0087] Figure 8 This is a block diagram illustrating a first image signal processor according to some embodiments. Figures 9 to 10 It is shown Figure 8 A block diagram illustrating the operation of the first corrector and merger. Figure 11 It is used for explanation Figure 8 The flowchart shows the operation of the first corrector and the merger.
[0088] Reference Figure 8 The first image signal processor 200 may include a first corrector 210, a combiner 220, a splitter 230, a second corrector 240, and a combiner 250. The first corrector 210, the combiner 220, the splitter 230, the second corrector 240, and the combiner 250 may be connected in this order, but the embodiments are not limited thereto.
[0089] The first corrector 210 can receive an image signal IS from the image sensor 100 and can generate feature values indicating whether to correct pixel values based on the received image signal IS. Furthermore, the first corrector 210 can provide a feature signal FS including the generated feature values to the merger 220.
[0090] Reference Figure 9 The image signal IS can be Figure 7 The image signal IS. For example, the image signal IS may include the image signal IS transmitted via... Figure 6 The pixel values sensed by the pixel array PA. For example, the image signal IS may include first green pixel values G1 to eighth green pixel values G8, first blue pixel values B1 to fourth blue pixel values B4, and first red pixel values R1 to fourth red pixel values R4.
[0091] In this configuration, the image signal IS may include global pixel values sensed during a unit frame interval. In some embodiments, the global pixel values may include all pixel values sensed by the image sensor 100. For example, the global pixel values may include all pixel values sensed during a unit frame interval. However, embodiments are not limited to this; the global pixel values may include only some pixel values sensed during a unit frame interval. For example, the global pixel values may include pixel values sampled by an encoder, etc.
[0092] Furthermore, the image signal IS may include local pixel values as part of the global pixel values. Local pixel values may be pixel values generated only from a portion of the pixel array PA. In an embodiment, local pixel values may be pixel values sampled from pixel values generated by the pixel array PA.
[0093] Assuming the image signal IS includes all pixel values sensed during a unit frame interval, if the image sensor 100 generates the image signal IS at 60 frames per second (fps), then the image signal IS may include all pixel values sensed within 1 / 60th of a second (which may be a unit frame interval corresponding to 60 fps). That is, the image signal IS may include the first green pixel values G1 to the eighth green pixel values G8, the first blue pixel values B1 to the fourth blue pixel values B4, and the first red pixel values R1 to the fourth red pixel values R4 sensed during a unit frame interval. However, the embodiment is not limited to this; the image signal IS may include some of the first green pixel values G1 to the eighth green pixel values G8, the first blue pixel values B1 to the fourth blue pixel values B4, and the first red pixel values R1 to the fourth red pixel values R4 sensed during a unit frame interval.
[0094] In one embodiment, the first image signal processor 200 may include a frame memory. The frame memory may be connected to the first corrector 210 and may receive image signals IS from the image sensor 100 for temporary storage. In this case, the frame memory may store pixel values sensed during a unit frame interval. The frame memory may then provide the temporarily stored image signals IS to the first corrector 210. However, the embodiment is not limited to this; the first image signal processor 200 may include row memory or block memory instead of a frame memory.
[0095] Return to reference Figure 9 The first corrector 210 may include an encoder 211, a first neural network 212, and a decoder 213.
[0096] Encoder 211 can receive image signal IS. (See reference...) Figure 11The encoder 211 can downscale the image signal IS in operation S300. The encoder 211 can generate the downscaled image signal IS_D by encoding the image signal IS. For example, the encoder 211 can generate the downscaled image signal IS_D by sampling.
[0097] For example, encoder 211 can encode first green pixel values G1 to eighth green pixel values G8, first blue pixel values B1 to fourth blue pixel values B4, and first red pixel values R1 to fourth red pixel values R4 to generate reduced pixel values Gd1, Rd, Bd, and Gd2. In this case, since the first green pixel values G1 to eighth green pixel values G8, the first blue pixel values B1 to fourth blue pixel values B4, and the first red pixel values R1 to fourth red pixel values R4 are global pixel values sensed during a unit frame interval, and the reduced pixel values Gd1, Rd, Bd, and Gd2 are generated by reducing the global pixel values, the reduced pixel values Gd1, Rd, Bd, and Gd2 can correspond to the global pixel values sensed during a unit frame interval. Alternatively, the reduced pixel values Gd1, Rd, Bd, and Gd2 can be local pixel values sensed during a unit frame interval.
[0098] The first neural network 212, along with the neural processor 320 and graphics processor 330 performing the neuralization, may have memory limitations when processing images. For example, the processing performance of the first neural network 212 may degrade when it processes an image signal IS that includes global pixel values sensed during a unit frame interval without altering it (e.g., without downscaling). Therefore, the processing performance of the first neural network 212 can be further enhanced when the image signal IS is downscaled or reduced by encoding through the encoder 211.
[0099] The first neural network 212 can receive a scaled-down image signal IS_D. The first neural network 212 may include multiple convolutional layers 212L. (See reference...) Figure 11 In operation S301, the first corrector 210 can extract the original feature signal OF from the image signal IS through the first neural network 212. The first neural network 212 can use the convolutional layer 212L to process the scaled-down image signal IS_D, and as a result, can output the original feature signal OF.
[0100] For example, the scaled pixel values Gd1, Rd, Bd, and Gd2 included in the scaled image signal IS_D can be passed through a convolutional layer 212L to output first feature values F1 to fourth feature values F4. In this case, first feature values F1 to fourth feature values F4 can be extracted from the scaled pixel values Gd1, Rd, Bd, and Gd2.
[0101] The first feature value F1 to the fourth feature value F4 may correspond to the reduced pixel values Gd1, Rd, Bd, and Gd2. For example, the first feature value F1 to the fourth feature value F4 may include feature values indicating whether the reduced pixel values Gd1, Rd, Bd, and Gd2 are corrected. For example, the first feature value F1 may represent the probability that the reduced pixel value Gd1 includes flares or haze. For example, if the reduced pixel value Gd1 includes flares, then the first feature value F1 may be 1. Alternatively, if the reduced pixel value Gd1 does not include flares, then the first feature value F1 may be 0. However, the embodiments are not limited to this.
[0102] Decoder 213 can receive the raw feature signal OF. (See reference...) Figure 11 Decoder 213 can upscale the original feature signal OF in operation S302. Decoder 213 can decode the original feature signal OF to amplify it, thereby generating the feature signal FS. For example, decoder 213 can use interpolation to generate the feature signal FS.
[0103] For example, decoder 213 can generate an amplified feature signal FS by decoding the first feature value F1 to the fourth feature value F4. In this case, the feature values of the feature signal FS may include the first feature value F1 to the fourth feature value F4, but the embodiment is not limited thereto.
[0104] The feature signal FS can have the same size as the image signal IS. That is, the decoder 213 can generate a feature signal FS with the same size as the image signal IS by adjusting the size of the original feature signal OF. In other words, the feature values of the feature signal FS can correspond to the global pixel values of the image signal IS sensed during a unit frame interval.
[0105] Through encoding by encoder 211 in the first corrector 210, feature value extraction by first neural network 212, and decoding by decoder 213, a feature signal FS can be generated, including feature values indicating whether flare or haze needs to be corrected for processing the entire image signal IS. Therefore, the feature signal FS can be generated efficiently without being limited by the performance of the first neural network 212, neural processor 320, and graphics processor 330.
[0106] Reference Figure 10 The combiner 220 can receive the feature signal FS and the image signal IS. (See reference...) Figure 11 In operation S303, the merger 220 can merge the feature signal FS and the image signal IS. That is, the merger 220 can output an image signal IS' which is a combination of the feature signal FS and the image signal IS.
[0107] The merger 220 allows the feature values of the feature signal FS to correspond to the pixel values of the image signal IS. For example, the first green pixel values G1 to the eighth green pixel values G8, the first blue pixel values B1 to the fourth blue pixel values B4, and the first red pixel values R1 to the fourth red pixel values R4 of the image signal IS can correspond to the first pixel values F1 to the fourth pixel values F4 of the feature signal FS. In this case, since the image signal IS has the same size as the feature signal FS, each pixel value can correspond to each feature value. However, the embodiment is not limited to this, and the merger 220 can merge the image signal IS and the feature signal FS in a simpler way.
[0108] Figure 12 It is shown Figure 8 A block diagram illustrating the operation of the splitter, the second corrector, and the combiner. Figure 13 It is used for explanation Figure 8 The flowchart shows the operation of the splitter, the second corrector, and the combiner.
[0109] Reference Figure 12 and Figure 13 The splitter 230 can receive the image signal IS'. During operation S310, the splitter 230 can split the image signal IS' into split image signals IS'_S and output them. As described above, the image signal IS' may include global pixel values sensed during a unit frame interval and feature values indicating whether to correct the global pixel values sensed during the unit frame interval. The splitter 230 can split the global pixel values sensed during the unit frame interval into different groups of sub-pixel values. That is, each group of sub-pixel values may include local pixel values. Additionally, the splitter 230 can split the feature values indicating whether to correct the global pixel values sensed during the unit frame interval into different groups of sub-feature values.
[0110] For example, pixel values included in image signal IS' can be split into first sub-pixel values SP1 and second sub-pixel values SP2. Similarly, feature values included in image signal IS' can be split into first sub-feature values SF1 and second sub-feature values SF2.
[0111] The splitter 230 can output a first split image signal IS'_S1 including a first sub-pixel value SP1 and a first sub-feature value SF1, and a second split image signal IS'_S2 including a second sub-pixel value SP2 and a second sub-feature value SF2. In this case, the first sub-pixel value SP1 can correspond to the first sub-feature value SF1, and the second sub-pixel value SP2 can correspond to the second sub-feature value SF2.
[0112] The splitter 230 can sequentially provide the first split image signal IS'_S1 and the second split image signal IS'_S2 to the second corrector 240. Since the image signal IS' includes global pixel values sensed during a unit frame interval and feature values indicating whether to correct pixel values, its capacity can be large, and the second corrector 240 can have memory limitations during processing. Therefore, by sequentially providing the first split image signal IS'_S1 and the second split image signal IS'_S2 to the second corrector 240 after splitting, the performance of the image signal processor 200 can be enhanced.
[0113] The second corrector 240 can perform correction based on the sequentially provided first split image signal IS'_S1 and second split image signal IS'_S2. In operation S311, the second corrector 240 can output a corrected image signal IS'_C based on sub-pixel values and sub-feature values via a second neural network. For example, the second corrector 240 can output a first corrected image signal IS'_C1 based on the first sub-pixel value SP1 and the first sub-feature value SF1 included in the first split image signal IS'_S1. Similarly, the second corrector 240 can output a second corrected image signal IS'_C2 based on the second sub-pixel value SP2 and the second sub-feature value SF2 included in the second split image signal IS'_S2. In this case, the first corrected image signal IS'_C1 and the second corrected image signal IS'_C2 can be output sequentially.
[0114] Furthermore, the second corrector 240 can use the first sub-pixel value SP1 and the first sub-feature value SF1 to perform correction for at least one of flare, haze, blur, and ghosting on the first sub-pixel value SP1. Additionally, the second corrector 240 can use the second sub-pixel value SP2 and the second sub-feature value SF2 to perform correction for at least one of flare, haze, blur, and ghosting on the second sub-pixel value SP2. However, the embodiments are not limited thereto.
[0115] In this scenario, the pixel values processed by the first corrector 210 may involve information about the entire image (e.g., flares or haze), while the pixel values processed by the second corrector 240 may involve information about a portion of the image (e.g., blur or ghosting). That is, the first corrector 210 can process global pixel values sensed during a unit frame interval, while the second corrector 240 can perform correction by processing split pixel values and split feature values.
[0116] In operation S312, the first image signal processor 200 can determine whether all image signals have been corrected. For example, if the correction of global subpixel values is complete (operation S312 is yes), then in operation S313, the combiner 250 can combine the corrected image signals IS'_C. For example, the combiner 250 can combine the first corrected image signal IS'_C1 and the second corrected image signal IS'_C2. The combiner 250 can output the image signal IS'' obtained by combining the first corrected image signal IS'_C1 and the second corrected image signal IS'_C2. In an embodiment, if not all image signals have been corrected (operation S312 is no), operation S311 can be repeated.
[0117] In operation S314, combiner 250 can send the image signal IS” to application processor 300. (See reference...) Figure 4 The image signal IS” can be sent to a second image signal processor 310 included in the application processor 300. In this case, the image signal IS” may include an image corrected for flare, haze, blur, and / or ghosting. The second image signal processor 310 can perform image processing on the image signal IS”. For example, the second image signal processor 310 can perform black level correction, defect pixel correction, shadow correction, automatic exposure (AE) correction, automatic white balance (AWB) correction, etc. on the image signal IS”. In addition, the second image signal processor 310 can perform de-mosaicing, edge enhancement, gamma correction, etc. on the image signal IS”.
[0118] The first image signal processor 200 can output an image signal IS with reduced image quality degradation, such as flare, haze, blur, or ghosting. Therefore, an electronic device 1 and an image sensing system can be provided to improve product reliability by reducing image quality degradation.
[0119] The following will refer to Figure 14 and Figure 15 An example of a first image signal processor 200 according to some embodiments is described.
[0120] Figure 14 This is a block diagram illustrating a first image signal processor according to some embodiments. Figure 15 It is shown Figure 14 A flowchart of the operation of the first image signal processor. For simplicity, details regarding... Figures 1 to 13 Some redundant descriptions.
[0121] Reference Figure 14 The first image signal processor 200 may include a memory 260. The memory 260 may be connected to the first corrector 210. In addition, the memory 260 may be connected to the image sensor 100 to receive image signals IS.
[0122] Reference Figure 14 and Figure 15 During operation S320, the first image signal processor 200 can determine whether the electronic device 1 requires or is using real-time processing. For example, video recording may correspond to real-time processing, but taking a picture may not.
[0123] When electronic device 1 requires real-time processing (operation S320 is yes), in operation S321, memory 260 can store image signal IS, and first image signal processor 200 can perform image processing on image signal IS. For example, image processing on image signal IS can be performed using only combiner 220, splitter 230, second corrector 240, and combiner 250, without first corrector 210. In this case, image signal IS stored in memory 260 may include global pixel values sensed during unit frame intervals.
[0124] When electronic device 1 does not require real-time processing (operation S320 is not), in operation S324, the first image signal processor 200 can use the image signal IS to perform correction.
[0125] After processing of the image signal IS is completed, in operation S322, the first image signal processor 200 can perform correction on the stored image signal. For example, the memory 260 can provide the previous frame image signal IS_PF to the first corrector 210. The first corrector 210 can then provide the feature signal FS to the merger 220 based on this signal.
[0126] In operation S323, the merger 220 can merge the feature signal FS generated based on the previous frame image signal IS_PF with the image signal IS provided during the next unit frame interval. (Refer to reference...) Figures 1 to 13 Unlike the first image signal processor 200 described, the merger 220 in this embodiment can merge the feature signal FS based on the previous frame with the image signal IS based on the current frame. Therefore, in the case of video recording rather than taking a picture, the first image signal processor 200 can correct the image signal IS based on the previous frame image signal IS_PF and the image signal IS.
[0127] The following will refer to Figure 16 and Figure 17 A communication system 3 is described according to some embodiments.
[0128] Figure 16 This is a block diagram illustrating a communication system according to some embodiments. Figure 17 It is shown Figure 16 A block diagram illustrating the operation of the communication system. For simplicity, details regarding... Figures 1 to 13 Some redundant descriptions.
[0129] Reference Figure 16 The communication system 3 may include an electronic device 1 and a data center 400. The electronic device 1 may include a communication processor 340. The data center 400 may include a neural network 410 and a memory 420. The electronic device 1 may exchange signals with the data center 400 through the communication processor 340.
[0130] The first corrector 210 may use the first weight W1 to extract the feature signal FS. For example, the first neural network 212 may use the first weight W1. The second corrector 240 may use the second weight W2 to generate the corrected image signal IS'_C. For example, the second neural network of the second corrector 240 may use the second weight W2.
[0131] The application processor 300 can receive a first weight W1 from the first corrector 210, a second weight W2 from the second corrector 240, and image signals IS and IS' from the first image signal processor 200.
[0132] In operation S330, application processor 300 may determine whether the image corrected by first image signal processor 200 has been properly corrected. For example, application processor 300 may determine whether image signal IS” includes the properly corrected image based on the received image signals IS and IS”. That is, application processor 300 may compare image signal IS with image signal IS”.
[0133] When the corrected image signal IS” is inappropriate (operation S330 is not enabled), in operation S331, electronic device 1 can send the image signal and weights to data center 400. For example, application processor 300 can send the image signals IS and IS”, as well as the first weight W1 and the second weight W2, to communication processor 340. Communication processor 340 can send the image signals IS and IS”, as well as the first weight W1 and the second weight W2, to data center 400 via an antenna.
[0134] The memory 420 of the data center 400 can store the received image signals IS and IS”, as well as the first weight W1 and the second weight W2. In operation S332, the data center 400 can generate a first weight W1' and a second weight W2' that have been corrected based on the image signals IS and IS” and the first weight W1 and the second weight W2 through the neural network 410.
[0135] In operation S333, the data center 400 can provide the corrected first weight W1' and second weight W2' to the electronic device 1. The communication processor 340 can send the received first weight W1' and second weight W2' to the application processor 300. The application processor 300 can send the first weight W1' to the first corrector 210 and the second weight W2' to the second corrector 240.
[0136] Through this operation, the first weight W1 and the second weight W2 can be updated in operation S334 using the received first weight W1' and second weight W2'. That is, when the output image of the first image signal processor 200 applying the first weight W1 and the second weight W2 is inappropriate, the first weight W1 and the second weight W2 can be updated using the first weight W1' and the second weight W2'. Therefore, image quality degradation can be reduced, thus providing an electronic device 1 and a communication system 3 including the same with improved product reliability.
[0137] When the corrected image signal IS” is appropriate (operation S330 is yes), operations S331 to S334 can be skipped or not performed.
[0138] The following will refer to Figure 18 An image sensing system 4 is described according to some embodiments.
[0139] Figure 18 This is a block diagram illustrating an image sensing system according to some embodiments. For the sake of simplicity, details regarding... Figures 1 to 13 Redundant description.
[0140] Reference Figure 18 Application processor 300 can receive image signal IS from image sensor 100. For example, second image signal processor 310 can receive image signal IS from image sensor 100. Second image signal processor 310 can perform correction on image signal IS and can provide corrected image signal IS_P to first image signal processor 200. (Refer to reference...) Figures 1 to 13 Unlike the image sensing system 2 described, the first image signal processor 200 of the image sensing system 4 can receive the image signal IS_P that has been corrected by the second image signal processor 310.
[0141] The first image signal processor 200 can return the image signal IS” obtained by correcting the image signal IS_P to the second image signal processor 310.
[0142] The following will refer to Figure 19 The image sensing system 5 is described according to some other embodiments.
[0143] Figure 19 This is a block diagram illustrating an image sensing system according to some embodiments. For the sake of simplicity, details regarding... Figures 1 to 13 Some redundant descriptions.
[0144] Reference Figure 19 The image sensing system 5 may include an image sensor 100 and an application processor 300. In this case, the application processor 300 may include a first image signal processor 200. That is, compared with the reference... Figures 1 to 13 The image sensing system 2, which is separate from the first image signal processor 200 and the application processor 300, is different in that the first image signal processor 200 may be included in the application processor 300 in the image sensing system 5.
[0145] The following will refer to Figure 20 and Figure 21 The electronic device 6 is described according to some other embodiments.
[0146] Figure 20 This is a block diagram used to describe an electronic device including a multi-camera module according to some embodiments. Figure 21 yes Figure 20 A detailed block diagram of the camera module. For simplicity, details regarding... Figures 1 to 19 Some redundant descriptions.
[0147] Reference Figure 20 The electronic device 6 may include a camera module group 1100, an application processor 1200, a power management integrated circuit (PMIC) 1300, an external memory 1400, and a display 1500.
[0148] Camera module group 1100 may include multiple camera modules 1100a, 1100b, and 1100c. Although the accompanying drawings illustrate an embodiment with three camera modules 1100a, 1100b, and 1100c, the embodiments are not limited thereto. In some embodiments, camera module group 1100 may be modified to include only two camera modules. Additionally, in some embodiments, camera module group 1100 may be modified to include n camera modules, where n can be a natural number greater than 3.
[0149] In this embodiment, one of the three camera modules 1100a, 1100b, and 1100c may be a reference camera module. Figures 1 to 19 The image sensor 100 described is a camera module. Alternatively, one of the three camera modules 1100a, 1100b, and 1100c may include the referenced... Figures 1 to 19 The image sensor 100 and the camera module of the first image signal processor 200 are described. In an embodiment, one of the three camera modules 1100a, 1100b, and 1100c may be a camera module including a reference. Figures 1 to 3 The image sensor 101 described is a camera module.
[0150] The following will refer to Figure 21 The detailed configuration of camera module 1100b is described below. The following description can also be applied to other camera modules 1100a and 1100c according to the embodiments.
[0151] Reference Figure 21The camera module 1100b may include a prism 1105, an optical path folding element (hereinafter referred to as "OPFE") 1110, an actuator 1130, an image sensing unit 1140, and a storage unit 1150.
[0152] The prism 1105 may include a reflective surface 1107 having a light-reflecting material, and change the path of light L incident from the outside.
[0153] In some embodiments, prism 1105 can change the path of light L incident in direction X to a direction Y perpendicular to direction X. Furthermore, prism 1105 can rotate the reflective surface 1107, which has a light-reflecting material, relative to the central axis 1106 in direction A, or rotate the central axis 1106 in direction B, thereby changing the path of light L incident in direction X to a direction Y perpendicular to it. In this case, OPFE 1110 can also move in direction Z, which is perpendicular to both directions X and Y.
[0154] In some embodiments, as shown in the figure, the maximum rotation angle of prism 1105 in direction A may be 15 degrees or less in the positive (+)A direction and greater than 15 degrees in the negative (-)A direction. However, the embodiments are not limited thereto.
[0155] In some embodiments, the prism 1105 may be moved between approximately 20 degrees, 10 and 20 degrees, or 15 and 20 degrees in the positive (+)B direction or the negative (-)B direction. In this case, the angle of movement may be the same in the positive (+)B direction or the negative (-)B direction, or may be almost the same in the positive (+)B direction or the negative (-)B direction, with a difference of approximately 1 degree.
[0156] In some embodiments, the prism 1105 may move the reflective surface 1107, which has a light-reflecting material, in a third direction (e.g., direction Z) parallel to the extension direction of the central axis 1106.
[0157] OPFE 1110 may include, for example, m optical lenses, where m may be a natural number. The m lenses may be moved in a second direction Y to change the optical zoom ratio of camera module 1100b. For example, if we assume that the basic optical zoom ratio of camera module 1100b is Z, then as the m optical lenses included in OPFE 1110 move, the optical zoom ratio of camera module 1100b may be 3Z, 5Z or greater.
[0158] Actuator 1130 can move an optical lens or OPFE 1110 (hereinafter referred to as "optical lens") to a specific position. For example, actuator 1130 can adjust the position of the optical lens so that image sensor 1142 can be positioned at the focal length of the optical lens for accurate sensing.
[0159] The image sensing unit 1140 may include an image sensor 1142, control logic 1144, and memory 1146. The image sensor 1142 may use light L provided by an optical lens to sense an image of the target. In some embodiments, the image sensor 1142 may include the image sensor 100 described above.
[0160] Control logic 1144 can control the overall operation of camera module 1100b. For example, control logic 1144 can control the operation of camera module 1100b according to control signals provided via control signal line CSLb.
[0161] Memory 1146 may store information required for the operation of camera module 1100b (e.g., calibration data 1147). Calibration data 1147 may include information required by camera module 1100b to generate image data using light L provided from an external source. For example, calibration data 1147 may include information about the optical axis, information about the degree of rotation, and information about the aforementioned focal length. When camera module 1100b is implemented as a multi-mode camera where the focal length varies depending on the position of the optical lens, calibration data 1147 may include information about autofocus and focal length values for each position (or state) of the optical lens.
[0162] Storage unit 1150 can store image data sensed by image sensor 1142. Storage unit 1150 may be disposed outside image sensing unit 1140 and may be implemented in a manner that stacks with a sensor chip included in image sensing unit 1140. In some embodiments, storage unit 1150 may be implemented as electrically erasable programmable read-only memory (EEPROM), but embodiments are not limited thereto.
[0163] Refer to together Figure 20 and Figure 21 In some embodiments, each of camera modules 1100a, 1100b, and 1100c may include an actuator 1130. Therefore, camera modules 1100a, 1100b, and 1100c may each include the same or different calibration data 1147 depending on the operation of the actuator 1130 included therein.
[0164] In some embodiments, one of the camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100b) may be a folding lens type camera module including the aforementioned prism 1105 and OPFE 1110, and the remaining camera modules (e.g., camera modules 1100a and 1100c) may be vertical type camera modules excluding the prism 1105 and OPFE 1110. However, the embodiments are not limited thereto.
[0165] In some embodiments, one of the camera modules 1100a, 1100b, and 1100c (camera module 1100c) may be a vertical depth camera that extracts depth information using, for example, infrared (IR). In this case, the application processor 1200 may merge image data provided from the depth camera with image data provided from another camera module (e.g., camera module 1100a or 1100b) to generate a three-dimensional (3D) depth image.
[0166] In some embodiments, at least two camera modules (e.g., camera modules 1100a and 1100c) may have different fields of view (angle of view). In this case, at least two camera modules (e.g., camera modules 1100a and 1100c) may have different optical lenses, but are not limited thereto.
[0167] Furthermore, in some embodiments, camera modules 1100a, 1100b, and 1100c may have different viewing angles. In this case, the optical lenses included in the respective camera modules 1100a, 1100b, and 1100c may also be different, but this disclosure is not limited thereto.
[0168] In some embodiments, camera modules 1100a, 1100b, and 1100c may be configured to be physically separate from each other. That is, instead of the sensing area of one image sensor 1142 being divided and used by camera modules 1100a, 1100b, and 1100c, an independent image sensor 1142 may be provided within each of camera modules 1100a, 1100b, and 1100c.
[0169] Return to reference Figure 20 The application processor 1200 may include an image processing unit 1210, a memory controller 1220, and internal memory 1230. The application processor 1200 may be implemented separately from the camera modules 1100a, 1100b, and 1100c. For example, the application processor 1200 and the camera modules 1100a, 1100b, and 1100c may be implemented as separate semiconductor chips.
[0170] The image processing unit 1210 may include a plurality of sub-image processors 1212a, 1212b and 1212c, an image generator 1214 and a camera module controller 1216.
[0171] The image processing unit 1210 may include sub-image processors 1212a, 1212b, and 1212c, the number of which corresponds to the number of camera modules 1100a, 1100b, and 1100c. In some embodiments, each of the sub-image processors 1212a, 1212b, and 1212c may include the first image signal processor 200 described above.
[0172] Image data generated from the respective camera modules 1100a, 1100b, and 1100c can be provided to the corresponding sub-image processors 1212a, 1212b, and 1212c via separate image signal lines ISLa, ISLb, and ISLc. For example, image data generated from camera module 1100a can be provided to sub-image processor 1212a via image signal line ISLa, image data generated from camera module 1100b can be provided to sub-image processor 1212b via image signal line ISLb, and image data generated from camera module 1100c can be provided to sub-image processor 1212c via image signal line ISLc. For example, this image data transmission can be performed using the Camera Serial Interface (CSI) based on the Mobile Industrial Processor Interface (MIPI), but the embodiments are not limited thereto.
[0173] In some embodiments, a sub-image processor may be arranged to correspond to multiple camera modules. For example, sub-image processors 1212a and 1212c may be integrated into a single sub-image processor instead of being separate from each other as shown, and image data provided from camera modules 1100a and 1100c may be selected by a data selector (e.g., a multiplexer) and then provided to the integrated sub-image processor.
[0174] Image data provided to each of the sub-image processors 1212a, 1212b, and 1212c may be provided to the image generator 1214. The image generator 1214 may use the image data provided from each of the sub-image processors 1212a, 1212b, and 1212c to generate an output image based on image generation information or a mode signal.
[0175] Specifically, based on image generation information or pattern signals, image generator 1214 can generate an output image by merging at least some image data generated from camera modules 1100a, 1100b, and 1100c with different viewpoints. Furthermore, based on image generation information or pattern signals, image generator 1214 can generate an output image by selecting any one of the image data generated from camera modules 1100a, 1100b, and 1100c with different viewpoints.
[0176] In some embodiments, the image data provided to the image generator 1214 may include the image signal IS output by the first image signal processor 200 described above.
[0177] In some embodiments, image generation information may include a zoom signal (or zoom factor). Furthermore, in some embodiments, the mode signal may be a signal based on, for example, a mode selected by the user.
[0178] When the image generation information is a zoom signal or zoom factor, and camera modules 1100a, 1100b, and 1100c have different fields of view or angles of view, image generator 1214 can perform different operations depending on the type of zoom signal. For example, when the zoom signal is a first signal, image data output from camera module 1100a can be merged with image data output from camera module 1100c, and then the merged image signal, along with image data output from camera module 1100b (which is not used for merging), can be used to generate an output image. When the zoom signal is a second signal different from the first signal, image generator 1214 does not perform image data merging, but can instead generate an output image by selecting any one of the image data output from camera modules 1100a, 1100b, and 1100c. However, the embodiments are not limited to this, and the method of processing image data can be modified in various ways as needed.
[0179] In some embodiments, the image generator 1214 may receive multiple image data with different exposure times from at least one of the sub-image processors 1212a, 1212b and 1212c, and perform high dynamic range (HDR) processing on the multiple image data to generate merged image data with increased dynamic range.
[0180] The camera module controller 1216 can provide control signals to each of the camera modules 1100a, 1100b, and 1100c. The control signals generated from the camera module controller 1216 can be provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.
[0181] Based on a mode signal or image generation information including a zoom signal, any one of camera modules 1100a, 1100b, and 1100c (e.g., camera module 1100a) can be designated as the master camera, and the remaining camera modules (e.g., camera modules 1100b and 1100c) can be designated as slave cameras. This information can be included in control signals to be provided to the corresponding camera modules 1100a, 1100b, and 1100c via separate control signal lines CSLa, CSLb, and CSLc.
[0182] The camera module can operate as both a master and slave based on the zoom factor or operating mode signal. For example, when the field of view of camera module 1100a is wider than that of camera module 1100c and the zoom factor indicates a low zoom ratio, camera module 1100c can operate as a master and camera module 1100a can operate as a slave. Conversely, when the zoom factor indicates a high zoom ratio, camera module 1100a can operate as a master and camera module 1100c can operate as a slave.
[0183] In some embodiments, control signals provided from camera module controller 1216 to each of camera modules 1100a, 1100b, and 1100c may include a synchronization enable signal. For example, when camera module 1100b is the main camera and camera modules 1100a and 1100c are the slave cameras, camera module controller 1216 may send a synchronization enable signal to camera module 1100b. Having received the synchronization enable signal, camera module 1100b may generate a synchronization signal based on the received synchronization enable signal and send the generated synchronization signal to camera modules 1100a and 1100c via synchronization signal line SSL. Based on the synchronization signal, camera modules 1100a, 1100b, and 1100c may synchronously send image data to application processor 1200.
[0184] In some embodiments, control signals provided from camera module controller 1216 to camera modules 1100a, 1100b, and 1100c may include mode information based on mode signals. Based on the mode information, camera modules 1100a, 1100b, and 1100c may operate in a first operating mode or a second operating mode with respect to the sensing rate.
[0185] In the first operating mode, camera modules 1100a, 1100b, and 1100c can generate image signals at a first rate (e.g., a first frame rate), encode them at a second rate higher than the first rate (e.g., a second frame rate higher than the first frame rate), and send the encoded image signals to application processor 1200. In this case, the second rate may be 30 times or less than the first rate.
[0186] Application processor 1200 can store the received image signal (i.e., the encoded image signal) in memory 1230 disposed therein or in memory 1400 disposed externally to application processor 1200. Application processor 1200 can then read the encoded image signal from memory 1230 or memory 1400 to decode it and display image data generated based on the decoded image signal. For example, a corresponding subprocessor among the sub-image processors 1212a, 1212b, and 1212c of image processing unit 1210 can perform decoding and can also perform image processing on the decoded image signal. For example, image data generated based on the decoded image signal can be displayed on display 1500.
[0187] In the second operating mode, camera modules 1100a, 1100b, and 1100c can generate image signals at a third rate lower than the first rate (e.g., a third frame rate lower than the first frame rate) and send the image signals to application processor 1200. The image signals provided to application processor 1200 can be unencoded signals. Application processor 1200 can perform image processing on the received image signals or store the image signals in memory 1230 or memory 1400.
[0188] PMIC 1300 can supply power (e.g., power supply voltage) to each of camera modules 1100a, 1100b, and 1100c. For example, under the control of application processor 1200, PMIC 1300 can supply a first power to camera module 1100a via power signal line PSLa, a second power to camera module 1100b via power signal line PSLb, and a third power to camera module 1100c via power signal line PSLc.
[0189] The PMIC 1300 can generate power corresponding to each of the camera modules 1100a, 1100b, and 1100c in response to a power control signal PCON from the application processor 1200, and can also adjust the power level. The power control signal PCON can include power adjustment signals for various operating modes of the camera modules 1100a, 1100b, and 1100c. For example, the operating mode may include a low-power mode, and in this case, the power control signal PCON can include information about the camera module operating in low-power mode and the set power level. The power levels supplied to the respective camera modules 1100a, 1100b, and 1100c can be the same or different. Furthermore, the power levels can be changed dynamically.
[0190] In summary, those skilled in the art will understand that many variations and modifications can be made to the embodiments without substantially departing from the principles of this disclosure. Therefore, the embodiments are used in a general descriptive sense only and are not intended to be limiting.
Claims
1. An image processing apparatus, comprising: Memory; as well as At least one image signal processor is configured to: The first neural network is used to generate feature values that indicate whether to correct global pixel values sensed during a unit frame interval, and a feature signal including said feature values is generated. An image signal is generated by merging the global pixel values with the feature signal; The pixel values included in the image signal are split into first sub-pixel values and second sub-pixel values; the frame feature signals included in the image signal are split into first sub-feature values corresponding to the first sub-pixel values and second sub-feature values corresponding to the second sub-pixel values; and a first sub-image signal including the first sub-pixel values and the first sub-feature values and a second sub-image signal including the second sub-pixel values and the second sub-feature values are generated; and The first sub-image signal and the second sub-image signal are corrected sequentially using a second neural network.
2. The image processing apparatus according to claim 1, wherein, The at least one image signal processor is further configured to generate reduced pixel values by encoding the global pixel values.
3. The image processing apparatus according to claim 2, wherein, The at least one image signal processor is also configured to extract original feature values from the reduced pixel values using the first neural network.
4. The image processing apparatus according to claim 3, wherein, The at least one image signal processor is further configured to decode the extracted raw feature values to generate feature values indicating whether the global pixel values have been corrected.
5. The image processing apparatus according to claim 1, wherein, The memory is configured to store the global pixel values and to provide the stored global pixel values to the at least one image signal processor.
6. The image processing apparatus according to claim 1, wherein, The first neural network includes multiple layers for processing the global pixel values. The second neural network includes multiple layers for processing the first sub-image signal and the second sub-image signal, and The first neural network is different from the second neural network.
7. The image processing apparatus according to claim 1, wherein, The feature value indicates whether flares or haze appear in the global pixel values.
8. The image processing apparatus according to claim 7, wherein, The at least one image signal processor is further configured to perform correction for blurring or ghosting in the first sub-image signal and the second sub-image signal.
9. The image processing apparatus according to claim 1, wherein, The memory is also configured to store the first frame image and provide the stored first frame image to the at least one image signal processor, and The at least one image signal processor is further configured to generate a corrected feature value indicating whether the first frame image has been corrected, and to generate a corrected feature signal including the corrected feature value, and to merge the corrected feature signal with a second frame image following the first frame image.
10. The image processing apparatus according to claim 1, wherein, The at least one image signal processor is further configured to generate a signal obtained by combining the first sub-image signal and the second sub-image signal.
11. The image processing apparatus according to claim 10, wherein, The signal obtained by combining the first sub-image signal and the second sub-image signal is sent to the application processor.
12. An electronic device comprising: An image sensor is configured to sense light passing through the display and output a first image signal including global pixel values sensed during a unit frame interval; as well as An image signal processor, which is connected to the image sensor and is configured to: Receive the first image signal; A second image signal is generated using a first neural network by correcting the global pixel values included in the first image signal; The first image signal is split into a first sub-image signal and a second sub-image signal, and the second image signal is split into a third sub-image signal and a fourth sub-image signal; as well as A first signal is generated using a second neural network based on the first sub-image signal and the third sub-image signal, and a second signal is generated based on the second sub-image signal and the fourth sub-image signal.
13. The electronic device according to claim 12, wherein, The electronic device includes a front surface and a rear surface. The display is disposed on the front surface, and The image sensor is configured to sense light incident on the front surface.
14. The electronic device according to claim 12, wherein, The image signal processor is further configured to generate a third signal by combining the first signal and the second signal, and The image generated based on the third signal is output to the display.
15. The electronic device according to claim 12, wherein, The image sensor is configured to sense light incident through a hole formed in the display.
16. The electronic device of claim 12, further comprising an application processor configured to receive the first signal and the second signal, and to perform image processing on the first signal and the second signal.
17. The electronic device of claim 12, further comprising an application processor configured to connect the image sensor to the image signal processor, and in, The application processor is configured to receive the first image signal from the image sensor and provide the processed first image signal to the image signal processor, wherein the processed first image signal has undergone image processing.
18. An electronic system comprising: Electronic devices, including image sensors, image signal processors, and communication processors; as well as The data center is configured to exchange data with the communication processor. The image sensor is configured to output a first image signal including global pixel values sensed during a unit frame interval, and to provide the first image signal to an image signal processor. The image signal processor is configured to generate a second image signal using the first neural network based on first weights applied to a first neural network and a first image signal provided from the image sensor, and to generate a third image signal using the second neural network based on second weights applied to a second neural network different from the first neural network, the first image signal, and the second image signal. The communication processor is configured to provide the first weight, the second weight, the first image signal, and the third image signal to the data center. The data center is configured to generate a third weight obtained by correcting the first weight and a fourth weight obtained by correcting the second weight based on the first weight, the second weight, the first image signal and the third image signal, and to provide the third weight and the fourth weight to the electronic device.
19. The electronic system according to claim 18, wherein, The first weight is updated based on the provided third weight, and the second weight is updated based on the provided fourth weight.
20. The electronic system according to claim 18, wherein, The data center includes a memory configured to store the third weight and the fourth weight.
Citation Information
Patent Citations
Microbubble generating device
KR1020200119874A
Image processing apparatus and method
CN101625849A
Column output circuits for image sensors
CN102656880A
