Image processing method, apparatus and electronic device

Abstract

The application discloses an image processing method, device and electronic equipment, and belongs to the technical field of image processing. The image processing method is applied to an image sensor, and the image sensor comprises a first light filtering assembly and a second light filtering assembly. The first light filtering assembly and the second light filtering assembly are both provided with a plurality of light filtering pixel units arranged in different color arrays. The method comprises the following steps: collecting a first light beam incident to the image sensor to obtain a first image corresponding to the first light filtering assembly and a second image corresponding to the second light filtering assembly when the image sensor is located at a first position; controlling the image sensor to be shifted to a second position in the pixel dimension; collecting a second light beam incident to the image sensor to obtain a third image corresponding to the first light filtering assembly and a fourth image corresponding to the second light filtering assembly when the image sensor is located at the second position; and synthesizing a target image according to the first image, the second image, the third image and the fourth image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, and electronic device. Background Technology

[0002] Image sensors are one of the key components of a camera for capturing images. They receive light signals, convert them into electrical signals through photoelectric conversion, then into digital signals, and finally into the image we see. While image sensors can receive light signals, they can only distinguish their energy levels, not colors. This led to the development of the Bayer array. It is thanks to the Bayer array, along with image interpolation algorithms, that we can reproduce the colors we see in the real world. However, the Bayer array also has obvious drawbacks: each pixel on the sensor only receives information about one color through a color filter, and the remaining colors must be guessed and supplemented by algorithms, resulting in a poor final image quality. Summary of the Invention

[0003] The purpose of this application is to provide an image processing method, apparatus, and electronic device that can solve the problem in the prior art where images acquired using image sensors cannot reproduce true colors, resulting in poor image quality.

[0004] In a first aspect, embodiments of this application provide an image processing method applied to an image sensor, the image sensor including a first filter component and a second filter component; both the first filter component and the second filter component are provided with a plurality of filter pixel units, and the color array arrangements of the plurality of filter pixel units of the first filter component and the second filter component are different; the method includes:

[0005] When the image sensor is in the first position, a first light beam incident on the image sensor is acquired to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component;

[0006] Control the image sensor to shift to a second position in the pixel dimension;

[0007] When the image sensor is located in the second position, a second light beam incident on the image sensor is acquired to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component;

[0008] A target image is synthesized based on the first image, the second image, the third image, and the fourth image; wherein the color array arrangements of the first image, the second image, the third image, and the fourth image are different.

[0009] Secondly, embodiments of this application provide an image processing apparatus, including an image sensor, a control module, and an image synthesis module. The image sensor includes a first filter component and a second filter component. Both the first filter component and the second filter component are provided with multiple filter pixel units, and the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component are different. The first filter component is used to collect a first light beam incident on the image sensor when the image sensor is in a first position to obtain a first image; and to collect a second light beam incident on the image sensor when the image sensor is in a second position to obtain a third image.

[0010] The second filter component is used to acquire the first light beam to obtain a second image when the image sensor is located at the first position; and to acquire the second light beam to obtain a fourth image when the image sensor is located at the second position.

[0011] The control module is used to control the image sensor to shift from the first position to the second position in the pixel dimension;

[0012] The image synthesis module is used to synthesize a target image based on the first image, the second image, the third image, and the fourth image; wherein the color array arrangements of the first image, the second image, the third image, and the fourth image are different.

[0013] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0016] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0017] In this embodiment, the image processing method is applied to an image sensor, which includes a first filter component and a second filter component. Both the first and second filter components have multiple filter pixel units, and the color array arrangements of these pixel units are different. When the image sensor is in a first position, a first light beam incident on the image sensor is acquired to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component. The image sensor is then shifted in the pixel dimension to a second position. Then, when the image sensor is in the second position, a second light beam incident on the image sensor is acquired to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component. A target image is then synthesized based on the first, second, third, and fourth images. As can be seen, since the color array arrangement of the multiple filter pixel units of the first and second filter components of the image sensor is different, the color array arrangement in the images acquired by the first and second filter components is also different. That is, only one shift of the image sensor and two acquisitions of the incident beam are needed to obtain four images of the incident beam. The color array arrangements of the first, second, third, and fourth images are all different, so that each pixel in the target image obtained by synthesizing the multiple images has rich color information. This avoids a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed and large image restoration deviation caused by shift. Therefore, this application not only improves the color restoration accuracy of the images acquired by the image sensor, but also reduces the technical difficulty of restoring the true color of the image based on the image sensor. Attached Figure Description

[0018] Figure 1 This is a schematic structural diagram of an image sensor according to an embodiment of this application;

[0019] Figure 2 This is a schematic flowchart of an image processing method provided according to an embodiment of this application;

[0020] Figure 3 This is a schematic structural diagram of an image sensor according to another embodiment of this application;

[0021] Figure 4 This is a schematic diagram illustrating the principle of an image sensor acquiring images according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram illustrating the principle of an image sensor acquiring images according to another embodiment of this application;

[0023] Figure 6This is a schematic diagram illustrating the principle of an image sensor acquiring images according to another embodiment of this application;

[0024] Figure 7 This is a schematic diagram illustrating the principle of image synthesis using an image sensor according to an embodiment of this application;

[0025] Figure 8 This is a schematic diagram illustrating the principle of an image sensor acquiring images according to another embodiment of this application;

[0026] Figure 9 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application;

[0027] Figure 10 This is a schematic block diagram of an electronic device according to an embodiment of this application;

[0028] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The image processing method and electronic device provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0032] The image processing method provided in this application is applied to an image sensor, wherein the image sensor includes a first filter component and a second filter component. Both the first and second filter components have multiple filter pixel units, and the color array arrangements of the multiple filter pixel units in the first and second filter components are different. When acquiring a light beam incident on the image sensor, the first and second filter components are used to process the incident light beam according to their respective color array arrangements, resulting in two images with different pixel color arrangements corresponding to the incident light beam.

[0033] Figure 1 This is a schematic structural diagram of an image sensor according to an embodiment of this application. Figure 1 As shown, the image sensor includes a first filter component 110 and a second filter component 120; the first filter component 110 is provided with a plurality of filter pixel units 111, and the second filter component 120 is provided with a plurality of filter pixel units 121, and the color array arrangements of the plurality of filter pixel units 111 and the plurality of filter pixel units 121 are different.

[0034] Figure 2 This is a schematic flowchart of an image processing method according to an embodiment of this application, such as... Figure 2 As shown, the image processing method includes the following steps S202-S208:

[0035] S202, with the image sensor in the first position, the first light beam incident on the image sensor is collected to obtain the first image corresponding to the first filter component and the second image corresponding to the second filter component.

[0036] In one embodiment, when acquiring a first light beam incident on an image sensor to obtain a first image and a second image, the following actions can be performed: acquiring a first sub-light beam incident on a first filter component to obtain a first incident light; and acquiring a second sub-light beam incident on a second filter component to obtain a second incident light. Furthermore, optical signal processing is performed on the first incident light to obtain the first image; and optical signal processing is performed on the second incident light to obtain the second image. The first light beam includes a first sub-light beam and a second sub-light beam.

[0037] In one embodiment, the incident direction of the first light beam incident on the image sensor can be determined according to the configuration of the first filter component 110 and the second filter component 120 in the image sensor. Optionally, a beam splitter can be configured in the image sensor to split the first light beam into a first sub-beam and a second sub-beam, wherein the first sub-beam is incident on the first filter component 110 and the second sub-beam is incident on the second filter component 120.

[0038] Optionally, a reflector can be configured in the image sensor. The interaction between the beam splitter and the reflector changes the incident direction of the light beam incident on the image sensor. The first light beam is incident on the beam splitter, which splits the first light beam to obtain a first sub-beam and a third sub-beam. The first sub-beam is incident on the first filter assembly 110, and the third sub-beam is incident on the reflector. The reflector reflects the third sub-beam to obtain a second sub-beam that can be incident on the second filter assembly 120.

[0039] S204 controls the image sensor to shift to a second position in the pixel dimension.

[0040] In one embodiment, the image sensor can be shifted to a second position in the pixel dimension according to a preset shifting method. The shifting method may include at least one of the following: the direction of movement and the number of pixels moved.

[0041] Optionally, the image sensor can be controlled to move a unit pixel distance in a specified direction to position the image sensor in a second position. The specified direction of movement can be any of upward, downward, leftward, or rightward. The unit pixel can be one pixel. The direction of movement of the image sensor can be set according to the different structures of the image sensor, with the aim of enabling the images passing through the first and second filter components to have different color arrays. For example, when the light-incident surfaces of the first and second filter components in the image sensor are on the same horizontal plane, the image sensor can be controlled to move downward or upward by one pixel.

[0042] Depending on the shifting method, the second position will also differ. For example, if the image sensor is moved downwards by one pixel, the second position will be located below the first position and separated from it by one pixel. If the image sensor is moved upwards by one pixel, the second position will be located above the first position and separated from it by one pixel.

[0043] S206, with the image sensor in the second position, the second light beam incident on the image sensor is collected to obtain the third image corresponding to the first filter component and the fourth image corresponding to the second filter component.

[0044] In particular, when acquiring the second beam incident on the image sensor, the image obtained will be different because the position of the image sensor changes.

[0045] Optionally, the second beam includes a fourth sub-beam and a fifth sub-beam. When acquiring the second beam incident on the image sensor to obtain the third and fourth images, the following steps can be specifically performed: acquiring the fourth sub-beam incident on the first filter component to obtain the third incident light; acquiring the fifth sub-beam incident on the second filter component to obtain the fourth incident light. The third image can be obtained by performing optical signal processing on the third incident light. The fourth image can be obtained by performing optical signal processing on the fourth incident light.

[0046] Optionally, a beam splitter is configured in the image sensor to split the second beam into a fourth sub-beam and a fifth sub-beam. The fourth sub-beam is incident on the first filter assembly 110, and the fifth sub-beam is incident on the second filter assembly 120.

[0047] Optionally, a reflector can be configured in the image sensor. The interaction between the beam splitter and the reflector changes the incident direction of the light beam incident on the image sensor. The second light beam is incident on the beam splitter, which splits the second light beam to obtain a fourth sub-beam and a sixth sub-beam. The fourth sub-beam is incident on the first filter assembly 110, and the sixth sub-beam is incident on the reflector. The sixth sub-beam is reflected by the reflector to obtain a fifth sub-beam that can be incident on the second filter assembly 120.

[0048] S208, synthesize the target image based on the first image, the second image, the third image, and the fourth image.

[0049] The color array arrangements of the first, second, third, and fourth images are different.

[0050] If the multiple filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1, then when synthesizing the target image, for each of the first image, the second image, the third image and the fourth image, the pixels at the corresponding positions of each image can be superimposed to obtain the color information of each pixel in the N color channels, and then the target image can be generated based on the color information of each pixel in the N color channels.

[0051] For example, when N is 4, the color array arrangement of the multiple filter pixel units 111 in the first filter component 110 is GRBG, and the color array arrangement of the multiple filter pixel units 121 in the second filter component 120 is RGGB. When the image sensor is in the first position, the color array arrangement of the acquired first image is the same as that of the multiple filter pixel units 111, i.e., GRBG. The color array arrangement of the acquired second image is the same as that of the multiple filter pixel units 121, i.e., RGGB. Assuming that the image sensor is moved down by one pixel, after the light beam incident on the image sensor is collected again, the color array arrangement of the third image obtained through the first filter component 110 will be BGGR, and the color array arrangement of the fourth image obtained through the second filter component 120 will be GBRG. It can be seen that each pixel in the four images obtained by collecting the light beam incident on the image sensor twice has color information in four color channels (including one red, two greens and one blue). By superimposing pixels at corresponding positions in four images, each pixel of the generated target image can contain color information from four color channels (including one red, two green, and one blue), thus restoring the true colors of the image to a greater extent.

[0052] In this embodiment, the image processing method is applied to an image sensor, which includes a first filter component and a second filter component. Both the first and second filter components have multiple filter pixel units, and the color array arrangements of these pixel units are different. When the image sensor is in a first position, a first light beam incident on the image sensor is acquired to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component. The image sensor is then shifted in the pixel dimension to a second position. Then, when the image sensor is in the second position, a second light beam incident on the image sensor is acquired to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component. A target image is then synthesized based on the first, second, third, and fourth images. As can be seen, since the color array arrangement of the multiple filter pixel units of the first and second filter components of the image sensor is different, the color array arrangement in the images acquired by the first and second filter components is also different. That is, only one shift of the image sensor and two acquisitions of the incident beam are needed to obtain four images of the incident beam. The color array arrangements of the first, second, third, and fourth images are all different, so that each pixel in the target image obtained by synthesizing the multiple images has rich color information. This avoids a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed and large image restoration deviation caused by shift. Therefore, this application not only improves the color restoration accuracy of the images acquired by the image sensor, but also reduces the technical difficulty of restoring the true color of the image based on the image sensor.

[0053] Figure 3 This is a schematic structural diagram of an image sensor according to another embodiment of this application. Figure 3 As shown, the image sensor includes: a light-transmitting lens, a first filter assembly 110, a second filter assembly 120, a beam splitter 130, and a reflector 140. After the first light beam is incident through the light-transmitting lens, it is split by the beam splitter 130 to obtain a first sub-beam and a third sub-beam. The first sub-beam is incident on the first filter assembly 110, and the third sub-beam is incident on the reflector 140 and reflected to become the second sub-beam. The second sub-beam is then incident on the second filter assembly 120.

[0054] Figure 3The diagram shows the acquisition of the first light beam when the image sensor is in the first position. Similarly, when the image sensor is in the second position, the principle of acquiring the second light beam is the same as that of the first light beam. After the second light beam is incident through the light-transmitting lens, it is split by the beam splitter 130 to obtain the fourth sub-beam and the sixth sub-beam. The fourth sub-beam is incident on the first filter component 110, and the sixth sub-beam is incident on the reflector 140 and reflected to become the fifth sub-beam. The fifth sub-beam is incident on the second filter component 120.

[0055] Figure 4 This is a schematic diagram illustrating the principle of an image sensor acquiring images according to an embodiment of this application. (P) ij This indicates that the filter pixel unit 111 located in the i-th row and j-th column within the first filter assembly 110 is Q. ij This refers to the filter pixel unit 121 located in the i-th row and j-th column within the second filter assembly 120. To ensure that the pixel colors at corresponding positions in the two images obtained after passing through the first filter assembly 110 and the second filter assembly 120 are different, then P... ij The color of the incident light used to pass through and Q ij The incident light used for transmission is of different colors, ensuring that the color received by any filter pixel unit 111 in the first filter assembly 110 is different from the color received by any filter pixel unit 121 in the second filter assembly 120. Therefore, when the images generated by the first filter assembly 110 and the second filter assembly 120 are subsequently superimposed, it ensures that each pixel in the superimposed image receives two different colors, reducing the number of colors that need to be calculated for each pixel. This simplifies the calculation algorithm and reduces the probability of incorrect colors, thus improving the realism of the image.

[0056] like Figure 4 As shown, the color array arrangement of the plurality of filter pixel units 111 in the first filter component 110 is GRBG, and the color array arrangement of the plurality of filter pixel units 121 in the second filter component 120 is RGGB. The first filter component 110 and the second filter component 120 are located in two different areas on the image sensor, for example... Figure 3The two regions shown are the same size, differing only in the color array arrangement of their filter pixel units. Therefore, the image obtained after passing through the first filter assembly 110 has a GRBG color array arrangement, and the image obtained after passing through the second filter assembly 120 has an RGGB color array arrangement. That is, the first filter assembly 110 and the second filter assembly 120 in the image sensor are used to receive two incident light beams from the beam splitting structure (including the beam splitter 130 and the emitting mirror 140), the difference being the different color array arrangements at the receiving points of these two filter assemblies. By adjusting the position of the incident light on the image sensor, images with different color arrays can be obtained from two regions of the same size.

[0057] Below, in Figure 3 and Figure 4 Based on the structure of the image sensor shown, the image processing method provided in this application will be illustrated through a specific embodiment.

[0058] First, when the image sensor is not moved, it is in its initial position. The electronic device takes its first image of the target object in the actual scene, such as... Figure 5 As shown, the area within the box at the target object location refers to the region that the image sensor can capture. The first light beam first passes through the lens assembly and enters the beam splitting structure (including beam splitter 130 and emitting mirror 140). The beam splitting structure splits the first light beam to obtain a first sub-beam and a second sub-beam. The first sub-beam is incident on the first filter assembly 110, and the second sub-beam is incident on the second filter assembly 120. That is, two filter assemblies with different color array arrangements in the image sensor receive the incident light at the same time. After photoelectric conversion, they are then digitally converted to obtain a digital image. The digital image is then processed, and the required intermediate area is cropped to obtain the first image and the second image, respectively. In this embodiment, taking a Bayer array as an example, the color array arrangements of the first image and the second image are different Bayer arrays. Figure 5 As shown, the color array arrangement of the first image is GRBG, and the color array arrangement of the second image is RGGB. It should be noted that... Figure 5 In the images shown, the numbers at each pixel position represent the feature value of the corresponding pixel in the actual scene. For example, the first pixel in the top left corner of the first image has a green intensity of 7, and the first pixel in the top left corner of the second image has a red intensity of 7. To clearly represent the feature value of each pixel, Figure 5 The image does not use shading to distinguish the color information of each pixel. The color information of each pixel in the image can be found in [reference needed]. Figure 4 As shown.

[0059] Next, the image sensor is moved down one pixel to the second position. After the image sensor moves down one pixel, the corresponding pixel and the filter components (including the first filter component 110 and the second filter component 120) also move down one pixel. At this time, the electronic device takes a second picture of the target object in the actual scene, such as... Figure 6 As shown, the area within the box at the target object location refers to the region that the image sensor can capture. The second beam first passes through the lens assembly and enters the beam-splitting structure (including beam splitter 130 and emitter 140). The beam-splitting structure splits the second beam to obtain a fourth sub-beam and a fifth sub-beam. The fourth sub-beam is incident on the first filter assembly 110, and the fifth sub-beam is incident on the second filter assembly 120. After photoelectric conversion, they are then digitally converted to obtain a digital image. The digital image is then processed, and the required intermediate area is cropped to obtain the third and fourth images. In this embodiment, using a Bayer array as an example, the color arrays of the third and fourth images are arranged in different Bayer arrays. Figure 6 As shown, the color array arrangement of the third image is BGGR, and the color array arrangement of the fourth image is GBRG. It should be noted that... Figure 6 In the images shown, the numbers at each pixel position represent the feature value of the corresponding pixel in the actual scene. For example, the first pixel in the top left corner of the third image has a blue intensity of 7, and the first pixel in the top left corner of the fourth image has a green intensity of 7. To clearly represent the feature value of each pixel, Figure 6 The text does not use shadows to distinguish the color information of each pixel position.

[0060] Next, the four captured images (i.e., the first, second, third, and fourth images) are combined to obtain the target image corresponding to the target object. When combining the four images, the color information of each pixel can be organized according to a predetermined color array arrangement, for example... Figure 7 As shown, after organizing the color information of each pixel according to the RGGB color array arrangement, four images with R, G, G, and B are obtained. Thus, each pixel obtains four colors capable of reproducing the true colors, enabling the final generated target image to reproduce the true colors of the target object to a greater extent.

[0061] It should be noted that the downward movement of the image sensor by one pixel in this embodiment is merely one exemplary shifting method. In practical applications, the image sensor shifting method can be customized according to shooting requirements. For example... Figure 8As shown, after controlling the image sensor to move upward by one pixel, the target object is photographed. The resulting digital images are somewhat different, but after image processing and cropping, the third and fourth images are the same as the third and fourth images obtained when the image sensor moves downward by one pixel.

[0062] As can be seen, in this embodiment, only one shift of the image sensor and two captures by the electronic device are needed to obtain four images of the target object. The color array arrangements of the first, second, third, and fourth images are all different, so that each pixel in the target image obtained by synthesizing multiple images has rich color information. This avoids a series of problems caused by excessive shifting of the image sensor, such as slow image acquisition speed and large image restoration deviation caused by shifting. Therefore, it not only improves the color restoration accuracy of the image acquired by the image sensor, but also reduces the technical difficulty of restoring the true color of the image based on the image sensor.

[0063] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0064] Figure 9 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application. In this embodiment, as... Figure 9 As shown, the image processing device includes an image sensor 91, a control module 92, and an image synthesis module 93. The image sensor 91 includes a first filter component 110 and a second filter component 120. Both the first filter component 110 and the second filter component 120 are provided with multiple filter pixel units, and the color array arrangements of the multiple filter pixel units of the first filter component 110 and the second filter component 120 are different.

[0065] The first filter component 110 is used to collect a first light beam incident on the image sensor 91 when the image sensor 91 is in a first position to obtain a first image; and to collect a second light beam incident on the image sensor 91 when the image sensor 91 is in a second position to obtain a third image.

[0066] The second filter component 120 is used to acquire the first light beam to obtain a second image when the image sensor 91 is located in the first position; and to acquire the second light beam to obtain a fourth image when the image sensor 91 is located in the second position.

[0067] The control module 92 is used to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the image synthesis module 93 is used to synthesize a target image based on the first image, the second image, the third image and the fourth image.

[0068] In one embodiment, the image sensor 91 further includes a beam splitter 130;

[0069] The beam splitter 130 is used to split the first beam to obtain the first sub-beam and the second sub-beam.

[0070] In one embodiment, the image sensor further includes: a reflector 140;

[0071] The beam splitter 130 is also used to split the first beam to obtain the first sub-beam and the third sub-beam incident on the reflector 140.

[0072] The reflector 140 is used to reflect the third sub-beam to obtain the second sub-beam.

[0073] In one embodiment, the control module 92 is further configured to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: the direction of movement and the number of pixels moved.

[0074] In one embodiment, the plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1;

[0075] The image synthesis module 93 is further configured to superimpose the pixels at corresponding positions in each of the first image, the second image, the third image, and the fourth image to obtain the color information of each pixel in the N color channels; and generate the target image based on the color information of each pixel in the N color channels.

[0076] In this embodiment, when the image sensor is in a first position, a first light beam incident on the image sensor is collected to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is in the second position, a second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. As can be seen, since the color array arrangement of the multiple filter pixel units of the first and second filter components of the image sensor is different, the color array arrangement in the images acquired by the first and second filter components is also different. That is, only one shift of the image sensor and two acquisitions of the incident beam are needed to obtain four images of the incident beam. The color array arrangements of the first, second, third, and fourth images are all different, so that each pixel in the target image obtained by synthesizing the multiple images has rich color information. This avoids a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed and large image restoration deviation caused by shift. Therefore, this application not only improves the color restoration accuracy of the images acquired by the image sensor, but also reduces the technical difficulty of restoring the true color of the image based on the image sensor.

[0077] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0078] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0079] The image processing apparatus provided in this application embodiment can implement the various processes implemented in any of the above-described image processing method embodiments. To avoid repetition, these processes will not be described again here.

[0080] Optionally, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001 and a memory 1002. The memory 1002 stores a program or instructions that can run on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0081] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0082] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0083] The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0084] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0085] The processor 1110 is configured to: acquire a first light beam incident on the image sensor when the image sensor is located at a first position; obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; control the image sensor to shift to a second position in the pixel dimension; acquire a second light beam incident on the image sensor when the image sensor is located at the second position; obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component; and synthesize a target image based on the first image, the second image, the third image, and the fourth image; wherein the color array arrangements of the first image, the second image, the third image, and the fourth image are different.

[0086] Optionally, the processor 1110 is further configured to acquire a first sub-beam incident on the first filter component to obtain incident light; acquire a second sub-beam incident on the second filter component to obtain second incident light; the first beam includes the first sub-beam and the second sub-beam; perform optical signal processing on the first incident light to obtain the first image; and perform optical signal processing on the second incident light to obtain the second image.

[0087] Optionally, the processor 1110 is further configured to perform beam splitting processing on the first beam through the beam splitter to obtain the first sub-beam and the second sub-beam.

[0088] Optionally, the processor 1110 is further configured to perform beam splitting processing on the first beam through the beam splitter to obtain the first sub-beam and the third sub-beam incident on the reflector; and to reflect the third sub-beam through the reflector to obtain the second sub-beam.

[0089] Optionally, the processor 1110 is further configured to control the image sensor to shift to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: the direction of movement and the number of pixels moved.

[0090] Optionally, the processor 1110 is further configured to, for each of the first image, the second image, the third image, and the fourth image, superimpose pixels at corresponding positions in each image to obtain color information of each pixel in the N color channels; and generate the target image based on the color information of each pixel in the N color channels. The plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1.

[0091] In this embodiment, when the image sensor is in a first position, a first light beam incident on the image sensor is collected to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is in the second position, a second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. As can be seen, since the color array arrangement of the multiple filter pixel units of the first and second filter components of the image sensor is different, the color array arrangement in the images acquired by the first and second filter components is also different. That is, only one shift of the image sensor and two acquisitions of the incident beam are needed to obtain four images of the incident beam. The color array arrangements of the first, second, third, and fourth images are all different, so that each pixel in the target image obtained by synthesizing the multiple images has rich color information. This avoids a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed and large image restoration deviation caused by shift. Therefore, this application not only improves the color restoration accuracy of the images acquired by the image sensor, but also reduces the technical difficulty of restoring the true color of the image based on the image sensor.

[0092] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0093] The memory 1109 can be used to store software programs and various data. The memory 1109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0094] Processor 1110 may include one or more processing units; optionally, processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1110.

[0095] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0096] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0097] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0098] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0099] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0100] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0102] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method is applied to an image sensor, which includes a beam splitter, a first filter component, and a second filter component; both the first filter component and the second filter component are provided with multiple filter pixel units, and the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component are different; the method includes: When the image sensor is in the first position, a first light beam incident on the image sensor is acquired to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; Control the image sensor to shift to a second position in the pixel dimension; When the image sensor is located in the second position, a second light beam incident on the image sensor is acquired to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component; A target image is synthesized based on the first image, the second image, the third image, and the fourth image; wherein the color array arrangements of the first image, the second image, the third image, and the fourth image are different; The process of acquiring the first light beam incident on the image sensor to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component includes: The first beam is split by the beam splitter to obtain a first sub-beam and a second sub-beam. A first sub-beam incident on the first filter component is collected to obtain a first incident light; a second sub-beam incident on the second filter component is collected to obtain a second incident light; The first incident light is processed to obtain the first image; the second incident light is processed to obtain the second image.

2. The method according to claim 1, characterized in that, The image sensor also includes: a reflector; The step of splitting the first beam using the beam splitter to obtain the first sub-beam and the second sub-beam includes: The first beam is split by the beam splitter to obtain the first sub-beam and the third sub-beam incident on the reflector. The third sub-beam is reflected by the reflector to obtain the second sub-beam.

3. The method according to claim 1, characterized in that, The control of shifting the image sensor to a second position in the pixel dimension includes: The image sensor is controlled to shift to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: the direction of movement and the number of pixels moved.

4. The method according to claim 1, characterized in that, The first filter component and the second filter component each have multiple filter pixel units including N color channels, where N is an integer greater than 1; The step of synthesizing a target image based on the first image, the second image, the third image, and the fourth image includes: For each of the first image, the second image, the third image, and the fourth image, the pixels at corresponding positions in each image are superimposed to obtain the color information of each pixel in the N color channels; The target image is generated based on the color information of each pixel in the N color channels.

5. An image processing apparatus, characterized in that, The system includes an image sensor, a control module, and an image synthesis module. The image sensor includes a beam splitter, a first filter component, and a second filter component. Both the first filter component and the second filter component are provided with multiple filter pixel units, and the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component are different. The first filter component is used to collect a first light beam incident on the image sensor when the image sensor is in a first position, thereby obtaining a first image; With the image sensor in the second position, a second light beam incident on the image sensor is acquired to obtain a third image; The second filter component is used to acquire the first light beam to obtain a second image when the image sensor is located at the first position; and to acquire the second light beam to obtain a fourth image when the image sensor is located at the second position. The control module is used to control the image sensor to shift from the first position to the second position in the pixel dimension; The image synthesis module is used to synthesize a target image based on the first image, the second image, the third image, and the fourth image; wherein the color array arrangements of the first image, the second image, the third image, and the fourth image are different; The image sensor further includes: a beam splitter; The beam splitter is used to split the first beam to obtain a first sub-beam and a second sub-beam. The first filter component is further configured to acquire a first sub-beam incident on the first filter component to obtain a first incident light; and to perform optical signal processing on the first incident light to obtain the first image; The second filter component is further configured to acquire the second sub-beam incident on the second filter component to obtain the second incident light; and to perform optical signal processing on the second incident light to obtain the second image.

6. The apparatus according to claim 5, characterized in that, The image sensor also includes: a reflector; The beam splitter is also used to split the first beam to obtain the first sub-beam and the third sub-beam incident on the reflector. The reflector is used to reflect the third sub-beam to obtain the second sub-beam.

7. The apparatus according to claim 5, characterized in that, The control module is further configured to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: the direction of movement and the number of pixels moved.

8. The apparatus according to claim 5, characterized in that, The first filter component and the second filter component each have multiple filter pixel units including N color channels, where N is an integer greater than 1; The image synthesis module is further configured to superimpose the pixels at corresponding positions in each of the first image, the second image, the third image, and the fourth image to obtain the color information of each pixel in the N color channels; The target image is generated based on the color information of each pixel in the N color channels.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1-4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1-4.

11. A computer program product, characterized in that, The program product is stored in a storage medium, and the program product includes instructions that are executed by at least one processor to implement the steps of the image processing method as described in any one of claims 1-4.

Images