Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of image processing. The image processing method is applied to the image sensor, and the image sensor comprises a first light filtering assembly and a second light filtering assembly. The first light filtering assembly and the second light filtering assembly are both provided with a plurality of light filtering pixel units which are arranged in different color arrays; the method comprises the following steps: collecting a first light beam incident to an image sensor under the condition that the image sensor is located at a first position, and obtaining a first image corresponding to a first light filtering assembly and a second image corresponding to a second light filtering assembly; controlling the image sensor to shift to a second position in the pixel dimension; collecting a second light beam incident to the image sensor under the condition that the image sensor is located at a second position to obtain a third image corresponding to the first light filtering assembly and a fourth image corresponding to the second light filtering assembly; and synthesizing a target image according to the first image, the second image, the third image and the fourth image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device and electronic equipment. Background Art

[0002] The image sensor is one of the key components of camera imaging. It receives light signals, converts them into electrical signals, and then converts them into digital signals, which finally become the pictures we see. Although the image sensor can receive light signals, it can only distinguish the energy size, but not the color, so the Bayer array was created. It is precisely because of the Bayer array and the calculation of the image interpolation algorithm that the colors we see in the real world can be restored. However, the disadvantage of the Bayer array is also obvious, that is, each pixel on the sensor can only obtain information about one color through the color filter, and the remaining colors can only be supplemented by guessing the color through the algorithm, resulting in poor final synthesized picture effect. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing method, device and electronic device, which can solve the problem in the prior art that images acquired by using image sensors cannot restore the true colors, resulting in poor image quality.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an image sensor, wherein the image sensor comprises a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently; the method comprises: When the image sensor is located at a first position, collecting a first light beam incident on the image sensor to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; Controlling the image sensor to shift to a second position in a pixel dimension; When the image sensor is located at the second position, collecting the second light beam incident on the image sensor to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component; A target image is synthesized according to the first image, the second image, the third image and the fourth image; wherein the color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

[0005] In a second aspect, an embodiment of the present application provides an image processing device, including an image sensor, a control module, and an image synthesis module, wherein the image sensor includes a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently; the first filter component is used to collect a first light beam incident on the image sensor to obtain a first image when the image sensor is located at a first position; and to collect a second light beam incident on the image sensor to obtain a third image when the image sensor is located at a second position; The second filter component is used to collect the first light beam to obtain a second image when the image sensor is located at the first position; and to collect the second light beam to obtain a fourth image when the image sensor is located at the second position; The control module is used to control the image sensor to shift from the first position to the second position in the pixel dimension; The image synthesis module is used to synthesize a target image according to the first image, the second image, the third image and the fourth image; wherein the color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0010] In an embodiment of the present application, an image processing method is applied to an image sensor, and the image sensor includes a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently. When the image sensor is located at a first position, a first light beam incident on the image sensor is collected to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is located at the second position, a second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. It can be seen that since the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component of the image sensor are different, the color array arrangements in the images respectively captured by the first filter component and the second filter component are also different. That is, four images of the incident light beam can be obtained by controlling the image sensor to shift once and capturing the incident light beam twice, wherein the color array arrangements of the first image, the second image, the third image and the fourth image are all different, so that each pixel in the target image obtained by synthesizing the captured multiple images has rich color information, avoiding a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed, large image restoration deviation caused by shifting, etc. Therefore, the present application not only improves the color restoration accuracy of the image captured by the image sensor, but also reduces the technical difficulty of restoring the real image color based on the image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic structural diagram of an image sensor provided according to an embodiment of the present application; Figure 2 is a schematic flow chart of an image processing method provided according to an embodiment of the present application; Figure 3 is a schematic structural diagram of an image sensor provided according to another embodiment of the present application; Figure 4 is a schematic principle diagram of an image sensor collecting an image according to an embodiment of the present application; Figure 5 is a schematic principle diagram of an image sensor collecting an image according to another embodiment of the present application; Figure 6 is a schematic principle diagram of an image sensor collecting an image according to another embodiment of the present application; Figure 7is a schematic principle diagram of synthesizing an image according to an image sensor in an embodiment of the present application; Figure 8 is a schematic principle diagram of an image sensor collecting an image according to another embodiment of the present application; Fig. 9 is a schematic block diagram of an image processing device provided according to an embodiment of the present application; Fig.10 is a schematic block diagram of an electronic device provided according to an embodiment of the present application; Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] The image processing method and electronic device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0015] The image processing method provided by the embodiment of the present application is applied to an image sensor, wherein the image sensor includes a first filter component and a second filter component, the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color array arrangements of the plurality of filter pixel units of the first filter component and the second filter component are different. When collecting a light beam incident to the image sensor, the first filter component and the second filter component are respectively used to process the incident light beam according to their respective color array arrangements, and obtain two images with different pixel color arrangements corresponding to the incident light beam.

[0016] Figure 1FIG. 1 is a schematic structural diagram of an image sensor provided according to an embodiment of the present application. Figure 1 As shown, the image sensor includes a first filter component 110 and a second filter component 120; the first filter component 110 is provided with a plurality of filter pixel units 111, and the second filter component 120 is provided with a plurality of filter pixel units 121, and the color arrays of the plurality of filter pixel units 111 and the plurality of filter pixel units 121 are arranged differently.

[0017] Figure 2 is a schematic flow chart of an image processing method provided according to an embodiment of the present application, such as Figure 2 As shown, the image processing method includes the following steps S202-S208: S202 , when the image sensor is located at a first position, collecting a first light beam incident on the image sensor to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component.

[0018] In one embodiment, when collecting a first light beam incident on an image sensor to obtain a first image and a second image, the following actions may be performed: collecting a first sub-beam incident on a first filter component to obtain a first incident light. And collecting a second sub-beam incident on a second filter component to obtain a second incident light. Further, performing optical signal processing on the first incident light to obtain a first image; performing optical signal processing on the second incident light to obtain a second image. The first light beam includes a first sub-beam and a second sub-beam.

[0019] In one embodiment, how to change the incident direction of the first light beam incident on the image sensor can be determined according to the configuration structure of the first filter component 110 and the second filter component 120 in the image sensor. Optionally, a beam splitter can be configured in the image sensor to perform beam splitting processing on the first light beam to obtain a first sub-beam and a second sub-beam, wherein the first sub-beam is incident on the first filter component 110, and the second sub-beam is incident on the second filter component 120.

[0020] Optionally, a reflector may be configured in the image sensor, and the incident direction of the light beam incident on the image sensor is changed through the interaction between the beam splitter and the reflector. The first light beam is incident on the beam splitter, and the first light beam is split by the beam splitter to obtain a first sub-beam and a third sub-beam. The first sub-beam is incident on the first filter component 110, and the third sub-beam is incident on the reflector. The third sub-beam is reflected by the reflector to obtain a second sub-beam that can be incident on the second filter component 120.

[0021] S204, controlling the image sensor to shift to a second position in the pixel dimension.

[0022] In one embodiment, the image sensor may be controlled to shift to the second position in the pixel dimension according to a preset shifting method. The shifting method may include at least one of the following: a shifting direction and a number of pixels shifted.

[0023] Optionally, the image sensor is controlled to move a unit pixel distance in a specified moving direction so that the image sensor is in the second position. The specified moving direction may be any direction of upward, downward, leftward, and rightward. The unit pixel may be one pixel. The moving direction of the image sensor may be set according to different structures of the image sensor, so as to enable the image passing through the first filter component and the second filter component to have different color arrays. For example, when the light incident surfaces of the first filter component and the second filter component in the image sensor are the same horizontal plane, the image sensor may be controlled to move downward or upward by a distance of one pixel.

[0024] The second position is different based on the different shifting methods. For example, if the image sensor is controlled to move downward by one pixel, the second position is located below the first position and is separated from the first position by one pixel. If the image sensor is controlled to move upward by one pixel, the second position is located above the first position and is separated from the first position by one pixel.

[0025] S206, when the image sensor is located at the second position, collecting the second light beam incident on the image sensor to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component.

[0026] When collecting the second light beam incident on the image sensor, the obtained image will be different because the position of the image sensor changes.

[0027] Optionally, the second light beam includes a fourth sub-beam and a fifth sub-beam. When collecting the second light beam incident on the image sensor to obtain the third image and the fourth image, the following steps can be specifically performed: collecting the fourth sub-beam incident on the first filter component to obtain the third incident light; collecting the fifth sub-beam incident on the second filter component to obtain the fourth incident light. By performing optical signal processing on the third incident light, the third image can be obtained. By performing optical signal processing on the fourth incident light, the fourth image can be obtained.

[0028] Optionally, a beam splitter is configured in the image sensor, and the second light beam is split by the beam splitter to obtain a fourth sub-beam and a fifth sub-beam, wherein the fourth sub-beam is incident on the first filter component 110 and the fifth sub-beam is incident on the second filter component 120 .

[0029] Optionally, a reflector may be configured in the image sensor, and the incident direction of the light beam incident on the image sensor is changed through the interaction between the beam splitter and the reflector. The second light beam is incident on the beam splitter, and the second light beam is split by the beam splitter to obtain a fourth sub-beam and a sixth sub-beam. The fourth sub-beam is incident on the first filter component 110, and the sixth sub-beam is incident on the reflector. The sixth sub-beam is reflected by the reflector to obtain a fifth sub-beam that can be incident on the second filter component 120.

[0030] S208, synthesizing a target image according to the first image, the second image, the third image and the fourth image.

[0031] The color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

[0032] If the multiple filter pixel units of the first filter component and the second filter component both include N color channels, where N is an integer greater than 1, then when synthesizing the target image, for each of the first image, the second image, the third image, and the fourth image, the pixels at the corresponding positions of each image can be superimposed to obtain the color information of each pixel in the N color channels, and then the target image can be generated based on the color information of each pixel in the N color channels.

[0033] For example, N is 4, the color array arrangement of the plurality of filter pixel units 111 in the first filter component 110 is GRBG, and the color array arrangement of the plurality of filter pixel units 121 in the second filter component 120 is RGGB. When the image sensor is located at the first position, the color array arrangement of the first image collected is the same as the color array arrangement of the plurality of filter pixel units 111, that is, GRBG. The color array arrangement of the second image collected is the same as the color array arrangement of the plurality of filter pixel units 121, that is, RGGB. Assuming that the image sensor is controlled to move downward by a distance of one pixel, after collecting the light beam incident to the image sensor again, the color array arrangement of the third image obtained by the first filter component 110 will be BGGR, and the color array arrangement of the fourth image obtained by the second filter component 120 will be GBRG. It can be seen that each pixel in the four images obtained by collecting the light beam incident to the image sensor twice has color information in four color channels (including one red, two green and one blue). By superimposing the pixels at corresponding positions in the four images, each pixel of the generated target image can contain color information in four color channels (including one red, two green and one blue), thereby restoring the true color of the image to a greater extent.

[0034] In an embodiment of the present application, an image processing method is applied to an image sensor, and the image sensor includes a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently. When the image sensor is located at a first position, a first light beam incident on the image sensor is collected to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is located at the second position, a second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. It can be seen that since the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component of the image sensor are different, the color array arrangements in the images respectively captured by the first filter component and the second filter component are also different. That is, four images of the incident light beam can be obtained by controlling the image sensor to shift once and capturing the incident light beam twice, wherein the color array arrangements of the first image, the second image, the third image and the fourth image are all different, so that each pixel in the target image obtained by synthesizing the captured multiple images has rich color information, avoiding a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed, large image restoration deviation caused by shifting, etc. Therefore, the present application not only improves the color restoration accuracy of the image captured by the image sensor, but also reduces the technical difficulty of restoring the real image color based on the image sensor.

[0035] Figure 3 FIG. 1 is a schematic structural diagram of an image sensor provided according to another embodiment of the present application. Figure 3 As shown, the image sensor includes: a light transmitting mirror, a first filter component 110, a second filter component 120, a beam splitter 130 and a reflector 140. After the first light beam is incident on the light transmitting mirror, it is split by the beam splitter 130 to obtain a first sub-beam and a third sub-beam, wherein the first sub-beam is incident on the first filter component 110, the third sub-beam is incident on the reflector 140 and then reflected to be a second sub-beam, and the second sub-beam is incident on the second filter component 120.

[0036] Figure 3What is shown is the situation of collecting the first light beam when the image sensor is located at the first position. Similarly, when the image sensor is located at the second position, the principle of collecting the second light beam is the same as that of the first light beam. After the second light beam is incident on the light-transmitting mirror, it is split by the spectroscope 130 to obtain a fourth sub-beam and a sixth sub-beam, wherein the fourth sub-beam is incident on the first filter component 110, the sixth sub-beam is incident on the reflector 140 and then reflected as a fifth sub-beam, and the fifth sub-beam is incident on the second filter component 120.

[0037] Figure 4 FIG. 1 is a schematic diagram of an image sensor collecting an image according to an embodiment of the present application. ij represents the filter pixel unit 111 located at the i-th row and the j-th column in the first filter component 110, with Q ij represents the filter pixel unit 121 located at the i-th row and the j-th column in the second filter component 120. In order to make the pixel colors at corresponding positions in the two images obtained after passing through the first filter component 110 and the second filter component 120 different, P ij The color of the incident light used to pass through is related to Q ij The colors of the incident light passing through are different, so that the color received by any filter pixel unit 111 in the first filter component 110 is different from the color received by any filter pixel unit 121 in the second filter component 120. Therefore, after the images generated by the first filter component 110 and the second filter component 120 are superimposed, it can be ensured that two different colors are obtained at each pixel in the superimposed image, reducing the number of colors that need to be calculated at each pixel, making the calculation algorithm more concise to a certain extent, and helping to reduce the probability of wrong colors, so that the authenticity of the image can be improved.

[0038] like Figure 4 As shown, the color array arrangement of the plurality of filter pixel units 111 in the first filter component 110 is GRBG, and the color array arrangement of the plurality of filter pixel units 121 in the second filter component 120 is RGGB. The first filter component 110 and the second filter component 120 are located in two different areas on the image sensor, for example Figure 3The left and right areas shown in the figure have the same size, and the only difference is that the color array arrangement of the filter pixel unit is different. Therefore, the color array arrangement of the image obtained after passing through the first filter component 110 is GRBG, and the color array arrangement of the image obtained after passing through the second filter component 120 is RGGB. That is, the first filter component 110 and the second filter component 120 in the image sensor are respectively used to receive two incident light beams from the splitting structure (including the splitter 130 and the transmitting mirror 140), and the difference is that the color array arrangements at the receiving points of the two filter components are different. By adjusting the position of the incident light on the image sensor, the two left and right areas of the same size can obtain images with different color arrays.

[0039] Below, in Figure 3 and Figure 4 Based on the structure of the image sensor shown, the image processing method provided by the present application is described through a specific embodiment.

[0040] First, when the image sensor is not shifted, the image sensor is located at the first position. The electronic device takes a first shot of the target object in the actual scene, such as Figure 5 As shown, the box at the position of the target object refers to the area that can be photographed by the image sensor. The first light beam first passes through the lens group and enters the spectroscopic structure (including the spectroscope 130 and the transmitting mirror 140). The spectroscopic structure performs spectroscopic processing on the first light beam to obtain a first sub-beam and a second sub-beam. The first sub-beam is incident on the first filter component 110, and the second sub-beam is incident on the second filter component 120. That is, the two filter components with different color array arrangements in the image sensor receive the incident light at the same time respectively, undergo photoelectric conversion, and then perform digital conversion to obtain a digital image. The digital image is then processed and the required middle area is cut out to obtain the first image and the second image respectively. In this embodiment, taking the Bayer array as an example, the color arrays of the first image and the second image are arranged as different Bayer arrays. As Figure 5 As shown, the color array arrangement of the first image is GRBG, and the color array arrangement of the second image is RGGB. Figure 5 In the images shown, the numbers at each pixel position represent the characteristic values ​​of the corresponding pixels in the actual scene. For example, the first pixel in the upper left corner of the first image obtains a green intensity with a characteristic of 7, and the first pixel in the upper left corner of the second image obtains a red intensity with a characteristic of 7. To clearly indicate the characteristic value of each pixel, Figure 5 The shading method is not used to distinguish the color information of each pixel position. The color information of each pixel position in the image can be referred to Figure 4 shown.

[0041] Afterwards, the image sensor is controlled to move downward by a distance of one pixel to the second position. After the image sensor moves downward by a distance of one pixel, the corresponding pixel and the filter component (including the first filter component 110 and the second filter component 120) also move downward by a distance of one pixel. At this time, the electronic device takes a second shot of the target object in the actual scene, such as Figure 6 As shown, the box at the position of the target object refers to the area that can be photographed by the image sensor. The second light beam first passes through the lens group and enters the spectroscopic structure (including the spectroscope 130 and the transmitting mirror 140). The spectroscopic structure performs spectroscopic processing on the second light beam to obtain the fourth sub-beam and the fifth sub-beam. The fourth sub-beam is incident on the first filter component 110, and the fifth sub-beam is incident on the second filter component 120. After photoelectric conversion and digital conversion, a digital image is obtained. The digital image is then processed and the required middle area is cut out to obtain the third image and the fourth image respectively. In this embodiment, taking the Bayer array as an example, the color arrays of the third image and the fourth image are arranged in different Bayer arrays. As shown Figure 6 As shown, the color array arrangement of the third image is BGGR, and the color array arrangement of the fourth image is GBRG. Figure 6 In the images shown, the numbers at each pixel position represent the characteristic values ​​of the corresponding pixels in the actual scene. For example, the first pixel in the upper left corner of the third image obtains a blue intensity with a characteristic of 7, and the first pixel in the upper left corner of the fourth image obtains a green intensity with a characteristic of 7. To clearly indicate the characteristic value of each pixel, Figure 6 No shading is used to distinguish the color information of each pixel position.

[0042] Afterwards, the four images (i.e., the first image, the second image, the third image, and the fourth image) are synthesized to obtain a target image corresponding to the target object. When synthesizing the four images, the color information of each pixel can be sorted according to a predetermined color array arrangement, for example Figure 7 As shown in the figure, after the color information of each pixel is sorted according to the color array arrangement of RGGB, four images of R, G, G, and B can be obtained. As a result, each pixel obtains four colors that can restore the true color, so that the final target image can restore the true color of the target object to a greater extent.

[0043] It should be noted that in this embodiment, controlling the image sensor to move downward by a distance of one pixel is only one exemplary shifting method. In practical applications, the shifting method of the image sensor can be customized according to shooting requirements. Figure 8As shown, after the image sensor is controlled to move upward by a distance of one pixel, the target object is photographed and the resulting digital images are different. However, after image processing and cropping, the resulting third and fourth images are the same as the third and fourth images obtained when the image sensor is moved downward by a distance of one pixel.

[0044] It can be seen that in the present embodiment, it is only necessary to control the image sensor to shift once and the electronic device to shoot twice to obtain four images of the target object, wherein the color array arrangements of the first image, the second image, the third image and the fourth image are all different, so that each pixel in the target image obtained by synthesizing the multiple images collected has rich color information, thereby avoiding a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed, large image restoration deviation caused by shifting, etc. Therefore, not only the color restoration accuracy of the image collected by the image sensor is improved, but also the technical difficulty of restoring the real image color based on the image sensor is reduced.

[0045] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0046] Fig. 9 is a schematic block diagram of an image processing device according to an embodiment of the present application. In this embodiment, Fig. 9 As shown, the image processing device includes an image sensor 91, a control module 92 and an image synthesis module 93. The image sensor 91 includes a first filter component 110 and a second filter component 120. The first filter component 110 and the second filter component 120 are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component 110 and the second filter component 120 are arranged differently. The first filter component 110 is used to collect the first light beam incident to the image sensor 91 to obtain a first image when the image sensor 91 is located at the first position; and to collect the second light beam incident to the image sensor 91 to obtain a third image when the image sensor 91 is located at the second position; The second filter component 120 is used to collect the first light beam to obtain a second image when the image sensor 91 is located at the first position; and to collect the second light beam to obtain a fourth image when the image sensor 91 is located at the second position; The control module 92 is used to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the image synthesis module 93 is used to synthesize a target image based on the first image, the second image, the third image and the fourth image.

[0047] In one embodiment, the image sensor 91 further includes: a beam splitter 130; The beam splitter 130 is used to perform beam splitting processing on the first light beam to obtain the first sub-beam and the second sub-beam.

[0048] In one embodiment, the image sensor further includes: a reflector 140; The beam splitter 130 is further used to perform beam splitting processing on the first light beam to obtain the first sub-beam and a third sub-beam incident on the reflector 140; The reflector 140 is used to reflect the third sub-beam to obtain the second sub-beam.

[0049] In one embodiment, the control module 92 is further used to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: a moving direction, and a number of pixels moved.

[0050] In one embodiment, the plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1; The image synthesis module 93 is also used to superimpose the pixels at the corresponding positions of each image in the first image, the second image, the third image and the fourth image to obtain the color information of each pixel in the N color channels; and generate the target image according to the color information of each pixel in the N color channels.

[0051] In an embodiment of the present application, a first light beam incident on an image sensor is collected when the image sensor is located at a first position to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is located at the second position, the second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. It can be seen that since the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component of the image sensor are different, the color array arrangements in the images respectively captured by the first filter component and the second filter component are also different. That is, four images of the incident light beam can be obtained by controlling the image sensor to shift once and capturing the incident light beam twice, wherein the color array arrangements of the first image, the second image, the third image and the fourth image are all different, so that each pixel in the target image obtained by synthesizing the captured multiple images has rich color information, avoiding a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed, large image restoration deviation caused by shifting, etc. Therefore, the present application not only improves the color restoration accuracy of the image captured by the image sensor, but also reduces the technical difficulty of restoring the real image color based on the image sensor.

[0052] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0053] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0054] The image processing device provided in the embodiment of the present application can implement each process implemented by the image processing method embodiment in any of the above embodiments, and will not be described again here to avoid repetition.

[0055] Alternatively, if Fig.10 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001 and a memory 1002, wherein the memory 1002 stores programs or instructions that can be executed on the processor 1001, and when the program or instructions are executed by the processor 1001, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0056] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0057] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. The electronic device 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109, and a processor 1110 and other components.

[0058] Those skilled in the art will appreciate that the electronic device 1100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 1110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here. Among them, the processor 1110 is used to collect the first light beam incident on the image sensor when the image sensor is located at the first position, and obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; control the image sensor to shift to the second position in the pixel dimension; when the image sensor is located at the second position, collect the second light beam incident on the image sensor, and obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component; synthesize the target image according to the first image, the second image, the third image and the fourth image; wherein the color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

[0059] Optionally, the processor 1110 is further used to collect a first sub-beam incident on the first filter component to obtain an incident light; collect a second sub-beam incident on the second filter component to obtain a second incident light; the first light beam includes the first sub-beam and the second sub-beam; perform optical signal processing on the first incident light to obtain the first image; and perform optical signal processing on the second incident light to obtain the second image.

[0060] Optionally, the processor 1110 is further configured to perform spectroscopic processing on the first light beam through the spectroscope to obtain the first sub-beam and the second sub-beam.

[0061] Optionally, the processor 1110 is further configured to perform spectroscopic processing on the first light beam through the spectroscope to obtain the first sub-beam and a third sub-beam incident on the reflector; and reflect the third sub-beam through the reflector to obtain the second sub-beam.

[0062] Optionally, the processor 1110 is further configured to control the image sensor to shift to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: a moving direction, and a number of pixels moved.

[0063] Optionally, the processor 1110 is further configured to, for each of the first image, the second image, the third image, and the fourth image, superimpose pixels at corresponding positions of each image to obtain color information of each pixel in the N color channels; and generate the target image according to the color information of each pixel in the N color channels. The plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1.

[0064] In an embodiment of the present application, a first light beam incident on an image sensor is collected when the image sensor is located at a first position to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; the image sensor is controlled to shift to a second position in the pixel dimension, and then when the image sensor is located at the second position, the second light beam incident on the image sensor is collected to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component, and a target image is synthesized based on the first image, the second image, the third image and the fourth image. It can be seen that since the color array arrangements of the multiple filter pixel units of the first filter component and the second filter component of the image sensor are different, the color array arrangements in the images respectively captured by the first filter component and the second filter component are also different. That is, four images of the incident light beam can be obtained by controlling the image sensor to shift once and capturing the incident light beam twice, wherein the color array arrangements of the first image, the second image, the third image and the fourth image are all different, so that each pixel in the target image obtained by synthesizing the captured multiple images has rich color information, avoiding a series of problems caused by too many shifts of the image sensor, such as slow image acquisition speed, large image restoration deviation caused by shifting, etc. Therefore, the present application not only improves the color restoration accuracy of the image captured by the image sensor, but also reduces the technical difficulty of restoring the real image color based on the image sensor.

[0065] It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processor (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0066] The memory 1109 can be used to store software programs and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory, or the memory 1109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0067] The processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

[0068] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0069] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0070] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0071] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0072] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0073] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0075] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to an image sensor, the image sensor comprises a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently; the method comprises: When the image sensor is located at a first position, collecting a first light beam incident on the image sensor to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component; Controlling the image sensor to shift to a second position in a pixel dimension; When the image sensor is located at the second position, collecting the second light beam incident on the image sensor to obtain a third image corresponding to the first filter component and a fourth image corresponding to the second filter component; A target image is synthesized according to the first image, the second image, the third image and the fourth image; wherein the color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

2. The method according to claim 1, characterized in that The collecting the first light beam incident on the image sensor to obtain a first image corresponding to the first filter component and a second image corresponding to the second filter component includes: Collecting a first sub-beam incident on the first filter component to obtain a first incident light; collecting a second sub-beam incident on the second filter component to obtain a second incident light; the first light beam includes the first sub-beam and the second sub-beam; The first incident light is subjected to optical signal processing to obtain the first image; and the second incident light is subjected to optical signal processing to obtain the second image.

3. The method according to claim 2, characterized in that The image sensor further includes: a spectroscope; Before collecting the first sub-beam incident on the first filter component to obtain the first incident light and collecting the second sub-beam incident on the second filter component to obtain the second incident light, the method further includes: The first light beam is split by the beam splitter to obtain the first sub-beam and the second sub-beam.

4. The method according to claim 3, characterized in that The image sensor further includes: a reflector; The step of performing a spectroscopic process on the first light beam to obtain the first sub-beam and the second sub-beam comprises: The first light beam is subjected to a splitting process by the beam splitter to obtain the first sub-beam and a third sub-beam incident on the reflector; The third sub-beam is reflected by the reflecting mirror to obtain the second sub-beam.

5. The method according to claim 1, characterized in that The controlling the image sensor to shift to a second position in the pixel dimension comprises: According to a preset shifting method, the image sensor is controlled to shift to the second position in the pixel dimension; the shifting method includes at least one of the following: a moving direction and a number of pixels moved.

6. The method according to claim 1, characterized in that The plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1; The step of synthesizing a target image according to the first image, the second image, the third image, and the fourth image comprises: For each of the first image, the second image, the third image, and the fourth image, superimpose pixels at corresponding positions of each image to obtain color information of each pixel in the N color channels; The target image is generated according to the color information of each pixel in the N color channels.

7. An image processing device, characterized in that: The invention comprises an image sensor, a control module and an image synthesis module, wherein the image sensor comprises a first filter component and a second filter component; the first filter component and the second filter component are both provided with a plurality of filter pixel units, and the color arrays of the plurality of filter pixel units of the first filter component and the second filter component are arranged differently; The first filter component is used for collecting a first light beam incident on the image sensor to obtain a first image when the image sensor is located at a first position; When the image sensor is located at the second position, collecting a second light beam incident on the image sensor to obtain a third image; The second filter component is used to collect the first light beam to obtain a second image when the image sensor is located at the first position; and to collect the second light beam to obtain a fourth image when the image sensor is located at the second position; The control module is used to control the image sensor to shift from the first position to the second position in the pixel dimension; The image synthesis module is used to synthesize a target image according to the first image, the second image, the third image and the fourth image; wherein the color arrays of the first image, the second image, the third image and the fourth image are arranged differently.

8. The device according to claim 7, characterized in that The image sensor further includes: a spectroscope; The beam splitter is used to perform beam splitting processing on the first light beam to obtain the first sub-beam and the second sub-beam.

9. The device according to claim 8, characterized in that The image sensor further includes: a reflector; The beam splitter is further used to perform beam splitting processing on the first light beam to obtain the first sub-beam and a third sub-beam incident on the reflector; The reflector is used to reflect the third sub-beam to obtain the second sub-beam.

10. The device according to claim 7, characterized in that The control module is further used to control the image sensor to shift from the first position to the second position in the pixel dimension according to a preset shifting method; the shifting method includes at least one of the following: a moving direction and a number of pixels moved.

11. The device according to claim 7, characterized in that The plurality of filter pixel units of the first filter component and the second filter component each include N color channels, where N is an integer greater than 1; The image synthesis module is further used to superimpose pixels at corresponding positions of each of the first image, the second image, the third image and the fourth image to obtain color information of each pixel in the N color channels; The target image is generated according to the color information of each pixel in the N color channels.

12. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

14. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 6.

Images