Image processing method and related device

Abstract

Embodiments of the present application provide an image processing method and related equipment, wherein the method comprises: obtaining multiple original images, wherein the multiple original images are images sensed by a photosensitive element at multiple imaging positions, and the distance between the two closest imaging positions in the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; and superimposing the multiple original images based on the multiple imaging positions to obtain a composite image, wherein the pixels in the composite image are sub-pixels divided in the superimposition process of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images. By using the embodiments of the present application, the image resolution can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and related equipment. Background Technology

[0002] Existing methods for improving image resolution rely on the minimum distance between two image capture positions being smaller than the distance *d* between any two adjacent pixels in a photosensitive element (also known as a photosensitive device, image sensor, or image sensor). Therefore, during image capture, the minimum distance the photosensitive element moves each time must be less than the distance between two adjacent pixels, for example, d / 2, d / 3, d / 5, etc. Because the minimum distance the photosensitive element moves each time is so small, a very high-precision actuator is required to achieve this. However, existing actuators cannot yet achieve this, resulting in extremely low yield rates for improving image resolution. Summary of the Invention

[0003] This application discloses an image processing method and related equipment, which can effectively improve image resolution.

[0004] In a first aspect, embodiments of this application disclose an image processing method, comprising: acquiring multiple original images, wherein the multiple original images are images sensed by a photosensitive element at multiple imaging positions respectively, the distance between the two imaging positions with the smallest distance among the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; superimposing the multiple original images based on the multiple imaging positions to obtain a composite image, wherein the pixels in the composite image are determined according to the sub-pixels obtained during the superposition of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images.

[0005] In this embodiment, a photosensitive element senses at multiple imaging positions, and an original image is obtained at each imaging position. Thus, multiple original images can be sensed from multiple imaging positions, and there is a one-to-one correspondence between the multiple imaging positions and the multiple original images. Then, the multiple original images are superimposed based on the multiple imaging positions during sensing, that is, the multiple original images are superimposed in a staggered manner on the same coordinate system (or plane), and when superimposed on the same coordinate system (or plane), any one of the multiple original images is superimposed at its corresponding imaging position. During the superposition of multiple original images, a pixel in any one of the original images is divided into multiple sub-pixels. The sub-pixels obtained during the superposition process can be used as pixels to form a composite image, so a pixel in the original image corresponds to multiple pixels in the composite image. Therefore, the resolution of the composite image is greater than the resolution of any one of the multiple original images. Since the distance between the two closest imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element, meaning the minimum distance the photosensitive element moves each time during imaging is greater than the distance between any two adjacent sensing pixels, the precision requirement for the actuator is not high. Imaging and superimposing images using existing actuators can effectively improve image resolution. It should be noted that the size of the composite image can be the same as the original image, or it can be slightly smaller. This is because multiple original images are stacked in a staggered manner, inevitably resulting in non-overlapping areas. For these non-overlapping areas, pixels in the original image are not divided into sub-pixels, and can be directly used as pixels in the composite image. In this case, the pixels in the composite image originate from non-overlapping areas, and the size of the composite image is the same as the original image. However, the number of pixels directly used as composite images is far less than the total number of pixels in the composite image, so the impact on the composite image (its application) is negligible. If the pixels in the composite image only originate from overlapping areas, the size of the composite image is slightly smaller than the size of the original image. It should be further noted that, compared to the prior art, the minimum distance that the photosensitive element moves each time is less than the distance between two adjacent pixels. In the embodiments of this application, the minimum distance that the photosensitive element moves each time is greater than the distance between two adjacent pixels. Therefore, the area of ​​the non-overlapping region caused by the misalignment and overlay of the original image in the embodiments of this application is larger than that in the prior art. That is, the number of pixels in the original image that are not divided into sub-pixels in the embodiments of this application is greater than that in the prior art. However, compared to the prior art, the number of pixels that are not divided into sub-pixels in the added part is far less than the total number of pixels in the synthesized image. Even if these pixels that are not divided into sub-pixels in the added part are directly used as pixels in the synthesized image, the impact on the synthesized image (application) can still be ignored.It should be understood that any sensing pixel in a photosensitive element can only sense one wavelength range of the spectrum, and the original image is an image directly sensed by the photosensitive element. Therefore, any pixel in the original image contains only one color information. When the photosensitive element senses only one wavelength range of the spectrum, that is, when all sensing pixels in the photosensitive element sense the same wavelength range of the spectrum, all pixels in the original image are the same color. When the photosensitive element senses three wavelength ranges of the spectrum, for example, taking four adjacent 2×2 sensing pixels as a spectral sensing unit, and the adjacent and regularly repeated spectral sensing units constitute the overall photosensitive element, the four sensing pixels in the spectral sensing unit can sense three wavelength ranges of the spectrum, that is, the four sensing pixels in the spectral sensing unit can sense three color spectra. Therefore, the original image includes pixels of three colors, and the color arrangement of all pixels in the original image corresponds to the arrangement of the spectral sensing units in the photosensitive element. When the photosensitive element senses three wavelength ranges of the spectrum, the vast majority of pixels in the synthesized image are sub-pixels obtained during the superposition process. The superposition process results in each sub-pixel having three colors superimposed; therefore, the vast majority of pixels in the synthesized image contain three color information. In contrast, pixels directly used from the original image as pixels in the synthesized image contain only one color information. Since the number of these pixels is far less than the total number of pixels in the synthesized image, it can be considered negligible. Therefore, it can be assumed that any pixel in the synthesized image contains three color information. For ease of description, the following assumes that any pixel in the synthesized image contains three color information. In summary, the embodiments of this application can effectively improve the spatial resolution and color resolution of the image.

[0006] In one possible implementation, the distance s between the two image capture positions with the smallest distance among the plurality of image capture positions is [(a×n+b) / n]×d; where a is a positive integer; n is a prime number; 1≤b≤n-1, where b is an integer; and d is the distance between any two adjacent sensing pixels in the photosensitive element.

[0007] In the embodiments of this application, the distance s between the two closest image capturing positions among the multiple image capturing positions is [(a×n+b) / n]×d, that is, the minimum moving distance s of the photosensitive element during each image capturing operation is (a+b / n)×d. Since 1≤b≤n-1, b / n is a fraction; and a is a positive integer, so a+b / n is greater than 1 and a+b / n is not an integer. In this way, the minimum moving distance s of the photosensitive element during each image capturing operation can be greater than the distance d between any two adjacent sensing pixels in the photosensitive element, and s is not an even multiple of d. In the embodiments of this application, the misalignment superposition of multiple original images requires that the pixels in the original images are divided into a plurality of sub-pixels, and the superposition requires that the pixels in different original images do not completely overlap. In the embodiments of this application, s is not an even multiple of d, which can prevent the pixels in different original images from completely overlapping during superposition, ensure that the pixels in the original images can be divided into sub-pixels in the superposition process, and is beneficial to ensuring that the sub-pixels have superposition of three color information. It should be noted that n is a prime number, n can be selected according to the multiple of image resolution to be improved as required, and the values of a and b can be selected according to the controllable accuracy of the actuator.

[0008] In a possible implementation, n is not 2 and n<f×(n-1), and the coordinates of any one of the multiple image capturing positions in the same coordinate system are (i×s+W / 2, j×s+H / 2); where

[0009] said m is the remainder of a×n+b divided by 2n;

[0010] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), said i and said j do not change at the same time, said i and said j both change monotonically, and said i and said j are integers; said W is the number of sensing pixels in the width direction of said photosensitive element; said H is the number of sensing pixels in the height direction of said photosensitive element; said photosensitive element senses spectra of three wavelength ranges, the spectra of said three wavelength ranges correspond to three color information, and the determining of the pixels in the synthesized image according to the sub-pixels obtained by division during the superposition process of the multiple original images includes: the pixels in the synthesized image are determined according to first sub-pixels obtained by division during the superposition process of the multiple original images, and said first sub-pixels contain said three color information.

[0011] In the embodiments of the present application, the image capturing position may be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at the image capturing position; the coordinates of any one of the plurality of image capturing positions in the same coordinate system are (i×s+W / 2, j×s+H / 2), where s is the minimum moving distance of the photosensitive element each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. When n is not 2 and n<f×(n-1), 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, both i and j change monotonically (for example, change monotonically increasing or change monotonically decreasing), and i and j are integers, so that the total number of the plurality of image capturing positions is 2n-1, where when n>m, f=m, and m is the remainder of a×n+b divided by 2n; when n<m, f=2n-m. In this way, one original image is obtained by sensing at each of the 2n-1 image capturing positions, so that a total of 2n-1 original images are obtained, and a composite image can be obtained by superimposing the 2n-1 original images. In the superimposing process, one pixel in the original image is divided into n×n sub-pixels, so that the resolution of the composite image can be greater than the resolution of the original image. Moreover, since the photosensitive element senses spectrums of three wavelength ranges, that is, the photosensitive element senses spectrums of three colors, the original image includes pixels of three colors, and in the process of superimposing the 2n-1 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three types of color information, and the first sub-pixels are used as pixels in the composite image, so that any pixel in the composite image can contain three types of color information.

[0012] In a possible implementation, said n is 2 or n>f×(n-1), and the coordinates of any one of said plurality of image capturing positions in the same coordinate system are (i×s+W / 2, j×s+H / 2); where,

[0013] said m is the remainder of a×n+b divided by 2n;

[0014] 0≤i≤n and 0≤j≤n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, the three wavelength ranges of spectrum correspond to three color information, and the pixel determination in the composite image based on the sub-pixel division obtained during the superposition of multiple original images includes: the pixel determination in the composite image based on the first sub-pixel division obtained during the superposition of multiple original images, the first sub-pixel containing the three color information.

[0015] In the embodiments of the present application, the imaging position may be represented by the coordinates of the center of the photosensitive element when the photosensitive element performs imaging at this imaging position; the coordinates of any one of the multiple imaging positions under the same coordinate system are (i×s+W / 2, j×s+H / 2), where s is the minimum moving distance of the photosensitive element each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. When n is 2, 0 ≤ i ≤ 2 and 0 ≤ j ≤ 2, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change at the same time, both i and j change monotonically (for example, monotonically increasing or monotonically decreasing), and i and j are integers, so there are a total of 5 imaging positions among the multiple imaging positions; in this way, one original image is sensed and obtained at each of the 5 imaging positions, so that a total of 5 original images are obtained, and the 5 original images can be superimposed to obtain a composite image. In the superimposition process, one pixel in the original image is divided into 2×2 sub-pixels, so that the resolution of the composite image can be greater than the resolution of the original image; moreover, since the photosensitive element senses spectra in three wavelength ranges, that is, the photosensitive element senses spectra of three colors, the original image includes pixels of three colors. In the superimposition process of the 5 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three color information, and the first sub-pixels are used as the pixels in the composite image, so that any pixel in the composite image can contain three color information. Alternatively, when n > f×(n-1), 0 ≤ i ≤ n and 0 ≤ j ≤ n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change at the same time, both i and j change monotonically (for example, monotonically increasing or monotonically decreasing), and i and j are integers, so there are a total of 2n+1 imaging positions among the multiple imaging positions, wherein when n > m, f = m, and m is the remainder of a×n+b divided by 2n; when n < m, f = 2n - m; in this way, one original image is sensed and obtained at each of the 2n+1 imaging positions, so that a total of 2n+1 original images are obtained, and the 2n+1 original images can be superimposed to obtain a composite image. In the superimposition process, one pixel in the original image is divided into n×n sub-pixels, and each sub-pixel has three superimposed color information, so that the resolution of the composite image can be greater than the resolution of the original image; moreover, since the photosensitive element senses spectra in three wavelength ranges, that is, the photosensitive element senses spectra of three colors, the original image includes pixels of three colors. In the superimposition process of the 2n+1 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three color information, and the first sub-pixels are used as the pixels in the composite image, so that any pixel in the composite image can contain three color information.

[0016] In one possible implementation, the determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: determining pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixel lacks one or two of the three color information; determining any missing color information of the second sub-pixel based on any color information of the third sub-pixel, wherein the third sub-pixel is a sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is a sub-pixel near the second sub-pixel that contains any of the color information.

[0017] In this embodiment of the application, in the process of superimposing multiple original images, in addition to dividing a first sub-pixel containing three colors, a second sub-pixel containing one or two colors can also be divided. For any color information missing in the second sub-pixel, it can be estimated based on the color information of the nearby sub-pixels containing that color information. After estimating the color information missing in the second sub-pixel, the first and second sub-pixels can be used as pixels of the synthesized image, so that any pixel in the synthesized image can also contain three colors.

[0018] In one possible implementation, n is 2 or n > f × (n-1), and the coordinates of any one of the plurality of image-taking positions are (i × s + W / 2, j × s + H / 2) in the same coordinate system; wherein,

[0019] The m is the remainder when a×n+b is divided by 2n;

[0020] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the pixels in the synthesized image are determined according to the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined according to the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel is missing one or two of the three color information; any missing color information of the second sub-pixel is determined according to any color information of the third sub-pixel, the third sub-pixel is the sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is the sub-pixel near the second sub-pixel that contains the any color information.

[0021] In this embodiment of the application, the image capture position can be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at that position; the coordinates of any one of the multiple image capture positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system, where s is the minimum distance the photosensitive element moves each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. When n is 2, 0 ≤ i ≤ 1 and 0 ≤ j ≤ 1. In any update of (i × s + W / 2, j × s + H / 2), i and j do not change simultaneously; both i and j are monotonically changing (e.g., monotonically increasing or monotonically decreasing). i and j are integers, so there are a total of 3 imaging positions. Thus, one original image is sensed at each of the above 3 imaging positions, resulting in a total of 3 original images. These 3 original images are superimposed to obtain a composite image. The superposition process divides a pixel in the original image into 2 × 2 sub-pixels. However, some sub-pixels have the superposition of three color information, while others do not. For the second sub-pixel that does not have the superposition of three color information, the missing color information is estimated through its neighboring sub-pixels. Specifically, the second... Any color information missing in a sub-pixel is estimated based on the same color information of a third sub-pixel near the second sub-pixel that contains that color information. For example, the color information of the third sub-pixel can be directly used as the color information of the second sub-pixel. If there are multiple third sub-pixels near the second sub-pixel, the color information of the second sub-pixel can be estimated using the color information of these multiple third sub-pixels. For example, the average of the color information of these multiple third sub-pixels can be used as the color information of the second sub-pixel. It should be understood that after estimating the color information of the sub-pixels with missing color information, all the sub-pixels obtained by superposition contain three colors of information. Therefore, the resolution of the synthesized image can be greater than the resolution of the original image, and any pixel in the synthesized image contains three colors of information.Alternatively, in the case of n>f×(n-1), where 0≤i≤n-1 and 0≤j≤n-1, during any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, both i and j change monotonically (for example, monotonically increasing or monotonically decreasing), and i and j are integers, so that the total number of the multiple imaging positions is 2n-1, wherein when n>m, f=m, and m is the remainder of a×n+b divided by 2n; when n<m, f=2n-m. In this way, one original image is sensed at each of the 2n-1 imaging positions described above, so that a total of 2n-1 original images are obtained. The 2n-1 original images can be superimposed to obtain a composite image. In the superimposition process, one pixel in the original image is divided into n×n sub-pixels, but some sub-pixels have three types of color information superimposed, and some sub-pixels do not have three types of color information superimposed. For a second sub-pixel that does not have three types of color information superimposed, any missing color information thereof is estimated through sub-pixels in the vicinity thereof. Specifically, any missing color information of the second sub-pixel is estimated according to the corresponding color information of a third sub-pixel that is in the vicinity of the second sub-pixel and contains the any color information, for example, directly using the any color information of the third sub-pixel as the any color information of the second sub-pixel. If there are a plurality of third sub-pixels in the vicinity of the second sub-pixel, the any color information of the second sub-pixel can be estimated through the any color information of the plurality of third sub-pixels, for example, taking the average value of the any color information of the plurality of third sub-pixels as the any color information of the second sub-pixel. It should be understood that after the color information estimation is performed on the sub-pixels with missing color information, all the sub-pixels obtained by superimposition contain three types of color information, so that the resolution of the composite image can be greater than the resolution of the original image, and any pixel in the composite image contains three types of color information.

[0022] In a possible implementation, the photosensitive element senses a spectrum in one wavelength range, and the coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system, wherein 0≤i≤n-1 and 0≤j≤n-1, during any update of (i×s+W / 2, j×s+H / 2), the i and the j do not change simultaneously, both the i and the j change monotonically, and the i and the j are integers; the W is the number of sensing pixels in the width direction of the photosensitive element; and the H is the number of sensing pixels in the height direction of the photosensitive element.

[0023] In this embodiment, the photosensitive element senses a spectrum of a wavelength range. The imaging position can be represented by the coordinates of the center of the photosensitive element when the image is captured at that imaging position. The coordinates of any imaging position among multiple imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system, where s is the minimum distance the photosensitive element moves each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. Since 0 ≤ i ≤ n-1 and 0 ≤ j ≤ n-1, in any update of (i × s + W / 2, j × s + H / 2), i and j do not change simultaneously; both i and j are monotonically changing (e.g., monotonically increasing or monotonically decreasing). Since i and j are integers, there are a total of 2n-1 imaging positions. Thus, one original image is sensed at each of the above 2n-1 imaging positions, resulting in a total of 2n-1 original images. These 2n-1 original images are then superimposed to obtain a composite image. The superposition process divides one pixel in the original image into n × n sub-pixels, thereby making the resolution of the composite image greater than that of the original image.

[0024] In one possible implementation, n is 2, the plurality of imaging positions are two imaging positions, and the coordinates of the two imaging positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein, W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the determination of pixels in the composite image based on sub-pixels obtained during the superposition of the plurality of original images includes: the pixels in the composite image are determined based on second sub-pixels obtained during the superposition of the plurality of original images, the second sub-pixel lacking one or two of the three color information; any missing color information of the second sub-pixel is determined based on any color information of the third sub-pixel, the third sub-pixel being a sub-pixel obtained during the superposition of the plurality of original images, and the third sub-pixel being a sub-pixel near the second sub-pixel containing the any color information.

[0025] In this embodiment, the image acquisition position can be represented by the coordinates of the center of the photosensitive element when it acquires an image at that position. When n is 2, multiple image acquisition positions can be represented by two positions, with coordinates (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, where W is the number of sensed pixels in the width direction of the photosensitive element, and H is the number of sensed pixels in the height direction. Thus, one original image is obtained from each of the two image acquisition positions, resulting in two original images. These two original images are then superimposed to obtain a composite image. The superposition process divides one pixel in the original image into 2×2 sub-pixels, thereby allowing the resolution of the composite image to be greater than that of the original image. However, since subpixels do not have the superposition of the three color information, the missing color information is estimated from the subpixels near them. Specifically, the missing color information of the second subpixel is estimated from the color information of the third subpixel near the second subpixel that contains the missing color information. For example, the missing color information of the third subpixel can be directly used as the missing color information of the second subpixel. If there are multiple third subpixels near the second subpixel, the missing color information of the second subpixel can be estimated from the missing color information of these multiple third subpixels. For example, the average of the missing color information of these multiple third subpixels can be used as the missing color information of the second subpixel. It should be understood that after estimating the color information of the subpixels with missing color information, all the subpixels obtained by superposition contain the three color information, so that any pixel in the synthesized image can contain the three color information.

[0026] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, where n is 2, and the plurality of imaging positions are two imaging positions, the coordinates of which are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0027] In the embodiments of the present application, the photosensitive element senses a spectrum in a wavelength range, and the image capturing position can be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at the image capturing position; when n is 2, the multiple image capturing positions may be two image capturing positions, and the coordinates of the two image capturing positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system respectively, where W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. In this way, one original image is sensed and obtained at each of the above two image capturing positions, so that a total of two original images are obtained. A synthesized image can be obtained by superimposing the above two original images. In the superimposing process, one pixel in the original image is divided into 2×2 sub-pixels, so that the resolution of the synthesized image can be greater than the resolution of the original image.

[0028] In a second aspect, the embodiments of the present application disclose an image processing apparatus, the beneficial effects of which can be referred to the corresponding description in the first aspect. The image processing apparatus includes: an acquisition unit, configured to acquire a plurality of original images, where the plurality of original images are images obtained by sensing respectively at a plurality of image capturing positions by a photosensitive element, and the distance between two image capturing positions with the smallest distance among the plurality of image capturing positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; and a superimposition unit, configured to superimpose the plurality of original images based on the plurality of image capturing positions to obtain a synthesized image, where pixels in the synthesized image are determined according to sub-pixels obtained by dividing during the process of superimposing the plurality of original images, and the resolution of the synthesized image is greater than the resolution of any one of the plurality of original images.

[0029] In a possible implementation, the distance s between the two image capturing positions with the smallest distance among the plurality of image capturing positions is [(a×n+b) / n]×d; the a is a positive integer; the n is a prime number; 1≤b≤n-1, and the b is an integer; the d is the distance between any two adjacent sensing pixels in the photosensitive element.

[0030] In a possible implementation, the n is not 2 and n<f×(n-1), and the coordinates of any one of the plurality of image capturing positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where,

[0031] the m is the remainder of a×n+b divided by 2n;

[0032] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, the three wavelength ranges of spectrum correspond to three color information, and the pixel determination in the composite image based on the sub-pixel division obtained during the superposition of multiple original images includes: the pixel determination in the composite image based on the first sub-pixel division obtained during the superposition of multiple original images, the first sub-pixel containing the three color information.

[0033] In one possible implementation, n is 2 or n > f × (n-1), and the coordinates of any one of the plurality of image-taking positions are (i × s + W / 2, j × s + H / 2) in the same coordinate system; wherein,

[0034] The m is the remainder when a×n+b is divided by 2n;

[0035] 0≤i≤n and 0≤j≤n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, the three wavelength ranges of spectrum correspond to three color information, and the pixel determination in the composite image based on the sub-pixel division obtained during the superposition of multiple original images includes: the pixel determination in the composite image based on the first sub-pixel division obtained during the superposition of multiple original images, the first sub-pixel containing the three color information.

[0036] In one possible implementation, the determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: determining pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixel lacks one or two of the three color information; determining any missing color information of the second sub-pixel based on any color information of the third sub-pixel, wherein the third sub-pixel is a sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is a sub-pixel near the second sub-pixel that contains any of the color information.

[0037] In one possible implementation, n is 2 or n > f × (n-1), and the coordinates of any one of the plurality of image-taking positions are (i × s + W / 2, j × s + H / 2) in the same coordinate system; wherein,

[0038] The m is the remainder when a×n+b is divided by 2n;

[0039] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the pixels in the synthesized image are determined according to the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined according to the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel is missing one or two of the three color information; any missing color information of the second sub-pixel is determined according to any color information of the third sub-pixel, the third sub-pixel is the sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is the sub-pixel near the second sub-pixel that contains the any color information.

[0040] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, and the coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein 0≤i≤n-1 and 0≤j≤n-1, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0041] In one possible implementation, n is 2, the plurality of imaging positions are two imaging positions, and the coordinates of the two imaging positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein, W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the determination of pixels in the composite image based on sub-pixels obtained during the superposition of the plurality of original images includes: the pixels in the composite image are determined based on second sub-pixels obtained during the superposition of the plurality of original images, the second sub-pixel lacking one or two of the three color information; any missing color information of the second sub-pixel is determined based on any color information of the third sub-pixel, the third sub-pixel being a sub-pixel obtained during the superposition of the plurality of original images, and the third sub-pixel being a sub-pixel near the second sub-pixel containing the any color information.

[0042] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, where n is 2, and the plurality of imaging positions are two imaging positions, the coordinates of which are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0043] Thirdly, embodiments of this application disclose an image processing apparatus, which includes a processor and a photosensitive element; the processor is configured to sense at multiple imaging positions using the photosensitive element to obtain multiple original images, wherein the distance between the two imaging positions with the smallest distance among the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; and to superimpose the multiple original images based on the multiple imaging positions to obtain a composite image, wherein the pixels in the composite image are determined according to the sub-pixels obtained during the superposition of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images.

[0044] Fourthly, embodiments of this application provide an image processing apparatus, including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing steps in the method as described in any one of the first aspects.

[0045] Fifthly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing steps in the method as described in any one of the first aspects.

[0046] In a sixth aspect, embodiments of this application provide a chip, including: a processor for calling and running a computer program from a memory, causing a device on which the chip is mounted to perform the method as described in any one of the first aspects.

[0047] In a seventh aspect, embodiments of this application provide a computer-readable storage medium comprising a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of the first to fourth aspects above.

[0048] Eighthly, embodiments of this application provide a computer program product comprising a computer program that, when run on a computer or processor, causes the computer or processor to perform the method as described in any one of the first aspects above. Attached Figure Description

[0049] The accompanying drawings used in the embodiments of this application are described below.

[0050] Figure 1A This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0051] Figure 1B This is a schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0052] Figure 1C This is a schematic diagram of the structure of the camera provided in an embodiment of this application;

[0053] Figure 2 This is a schematic diagram of a user interface for taking photos provided in an embodiment of this application;

[0054] Figure 3 This is a schematic diagram of an image overlay provided in an embodiment of this application;

[0055] Figure 4 This is a schematic diagram of another image overlay provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a composite image provided in an embodiment of this application;

[0057] Figure 6is a schematic diagram of a user interface for an album provided by an embodiment of the present application;

[0058] Figure 7 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0059] Figures 8A to 8D is a schematic diagram of possible values of i and j provided by an embodiment of the present application;

[0060] Figure 9 is a tree diagram of the values of i and j when n is 3 under the condition of n<f×(n-1) provided by an embodiment of the present application;

[0061] Figure 10 is a schematic diagram of a coordinate system provided by an embodiment of the present application;

[0062] Figure 11 is a tree diagram of the values of i and j when n is 3 under the condition of n>f×(n-1) provided by an embodiment of the present application;

[0063] Figure 12 is a schematic diagram of another coordinate system provided by an embodiment of the present application;

[0064] Figure 13 is a schematic diagram of a photosensitive element provided by an embodiment of the present application;

[0065] Figure 14 is a schematic diagram of division of sensing sub-pixels in a photosensitive element provided by an embodiment of the present application;

[0066] Figure 15 is a schematic diagram of dividing sub-pixels in the image row direction provided by an embodiment of the present application;

[0067] Figure 16 is a schematic diagram of another way of dividing sub-pixels in the image row direction provided by an embodiment of the present application;

[0068] Figure 17 is a schematic diagram of dividing sub-pixels in the image column direction provided by an embodiment of the present application;

[0069] Figure 18 is a schematic diagram of another way of dividing sub-pixels in the image column direction provided by an embodiment of the present application;

[0070] Figure 19 is a schematic diagram of image composition provided by an embodiment of the present application;

[0071] Figure 20 is a schematic diagram of division of sensing sub-pixels in another photosensitive element provided by an embodiment of the present application;

[0072] Figure 21This is a schematic diagram illustrating another method of dividing sub-pixels in the row direction of an image, provided in an embodiment of this application.

[0073] Figure 22 This is another schematic diagram of dividing sub-pixels in the row direction of an image provided in the embodiments of this application;

[0074] Figure 23 This is a schematic diagram illustrating the division of sensing sub-pixels in another photosensitive element provided in an embodiment of this application;

[0075] Figure 24 This is another schematic diagram of dividing sub-pixels in the row direction of an image provided in the embodiments of this application;

[0076] Figure 25 This is another schematic diagram of dividing sub-pixels in the row direction of an image provided in the embodiments of this application;

[0077] Figure 26 This is another schematic diagram of dividing sub-pixels in the row direction of an image provided in the embodiments of this application;

[0078] Figure 27 This is another schematic diagram of dividing sub-pixels in the row direction of an image provided in the embodiments of this application;

[0079] Figure 28 This is a schematic diagram of an original image provided in an embodiment of this application;

[0080] Figure 29 This is a schematic diagram of another original image provided in an embodiment of this application;

[0081] Figure 30 This is a schematic diagram of an overlay of an original image provided in an embodiment of this application;

[0082] Figure 31 This is a schematic diagram of another image synthesis provided in an embodiment of this application;

[0083] Figure 32 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0084] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0085] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0086] In this specification, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0087] First, an exemplary electronic device provided in the following embodiments of this application is introduced.

[0088] Please see Figure 1A , Figure 1A This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0089] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0090] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0091] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0092] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0093] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0094] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0095] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0096] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0097] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0098] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0099] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0100] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0101] The modem processor may include a modulator and a demodulator. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0102] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0103] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0104] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0105] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0106] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0107] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0108] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0109] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0110] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0111] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0112] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0113] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0114] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0115] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0116] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0117] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0118] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0119] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0120] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0121] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100.

[0122] The 180C barometric pressure sensor is used to measure barometric pressure.

[0123] The magnetic sensor 180D includes a Hall sensor.

[0124] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0125] Distance sensor 180F is used to measure distance.

[0126] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0127] The 180L ambient light sensor is used to detect ambient light intensity.

[0128] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0129] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy.

[0130] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0131] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0132] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0133] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to different touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0134] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0135] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0136] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0137] Please see Figure 1B , Figure 1B This is a schematic diagram of the software structure of the electronic device 100 provided in the embodiments of this application.

[0138] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0139] The application layer can include a series of application packages.

[0140] like Figure 1B As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0141] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0142] like Figure 1BAs shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, event manager, etc.

[0143] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0144] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0145] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0146] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0147] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0148] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0149] The event manager can be used to determine whether the touch coordinates of a user's touch operation are within the first area when the first control mode is enabled. If yes, the event for that touch operation is reported to the application layer; otherwise, no action is taken on the touch operation.

[0150] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0151] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0152] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0153] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0154] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0155] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0156] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0157] A 2D graphics engine is a graphics engine for 2D drawing. The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0158] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 100 when the first control mode is enabled.

[0159] With the first control mode enabled, when touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer, and the event manager determines whether the touch coordinates are within the first area. If so, it identifies the control corresponding to the raw input event. For example, if the touch operation is a single click and the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193. If not, no processing is performed on the raw input event.

[0160] Please see Figure 1C , Figure 1CThis is a schematic diagram of the structure of the camera provided in the embodiments of this application; as shown Figure 1C As shown, the camera 193 includes a actuator assembly 193A, a lens assembly 193B, and a photosensitive element assembly 193C.

[0161] The aforementioned lens assembly 193B includes at least one lens and at least one lens bracket. The optical axis of the lens is perpendicular to the plane of the sensing pixels in the aforementioned photosensitive element assembly 193C and is disposed in the lens bracket. The lens bracket is used to carry the lens and support the movement of the lens relative to the photosensitive element assembly 193C.

[0162] The aforementioned brake assembly 193A has the capability to move the relative position between the photosensitive element assembly 193C and the lens assembly 193B in at least two directions on a plane perpendicular to the optical axis of the lens. This capability includes the ability to move simultaneously in two mutually orthogonal directions and the ability to rotate about the optical axis on the plane perpendicular to the optical axis. When moving the relative position between the photosensitive element assembly 193C and the lens assembly 193B, the target of movement can be either the photosensitive element assembly 193C or the lens assembly 193B. The braking force generated by the brake assembly 193A when moving the relative position between the photosensitive element assembly 193C and the lens assembly 193B can be electromagnetic force, electrostatic force, or stress generated in the braking material due to temperature changes, etc.

[0163] The aforementioned photosensitive element assembly 193C includes at least one photosensitive element and at least one chip holder. The photosensitive element is used to sense light that passes through the lens and reaches the sensing pixels. The chip holder at least provides a structure to support the photosensitive element and supports the actuator assembly 193A to perform the function of moving the relative position of the photosensitive element assembly 193C and the lens assembly 193B. The sensing pixels of the photosensitive element can be sensing pixels that sense the same spectrum. Alternatively, at any position on the photosensitive element, four adjacent 2×2 sensing pixels can form a spectral sensing unit. Adjacent and regularly repeated spectral sensing units constitute the overall photosensitive element. The four sensing pixels in the spectral sensing unit can sense three or four wavelength ranges of the spectrum.

[0164] The total number of movements of the brake assembly 193A, the distance of movement relative to the position of each movement, the dwell time after moving to a certain position, the movement time from one position to another, and other relevant parameters of the brake assembly 193A can be set according to requirements.

[0165] Among them, after the brake assembly 193A moves to a certain position, the number of times the photosensitive element takes pictures, the exposure time of each picture, and other relevant parameters of the photosensitive element can be set according to requirements.

[0166] Secondly, an application scenario that may be involved in the embodiments of this application will be introduced by way of example.

[0167] Please see Figure 2 , Figure 2 This is a schematic diagram of a user interface (UI) for taking photos provided in an embodiment of this application; as shown. Figure 2 As shown, when a user takes a picture of a target using an electronic device, the user activates the device's camera function. The screen displays a camera interface. Once the camera is pointed at the target, the user presses the shutter button on the camera interface, and the device captures an image of the target using its camera. The method by which the electronic device captures the image of the target using its camera is as follows:

[0168] Example Method 1: First, before the brake assembly controls the movement of the photosensitive element or lens, the photosensitive element senses at least one original image at its original position; then, the brake assembly controls the photosensitive element or lens to move in one of two orthogonal directions on a plane perpendicular to the optical axis of the lens. While moving in one direction, the brake pauses at each specific position for a period of time, and the photosensitive element senses at least one original image at that specific position, until the movement in one direction ends; after the movement in one direction ends, movement begins in the other direction. Similarly, while moving in the other direction, the brake pauses at each specific position for a period of time, and the photosensitive element senses at least one original image at that specific position, until the movement in the other direction ends.

[0169] After capturing images in two orthogonal directions, the electronic device superimposes all the original images obtained from the capture onto the same plane according to the capture position of each original image. The superposition process divides the pixels in the original image into sub-pixels, and uses the sub-pixels as pixels to obtain the composite image.

[0170] For example, such as Figure 3As shown, the two orthogonal directions on the plane perpendicular to the optical axis of the lens are the x-direction and the y-direction. First, the photosensitive element senses an original image 0 at its original position. Then, the actuator assembly controls the photosensitive element or lens to move in the x-direction. During the x-direction movement, the photosensitive element senses an original image at n-1 or n positions, thus acquiring a total of n-1 or n original images. After the x-direction movement ends, the actuator assembly controls the photosensitive element or lens to move in the y-direction. During the y-direction movement, the photosensitive element senses an original image at n-1 or n positions, thus acquiring a total of n-1 or n original images. After multiple image acquisitions, 2n-1 or 2n+1 original images can be obtained. The electronic device superimposes the 2n-1 or 2n+1 original images onto the same plane according to the image acquisition position of each original image. The superposition process divides the pixels in the original images into sub-pixels, and uses the sub-pixels as pixels to obtain a composite image. The composite image is shown below. Figure 5 As shown.

[0171] Example Method 2: First, before the brake assembly controls the movement of the photosensitive element or lens, the photosensitive element senses at least one original image at its original position; then, the brake assembly controls the photosensitive element or lens to move at the same rate in two orthogonal directions on a plane perpendicular to the optical axis of the lens. Each time the lens moves to a specific position, the photosensitive element senses at least one original image at that specific position until the movement ends.

[0172] After image acquisition is completed, the electronic device will superimpose all the original images obtained from the image acquisition onto the same plane according to the image acquisition position of each original image. During the superposition process, the pixels in the original image are divided into sub-pixels, and the sub-pixels are used as pixels to obtain the composite image.

[0173] For example, such as Figure 4 As shown, the two orthogonal directions on the plane perpendicular to the optical axis of the lens are the x-direction and the y-direction. First, the photosensitive element senses an original image 0 at its original position; then, the actuator combination controls the photosensitive element or lens to move simultaneously at the same speed in the x-direction and the y-direction. When it moves to a certain position, the photosensitive element senses an original image 1 at that position. After two image captures, two original images are obtained. The electronic device superimposes the two original images onto the same plane according to the image capture position of each original image. The superposition process divides the pixels in the original images into sub-pixels, and uses the sub-pixels as pixels to obtain a composite image, as shown in the figure. Figure 5 As shown.

[0174] The aforementioned composite image can be considered as an image of the target captured by the camera. After the electronic device captures the image of the target using the exemplary method described above, it stores the composite image, which is then stored in the electronic device's photo album. Figure 6 As shown, the image of the target captured by the camera and displayed in the album's user interface is this composite image.

[0175] The technical solution provided in this application is described below with reference to specific implementation methods.

[0176] Please see Figure 7 , Figure 7 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The image processing method can be executed by an electronic device and includes, but is not limited to, the following steps:

[0177] 701. Acquire multiple original images, wherein the multiple original images are images sensed by a photosensitive element at multiple imaging positions respectively, and the distance between the two imaging positions with the smallest distance among the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element.

[0178] 702. The multiple original images are superimposed based on the multiple image capture positions to obtain a composite image, wherein the pixels in the composite image are determined according to the sub-pixels obtained during the superposition of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images.

[0179] In this embodiment, a photosensitive element senses at multiple imaging positions, and an original image is obtained at each imaging position. Thus, multiple original images can be sensed from multiple imaging positions, and there is a one-to-one correspondence between the multiple imaging positions and the multiple original images. Then, the multiple original images are superimposed based on the multiple imaging positions during sensing, that is, the multiple original images are superimposed in a staggered manner on the same coordinate system (or plane), and when superimposed on the same coordinate system (or plane), any one of the multiple original images is superimposed at its corresponding imaging position. During the superposition of multiple original images, a pixel in any one of the original images is divided into multiple sub-pixels. The sub-pixels obtained during the superposition process can be used as pixels to form a composite image, so a pixel in the original image corresponds to multiple pixels in the composite image. Therefore, the resolution of the composite image is greater than the resolution of any one of the multiple original images. Since the distance between the two closest imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element, meaning the minimum distance the photosensitive element moves each time during imaging is greater than the distance between any two adjacent sensing pixels, the precision requirement for the actuator is not high. Imaging and superimposing images using existing actuators can effectively improve image resolution. It should be noted that the size of the composite image can be the same as the original image, or it can be slightly smaller. This is because multiple original images are stacked in a staggered manner, inevitably resulting in non-overlapping areas. For these non-overlapping areas, pixels in the original image are not divided into sub-pixels, and can be directly used as pixels in the composite image. In this case, the pixels in the composite image originate from non-overlapping areas, and the size of the composite image is the same as the original image. However, the number of pixels directly used as composite images is far less than the total number of pixels in the composite image, so the impact on the composite image (its application) is negligible. If the pixels in the composite image only originate from overlapping areas, the size of the composite image is slightly smaller than the size of the original image. It should be further noted that, compared to the prior art, the minimum distance that the photosensitive element moves each time is less than the distance between two adjacent pixels. In the embodiments of this application, the minimum distance that the photosensitive element moves each time is greater than the distance between two adjacent pixels. Therefore, the area of ​​the non-overlapping region caused by the misalignment and overlay of the original image in the embodiments of this application is larger than that in the prior art. That is, the number of pixels in the original image that are not divided into sub-pixels in the embodiments of this application is greater than that in the prior art. However, compared to the prior art, the number of pixels that are not divided into sub-pixels in the added part is far less than the total number of pixels in the synthesized image. Even if these pixels that are not divided into sub-pixels in the added part are directly used as pixels in the synthesized image, the impact on the synthesized image (application) can still be ignored.It should be understood that any sensing pixel in a photosensitive element can only sense one wavelength range of the spectrum, and the original image is an image directly sensed by the photosensitive element. Therefore, any pixel in the original image contains only one color information. When the photosensitive element senses only one wavelength range of the spectrum, that is, when all sensing pixels in the photosensitive element sense the same wavelength range of the spectrum, all pixels in the original image are the same color. When the photosensitive element senses three wavelength ranges of the spectrum, for example, taking four adjacent 2×2 sensing pixels as a spectral sensing unit, and the adjacent and regularly repeated spectral sensing units constitute the overall photosensitive element, the four sensing pixels in the spectral sensing unit can sense three wavelength ranges of the spectrum, that is, the four sensing pixels in the spectral sensing unit can sense three color spectra. Therefore, the original image includes pixels of three colors, and the color arrangement of all pixels in the original image corresponds to the arrangement of the spectral sensing units in the photosensitive element. When the photosensitive element senses three wavelength ranges of the spectrum, the vast majority of pixels in the synthesized image are sub-pixels obtained during the superposition process. The superposition process results in each sub-pixel having three colors superimposed; therefore, the vast majority of pixels in the synthesized image contain three color information. In contrast, pixels directly used from the original image as pixels in the synthesized image contain only one color information. Since the number of these pixels is far less than the total number of pixels in the synthesized image, it can be considered negligible. Therefore, it can be assumed that any pixel in the synthesized image contains three color information. For ease of description, the following assumes that any pixel in the synthesized image contains three color information. In summary, the embodiments of this application can effectively improve the spatial resolution and color resolution of the image.

[0180] In one possible implementation, the distance s between the two image capture positions with the smallest distance among the plurality of image capture positions is [(a×n+b) / n]×d; where a is a positive integer; n is a prime number; 1≤b≤n-1, where b is an integer; and d is the distance between any two adjacent sensing pixels in the photosensitive element.

[0181] Specifically, the embodiments of the present application provide a solution for improving image resolution, which can be implemented by using an actuator with a movement accuracy greater than the distance between two adjacent sensing pixels in the photosensitive element. Assume that the distance between two adjacent sensing pixels in the photosensitive element is d, that is, the pixel size is d×d, the total number of sensing pixels (pixel number) in the photosensitive element is H×W, H is the number of sensing pixels in the height direction of the photosensitive element, and W is the number of sensing pixels in the width direction of the photosensitive element. The minimum moving distance of the photosensitive element each time is s, where s = [(a×n+b) / n]×d; in the foregoing formula, n is a prime number much smaller than H or W, and n can be selected according to the multiple of resolution to be improved, for example, n can be 2, 3, 5, 7, etc.; a is a positive integer; b is an integer greater than or equal to 1 and less than or equal to n-1; the values of a and b can be selected according to the controllable accuracy of the actuator.

[0182] In the embodiments of the present application, the distance s between the two closest image capturing positions among the multiple image capturing positions is [(a×n+b) / n]×d, that is, the minimum moving distance s of the photosensitive element each time during image capturing is (a+b / n)×d. Since 1≤b≤n-1, b / n is a fraction; and a is a positive integer, so a+b / n is greater than 1 and a+b / n is not an integer. In this way, the minimum moving distance s of the photosensitive element each time during image capturing can be made greater than the distance d between any two adjacent sensing pixels in the photosensitive element, and s is not an even multiple of d; in the embodiments of the present application, the misalignment superposition of multiple original images requires that the pixels in the original images are divided into a plurality of sub-pixels, and the pixels in different original images are required not to completely overlap during superposition. In the embodiments of the present application, s is not an even multiple of d, which can prevent the pixels in different original images from completely overlapping during superposition, ensure that the pixels in the original images can be divided into sub-pixels during the superposition process, and is beneficial to ensuring that the sub-pixels have superposition of three types of color information. It should be noted that n is a prime number, n can be selected according to the multiple of image resolution to be improved, and the values of a and b can be selected according to the controllable accuracy of the actuator.

[0183] In a possible implementation, n is not 2 and n<f×(n-1), and the coordinates of any one of the multiple image capturing positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein,

[0184] said m is the remainder of a×n+b divided by 2n;

[0185] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, both i and j change monotonically, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses spectra of three wavelength ranges, the spectra of the three wavelength ranges correspond to three color information, and the determining of the pixels in the composite image according to the sub-pixels obtained by division in the superposition process of the multiple original images comprises: the pixels in the composite image are determined according to the first sub-pixels obtained by division in the superposition process of the multiple original images, and the first sub-pixels contain the three color information.

[0186] Wherein m is a modulus, m is the remainder of a×n+b divided by 2n, which can be expressed as m=(a×n+b) mod (2×n), that is, performing a modulo operation on (a×n+b) with respect to (2×n); the reason for performing the modulo operation with respect to (2×n) is that the arrangement of pixels in rows / columns has a period of 2×n.

[0187] It should be noted that, for the coordinates of the imaging position (i×s+W / 2, j×s+H / 2), 0≤i≤n-1 and 0≤j≤n-1, i and j are integers, both i and j change monotonically, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously; that is to say, the values of i and j change in a monotonically increasing or monotonically decreasing manner, and in any update of the coordinates of the imaging position (i×s+W / 2, j×s+H / 2), only one of the values of i and j can change; moreover, when the values of i and j change in a monotonically increasing manner, if neither i nor j reaches the maximum value n-1, either i or j can be selected to change, and when one of i and j reaches the maximum value n-1, only the one that does not reach the maximum value n-1 can change; when the values of i and j change in a monotonically decreasing manner, if neither i nor j reaches the minimum value 0, either i or j can be selected to change, and when one of i and j reaches the minimum value 0, only the one that does not reach the minimum value 0 can change.

[0188] Figure 8A , Figure 8B , Figure 8C and Figure 8D show several possible values of i and j, and the present application is not limited thereto. It should be understood that different values of i and j correspond to different coordinates of the imaging position. Figure 9 shows a tree diagram of the values of i and j when n is 3 under the condition that n<f×(n-1).

[0189] Any one of the multiple imaging positions in the embodiments of this application can be represented by coordinates in the same coordinate system. Please refer to... Figure 10 , Figure 10 This is a schematic diagram of a coordinate system provided in an embodiment of this application. This coordinate system is established on the plane where the photosensitive element is located. Figure 10 As shown, a coordinate system xOy is established with the upper left vertex of the photosensitive element in its initial position as the origin O(0,0), the width direction of the photosensitive element as the x-axis, and the height direction of the photosensitive element as the y-axis. In this context, the coordinates of the top-left vertex of the photosensitive element at any imaging position can be represented as (i, j) in the xOy coordinate system. The x-coordinate (i, j) represents the distance of the photosensitive element from the y-axis at that imaging position, and the y-coordinate (j) represents the distance of the photosensitive element from the x-axis at that imaging position. The distance between the photosensitive element and the x-axis or y-axis can be represented by the distance between the center of the photosensitive element and the x-axis or y-axis. Therefore, the imaging position of the photosensitive element can be determined by the coordinates (i, j) of the top-left vertex of the photosensitive element, the number of sensing pixels H in the height direction of the photosensitive element, and the number of sensing pixels W in the width direction of the photosensitive element. That is, when the coordinates of the top-left vertex of the photosensitive element are (i, j), the imaging position of the photosensitive element is represented as (i + W / 2, j + H / 2). It should be noted that... Figure 10 Therefore Figure 8A The values ​​of i and j shown are used to determine the coordinates of the image acquisition position. As an example, all coordinates described in this article can be... Figure 10 Coordinates in the described coordinate system.

[0190] For example, the image acquisition process and the image composition process of this implementation are described below, wherein the moving target being the lens and the moving target being the photosensitive element are equivalent. The following description uses the moving photosensitive element as an example:

[0191] I. Image acquisition process.

[0192] In the coordinate system xOy, the photosensitive element takes images at the image-taking positions (W / 2, H / 2), (s+W / 2, H / 2), (2s+W / 2, H / 2), (3s+W / 2, H / 2), ..., ((n-2)s+W / 2, H / 2), ((n-1)s+W / 2, H / 2) and ((n-1)s+W / 2, s+H / 2), ((n-1)s+W / 2, 2s+H / 2), ((n-1)s+W / 2, 3s+H / 2), ..., ((n-1)s+W / 2, (n-2)s+H / 2), ((n-1)s+W / 2, (n-1)s+H / 2) to obtain 2n-1 original images. For example, the photosensitive element moves in the following order: first in one of the x-direction and y-direction, then in the other direction; for example, first in the x-direction, and after completing the movement in the x-direction, then in the y-direction. Here, the movement described in this application refers to translation, with a total of 2n-2 movements, including n-1 movements in the x-direction and n-1 movements in the y-direction; the total number of image captures is 2n-1, including one image capture at the original position, n-1 image captures in the x-direction, and n-1 image captures in the y-direction; the specific movement process is as follows:

[0193] 1) Take an image at the original position.

[0194] At time t(0), the photosensitive element takes an image at the original position, that is, the photosensitive element takes an image at the image position (W / 2, H / 2) to obtain the original image 0.

[0195] 2) Moving and capturing images in the x-direction.

[0196] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance s in the x direction, that is, from the image acquisition position (W / 2, H / 2) to the image acquisition position (s+W / 2, H / 2); the photosensitive element acquires an image at the position at time t(1)_x, that is, the photosensitive element acquires an image at the position (s+W / 2, H / 2) to obtain the original image x(1).

[0197] During the time interval t(1)_x to t(2)_x, the photosensitive element moves a distance s in the x direction, that is, from the image acquisition position (s+W / 2, H / 2) to the image acquisition position (2s+W / 2, H / 2); the photosensitive element acquires an image at the position at time t(2)_x, that is, the photosensitive element acquires an image at the position (2s+W / 2, H / 2) to obtain the original image x(2).

[0198] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance s in the x direction, that is, from the image acquisition position (2s+W / 2, H / 2) to the image acquisition position (3s+W / 2, H / 2); the photosensitive element acquires an image at the position at time t(3)_x, that is, the photosensitive element acquires an image at the position (3s+W / 2, H / 2) to obtain the original image x(3). ......

[0200] During the time interval t(n-3)_x to t(n-2)_x, the photosensitive element moves a distance s in the x direction, that is, from the image acquisition position ((n-3)s+W / 2, H / 2) to the image acquisition position ((n-2)s+W / 2, H / 2); the photosensitive element acquires an image at the position at time t(n-2)_x, that is, the photosensitive element acquires an image at the position ((n-2)s+W / 2, H / 2) to obtain the original image x(n-2).

[0201] During the time interval t(n-2)_x to t(n-1)_x, the photosensitive element moves a distance s in the x direction, that is, from the image acquisition position ((n-2)s+W / 2, H / 2) to the image acquisition position ((n-1)s+W / 2, H / 2); the photosensitive element acquires an image at the position at time t(n-1)_x, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, H / 2) to obtain the original image x(n-1).

[0202] At this point, the movement and image acquisition of the photosensitive element in the x-direction have been completed, and the movement and image acquisition in the y-direction will then begin.

[0203] 3) Move in the y direction and capture the image.

[0204] During the time interval from t(n-1)_x to t(1)_y, the photosensitive element moves a distance s in the y direction, that is, from the image acquisition position ((n-1)s+W / 2, H / 2) to the image acquisition position ((n-1)s+W / 2, s+H / 2); the photosensitive element acquires an image at the position at time t(1)_y, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, s+H / 2) to obtain the original image y(1).

[0205] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance s in the y direction, that is, from the image acquisition position ((n-1)s+W / 2, s+H / 2) to the image acquisition position ((n-1)s+W / 2, 2s+H / 2); the photosensitive element acquires an image at the position at time t(2)_y, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, 2s+H / 2) to obtain the original image y(2).

[0206] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance s in the y direction, that is, from the image acquisition position ((n-1)s+W / 2, 2s+H / 2) to the image acquisition position ((n-1)s+W / 2, 3s+H / 2); the photosensitive element acquires an image at the position at time t(3)_y, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, 3s+H / 2) to obtain the original image y(3). ......

[0208] During the time interval from t(n-3)_y to t(n-2)_y, the photosensitive element moves a distance s in the y direction, that is, from the image acquisition position ((n-1)s+W / 2, (n-3)s+H / 2) to the image acquisition position ((n-1)s+W / 2, (n-2)s+H / 2); the photosensitive element acquires an image at the position at time t(n-2)_y, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, (n-2)s+H / 2) to obtain the original image y(n-2).

[0209] During the time interval from t(n-2)_y to t(n-1)_y, the photosensitive element moves a distance s in the y direction, that is, from the image acquisition position ((n-1)s+W / 2, (n-2)s+H / 2) to the image acquisition position ((n-1)s+W / 2, (n-1)s+H / 2); the photosensitive element acquires an image at the position at time t(n-1)_y, that is, the photosensitive element acquires an image at the position ((n-1)s+W / 2, (n-1)s+H / 2) to obtain the original image y(n-1).

[0210] At this point, the movement of the photosensitive element in the x-direction and y-direction, as well as image acquisition, have all been completed.

[0211] It should be noted that the moving sequence of the photosensitive element described in this application is merely exemplary. Any moving sequence that can achieve the same effect as the imaging positions described in this application is within the scope of protection of this application. This application does not list all possible moving sequences of the photosensitive element, nor does it impose any specific limitations on the moving sequence of the photosensitive element.

[0212] II. Synthesis process.

[0213] It should be understood that image resolution includes image spatial resolution and image color resolution. The process of improving image spatial resolution and image color resolution in the embodiments of this application is described below.

[0214] 1) Improve spatial resolution.

[0215] Projecting all the original images (e.g., 2n-1 original images) obtained from the image acquisition positions onto the xOy coordinate system reveals overlapping areas between the original images. Specifically, since the moving distance s = [(a×n+b) / n]×d, after the photosensitive element moves multiple times to acquire images, and then projects all the original images obtained from the image acquisition positions onto the xOy coordinate system and superimposes them, the superposition process can divide a pixel in the original image (excluding pixels that do not overlap) into n×n sub-pixels, where the size of each sub-pixel is d / n×d / n, that is, dividing a pixel in the original image into n rows and n columns of sub-pixels. Therefore, in the process of integrating all the original images to obtain the composite image, each pixel in each of the original images can be divided into n×n sub-pixels, which is equivalent to one sensing pixel in the photosensitive element corresponding to n×n sub-pixels in the original image, or one sensing pixel in the photosensitive element corresponding to n×n pixels in the composite image. The total number of sensing pixels in the photosensitive element is H×W, so the total number of pixels in each original image is also H×W, resulting in a total of H×W×n×n sub-pixels in each original image. Each sub-pixel is used as a pixel in the composite image, resulting in a composite image with H×W×n×n pixels. It can be understood that compared to the original image, the total number of pixels in the composite image is n×n times that of the original image; the size of the composite image remains unchanged, but the pixel spacing is 1 / n of the original, meaning the spatial resolution of the composite image is n×n times that of the original image.

[0216] 2) Improve color resolution.

[0217] There are three types of subpixels separated during the overlay of all original images. Assuming the image is in ACCD format, A, C, and D represent different colors, that is, the colors corresponding to the three wavelength ranges of the spectrum are color A, color C, and color D, respectively. The following is an example using RGGB or RYYB format, described in detail below:

[0218] Type 1 subpixels: These consist of a superposition of three colors. When the original image is in RGGB or RYYB format, R, G, B, and Y represent red, green, blue, and yellow, respectively. For example, moving in the x-direction can yield R and G / Y, while moving in the y-direction can yield G / Y and B, or vice versa, and so on.

[0219] The second type of subpixel: has two colors superimposed. For example, R and G / Y can only be obtained when moving in the x direction, or G / Y and B can only be obtained when moving in the x direction, or R and G / Y can only be obtained when moving in the y direction, or G / Y and B can only be obtained when moving in the y direction, and so on.

[0220] The third type of subpixel: a superposition of only one color. For example, when moving in the x-direction and the y-direction, only one of the colors R, G, B, and Y can be obtained.

[0221] It should be noted that, from Figure 10 As can be seen, connecting all the image acquisition points forms a broken line with a 90° angle. The overall image acquisition position first moves in the x-direction and then in the y-direction. Therefore... Figure 10 The area enclosed by the solid line represents the total imaging area for 2n-1 imaging operations. This total imaging area can be divided into the original imaging area (i.e., the imaging area of ​​the original image 0) and the newly added imaging area. The original imaging area is the imaging area of ​​the original image 0, specifically the area where 0 ≤ x ≤ W and 0 ≤ y ≤ H. The imaging information of the output composite image originates entirely from the imaging information obtained within this original imaging area. Within the original imaging area, the area that the photosensitive element can capture in each imaging operation (the area that the photosensitive element can cover in each imaging operation), specifically (n-1)s ≤ x ≤ W and (n-1)s ≤ y ≤ H), is defined as the complete imaging area. The other areas within the original imaging area besides the complete imaging area, specifically the areas where 0 ≤ x ≤ (n-1)s and 0 ≤ y ≤ H, and the areas where (n-1)s ≤ x ≤ W and 0 ≤ y ≤ (n-1)s, are defined as incomplete imaging areas. The newly added imaging area refers to the area that increases compared to the original imaging area due to the movement of the photosensitive element. Specifically, it includes the regions where W ≤ x ≤ W + (n-1)s and 0 ≤ y ≤ H, and the regions where (n-1)s ≤ x ≤ W + (n-1)s and H ≤ y ≤ H + (n-1)s. Therefore, the first type of subpixels are mainly distributed in the complete imaging area, while the second type and the first type of subpixels are distributed in the incomplete imaging area and the newly added imaging area, respectively.

[0222] It should be further noted that the area of ​​the complete imaging region is much larger than the area of ​​the incomplete imaging region, and the area of ​​the complete imaging region is close to the area of ​​the original imaging region. Therefore, the incomplete imaging region can be ignored, and the complete imaging region can be approximately considered to be the original imaging region. It can also be approximately considered that all the imaging information of the composite image comes from the imaging information obtained by imaging in the complete imaging region.

[0223] As is known from the foregoing, the imaging information of the synthetic image is entirely derived from the imaging information obtained by performing imaging in the original imaging region, that is, the imaging information of the synthetic image is entirely derived from the imaging information obtained by performing imaging in the complete imaging region and the incomplete imaging region. When n is not 2 and n<f×(n-1), for the complete imaging region, one pixel in the original image is divided into n×n sub-pixels, wherein the size of each sub-pixel is d / n×d / n, each sub-pixel can acquire three pieces of color information, and a sub-pixel that has acquired three pieces of color information is denoted as a first sub-pixel. For the incomplete imaging region, each sub-pixel therein may not necessarily acquire three pieces of color information, and may acquire one or two pieces of color information, and a sub-pixel that has acquired one or two pieces of color information is denoted as a second sub-pixel; moreover, in the original image, there are still pixels in the incomplete imaging region that are not divided into sub-pixels, wherein the color information missing from the second sub-pixel needs to be estimated and supplemented. For example, for a missing color in a color-missing sub-pixel (that is, a sub-pixel whose color sampling cannot cover three colors), the information of the missing color of the color-missing sub-pixel can be estimated with reference to the imaging information of the sub-pixel that is closest to the color-missing sub-pixel and has the missing color. For instance, for a certain sub-pixel that misses the color R, the information of the color R is acquired from other sub-pixels closest to this sub-pixel, and is used as the information of the color R of this sub-pixel. Therefore, there may be various ways to select the pixels in the synthetic image, which will be described below by way of example.

[0224] Example 1: Only the sub-pixels in the complete imaging region are used as the pixels in the synthetic image.

[0225] In this case, since all the sub-pixels in the complete imaging region are the first sub-pixels and all contain three pieces of color information, any pixel in the synthetic image contains three pieces of color information.

[0226] Example 2: The sub-pixels in the complete imaging region and the sub-pixels in the incomplete imaging region are used as the pixels in the synthetic image.

[0227] In this case, all the sub-pixels in the complete imaging region are the first sub-pixels and all contain three pieces of color information; the sub-pixels in the incomplete imaging region include second sub-pixels, and the second sub-pixels miss one or two of the three pieces of color information, and the color information missing from the second sub-pixels needs to be estimated and supplemented; that is, any piece of color information missing from the second sub-pixel is determined according to that piece of color information of a third sub-pixel, and the third sub-pixel is a sub-pixel obtained by division during the superposition of multiple original images, and is a sub-pixel that is near the second sub-pixel and contains that piece of color information. In this way, any one of the sub-pixels in the complete imaging region and the sub-pixels in the incomplete imaging region contains three pieces of color information, so any pixel in the synthetic image contains three pieces of color information.

[0228] Alternatively, for the second sub-pixel lacking color information in the incomplete image region, the missing color information is not estimated or filled in; the second sub-pixel is directly used together with the first sub-pixel as pixels in the composite image. In this case, for the incomplete image region, each sub-pixel cannot obtain all three colors, and the image information of the upper and left edges of the composite image comes from the image information obtained from the incomplete image region. Therefore, pixels in the upper and left edges of the composite image will have color missing information. For the complete image region, each sub-pixel can obtain all three colors, and the image information of most areas in the composite image, except for the upper and left edges, comes from the image information obtained from the complete image region. Therefore, pixels in most areas of the composite image can obtain all three colors. However, since the upper and left edges of the composite image occupy a very small portion of the composite image area, the impact of color missing pixels in the upper and left edges on the composite image (and its application) is negligible. Therefore, the color resolution of the composite image is also improved compared to the original image.

[0229] Example 3: Subpixels in the complete image region, subpixels in the incomplete image region, and pixels in the incomplete image region that are not classified as subpixels are used as pixels in the composite image.

[0230] In this case, all sub-pixels in the complete image area are first sub-pixels, each containing three colors; the missing color information of the second sub-pixels in the incomplete image area can be estimated and filled in; pixels in the incomplete image area that are not classified as sub-pixels contain only one color, and the missing other two colors can be ignored.

[0231] It should be noted that, regarding the size of the synthesized image, the size of the synthesized image obtained in Example 1 is smaller than that obtained in Example 2, and the size of the synthesized image obtained in Example 2 is smaller than that obtained in Example 3. The size of the synthesized image obtained in Example 3 is the same as that of the original image. Furthermore, the sizes of the synthesized images obtained in Example 1 and Example 2 are also approximately equal to the size of the original image. In addition, the number of first sub-pixels containing all three colors accounts for the vast majority of the total pixels in the synthesized image. Therefore, regardless of which example is used to obtain the synthesized image, it can be assumed that all pixels in the synthesized image contain all three colors. In the following description, as long as the color sampling of the sub-pixels corresponding to the pixels in the vast majority of the synthesized image can cover all three colors, it is assumed that all pixels in the synthesized image contain all three colors.

[0232] In the embodiments of the present application, an image capturing position may be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at the image capturing position; the coordinates of any one of the plurality of image capturing positions in the same coordinate system are (i×s+W / 2, j×s+H / 2), where s is the minimum moving distance of the photosensitive element each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. Under the condition that n is not 2 and n<f×(n-1), 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change at the same time, both i and j change monotonically (for example, change monotonically increasing or change monotonically decreasing), i and j are integers, so the total number of the plurality of image capturing positions is 2n-1, where when n>m, f=m, and m is the remainder of a×n+b divided by 2n; when n<m, f=2n-m. In this way, one original image is obtained by sensing at each of the above 2n-1 image capturing positions, so that a total of 2n-1 original images are obtained, and a synthesized image can be obtained by superimposing the above 2n-1 original images. In the superimposition process, one pixel in the original image is divided into n×n sub-pixels, so that the resolution of the synthesized image can be greater than the resolution of the original image. Moreover, since the photosensitive element senses spectra of three wavelength ranges, that is, the photosensitive element senses spectra of three colors, the original image includes pixels of three colors. In the process of superimposing the 2n-1 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three color information, and the first sub-pixels are used as pixels in the synthesized image, so that any pixel in the synthesized image can contain three color information.

[0233] In a possible implementation, n is 2 or n>f×(n-1), and the coordinates of any one of the plurality of image capturing positions in the same coordinate system are (i×s+W / 2, j×s+H / 2); where

[0234] said m is the remainder of a×n+b divided by 2n;

[0235] 0 ≤ i ≤ n and 0 ≤ j ≤ n. In any update of (i × s + W / 2, j × s + H / 2), i and j do not change simultaneously, both i and j are monotonically changing, and i and j are integers. W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element. The photosensitive element senses three wavelength ranges of spectrum, which correspond to three color information. The determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images includes: the determination of pixels in the synthesized image based on first sub-pixels obtained during the superposition of multiple original images, where the first sub-pixels contain the three color information. In this case, a × n + b is an odd number.

[0236] It should be noted that when n is 2, the value of b can only be 1, and s = (a + 1 / 2) × d, for example, s can be 3d / 2, 5d / 2, 7d / 2, 9d / 2, etc. If the photosensitive element moves once in the x-direction and once in the y-direction, for example, according to the aforementioned implementation method, a total of 3 original images are obtained. At this time, the 3 original images are superimposed. For the complete imaging area, a pixel in the original image is divided into 2 × 2 sub-pixels, where the size of each sub-pixel is d / 2 × d / 2. In a region, there may be sub-pixels where color sampling cannot cover all three colors. Specifically, the following pattern exists in both the row and column directions: starting from the a+3 / 2th sub-pixel, sub-pixels where color sampling can cover all three colors and sub-pixels where color sampling cannot cover all three colors alternate until the a+3 / 2th sub-pixel from the end. Furthermore, the sub-pixels where color sampling cannot cover all three colors can only cover one color and are missing two colors. Among them, the color sampling of the a+3 / 2th sub-pixel from the end can cover all three colors.

[0237] Alternatively, when n > f × (n-1), for example, s is 4d / 3, 7d / 5, 11d / 5, etc., if the photosensitive element moves n-1 times in the x direction and n-1 times in the y direction, for example, by performing image acquisition in the aforementioned manner, a total of 2n-1 original images are acquired. At this time, the 2n-1 original images are superimposed. For the complete image acquisition area, a pixel in the original image is divided into n × n sub-pixels, where the size of each sub-pixel is d / n × d / n. In the complete image acquisition area, there will be sub-pixels whose color sampling cannot cover the three colors. Figure 11 The tree diagram shows the values ​​of i and j when n is 3, in the case of n>f×(n-1).

[0238] To ensure that all pixels in the composite image contain all three colors, it is necessary for all sub-pixels within the complete image area to contain all three colors. This can be achieved by adding one more movement in both the x and y directions. The specific movement process of the photosensitive element is described below:

[0239] I. When n is 2.

[0240] 1) Take an image at the original position.

[0241] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0242] 2) Moving and capturing images in the x-direction.

[0243] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (s+W / 2, H / 2) to obtain the original image x(1).

[0244] During the time interval from t(1)_x to t(2)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (2s+W / 2, H / 2) to obtain the original image x(2).

[0245] 3) Move in the y direction and capture the image.

[0246] During the time interval from t(n)_x to t(1)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (2s+W / 2, s+H / 2) to obtain the original image y(1).

[0247] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (2s+W / 2, 2s+H / 2) to obtain the original image y(2).

[0248] As shown above, after five image captures, five original images are obtained. By superimposing these five original images, a pixel in the original image can be divided into 2×2 sub-pixels, and each sub-pixel in the complete image capture area contains three colors.

[0249] Second, when n > f × (n-1).

[0250] 1) Take an image at the original position.

[0251] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0252] 2) Moving and capturing images in the x-direction.

[0253] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (s+W / 2, H / 2) to obtain the original image x(1).

[0254] During the time interval from t(1)_x to t(2)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (2s+W / 2, H / 2) to obtain the original image x(2).

[0255] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (3s+W / 2, H / 2) to obtain the original image x(3). ......

[0257] During the time interval from t(n-3)_x to t(n-2)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(n-2)_x, that is, the photosensitive element takes an image at the position ((n-2)s+W / 2, H / 2) to obtain the original image x(n-2).

[0258] During the time interval from t(n-2)_x to t(n-1)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(n-1)_x, that is, the photosensitive element takes an image at the position ((n-1)s+W / 2, H / 2) to obtain the original image x(n-1).

[0259] During the time interval from t(n-1)_x to t(n)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(n)_x, that is, the photosensitive element takes an image at the position (n×s+W / 2, H / 2) to obtain the original image x(n).

[0260] 3) Move in the y direction and capture the image.

[0261] During the time interval from t(n)_x to t(1)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, s+H / 2) to obtain the original image y(1).

[0262] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, 2s+H / 2) to obtain the original image y(2).

[0263] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, 3s+H / 2) to obtain the original image y(3). ......

[0265] During the time interval from t(n-3)_y to t(n-2)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(n-2)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, (n-2)s+H / 2) to obtain the original image y(n-2).

[0266] During the time interval from t(n-2)_y to t(n-1)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(n-1)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, (n-1)s+H / 2) to obtain the original image y(n-1).

[0267] During the time interval from t(n-1)_y to t(n)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(n)_y, that is, the photosensitive element takes an image at the position (n×s+W / 2, n×s+H / 2) to obtain the original image y(n).

[0268] As shown above, after 2n+1 image captures, 2n+1 original images are obtained. By superimposing these 2n+1 original images, a pixel in the original image can be divided into n×n sub-pixels, and each sub-pixel in the complete image capture area contains three colors.

[0269] It should be understood that, in this implementation, the method of selecting pixels in the composite image can refer to the descriptions shown in Examples 1 to 3 above.

[0270] In the embodiments of the present application, the imaging position may be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at the imaging position; the coordinates of any one of the multiple imaging positions under the same coordinate system are (i×s+W / 2, j×s+H / 2), where s is the minimum moving distance of the photosensitive element each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. When n is 2, 0≤i≤2 and 0≤j≤2, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change at the same time, both i and j change monotonically (for example, monotonically increasing or monotonically decreasing), and i and j are integers, so the multiple imaging positions total 5 imaging positions; in this way, one original image is sensed and obtained at each of the above 5 imaging positions, so that a total of 5 original images are obtained, and the above 5 original images are superimposed to obtain a composite image. In the superimposition process, one pixel in the original image is divided into 2×2 sub-pixels, so that the resolution of the composite image can be greater than the resolution of the original image; moreover, since the photosensitive element senses spectra in three wavelength ranges, that is, the photosensitive element senses spectra of three colors, the original image includes pixels of three colors. In the process of superimposing the 5 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three color information, and the first sub-pixels are used as the pixels in the composite image, so that any pixel in the composite image can contain three color information. Alternatively, when n>f×(n-1), 0≤i≤n and 0≤j≤n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change at the same time, both i and j change monotonically (for example, monotonically increasing or monotonically decreasing), and i and j are integers, so the multiple imaging positions total 2n+1 imaging positions, where when n>m, f=m, and m is the remainder of a×n+b divided by 2n; when n<m, f=2n-m; in this way, one original image is sensed and obtained at each of the above 2n+1 imaging positions, so that a total of 2n+1 original images are obtained, and the above 2n+1 original images are superimposed to obtain a composite image. In the superimposition process, one pixel in the original image is divided into n×n sub-pixels, and each sub-pixel has three superimposed color information, so that the resolution of the composite image can be greater than the resolution of the original image; moreover, since the photosensitive element senses spectra in three wavelength ranges, that is, the photosensitive element senses spectra of three colors, the original image includes pixels of three colors. In the process of superimposing the 2n+1 original images, first sub-pixels with three superimposed colors can be obtained by division, that is, the first sub-pixels contain three color information, and the first sub-pixels are used as the pixels in the composite image, so that any pixel in the composite image can contain three color information.

[0271] In one possible implementation, the determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: determining pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixel lacks one or two of the three color information; determining any missing color information of the second sub-pixel based on any color information of the third sub-pixel, wherein the third sub-pixel is a sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is a sub-pixel near the second sub-pixel that contains any of the color information.

[0272] It should be understood that for sub-pixels obtained during the overlay of multiple original images, as long as a sub-pixel is missing any one or two of the three color information, it can be filled in by color estimation and filling in the missing information so that the sub-pixel contains the three color information.

[0273] In this embodiment of the application, in the process of superimposing multiple original images, in addition to dividing a first sub-pixel containing three colors, a second sub-pixel containing one or two colors can also be divided. For any color information missing in the second sub-pixel, it can be estimated based on the color information of the nearby sub-pixels containing that color information. After estimating the color information missing in the second sub-pixel, the first and second sub-pixels can be used as pixels of the synthesized image, so that any pixel in the synthesized image can also contain three colors.

[0274] In one possible implementation, n is 2 or n > f × (n-1), and the coordinates of any one of the plurality of image-taking positions are (i × s + W / 2, j × s + H / 2) in the same coordinate system; wherein,

[0275] The m is the remainder when a×n+b is divided by 2n;

[0276] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the pixels in the synthesized image are determined according to the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined according to the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel is missing one or two of the three color information; any missing color information of the second sub-pixel is determined according to any color information of the third sub-pixel, the third sub-pixel is the sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is the sub-pixel near the second sub-pixel that contains the any color information. In this case, a×n+b is an odd number.

[0277] It should be noted that when n is 2, the value of b can only be 1, and s = (a + 1 / 2) × d, for example, s can be 3d / 2, 5d / 2, 7d / 2, 9d / 2, etc. If the photosensitive element moves once in the x-direction and once in the y-direction, for example, according to the aforementioned implementation method, a total of 3 original images are obtained. At this time, the 3 original images are superimposed. For the complete imaging area, a pixel in the original image is divided into 2 × 2 sub-pixels, where the size of each sub-pixel is d / 2 × d / 2. In a region, there may be sub-pixels where color sampling cannot cover all three colors. Specifically, the following pattern exists in both the row and column directions: starting from the a+3 / 2th sub-pixel, sub-pixels where color sampling can cover all three colors and sub-pixels where color sampling cannot cover all three colors alternate until the a+3 / 2th sub-pixel from the end. Furthermore, the sub-pixels where color sampling cannot cover all three colors can only cover one color and are missing two colors. Among them, the color sampling of the a+3 / 2th sub-pixel from the end can cover all three colors. For example, when s is 3d / 2, the color sample of the 4th subpixel can cover three colors, the color sample of the 5th subpixel cannot cover three colors, the color sample of the 6th subpixel can cover three colors, the color sample of the 7th subpixel cannot cover three colors, the color sample of the 8th subpixel can cover three colors, the color sample of the 9th subpixel cannot cover three colors, and so on, until the (a+3 / 2)th subpixel from the end, where the color sample of the (a+3 / 2)th subpixel from the end can cover three colors.

[0278] Alternatively, when n > f × (n-1), for example, s is 4d / 3, 7d / 5, 11d / 5, etc., if the photosensitive element moves n-1 times in the x direction and n-1 times in the y direction, for example, by performing image acquisition in the aforementioned manner, a total of 2n-1 original images are acquired. At this time, the 2n-1 original images are superimposed. For the complete image acquisition area, a pixel in the original image is divided into n × n sub-pixels, where the size of each sub-pixel is d / n × d / n. In the complete image acquisition area, there will be sub-pixels whose color sampling cannot cover the three colors.

[0279] To ensure that all pixels in the synthesized image contain all three colors, it is necessary to estimate and fill in the color information of sub-pixels lacking color information within the complete image area. For example, for a sub-pixel missing a color (i.e., a sub-pixel whose color sampling cannot cover all three colors), the color information of the missing color can be estimated by referring to the image information of the nearest sub-pixel containing the missing color. For instance, for a sub-pixel missing a color R, the color information of R is obtained from the nearest other sub-pixels and used as the color information of the current sub-pixel.

[0280] It should be understood that after color information estimation and completion of sub-pixels lacking color information within the complete image capture area, each sub-pixel within the complete image capture area contains three colors. In this implementation, the method for selecting pixels in the composite image can be referred to the descriptions in Examples 1 to 3 above.

[0281] In this embodiment of the application, the image capture position can be represented by the coordinates of the center of the photosensitive element when the photosensitive element captures an image at that position; the coordinates of any one of the multiple image capture positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system, where s is the minimum distance the photosensitive element moves each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. When n is 2, 0 ≤ i ≤ 1 and 0 ≤ j ≤ 1. In any update of (i × s + W / 2, j × s + H / 2), i and j do not change simultaneously; both i and j are monotonically changing (e.g., monotonically increasing or monotonically decreasing). i and j are integers, so there are a total of 3 imaging positions. Thus, one original image is sensed at each of the above 3 imaging positions, resulting in a total of 3 original images. These 3 original images are superimposed to obtain a composite image. The superposition process divides a pixel in the original image into 2 × 2 sub-pixels. However, some sub-pixels have the superposition of three color information, while others do not. For the second sub-pixel that does not have the superposition of three color information, the missing color information is estimated through its neighboring sub-pixels. Specifically, the second... Any color information missing in a sub-pixel is estimated based on the same color information of a third sub-pixel near the second sub-pixel that contains that color information. For example, the color information of the third sub-pixel can be directly used as the color information of the second sub-pixel. If there are multiple third sub-pixels near the second sub-pixel, the color information of the second sub-pixel can be estimated using the color information of these multiple third sub-pixels. For example, the average of the color information of these multiple third sub-pixels can be used as the color information of the second sub-pixel. It should be understood that after estimating the color information of the sub-pixels with missing color information, all the sub-pixels obtained by superposition contain three colors of information. Therefore, the resolution of the synthesized image can be greater than the resolution of the original image, and any pixel in the synthesized image contains three colors of information.Alternatively, in the case of n>f×(n-1), 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, both i and j change monotonically (for example, change monotonically increasing or change monotonically decreasing), and i and j are integers. Therefore, there are a total of 2n-1 imaging positions, where when n>m, f=m, and m is the remainder of a×n+b divided by 2n; when n<m, f=2n-m. In this way, one original image is sensed at each of the 2n-1 imaging positions described above, so that a total of 2n-1 original images are obtained. A synthesized image can be obtained by superimposing the 2n-1 original images. In the superimposition process, one pixel in the original image is divided into n×n sub-pixels, but some sub-pixels have three types of color information superimposed, and some sub-pixels do not have three types of color information superimposed. For a second sub-pixel that does not have three types of color information superimposed, any missing color information thereof is estimated through sub-pixels in its vicinity. Specifically, any missing color information of the second sub-pixel is estimated according to the corresponding color information of a third sub-pixel that is in the vicinity of the second sub-pixel and contains the any color information, for example, directly taking the any color information of the third sub-pixel as the any color information of the second sub-pixel. If there are a plurality of third sub-pixels in the vicinity of the second sub-pixel, the any color information of the second sub-pixel may be estimated through the any color information of the plurality of third sub-pixels, for example, taking the average value of the any color information of the plurality of third sub-pixels as the any color information of the second sub-pixel. It should be understood that after the color information estimation is performed on the sub-pixels with missing color information, all the sub-pixels obtained by superimposition contain three types of color information, so that the resolution of the synthesized image can be greater than the resolution of the original image, and any pixel in the synthesized image contains three types of color information.

[0282] Combined with the foregoing, it can be seen that when n is 2 or n>f×(n-1), two methods can be used to make all sub-pixels in the complete imaging area contain three types of color information. The first method is to increase the number of imaging positions, that is, to increase the number of movements of the photosensitive element; the other method is to perform color information estimation and complementation. It should be noted that, for the first method, increasing the number of imaging positions will increase the number of movements of the photosensitive element and raise the imaging cost, but all the color information of the sub-pixels in the complete imaging area comes from actual imaging, so the authenticity of the synthesized image is guaranteed. For the second method, the number of movements of the photosensitive element is small, the number of imaging times is small, and the information of some sub-pixels needs to be estimated by means of the information of the surrounding sub-pixels, which will affect the authenticity of the synthesized image. However, a smaller number of movements can reduce the imaging time and lower the imaging cost, and the implementer can adopt a balanced solution between the effect of reducing the number of movements on the (application of the) synthesized image and the cost of performing the movements.

[0283] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, and the coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein 0≤i≤n-1 and 0≤j≤n-1, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0284] For a complete imaging area, a pixel in the original image is divided into n×n sub-pixels, where the size of each sub-pixel is d / n×d / n; when the sensing element senses only one wavelength range of the spectrum, the original image and the composite image can be grayscale images.

[0285] In the embodiments of this application, the photosensitive element senses a spectrum of a wavelength range, that is, any sensing pixel in the sensing element senses a spectrum of the same wavelength range or a spectrum of the same color; the image-taking position can be represented by the coordinates of the center of the photosensitive element when the photosensitive element takes an image at that image-taking position; the coordinates of any image-taking position among multiple image-taking positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system, where s is the minimum distance the photosensitive element moves each time, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. Since 0 ≤ i ≤ n-1 and 0 ≤ j ≤ n-1, in any update of (i × s + W / 2, j × s + H / 2), i and j do not change simultaneously; both i and j are monotonically changing (e.g., monotonically increasing or monotonically decreasing). Since i and j are integers, there are a total of 2n-1 imaging positions. Thus, one original image is sensed at each of the above 2n-1 imaging positions, resulting in a total of 2n-1 original images. These 2n-1 original images are then superimposed to obtain a composite image. The superposition process divides one pixel in the original image into n × n sub-pixels, thereby making the resolution of the composite image greater than that of the original image.

[0286] In one possible implementation, n is 2, the plurality of imaging positions are two imaging positions, and the coordinates of the two imaging positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein, W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the determination of pixels in the composite image based on sub-pixels obtained during the superposition of the plurality of original images includes: the pixels in the composite image are determined based on second sub-pixels obtained during the superposition of the plurality of original images, the second sub-pixel lacking one or two of the three color information; any missing color information of the second sub-pixel is determined based on any color information of the third sub-pixel, the third sub-pixel being a sub-pixel obtained during the superposition of the plurality of original images, and the third sub-pixel being a sub-pixel near the second sub-pixel containing the any color information.

[0287] When n is 2, b can only be 1. The photosensitive element moves simultaneously in the x and y directions by s = (a + 1 / 2) × d, which is the distance it moves along the diagonal of the angle formed by the x and y axes. like Figure 12 As shown, in this case, the region s≤x≤W and s≤y≤H is the complete imaging region, while the regions 0≤x≤s and 0≤y≤H, and s≤x≤W and 0≤y≤s are incomplete imaging regions. That is, the regions W≤x≤W+s and 0≤y≤H, and s≤x≤W+s and H≤y≤H+s are newly added imaging regions. The specific movement process of the photosensitive element is described below:

[0288] 1) Take an image at the original position.

[0289] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0290] 2) Take the image after moving a distance of s = (a + 1 / 2) × d in both the x and y directions.

[0291] During the time interval from t(0) to t(1), the photosensitive element moves from the original position (W / 2, H / 2) to the image acquisition position (s+W / 2, s+W / 2); at time t(1), the photosensitive element acquires an image at the position (s+W / 2, s+W / 2) to obtain the original image 1.

[0292] For a complete image region, by superimposing the original image 0 and the original image 1 according to their image positions, a pixel in either image 0 or image 1 can be divided into 2×2 sub-pixels, with a sub-pixel size of d / 2×d / 2. Any sub-pixel can cover either two colors or one color. For a sub-pixel that covers two colors, one color originates from the original image 0, and the other color originates from the original image 1. For a sub-pixel that covers only one color, the color it covers originates from both the original image 0 and the original image 1; that is, for this type of sub-pixel, the colors of the original image 0 and the original image 1 are the same.

[0293] During the synthesis process, for any sub-pixel, the missing color information can be estimated based on the sub-pixels near it. For example, if the missing color of the sub-pixel is R, then the R color information of the sub-pixel is estimated by using the R color information of the nearest sub-pixel with color R. If there are multiple sub-pixels with color R that are closest to the sub-pixel, then the average of the R color information of these multiple sub-pixels is calculated, and the average of the R color information of these multiple sub-pixels is taken as the R color information of the sub-pixel.

[0294] It should be understood that, in this implementation, the method of selecting pixels in the composite image can refer to the descriptions shown in Examples 1 to 3 above.

[0295] In this embodiment, the image acquisition position can be represented by the coordinates of the center of the photosensitive element when it acquires an image at that position. When n is 2, multiple image acquisition positions can be represented by two positions, with coordinates (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, where W is the number of sensed pixels in the width direction of the photosensitive element, and H is the number of sensed pixels in the height direction. Thus, one original image is obtained from each of the two image acquisition positions, resulting in two original images. These two original images are then superimposed to obtain a composite image. The superposition process divides one pixel in the original image into 2×2 sub-pixels, thereby allowing the resolution of the composite image to be greater than that of the original image. However, since subpixels do not have the superposition of the three color information, the missing color information is estimated from the subpixels near them. Specifically, the missing color information of the second subpixel is estimated from the color information of the third subpixel near the second subpixel that contains the missing color information. For example, the missing color information of the third subpixel can be directly used as the missing color information of the second subpixel. If there are multiple third subpixels near the second subpixel, the missing color information of the second subpixel can be estimated from the missing color information of these multiple third subpixels. For example, the average of the missing color information of these multiple third subpixels can be used as the missing color information of the second subpixel. It should be understood that after estimating the color information of the subpixels with missing color information, all the subpixels obtained by superposition contain the three color information, so that any pixel in the synthesized image can contain the three color information.

[0296] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, where n is 2, and the plurality of imaging positions are two imaging positions, the coordinates of which are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0297] When n is 2, b can only be 1. The photosensitive element moves simultaneously in the x and y directions by s = (a + 1 / 2) × d, which is the distance it moves along the diagonal of the angle formed by the x and y axes. The specific movement process of the photosensitive element is described as follows:

[0298] 1) Take an image at the original position.

[0299] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0300] 2) Take the image after moving a distance of s = (a + 1 / 2) × d in both the x and y directions.

[0301] During the time interval from t(0) to t(1), the photosensitive element moves from the original position (W / 2, H / 2) to the image acquisition position (s+W / 2, s+W / 2); at time t(1), the photosensitive element acquires an image at the position (s+W / 2, s+W / 2) to obtain the original image 1.

[0302] For the pixels of the original image in the complete imaging area, the superposition of the original image 0 and the original image 1 can divide it into 2×2 sub-pixels; when the sensing element senses only one wavelength range of spectrum, each sub-pixel contains the same color information, and the original image and the composite image can be grayscale images.

[0303] In this embodiment, the photosensitive element senses a spectrum within a wavelength range. The imaging position can be represented by the coordinates of the center of the photosensitive element when it takes an image at that position. When n is 2, multiple imaging positions can be two imaging positions, with coordinates (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, where W is the number of sensed pixels in the width direction of the photosensitive element, and H is the number of sensed pixels in the height direction of the photosensitive element. Thus, one original image is sensed at each of the two imaging positions, resulting in two original images. These two original images are then superimposed to obtain a composite image. The superposition process divides one pixel in the original image into 2×2 sub-pixels, thereby allowing the resolution of the composite image to be greater than that of the original image.

[0304] The technical solutions provided in the embodiments of this application will be described in detail below with reference to specific examples.

[0305] Please see Figure 13 , Figure 13 This is a schematic diagram of a photosensitive element provided in an embodiment of this application. The colors in the photosensitive element can be RGGB (three spectra or colors) or RYYB (three spectra or colors). The total number of sensing pixels of the photosensitive element is 10×10, that is, H=10, W=10.

[0306] Example 1: Using Figure 13 The photosensitive element shown, taking n=3, a=1, b=1, s=4d / 3 as an example, will be used to describe the technical solution provided in the embodiments of this application. Specifically, as follows:

[0307] In this case, each sensing pixel in the photosensitive element is divided into 3×3 sensing sub-pixels, where the size of each sensing sub-pixel is d / 3×d / 3, such as Figure 14as shown. m=(a×n+b)|(2×n)=(1×3+1)|(2×3)=4, so n<m; and f=2n-m=2×3-4=2, f×(n-1)=2×(3-1)=4, so n<f×(n-1); 2 movements in the x direction and 2 movements in the y direction are required, so that all sub-pixels in the complete imaging area contain three types of color information.

[0308] I. Imaging Process

[0309] On this xOy coordinate system, imaging is performed at positions (W / 2, H / 2), (4d / 3+W / 2, H / 2), (8d / 3+W / 2, H / 2), (8d / 3+W / 2, 4d / 3+H / 2), and (8d / 3+W / 2, 8d / 3+H / 2) respectively, to obtain 5 original images. The photosensitive element moves in the x direction first, and after the movement in the x direction is completed, it then moves in the y direction. The specific movement process is as follows:

[0310] 1) Perform imaging at the original position.

[0311] At time t(0), the photosensitive element performs imaging at the original position (W / 2, H / 2) to obtain original image 0.

[0312] 2) Movement in the x direction and imaging.

[0313] During the time period from t(0) to t(1)_x, the photosensitive element moves a distance of 4d / 3 in the x direction; the photosensitive element performs imaging at the position where it is located at time t(1)_x, that is, the photosensitive element performs imaging at the position (4d / 3+W / 2, H / 2) to obtain original image x(1).

[0314] During the time period from t(1)_x to t(2)_x, the photosensitive element moves a distance of 4d / 3 in the x direction; the photosensitive element performs imaging at the position where it is located at time t(2)_x, that is, the photosensitive element performs imaging at the position (8d / 3+W / 2, H / 2) to obtain original image x(2).

[0315] 3) Movement in the y direction and imaging.

[0316] During the time period from t(2)_x to t(1)_y, the photosensitive element moves a distance of 4d / 3 in the y direction; the photosensitive element performs imaging at the position where it is located at time t(1)_y, that is, the photosensitive element performs imaging at the position (8d / 3+W / 2, 4d / 3+H / 2) to obtain original image y(1).

[0317] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 4d / 3 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (8d / 3+W / 2, 8d / 3+H / 2) to obtain the original image y(2).

[0318] II. Synthesis Process

[0319] Since n=3, the distance moved each time is 4d / 3. It can be considered that the superposition process divides a pixel in the original image into 3×3 sub-pixels, where each sub-pixel has a size of d / 3×d / 3; or, it can be considered that each sensing pixel in the photosensitive element is divided into 3×3 sensing sub-pixels, where each sensing sub-pixel has a size of d / 3×d / 3, such as... Figure 14 As shown. For any row (same y-coordinate value) of sensing subpixels in the photosensitive element, the color of the first sensing subpixel in that row (smallest x-coordinate value) may be R or G / Y; for any column (same x-coordinate value) of sensing subpixels in the photosensitive element, the color of the first sensing subpixel in that column (smallest x-coordinate value) may be R or G / Y.

[0320] 1) The process of moving and superimposing the first sensed sub-pixel of color R in any row of sensed sub-pixels in the x-direction is as follows: Figure 15 As shown. For any row of sub-pixels acquired by the sensing sub-pixels, there are a total of 30 sub-pixels, numbered 1-30 in sequence; the three rows of sub-pixels acquired by the sensing sub-pixels are superimposed according to their positions on the coordinate system xOy, where these three rows of sub-pixels are in the original image 0, the original image x(1) and the original image x(2) respectively; thus, the superimposed row of sub-pixels has a total of 38 sub-pixels, numbered 1-38 in sequence; for the sub-pixels in this row, the sub-pixels numbered 8-31 can obtain two colors (R and G / Y), the sub-pixels numbered 5, 6, 33 and 34 can also obtain two colors, and the sub-pixels numbered 1-4, 7, 32 and 35-38 can only obtain one color.

[0321] 2) The process of moving and superimposing the first sensing sub-pixel with color G / Y in any row of sensing sub-pixels in the x-direction is as follows: Figure 16 As shown. Compared to the case where the first sensed sub-pixel is of color R in any row of sensed sub-pixels, the motion imaging and overlay rules are the same; only the color types that the sub-pixels can obtain differ. Sub-pixels numbered 8-31 can obtain two colors (G / Y and B), as detailed in the reference. Figure 15 The description of that will not be repeated here.

[0322] 3) The process of moving and capturing images of any column of the first sensing sub-pixel with color R in the y direction is as follows: Figure 17 As shown. For any column of sub-pixels acquired by the sensing sub-pixels, there are a total of 30 sub-pixels, numbered 1-30 in sequence; the three columns of sub-pixels acquired by the sensing sub-pixels are superimposed according to their positions on the coordinate system xOy, where these three columns of sub-pixels are respectively in the original image x(2), the original image y(1), and the original image y(2); thus, the superimposed column of sub-pixels has a total of 38 sub-pixels, numbered 1-38 in sequence; for this column of sub-pixels, sub-pixels numbered 8-31 can obtain two colors (R and G / Y), sub-pixels numbered 5, 6, 33, and 34 can also obtain two colors, and sub-pixels numbered 1-4, 7, 32, and 35-38 can only obtain one color.

[0323] 4) The first sensing sub-pixel with color G / Y, any column sensing sub-pixel, its image acquisition and superposition process in the y direction is as follows: Figure 18 As shown. Compared to the case where the first sensed sub-pixel is of color R in any column of sensed sub-pixels, the motion imaging and superposition rules are the same; only the color types that the sub-pixels can obtain differ. Sub-pixels numbered 8-31 can obtain two colors (G / Y and B), as detailed in the reference. Figure 17 The description of that will not be repeated here.

[0324] Figure 19 The image shows the result of superimposing the original image 0, original image x(1), original image x(2), original image y(1), and original image y(2) according to their image capture positions. Figure 19As shown, the size of the sub-pixels in the complete imaging area is d / 3×d / 3, and the sub-pixels in the complete imaging area can be used as the pixels in the composite image; there are sub-pixels of various sizes in the non-complete imaging area, such as d / 3×d / 3, d / 3×2d / 3, d / 3×d, 2d / 3×d, the sub-pixels in the complete imaging area and the sub-pixels in the non-complete imaging area can be used as the pixels in the composite image; there are also pixels not divided into sub-pixels in the non-complete imaging area, the sub-pixels in the complete imaging area, the sub-pixels in the non-complete imaging area can be used as the pixels in the composite image, and the pixels not divided into sub-pixels in the non-complete imaging area can also be used as the pixels in the composite image. Wherein, the sub-pixel with the size of d / 3×2d / 3 in the non-complete imaging area can be regarded as 1×2 sub-pixels with the size of d / 3×d / 3, the sub-pixel with the size of d / 3×d can be regarded as 1×3 sub-pixels with the size of d / 3×d / 3, the sub-pixel with the size of 2d / 3×d can be regarded as 2×3 sub-pixels with the size of d / 3×d / 3, and the pixel not divided into sub-pixels can be regarded as 3×3 sub-pixels with the size of d / 3×d / 3. Therefore, the pixels in the composite image are derived from the sub-pixels with row numbers 1 to 30 and column numbers 1 to 30. It can be found that after moving 8 sub-pixels in the x direction during the mobile imaging process, the movement starts in the y direction. Therefore, during the composition process, the pixels in columns 1 to 8 cannot be superimposed with the imaging information in the y direction, and their pixel information is only derived from the imaging in the x direction; similarly, the pixels in rows 31 to 38 are derived from the imaging in the y direction. Any sub-pixel among the pixels with row numbers 9 to 30 and column numbers 9 to 30 can obtain three colors, specifically R and G / Y derived from imaging in the x direction and G / Y and B derived from imaging in the y direction, or G / Y and B derived from imaging in the x direction and R and G / Y derived from imaging in the y direction. The sub-pixels with row numbers 31 to 38 and column numbers 31 to 38 are the sub-pixels in the newly added imaging area.

[0325] Example 2: Using Figure 13 The photosensitive element shown, taking n=5, a=1, b=1, s=6d / 5 as an example, to describe the technical solution provided by the embodiments of the present application. The details are as follows:

[0326] In this case, each sensing pixel in the photosensitive element is divided into 5×5 sensing sub-pixels, wherein the size of each sensing sub-pixel is d / 5×d / 5, as Figure 20 shown; m=(a×n+b)|(2×n)=(1×5+1)|(2×5)=6, so n<m; and f=2n-m=2×5-6=4, f×(n-1)=4×(5-1)=16, so n<f×(n-1); it is necessary to move 4 times in the x direction and 4 times in the y direction, so that all sub-pixels in the complete imaging area contain three color pieces of information.

[0327] I. Image Acquisition Process

[0328] In the coordinate system xOy, images are captured at positions (W / 2, H / 2), (6d / 5+W / 2, H / 2), (12d / 5+W / 2, H / 2), (18d / 5+W / 2, H / 2), (24d / 5+W / 2, H / 2), (24d / 5+W / 2, 6d / 5+H / 2), (24d / 5+W / 2, 12d / 5+H / 2), (24d / 5+W / 2, 18d / 5+H / 2), and (24d / 5+W / 2, 24d / 5+H / 2) to obtain nine original images. The photosensitive element moves first in the x-direction, and after completing the x-direction movement, it moves in the y-direction. The specific movement process is as follows:

[0329] 1) Take an image at the original position.

[0330] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0331] 2) Moving and capturing images in the x-direction.

[0332] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance of 6d / 5 in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (6d / 5+W / 2, H / 2) to obtain the original image x(1).

[0333] During the time interval t(1)_x to t(2)_x, the photosensitive element moves a distance of 6d / 5 in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (12d / 5+W / 2, H / 2) to obtain the original image x(2).

[0334] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance of 6d / 5 in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (18d / 5+W / 2, H / 2) to obtain the original image x(3).

[0335] During the time interval from t(3)_x to t(4)_x, the photosensitive element moves a distance of 6d / 5 in the x direction; the photosensitive element takes an image at the position at time t(4)_x, that is, the photosensitive element takes an image at the position (24d / 5+W / 2, H / 2) to obtain the original image x(4).

[0336] 3) Move in the y direction and capture the image.

[0337] During the time interval from t(4)_x to t(1)_y, the photosensitive element moves a distance of 6d / 5 in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (24d / 5+W / 2, 6d / 5+H / 2) to obtain the original image y(1).

[0338] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 6d / 5 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (24d / 5+W / 2, 12d / 5+H / 2) to obtain the original image y(2).

[0339] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance of 6d / 5 in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (24d / 5+W / 2, 18d / 5+H / 2) to obtain the original image y(3).

[0340] During the time interval t(3)_y to t(4)_y, the photosensitive element moves a distance of 6d / 5 in the y direction; the photosensitive element takes an image at the position at time t(4)_y, that is, the photosensitive element takes an image at the position (24d / 5+W / 2, 24d / 5+H / 2) to obtain the original image y(4).

[0341] II. Synthesis Process

[0342] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 21 As shown. For any row of sub-pixels collected by the sensing sub-pixels, there are a total of 50 sub-pixels, numbered 1-50 in sequence; after moving 4 times in the x direction, the row of sensing sub-pixels is collected into 5 rows of sub-pixels according to their positions on the coordinate system xOy. Among them, these 5 rows of sub-pixels are in the original image 0, the original image x(1), the original image x(2), the original image x(3), and the original image x(4), respectively; thus, the superimposed row of sub-pixels has a total of 74 sub-pixels, numbered 1-74 in sequence. There is a continuous segment of sub-pixels in this row that can obtain two colors of information, namely the sub-pixels numbered 12-63, so the requirements for obtaining the composite image are met.

[0343] Similarly, for any row of sensing sub-pixels where the color of the first sensing sub-pixel is G / Y, the process of moving for image capturing and superimposing in the x direction; and for any column of sensing sub-pixels where the color of the first sensing sub-pixel is R, the process of moving for image capturing in the y direction; and for any column of sensing sub-pixels where the color of the first sensing sub-pixel is G / Y, the process of moving for image capturing and superimposing in the y direction; all have the same principle or rule as the process of moving for image capturing and superimposing in the x direction for any row of sensing sub-pixels where the color of the first sensing sub-pixel is R, and reference may be made Figure 21 to the description shown, which will not be enumerated herein.

[0344] It should be understood that in this case, the principle or rule of the process of obtaining a composite image from sub-pixels is the same as the aforementioned Figure 19 principle or rule described, and reference may be made Figure 19 to the description shown, which will not be repeated herein.

[0345] Example 3: Using Figure 13 the photosensitive element shown, the technical solution provided by the embodiment of the present application is described by taking n=5, a=1, b=2, s=7d / 5 as an example. The details are as follows:

[0346] In this case, each sensing pixel in the photosensitive element is divided into 5×5 sensing sub-pixels, wherein the size of each sensing sub-pixel is d / 5×d / 5, as shown Figure 20 ; m=(a×n+b)|(2×n)=(1×5+2)|(2×5)=7, so n<m; and f=2n-m=2×5-7=3, f×(n-1)=3×(5-1)=12, so n<f×(n-1); 4 moves are required in the x direction and 4 moves are required in the y direction, so that all sub-pixels in the complete image capturing area contain three color pieces of information.

[0347] I. Image Capturing Process

[0348] On the coordinate system xOy, images are captured at positions (W / 2, H / 2), (7d / 5+W / 2, H / 2), (14d / 5+W / 2, H / 2), (21d / 5+W / 2, H / 2), (28d / 5+W / 2, H / 2), (28d / 5+W / 2, 7d / 5+H / 2), (28d / 5+W / 2, 14d / 5+H / 2), (28d / 5+W / 2, 21d / 5+H / 2), (28d / 5+W / 2, 28d / 5+H / 2) to obtain 9 original images. The photosensitive element moves in the x direction first, and after the movement in the x direction is completed, then moves in the y direction. The specific movement process is as follows:

[0349] 1) Capture an image at the original position.

[0350] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0351] 2) Moving and capturing images in the x-direction.

[0352] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance of 7d / 5 in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (7d / 5+W / 2, H / 2) to obtain the original image x(1).

[0353] During the time interval t(1)_x to t(2)_x, the photosensitive element moves a distance of 7d / 5 in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (14d / 5+W / 2, H / 2) to obtain the original image x(2).

[0354] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance of 7d / 5 in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (21d / 5+W / 2, H / 2) to obtain the original image x(3).

[0355] During the time interval t(3)_x to t(4)_x, the photosensitive element moves a distance of 7d / 5 in the x direction; the photosensitive element takes an image at the position at time t(4)_x, that is, the photosensitive element takes an image at the position (28d / 5+W / 2, H / 2) to obtain the original image x(4).

[0356] 3) Move in the y direction and capture the image.

[0357] During the time interval from t(4)_x to t(1)_y, the photosensitive element moves a distance of 7d / 5 in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (28d / 5+W / 2, 7d / 5+H / 2) to obtain the original image y(1).

[0358] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 7d / 5 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (28d / 5+W / 2, 14d / 5+H / 2) to obtain the original image y(2).

[0359] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance of 7d / 5 in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (28d / 5+W / 2, 21d / 5+H / 2) to obtain the original image y(3).

[0360] During the time interval from t(3)_y to t(4)_y, the photosensitive element moves a distance of 7d / 5 in the y direction; the photosensitive element takes an image at the position at time t(4)_y, that is, the photosensitive element takes an image at the position (28d / 5+W / 2, 28d / 5+H / 2) to obtain the original image y(4).

[0361] II. Synthesis Process

[0362] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 22 As shown. For any row of sub-pixels collected by the sensing sub-pixels, there are a total of 50 sub-pixels, numbered 1-50 in sequence; after moving 4 times in the x direction, the row of sensing sub-pixels is collected into 5 rows of sub-pixels according to their positions on the coordinate system xOy. Among them, these 5 rows of sub-pixels are in the original image 0, the original image x(1), the original image x(2), the original image x(3), and the original image x(4), respectively; thus, the superimposed row of sub-pixels has a total of 78 sub-pixels, numbered 1-78 in sequence. There is a continuous segment of sub-pixels in this row that can obtain two colors of information, namely the sub-pixels numbered 13-66, so the requirements for obtaining the composite image are met.

[0363] Similarly, the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color G / Y; and the process of moving and superimposing sub-pixels in the y-direction for any column of sub-pixels with the first sub-pixel color R; are all based on the same principle or pattern as the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color R. This can be referred to... Figure 22 The descriptions shown are not listed here.

[0364] It should be understood that, in this case, the principle or rule of obtaining a composite image based on sub-pixel density is similar to the aforementioned... Figure 19 The principles or rules described are the same and can be referred to. Figure 19 The description shown will not be repeated here.

[0365] Example 4: Using Figure 13The photosensitive element shown, taking n=2, a=1, b=1, s=3d / 2 as an example, will be used to describe the technical solution provided in the embodiments of this application. Specifically, as follows:

[0366] In this case, each sensing pixel in the photosensitive element is divided into 2×2 sensing sub-pixels, where the size of each sensing sub-pixel is d / 2×d / 2, such as Figure 23 As shown. If one movement is required in the x-direction and one movement is required in the y-direction, it is impossible to ensure that all sub-pixels in the complete image area contain all three color information. In order to ensure that all sub-pixels in the complete image area contain all three color information, this example requires two movements in the x-direction and two movements in the y-direction.

[0367] I. Image Acquisition Process

[0368] 1) Take an image at the original position.

[0369] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0370] 2) Moving and capturing images in the x-direction.

[0371] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (3d / 2+W / 2, H / 2) to obtain the original image x(1).

[0372] During the time interval from t(1)_x to t(2)_x, the photosensitive element moves a distance s in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (3d+W / 2, H / 2) to obtain the original image x(2).

[0373] 3) Move in the y direction and capture the image.

[0374] During the time interval from t(n)_x to t(1)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (3d+W / 2, 3d / 2+H / 2) to obtain the original image y(1).

[0375] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance s in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (3d+W / 2, 3d+H / 2) to obtain the original image y(2).

[0376] II. Synthesis Process

[0377] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 24 As shown. For any row of sub-pixels captured by the sensing sub-pixels, there are a total of 20 sub-pixels, numbered sequentially from 1 to 20; if only one movement is made in the x direction, the two rows of sub-pixels captured by the sensing sub-pixels are superimposed according to their positions on the coordinate system xOy, where these two rows of sub-pixels are in the original image 0 and the original image x(1), respectively; since there are sub-pixels with only the same color superimposed during the superposition process, such as Figure 24 The subpixels enclosed by the dashed box have only one color superposition. For example, the subpixel at row number 5 of the original image 0 and the subpixel at row number 2 of the original image x(1) have only one color superposition, the subpixel at row number 7 of the original image 0 and the subpixel at row number 4 of the original image x(1) have only one color superposition, and so on. Therefore, the resulting row of subpixels always contains intermittent subpixels that can only obtain one color information, and it is impossible to have a continuous segment of subpixels that can obtain two colors. For example... Figure 24 In the synthesis 1, the superimposed row of subpixels has a total of 23 subpixels, numbered 1-23. In this row of subpixels, there are always intermittent subpixels that can only obtain one color information, and there is no continuous segment of subpixels that can obtain two colors.

[0378] During the superposition process, each sub-pixel needs to acquire three colors, specifically two colors in the x-direction and two colors in the y-direction, which are R and G / Y / W from the x-direction image and G / Y / W and B from the y-direction image, or G / Y and B from the x-direction image and R and G / Y from the y-direction image. Therefore, in this case, an additional image acquisition in the x-direction is required to obtain the original image x(2). After moving twice in the x-direction, the row of sensed sub-pixels is acquired into three rows of sub-pixels and superimposed according to their positions on the coordinate system xOy. These three rows of sub-pixels are located in the original image 0, the original image x(1), and the original image x(2), respectively. Thus, when only the original image 0 and the original image x(1) are superimposed, the sub-pixels with only one color will be superimposed with the sub-pixels of the other color in the original image x(2). For example, the subpixel at row number 7 of the original image 0 and the subpixel at row number 4 of the original image x(1) have only one color superposition, while the subpixel at row number 1 of the original image x(2) has another color; when the subpixel at row number 7 of the original image 0, the subpixel at row number 4 of the original image x(1), and the subpixel at row number 1 of the original image x(2) are superimposed, there is a superposition of two colors. Therefore, a row of subpixels superimposed from the three rows of subpixels in the original image 0, the original image x(1), and the original image x(2) has a total of 26 subpixels (e.g., ... Figure 24 In the synthesis 2), the subpixels are numbered 1-26 in sequence. There is a continuous segment of subpixels in this row that can obtain two colors of information, namely the subpixels numbered 6-21. Therefore, the requirements for obtaining the synthesized image are met.

[0379] Similarly, the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color G / Y; and the process of moving and superimposing sub-pixels in the y-direction for any column of sub-pixels with the first sub-pixel color R; are all based on the same principle or pattern as the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color R. This can be referred to... Figure 24 The descriptions shown are not listed here.

[0380] It should be understood that, in this case, the principle or rule of obtaining a composite image based on sub-pixel density is similar to the aforementioned... Figure 19 The principles or rules described are the same and can be referred to. Figure 19 The description shown will not be repeated here.

[0381] Example 5: Using Figure 13For the photosensitive element shown, the technical solution provided by the embodiments of the present application is described by taking n=3, a=1, b=2, and s=5d / 3 as an example. The details are as follows:

[0382] In this case, each sensing pixel in the photosensitive element is divided into 3×3 sensing sub-pixels, wherein the size of each sensing sub-pixel is d / 3×d / 3, as Figure 14 shown. m=(a×n+b)|(2×n)=(1×3+2)|(2×3)=5, so n<m; and f=2n-m=2×3-5=1, f×(n-1)=1×(3-1)=2, so n>f×(n-1); if it is required to move 2 times in the x direction and 2 times in the y direction, the sub-pixels in the complete imaging area cannot all contain three types of color information. To make all sub-pixels in the complete imaging area contain three types of color information, in this example, it is required to move 3 times in the x direction and 3 times in the y direction.

[0383] I. Imaging Process

[0384] On the xOy coordinate system, imaging is performed at positions (W / 2, H / 2), (5d / 3+W / 2, H / 2), (10d / 3+W / 2, H / 2), (5d+W / 2, H / 2), (5d+W / 2, 5d / 3+H / 2), (5d+W / 2, 10d / 3+H / 2), and (5d+W / 2, 5d+H / 2) respectively, to obtain 7 original images. The photosensitive element moves in the x direction first, and after the movement in the x direction is completed, it then moves in the y direction. The specific movement process is as follows:

[0385] 1) Perform imaging at the original position.

[0386] At time t(0), the photosensitive element performs imaging at the original position (W / 2, H / 2) to obtain an original image 0.

[0387] 2) Move in the x direction and perform imaging.

[0388] During the time period from t(0) to t(1)_x, the photosensitive element moves a distance of 5d / 3 in the x direction; the photosensitive element performs imaging at the position where it is located at time t(1)_x, that is, the photosensitive element performs imaging at the position (5d / 3+W / 2, H / 2) to obtain an original image x(1).

[0389] During the time period from t(1)_x to t(2)_x, the photosensitive element moves a distance of 5d / 3 in the x direction; the photosensitive element performs imaging at the position where it is located at time t(2)_x, that is, the photosensitive element performs imaging at the position (10d / 3+W / 2, H / 2) to obtain an original image x(2).

[0390] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance of 5d / 3 in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (5d+W / 2, H / 2) to obtain the original image x(3).

[0391] 3) Move in the y direction and capture the image.

[0392] During the time interval from t(3)_x to t(1)_y, the photosensitive element moves a distance of 5d / 3 in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (5d+W / 2, 5d / 3+H / 2) to obtain the original image y(1).

[0393] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 5d / 3 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (5d+W / 2, 10d / 3+H / 2) to obtain the original image y(2).

[0394] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance of 5d / 3 in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (5d+W / 2, 5d+H / 2) to obtain the original image y(3).

[0395] II. Synthesis Process

[0396] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 25 As shown. For any row of sub-pixels captured by the sensing sub-pixels, there are a total of 30 sub-pixels, numbered sequentially from 1 to 30; if only moving twice in the x direction, the row of sensing sub-pixels is captured into 3 rows of sub-pixels according to their positions on the coordinate system xOy, where these 3 rows of sub-pixels are in the original image 0, the original image x(1), and the original image x(2), respectively; since there are sub-pixels with only the same color that are superimposed during the superposition process, such as Figure 25The subpixels enclosed by the dashed box have only one color superimposed. For example, the subpixel with row number 13 in the original image 0, the subpixel with row number 8 in the original image x(1), and the subpixel with row number 3 in the original image x(2) have only one color superimposed; the subpixel with row number 16 in the original image 0, the subpixel with row number 11 in the original image x(1), and the subpixel with row number 6 in the original image x(2) have only one color superimposed, and so on. Therefore, in the row of subpixels obtained by superposition, there are always intermittent subpixels that can only obtain one color information, and it is impossible to have a continuous segment of subpixels that can obtain two colors. Figure 25 In the synthesis 1, the superimposed row of subpixels has a total of 40 subpixels, numbered 1-40. In this row of subpixels, there are always intermittent subpixels that can only obtain one color information, and there is no continuous segment of subpixels that can obtain two colors.

[0397] During the superposition process, each sub-pixel needs to acquire three colors, specifically two colors in the x-direction and two colors in the y-direction, which are R and G / Y / W from the x-direction image and G / Y / W and B from the y-direction image, or G / Y and B from the x-direction image and R and G / Y from the y-direction image. Therefore, in this case, an additional image acquisition is needed in the x-direction to obtain the original image x(3). After moving 3 times in the x-direction, the row of sensed sub-pixels is acquired into 4 rows of sub-pixels and superimposed according to their positions on the coordinate system xOy. These 4 rows of sub-pixels are respectively in the original image 0, the original image x(1), the original image x(2), and the original image x(3). Thus, when only the original image 0, the original image x(1), and the original image x(2) are superimposed, the sub-pixels with only one color superimposed will be superimposed with the sub-pixels of the other color in the original image x(3). For example, subpixels with row number 16 in the original image 0, subpixels with row number 11 in the original image x(1), and subpixels with row number 6 in the original image x(2) have only one color superposition, while subpixels with row number 1 in the original image x(3) have another color; when subpixels with row number 16 in the original image 0, subpixels with row number 11 in the original image x(1), subpixels with row number 6 in the original image x(2), and subpixels with row number 1 in the original image x(3) are superimposed, there is a superposition of two colors. Thus, a row of superimposed subpixels has a total of 45 subpixels (e.g., Figure 25 In the synthesis 2), the sub-pixels are numbered 1-45 in sequence. There is a continuous segment of sub-pixels in this row that can obtain two colors of information, namely the sub-pixels numbered 14-32. Therefore, the requirements for obtaining the synthesized image are met.

[0398] Similarly, for any row of sensing sub-pixels where the color of the first sensing sub-pixel is G / Y, the process of moving for image capture and superposition in the x direction; and for any column of sensing sub-pixels where the color of the first sensing sub-pixel is R, the process of moving for image capture in the y direction; and for any column of sensing sub-pixels where the color of the first sensing sub-pixel is G / Y, the process of moving for image capture and superposition in the y direction; all share the same principle or rule as the process of moving for image capture and superposition in the x direction for any row of sensing sub-pixels where the color of the first sensing sub-pixel is R, and reference may be made thereto Figure 25 The description shown in the above will not be enumerated here.

[0399] It should be understood that in this case, the principle or rule of the process of obtaining a composite image according to the sub-pixels is the same as that of the aforementioned Figure 19 The principle or rule described in the above is the same, and reference may be made thereto Figure 19 The description shown in the above will not be repeated here.

[0400] Example 6: Using Figure 13 The photosensitive element shown in the above performs moving image capture with n=5, a=1, b=4, s=9d / 5 to describe the technical solution provided by the embodiments of the present application. The details are as follows:

[0401] In this case, each sensing pixel in the photosensitive element is divided into 5×5 sensing sub-pixels, wherein the size of each sensing sub-pixel is d / 5×d / 5, as shown in Figure 20 Shown in the above. m=(a×n+b)|(2×n)=(1×5+4)|(2×5)=9, so n<m; and f=2n-m=2×5-9=1, f×(n-1)=1×(5-1)=4, so n>f×(n-1); if 4 moves are required in the x direction and 4 moves are required in the y direction, it is impossible to make all sub-pixels in the complete image capture area contain three color information. In order to make all sub-pixels in the complete image capture area contain three color information, this example requires 5 moves in the x direction and 5 moves in the y direction.

[0402] 1. Image capture process

[0403] In the coordinate system xOy, images are captured at positions (W / 2, H / 2), (9d / 5+W / 2, H / 2), (18d / 5+W / 2, H / 2), (27d / 5+W / 2, H / 2), (36d / 5+W / 2, H / 2), (9d+W / 2, H / 2), (9d+W / 2, 9d / 5+H / 2), (9d+W / 2, 18d / 5+H / 2), (9d+W / 2, 27d / 5+H / 2), (9d+W / 2, 36d / 5+H / 2), and (9d+W / 2, 9d+H / 2), to obtain 11 original images. The photosensitive element moves first in the x-direction, and after completing the x-direction movement, it moves in the y-direction. The specific movement process is as follows:

[0404] 1) Take an image at the original position.

[0405] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0406] 2) Moving and capturing images in the x-direction.

[0407] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance of 9d / 5 in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (9d / 5+W / 2, H / 2) to obtain the original image x(1).

[0408] During the time interval t(1)_x to t(2)_x, the photosensitive element moves a distance of 9d / 5 in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (18d / 5+W / 2, H / 2) to obtain the original image x(2).

[0409] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance of 9d / 5 in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (27d / 5+W / 2, H / 2) to obtain the original image x(3).

[0410] During the time interval t(3)_x to t(4)_x, the photosensitive element moves a distance of 9d / 5 in the x direction; the photosensitive element takes an image at the position at time t(4)_x, that is, the photosensitive element takes an image at the position (36d / 5+W / 2, H / 2) to obtain the original image x(4).

[0411] During the time interval t(4)_x to t(5)_x, the photosensitive element moves a distance of 9d / 5 in the x direction; the photosensitive element takes an image at the position at time t(5)_x, that is, the photosensitive element takes an image at the position (9d+W / 2, H / 2) to obtain the original image x(5).

[0412] 3) Move in the y direction and capture the image.

[0413] During the time interval from t(4)_x to t(1)_y, the photosensitive element moves a distance of 9d / 5 in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (9d+W / 2, 9d / 5+H / 2) to obtain the original image y(1).

[0414] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 9d / 5 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (9d+W / 2, 18d / 5+H / 2) to obtain the original image y(2).

[0415] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance of 9d / 5 in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (9d+W / 2, 27d / 5+H / 2) to obtain the original image y(3).

[0416] During the time interval from t(3)_y to t(4)_y, the photosensitive element moves a distance of 9d / 5 in the y direction; the photosensitive element takes an image at the position at time t(4)_y, that is, the photosensitive element takes an image at the position (9d+W / 2, 36d / 5+H / 2) to obtain the original image y(4).

[0417] During the time interval from t(4)_y to t(5)_y, the photosensitive element moves a distance of 9d / 5 in the y direction; the photosensitive element takes an image at the position at time t(5)_y, that is, the photosensitive element takes an image at the position (9d+W / 2, 9d+H / 2) to obtain the original image y(5).

[0418] II. Synthesis Process

[0419] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 26As shown. For any row of sub-pixels captured by the sensing sub-pixels, there are a total of 50 sub-pixels, numbered sequentially from 1 to 50; if only moving 4 times in the x direction, the row of sensing sub-pixels is captured into 5 rows of sub-pixels according to their positions on the coordinate system xOy, where these 5 rows of sub-pixels are respectively in the original image 0, original image x(1), original image x(2), original image x(3), and original image x(4). Since there are sub-pixels of the same color that are superimposed during the superposition process, such as Figure 26 The subpixels enclosed by the dashed box have only one color superposition; for example, the subpixel with row number 41 in the original image 0, the subpixel with row number 32 in the original image x(1), the subpixel with row number 23 in the original image x(2), the subpixel with row number 14 in the original image x(3), and the subpixel with row number 5 in the original image x(4) have only one color superposition, the subpixel with row number 46 in the original image 0, the subpixel with row number 37 in the original image x(1), the subpixel with row number 28 in the original image x(2), the subpixel with row number 19 in the original image x(3), and the subpixel with row number 10 in the original image x(4) have only one color superposition, and so on. Therefore, in the row of subpixels obtained by superposition, there are always intermittent subpixels that can only obtain one color information, and it is impossible to have a continuous segment of subpixels that can obtain two colors.

[0420] During the superposition process, each sub-pixel needs to acquire three colors, specifically two colors in the x-direction and two colors in the y-direction, which are R and G / Y / W from the x-direction image and G / Y / W and B from the y-direction image, or G / Y and B from the x-direction image and R and G / Y from the y-direction image. Therefore, in this case, an additional image acquisition is needed in the x-direction to obtain the original image x(5). After moving 5 times in the x-direction, the row of sensed sub-pixels is acquired into 6 rows of sub-pixels and superimposed according to their positions on the coordinate system xOy. These 6 rows of sub-pixels are respectively in the original image 0, original image x(1), original image x(2), original image x(3), original image x(4), and original image x(5). Thus, when only the original images 0, original image x(1), original image x(2), original image x(3), and original image x(4) are superimposed, the sub-pixels with only one color superimposed will be superimposed with the sub-pixels of another color in the original image x(5). For example, subpixels with row number 46 in the original image 0, subpixels with row number 37 in the original image x(1), subpixels with row number 28 in the original image x(2), subpixels with row number 19 in the original image x(3), and subpixels with row number 10 in the original image x(4) have only one color superposition, while subpixels with row number 1 in the original image x(5) have another color; when subpixels with row number 46 in the original image 0, subpixels with row number 37 in the original image x(1), subpixels with row number 28 in the original image x(2), subpixels with row number 19 in the original image x(3), subpixels with row number 10 in the original image x(4), and subpixels with row number 1 in the original image x(5) are superimposed, there is a superposition of two colors. Thus, in the superimposed row of subpixels, there is a continuous segment of subpixels that can obtain information of both colors, thus satisfying the requirements for obtaining a composite image. It should be noted that, due to the large number of subpixels, Figure 26 It is impossible to list them all. Figure 26 Only a portion of the subpixel overlay is shown.

[0421] Similarly, the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color G / Y; and the process of moving and superimposing sub-pixels in the y-direction for any column of sub-pixels with the first sub-pixel color R; are all based on the same principle or pattern as the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color R. (Refer to...) Figure 26 The descriptions shown are not listed here.

[0422] It should be understood that, in this case, the principle or rule of obtaining a composite image based on sub-pixel density is similar to the aforementioned... Figure 19 The principles or rules described are the same and can be referred to. Figure 19 The description shown will not be repeated here.

[0423] Example 7: Using moving image acquisition with n=5, a=2, b=1, and s=11d / 5, the technical solution provided in the embodiments of this application is described. Specifically, it is as follows:

[0424] In this case, each sensing pixel in the photosensitive element is divided into 5×5 sensing sub-pixels, where the size of each sensing sub-pixel is d / 5×d / 5. m=(a×n+b)|(2×n)=(2×5+1)|(2×5)=1, therefore n>m; while f=m=1, f×(n-1)=1×(5-1)=4, therefore n>f×(n-1); if 4 movements are required in the x-direction and 4 movements are required in the y-direction, it is impossible to ensure that all sub-pixels in the complete image area contain three color information. To ensure that all sub-pixels in the complete image area contain three color information, this example requires 5 movements in the x-direction and 5 movements in the y-direction.

[0425] I. Image Acquisition Process

[0426] In the coordinate system xOy, images are captured at positions (W / 2, H / 2), (11d / 5+W / 2, H / 2), (22d / 5+W / 2, H / 2), (33d / 5+W / 2, H / 2), (44d / 5+W / 2, H / 2), (11d+W / 2, H / 2), (11d+W / 2, 11d / 5+H / 2), (11d+W / 2, 22d / 5+H / 2), (11d+W / 2, 33d / 5+H / 2), (11d+W / 2, 44d / 5+H / 2), and (11d+W / 2, 11d+H / 2), resulting in 11 original images. The photosensitive element moves first in the x-direction, and after completing the x-direction movement, it moves in the y-direction. The specific movement process is as follows:

[0427] 1) Take an image at the original position.

[0428] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0.

[0429] 2) Moving and capturing images in the x-direction.

[0430] During the time interval from t(0) to t(1)_x, the photosensitive element moves a distance of 11d / 5 in the x direction; the photosensitive element takes an image at the position at time t(1)_x, that is, the photosensitive element takes an image at the position (11d / 5+W / 2, H / 2) to obtain the original image x(1).

[0431] During the time interval from t(1)_x to t(2)_x, the photosensitive element moves a distance of 11d / 5 in the x direction; the photosensitive element takes an image at the position at time t(2)_x, that is, the photosensitive element takes an image at the position (22d / 5+W / 2, H / 2) to obtain the original image x(2).

[0432] During the time interval from t(2)_x to t(3)_x, the photosensitive element moves a distance of 11d / 5 in the x direction; the photosensitive element takes an image at the position at time t(3)_x, that is, the photosensitive element takes an image at the position (33d / 5+W / 2, H / 2) to obtain the original image x(3).

[0433] During the time interval t(3)_x to t(4)_x, the photosensitive element moves a distance of 11d / 5 in the x direction; the photosensitive element takes an image at the position at time t(4)_x, that is, the photosensitive element takes an image at the position (44d / 5+W / 2, H / 2) to obtain the original image x(4).

[0434] During the time interval t(4)_x to t(5)_x, the photosensitive element moves a distance of 11d / 5 in the x direction; the photosensitive element takes an image at the position where it is at time t(5)_x, that is, the photosensitive element takes an image at the position (11d+W / 2, H / 2) to obtain the original image x(5).

[0435] 3) Move in the y direction and capture the image.

[0436] During the time interval from t(4)_x to t(1)_y, the photosensitive element moves a distance of 11d / 5 in the y direction; the photosensitive element takes an image at the position at time t(1)_y, that is, the photosensitive element takes an image at the position (11d+W / 2, 11d / 5+H / 2) to obtain the original image y(1).

[0437] During the time interval from t(1)_y to t(2)_y, the photosensitive element moves a distance of 11d / 5 in the y direction; the photosensitive element takes an image at the position at time t(2)_y, that is, the photosensitive element takes an image at the position (11d+W / 2, 22d / 5+H / 2) to obtain the original image y(2).

[0438] During the time interval from t(2)_y to t(3)_y, the photosensitive element moves a distance of 11d / 5 in the y direction; the photosensitive element takes an image at the position at time t(3)_y, that is, the photosensitive element takes an image at the position (11d+W / 2, 33d / 5+H / 2) to obtain the original image y(3).

[0439] During the time interval from t(3)_y to t(4)_y, the photosensitive element moves a distance of 11d / 5 in the y direction; the photosensitive element takes an image at the position at time t(4)_y, that is, the photosensitive element takes an image at the position (11d+W / 2, 44d / 5+H / 2) to obtain the original image y(4).

[0440] During the time interval from t(4)_y to t(5)_y, the photosensitive element moves a distance of 11d / 5 in the y direction; the photosensitive element takes an image at the position at time t(5)_y, that is, the photosensitive element takes an image at the position (11d+W / 2, 11d+H / 2) to obtain the original image y(5).

[0441] II. Synthesis Process

[0442] The process of moving and superimposing the first sensing sub-pixel of color R in any row of sensing sub-pixels in the x-direction is as follows: Figure 27 As shown. For any row of sub-pixels captured by the sensing sub-pixels, there are a total of 70 sub-pixels, numbered sequentially from 1 to 70; if only moving 4 times in the x direction, the row of sensing sub-pixels is captured into 5 rows of sub-pixels according to their positions on the xOy coordinate system, where these 5 rows of sub-pixels are respectively in the original image 0, original image x(1), original image x(2), original image x(3), and original image x(4). Since there are sub-pixels of the same color that are superimposed during the superposition process, such as Figure 27 The subpixels enclosed by the dashed box have only one color superposition; for example, the subpixel with row number 60 in the original image 0, the subpixel with row number 49 in the original image x(1), the subpixel with row number 38 in the original image x(2), the subpixel with row number 27 in the original image x(3), and the subpixel with row number 16 in the original image x(4) have only one color superposition, the subpixel with row number 65 in the original image 0, the subpixel with row number 54 in the original image x(1), the subpixel with row number 43 in the original image x(2), the subpixel with row number 32 in the original image x(3), and the subpixel with row number 21 in the original image x(4) have only one color superposition, and so on. Therefore, in the row of subpixels obtained by superposition, there are always intermittent subpixels that can only obtain one color information, and it is impossible to have a continuous segment of subpixels that can obtain two colors.

[0443] During the superposition process, each sub-pixel needs to acquire three colors, specifically two colors in the x-direction and two colors in the y-direction, which are R and G / Y / W from the x-direction image and G / Y / W and B from the y-direction image, or G / Y and B from the x-direction image and R and G / Y from the y-direction image. Therefore, in this case, an additional image acquisition is needed in the x-direction to obtain the original image x(5). After moving 5 times in the x-direction, the row of sensed sub-pixels is acquired into 6 rows of sub-pixels and superimposed according to their positions on the coordinate system xOy. These 6 rows of sub-pixels are respectively in the original image 0, original image x(1), original image x(2), original image x(3), original image x(4), and original image x(5). Thus, when only the original images 0, original image x(1), original image x(2), original image x(3), and original image x(4) are superimposed, the sub-pixels with only one color superimposed will be superimposed with the sub-pixels of another color in the original image x(5). For example, subpixels with row number 60 in the original image 0, subpixels with row number 49 in the original image x(1), subpixels with row number 38 in the original image x(2), subpixels with row number 27 in the original image x(3), and subpixels with row number 16 in the original image x(4) have only one color superposition, while subpixels with row number 5 in the original image x(5) have another color; when subpixels with row number 60 in the original image 0, subpixels with row number 49 in the original image x(1), subpixels with row number 38 in the original image x(2), subpixels with row number 27 in the original image x(3), subpixels with row number 16 in the original image x(4), and subpixels with row number 5 in the original image x(5) are superimposed, there is a superposition of two colors. Thus, in the superimposed row of subpixels, there is a continuous segment of subpixels that can obtain information of both colors, thus satisfying the requirements for obtaining a composite image. It should be noted that, due to the large number of subpixels, Figure 27 It is impossible to list them all. Figure 27 Only a portion of the subpixel overlay is shown.

[0444] Similarly, the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color G / Y; and the process of moving and superimposing sub-pixels in the y-direction for any column of sub-pixels with the first sub-pixel color R; are all based on the same principle or pattern as the process of moving and superimposing sub-pixels in the x-direction for any row of sub-pixels with the first sub-pixel color R. (Refer to...) Figure 27 The descriptions shown are not listed here.

[0445] It should be understood that, in this case, the principle or rule of obtaining a composite image based on sub-pixel density is similar to the aforementioned... Figure 19 The principles or rules described are the same and can be referred to. Figure 19 The description shown will not be repeated here.

[0446] Example 8: Using Figure 13 The photosensitive element shown is used to move and capture images with n=2, a=1, b=1, and s=3d / 2 to describe the technical solution provided in the embodiments of this application. Specifically, it is as follows:

[0447] In this case, each sensing pixel in the photosensitive element is divided into 2×2 sensing sub-pixels, where the size of each sensing sub-pixel is d / 2×d / 2, such as Figure 23 As shown, moving a distance of s = 3d / 2 simultaneously and at the same rate in the x and y directions is insufficient to ensure that all sub-pixels in the complete image region contain all three colors. The missing color information of a certain sub-pixel needs to be estimated and filled in.

[0448] I. Image Acquisition Process

[0449] 1) Take an image at the original position.

[0450] At time t(0), the photosensitive element takes an image at the original position (W / 2, H / 2) to obtain the original image 0, as shown in the figure. Figure 28 As shown.

[0451] 2) Take the image after moving a distance of s = 3d / 2 at the same speed in both the x and y directions.

[0452] During the time interval t(0) to t(1), the photosensitive element moves from the original position (W / 2, H / 2) to the imaging position (3d / 2+W / 2, 3d / 2+H / 2); at time t(1), the photosensitive element captures an image at the position (3d / 2+W / 2, 3d / 2+H / 2) to obtain the original image 1, as shown in the figure. Figure 29 As shown.

[0453] II. Synthesis Process

[0454] Figure 30 The process of superimposing the original image 0 and the original image 1 according to their image capture positions is shown. Figure 30 The result is shown by superimposing the original image 0 and the original image 1 according to their image capture positions. For example... Figure 31As shown, the size of the subpixel in the complete image area is d / 2×d / 2; the incomplete image area contains subpixels of various sizes as well as pixels that are not classified as subpixels. The sizes of the subpixels in the incomplete image area include d×d / 2, d / 2×d, and 3 / 4 of the size of the sensing pixel. In the incomplete image area, a subpixel of size d×d / 2 can be regarded as 2×1 subpixels of size d / 2×d / 2, 3 / 4 of the size of the sensing pixel can be regarded as 3 subpixels of size d / 2×d / 2, and pixels that are not classified as subpixels can be regarded as 2×2 subpixels of size d / 2×d / 2. The newly added image area contains sub-pixels of various sizes, as well as pixels that are not classified as sub-pixels. The sizes of the sub-pixels in the newly added image area include d×d / 2, d / 2×d, and 3 / 4 of the size of the sensing pixel. In the newly added image area, a sub-pixel of size d×d / 2 can be regarded as 2×1 sub-pixels of size d / 2×d / 2, and a sub-pixel of size 3 / 4 of the size of the sensing pixel can be regarded as 3 sub-pixels of size d / 2×d / 2. Pixels that are not classified as sub-pixels can be regarded as 2×2 sub-pixels of size d / 2×d / 2.

[0455] The pixels in the composite image originate from sub-pixels numbered 1 to 20 in rows and 1 to 20 in columns. Specifically, sub-pixels numbered 4 to 20 in rows and 1 to 20 in columns are sub-pixels within the complete image region, while sub-pixels numbered 1 to 3 in rows and 1 to 3 in columns are sub-pixels within the incomplete image region. Sub-pixels numbered 21 to 23 in rows and 21 to 23 are sub-pixels within the newly added image region.

[0456] Subpixels within the complete image region can be used as pixels in the composite image; alternatively, subpixels within both the complete and incomplete image regions can be used; or, subpixels within the complete and incomplete image regions, along with pixels in the incomplete image region that were not classified as subpixels, can be used as pixels in the composite image. Subpixels within the complete image region contain one or two color information. When used as pixels in the composite image, the missing color information is obtained through color estimation and padding. The process of color estimation and padding is described previously and will not be repeated here.

[0457] Please see Figure 32 , Figure 32is a schematic structural diagram of an image processing apparatus 3200 provided in an embodiment of the present application, the image processing apparatus 3200 comprising: an obtaining unit 3201, configured to obtain a plurality of original images, where the plurality of original images are images respectively sensed by a photosensitive element at a plurality of imaging positions, and a distance between two closest imaging positions among the plurality of imaging positions is greater than a distance between any two adjacent sensing pixels in the photosensitive element; and a superimposition unit 3202, configured to superimpose the plurality of original images based on the plurality of imaging positions to obtain a composite image, where pixels in the composite image are determined according to sub-pixels obtained by division during the superimposition of the plurality of original images, and the resolution of the composite image is greater than the resolution of any one of the plurality of original images.

[0458] In a possible implementation, the distance s between the two closest imaging positions among the plurality of imaging positions is [(a×n+b) / n]×d; the a is a positive integer; the n is a prime number; 1≤b≤n-1, and the b is an integer; and the d is the distance between any two adjacent sensing pixels in the photosensitive element.

[0459] In a possible implementation, the n is not 2 and n<f×(n-1), and coordinates of any one imaging position among the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where,

[0460] the m is a remainder of a×n+b divided by 2n;

[0461] 0≤i≤n-1 and 0≤j≤n-1, in any update of the (i×s+W / 2, j×s+H / 2), the i and the j do not change at the same time, both the i and the j change monotonically, and the i and the j are integers; the W is the number of sensing pixels in the width direction of the photosensitive element; the H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses spectra in three wavelength ranges, the spectra in the three wavelength ranges correspond to three types of color information, and that the pixels in the composite image are determined according to the sub-pixels obtained by division during the superimposition of the plurality of original images comprises: the pixels in the composite image are determined according to first sub-pixels obtained by division during the superimposition of the plurality of original images, where the first sub-pixels comprise the three types of color information.

[0462] In a possible implementation, the n is 2 or n>f×(n-1), and coordinates of any one imaging position among the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where,

[0463] The m is the remainder when a×n+b is divided by 2n;

[0464] 0≤i≤n and 0≤j≤n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, the three wavelength ranges of spectrum correspond to three color information, and the pixel determination in the composite image based on the sub-pixel division obtained during the superposition of multiple original images includes: the pixel determination in the composite image based on the first sub-pixel division obtained during the superposition of multiple original images, the first sub-pixel containing the three color information.

[0465] In one possible implementation, the determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: determining pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixel lacks one or two of the three color information; determining any missing color information of the second sub-pixel based on any color information of the third sub-pixel, wherein the third sub-pixel is a sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is a sub-pixel near the second sub-pixel that contains any of the color information.

[0466] In one possible implementation, n is 2 or n > f × (n-1), and the coordinates of any one of the plurality of image-taking positions are (i × s + W / 2, j × s + H / 2) in the same coordinate system; wherein,

[0467] The m is the remainder when a×n+b is divided by 2n;

[0468] 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the pixels in the synthesized image are determined according to the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined according to the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel is missing one or two of the three color information; any missing color information of the second sub-pixel is determined according to any color information of the third sub-pixel, the third sub-pixel is the sub-pixel obtained during the superposition of multiple original images, and the third sub-pixel is the sub-pixel near the second sub-pixel that contains the any color information.

[0469] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, and the coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein 0≤i≤n-1 and 0≤j≤n-1, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0470] In one possible implementation, n is 2, the plurality of imaging positions are two imaging positions, and the coordinates of the two imaging positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein, W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information; the determination of pixels in the composite image based on sub-pixels obtained during the superposition of the plurality of original images includes: the pixels in the composite image are determined based on second sub-pixels obtained during the superposition of the plurality of original images, the second sub-pixel lacking one or two of the three color information; any missing color information of the second sub-pixel is determined based on any color information of the third sub-pixel, the third sub-pixel being a sub-pixel obtained during the superposition of the plurality of original images, and the third sub-pixel being a sub-pixel near the second sub-pixel containing the any color information.

[0471] In one possible implementation, the photosensitive element senses a spectrum of a wavelength range, where n is 2, and the plurality of imaging positions are two imaging positions, the coordinates of which are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

[0472] It should be noted that the implementation of each unit in the embodiments of this application can also refer to the relevant descriptions in the foregoing other embodiments, which will not be repeated here. Of course, the image processing apparatus 3200 provided in the embodiments of this application includes, but is not limited to, the above-mentioned unit modules. For example, the image processing apparatus 3200 may also include a storage unit 3203. The storage unit 3203 can be used to store the program code and data of the image processing apparatus 3200. It should be further noted that... Figure 32 The beneficial effects of the illustrated embodiments can be found in the descriptions of the other embodiments described above.

[0473] This application embodiment also provides an image processing apparatus, which includes a processor and a photosensitive element; the processor is configured to sense at multiple imaging positions using the photosensitive element to obtain multiple original images, wherein the distance between the two imaging positions with the smallest distance among the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; and to superimpose the multiple original images based on the multiple imaging positions to obtain a composite image, wherein the pixels in the composite image are determined according to the sub-pixels obtained during the superposition of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images.

[0474] This application also provides an image processing apparatus, including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing steps in the methods as described in the above embodiments.

[0475] This application also provides an electronic device including a processor, a memory, a communication interface, and one or more programs stored in the memory and configured to be executed by the processor. The programs include instructions for performing steps in the methods described in the above embodiments.

[0476] This application also provides a chip, including: a processor, for calling and running a computer program from a memory, causing a device on which the chip is installed to perform the methods as described in the above embodiments.

[0477] This application also provides a computer-readable storage medium comprising a computer program that, when executed on a computer or processor, causes the computer or processor to perform the methods described in the above embodiments.

[0478] This application also provides a computer program product, which includes a computer program that, when run on a computer or processor, causes the computer or processor to perform the methods described in the above embodiments.

[0479] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0480] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0481] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0482] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0483] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0484] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0485] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0486] The steps in the methods of this application can be adjusted, combined, or deleted according to actual needs. Furthermore, the terminology and explanations in the embodiments of this application can be referred to the corresponding descriptions in other embodiments.

[0487] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0488] The above description and embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, comprising: acquiring a plurality of original images, wherein the plurality of original images are images respectively sensed by a photosensitive element at a plurality of image capturing positions, and the distance between two image capturing positions with the smallest distance among the plurality of image capturing positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; superimposing the plurality of original images based on the plurality of image capturing positions to obtain a composite image, wherein pixels in the composite image are determined according to sub-pixels obtained by division during the superimposition of the plurality of original images, and the resolution of the composite image is greater than the resolution of any one of the plurality of original images; the distance s between the two image capturing positions with the smallest distance among the plurality of image capturing positions is [(a×n+b) / n]×d; the a is a positive integer; the n is a prime number; 1≤b≤n-1, and the b is an integer; the d is the distance between any two adjacent sensing pixels in the photosensitive element; n is selected according to the multiple of resolution to be increased, and a and b are selected according to the controllable accuracy of the actuator.

2. The method according to claim 1, characterized in that, the n is not 2 and n<f×(n-1), and the coordinate of any one of the plurality of image capturing positions is (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n-1 and 0≤j≤n-1, in any update of the (i×s+W / 2, j×s+H / 2), the i and the j do not change at the same time, both the i and the j change monotonically, and the i and the j are integers; the W is the number of sensing pixels in the width direction of the photosensitive element; the H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses spectrums of three wavelength ranges, the spectrums of the three wavelength ranges correspond to three types of color information, and that the pixels in the composite image are determined according to the sub-pixels obtained by division during the superimposition of the plurality of original images comprises: the pixels in the composite image are determined according to first sub-pixels obtained by division during the superimposition of the plurality of original images, and the first sub-pixels comprise the three types of color information.

3. The method according to claim 1, characterized in that, the n is 2 or n>f×(n-1), and the coordinate of any one of the plurality of image capturing positions is (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n and 0≤j≤n, in any update of the (i×s+W / 2, j×s+H / 2), the i and the j do not change at the same time, both the i and the j change monotonically, and the i and the j are integers; the W is the number of sensing pixels in the width direction of the photosensitive element; the H is the number of sensing pixels in the height direction of the photosensitive element; the photosensitive element senses spectrums of three wavelength ranges, the spectrums of the three wavelength ranges correspond to three types of color information, and that the pixels in the composite image are determined according to the sub-pixels obtained by division during the superimposition of the plurality of original images comprises: the pixels in the composite image are determined according to first sub-pixels obtained by division during the superimposition of the plurality of original images, and the first sub-pixels comprise the three types of color information.

4. The method according to claim 2 or 3, characterized in that, The determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: the determination of pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixels lack one or two of the three types of color information; The missing color information of the second sub-pixel is determined based on the missing color information of the third sub-pixel. The third sub-pixel is a sub-pixel obtained during the superposition of the multiple original images. The third sub-pixel is a sub-pixel near the second sub-pixel that contains the missing color information.

5. The method according to claim 1, characterized in that, The n is 2 or n>f×(n-1), and the coordinates of any one of the multiple image-taking positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; The photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information. The pixels in the synthesized image are determined based on the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined based on the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel lacks one or two of the three color information. The missing color information of the second sub-pixel is determined based on the missing color information of the third sub-pixel. The third sub-pixel is a sub-pixel obtained during the superposition of the multiple original images. The third sub-pixel is a sub-pixel near the second sub-pixel that contains the missing color information.

6. The method according to claim 1, characterized in that, The photosensitive element senses a spectrum within a wavelength range. The coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where 0≤i≤n-1 and 0≤j≤n-1, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

7. The method according to claim 1, characterized in that, The number of pixels (n) is 2, and the plurality of image acquisition positions are two image acquisition positions. The coordinates of the two image acquisition positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively. Wherein, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. The photosensitive element senses spectra in three wavelength ranges, the spectra in the three wavelength ranges correspond to three pieces of color information, and the pixel in the synthesized image is determined according to the sub-pixel obtained by division during the superposition of the multiple original images, which includes: the pixel in the synthesized image is determined according to the second sub-pixel obtained by division during the superposition of the multiple original images, and the second sub-pixel lacks one or two of the three pieces of color information; Any piece of color information missing from the second sub-pixel is determined according to the said any piece of color information of a third sub-pixel, the third sub-pixel is a sub-pixel obtained by division during the superposition of the multiple original images, and the third sub-pixel is a sub-pixel near the second sub-pixel that contains the said any piece of color information.

8. The method according to claim 1, characterized in that, The photosensitive element senses a spectrum in one wavelength range, said n is 2, the plurality of image capturing positions are two image capturing positions, and the coordinates of the two image capturing positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) respectively in the same coordinate system; wherein, said W is the number of sensing pixels in the width direction of the photosensitive element; said H is the number of sensing pixels in the height direction of the photosensitive element.

9. An image processing apparatus, characterized in that, comprises: an acquisition unit, configured to acquire a plurality of original images, wherein the plurality of original images are images obtained by sensing respectively at a plurality of image capturing positions via a photosensitive element, and the distance between two image capturing positions with the smallest distance among the plurality of image capturing positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element; a superposition unit, configured to superpose the plurality of original images based on the plurality of image capturing positions to obtain a synthesized image, wherein pixels in the synthesized image are determined according to sub-pixels obtained by division during the superposition of the plurality of original images, and the resolution of the synthesized image is greater than the resolution of any one of the plurality of original images; The distance s between the two image capturing positions with the smallest distance among the plurality of image capturing positions is [(a×n+b) / n]×d; said a is a positive integer; said n is a prime number; 1≤b≤n-1, and said b is an integer; said d is the distance between any two adjacent sensing pixels in the photosensitive element; n is selected according to the resolution multiplication factor to be increased, and a and b are selected according to the controllable precision of the actuator.

10. The image processing apparatus according to claim 9, characterized in that, Said n is not 2 and n<f×(n-1), and the coordinates of any one of the plurality of image capturing positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; wherein, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n-1 and 0≤j≤n-1, in any update of said (i×s+W / 2, j×s+H / 2), said i and said j do not change simultaneously, said i and said j both change monotonically, and said i and said j are integers; said W is the number of sensing pixels in the width direction of the photosensitive element; said H is the number of sensing pixels in the height direction of the photosensitive element; The photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information. The determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images includes: the determination of pixels in the synthesized image based on first sub-pixels obtained during the superposition of multiple original images, and the first sub-pixel contains the three color information.

11. The image processing apparatus according to claim 9, characterized in that, The n is 2 or n>f×(n-1), and the coordinates of any one of the multiple image-taking positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n and 0≤j≤n, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; The photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information. The determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images includes: the determination of pixels in the synthesized image based on first sub-pixels obtained during the superposition of multiple original images, and the first sub-pixel contains the three color information.

12. The image processing apparatus according to claim 10 or 11, characterized in that, The determination of pixels in the synthesized image based on sub-pixels obtained during the superposition of multiple original images further includes: the determination of pixels in the synthesized image based on second sub-pixels obtained during the superposition of multiple original images, wherein the second sub-pixels lack one or two of the three types of color information; The missing color information of the second sub-pixel is determined based on the missing color information of the third sub-pixel. The third sub-pixel is a sub-pixel obtained during the superposition of the multiple original images. The third sub-pixel is a sub-pixel near the second sub-pixel that contains the missing color information.

13. The image processing apparatus according to claim 9, characterized in that, The n is 2 or n>f×(n-1), and the coordinates of any one of the multiple image-taking positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where, The m is the remainder when a×n+b is divided by 2n; 0≤i≤n-1 and 0≤j≤n-1, in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; H is the number of sensing pixels in the height direction of the photosensitive element; The photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information. The pixels in the synthesized image are determined based on the sub-pixels obtained during the superposition of multiple original images, including: the pixels in the synthesized image are determined based on the first sub-pixel and the second sub-pixel obtained during the superposition of multiple original images, the first sub-pixel contains the three color information, and the second sub-pixel lacks one or two of the three color information. The missing color information of the second sub-pixel is determined based on the missing color information of the third sub-pixel. The third sub-pixel is a sub-pixel obtained during the superposition of the multiple original images. The third sub-pixel is a sub-pixel near the second sub-pixel that contains the missing color information.

14. The image processing apparatus according to claim 9, characterized in that, The photosensitive element senses a spectrum within a wavelength range. The coordinates of any one of the plurality of imaging positions are (i×s+W / 2, j×s+H / 2) in the same coordinate system; where 0≤i≤n-1 and 0≤j≤n-1, and in any update of (i×s+W / 2, j×s+H / 2), i and j do not change simultaneously, i and j are both monotonically changing, and i and j are integers; W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

15. The image processing apparatus according to claim 9, characterized in that, The number of pixels (n) is 2, and the plurality of image acquisition positions are two image acquisition positions. The coordinates of the two image acquisition positions are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively. Wherein, W is the number of sensing pixels in the width direction of the photosensitive element, and H is the number of sensing pixels in the height direction of the photosensitive element. The photosensitive element senses three wavelength ranges of spectrum, and the three wavelength ranges of spectrum correspond to three color information. The pixel determination in the composite image based on the sub-pixel division obtained during the superposition of multiple original images includes: the pixel determination in the composite image based on the second sub-pixel division obtained during the superposition of multiple original images, wherein the second sub-pixel lacks one or two of the three color information. The missing color information of the second sub-pixel is determined based on the missing color information of the third sub-pixel. The third sub-pixel is a sub-pixel obtained during the superposition of the multiple original images. The third sub-pixel is a sub-pixel near the second sub-pixel that contains the missing color information.

16. The image processing apparatus according to claim 9, characterized in that, The photosensitive element senses a spectrum of a wavelength range, where n is 2, and the plurality of imaging positions are two imaging positions, the coordinates of which are (W / 2, H / 2) and (s+W / 2, s+H / 2) in the same coordinate system, respectively; wherein, W is the number of sensing pixels in the width direction of the photosensitive element; and H is the number of sensing pixels in the height direction of the photosensitive element.

17. An image processing apparatus, characterized in that, The image processing device includes a processor and a photosensitive element; The processor is configured to sense at multiple imaging positions using the photosensitive element to obtain multiple original images, wherein the distance between the two imaging positions with the smallest distance among the multiple imaging positions is greater than the distance between any two adjacent sensing pixels in the photosensitive element. And the multiple original images are superimposed based on the multiple image capture positions to obtain a composite image, wherein the pixels in the composite image are determined according to the sub-pixels obtained during the superposition of the multiple original images, and the resolution of the composite image is greater than the resolution of any one of the multiple original images; The distance s between the two image acquisition positions with the smallest distance among the plurality of image acquisition positions is [(a×n+b) / n]×d; where a is a positive integer; n is a prime number; 1≤b≤n-1, where b is an integer; d is the distance between any two adjacent sensing pixels in the photosensitive element; n is selected according to the resolution multiple required to be increased, and a and b are selected according to the controllable accuracy of the actuator.

18. An image processing apparatus, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-8.

19. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1-8.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1-8.

Images