Method of processing an image and apparatus therefor

Abstract

The application discloses a kind of processing method of image and its device, which can be applied to image processing technical field, wherein, method includes: image sensor receives first instruction, first instruction is used to indicate the part of current picture of electronic equipment is amplified (step 201);According to first instruction, determine the selected area corresponding to zoom operation (step 202);The original image data of selected area is rearranged to pixel, and the output image data of image sensor for selected area is obtained (step 203);Selected area output image data is sent to image signal processor (step 204).Thus, when zooming selected area is needed, image sensor can support the pixel rearrangement of the selected area, rather than the pixel rearrangement of fixed area, and so that the image sensor can only output the selected area, which can reduce the power consumption of image sensor.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0002] In the image processing process, the image signal processor (ISP) is responsible for receiving the raw signal data from the image sensor and processing the raw signal data.

[0003] Zooming, especially optical zoom, is a common operation in mobile device cameras (such as smartphones). For example, when it's necessary to magnify a specific area of ​​the preview image, the image sensor and ISP system need to work together, utilizing their technologies to enlarge the area. However, to ensure uninterrupted ISP operation, even when magnifying a small area using the image sensor and ISP, the image sensor still needs to output a relatively large area, which is not optimal for overall power consumption. Summary of the Invention

[0004] This application provides an image processing method and apparatus that can be applied to the field of image processing technology. It enables the image sensor to support pixel rearrangement of any user-selected area, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only to the selected area, thereby reducing the power consumption of the image sensor.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method being executed by an image sensor, the method comprising:

[0006] Receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified.

[0007] Based on the first instruction, the selected area corresponding to the zoom-in operation is determined;

[0008] The original image data of the selected region is subjected to pixel rearrangement processing to obtain the output image data of the image sensor for the selected region.

[0009] The output image data of the selected area is sent to the image signal processor.

[0010] In this technical solution, when a selected area needs to be magnified, the image sensor can support pixel rearrangement of the selected area instead of pixel rearrangement of a fixed area, and enable the image sensor to output only to the selected area, which can reduce the power consumption of the image sensor.

[0011] In one implementation, performing pixel rearrangement processing on the original image data of the selected region to obtain the output image data of the image sensor for the selected region includes:

[0012] The original image data of the selected region is subjected to pixel rearrangement processing to obtain the intermediate image data of the selected region;

[0013] Based on the first instruction, determine the magnification factor corresponding to the magnification operation;

[0014] The intermediate image data is resolution adjusted according to the magnification, and the image data obtained after resolution adjustment is used as the output image data of the image sensor for the selected area.

[0015] In one possible implementation, performing pixel rearrangement processing on the original image data of the selected region to obtain intermediate image data of the selected region includes:

[0016] The original image data of the selected region is converted into Bayer format using a re-mosaic algorithm to obtain intermediate image data of the selected region.

[0017] In one implementation, adjusting the resolution of the intermediate image data according to the magnification factor includes: magnifying the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the magnification factor.

[0018] In one possible implementation, the step of magnifying the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the magnification factor includes:

[0019] Based on the magnification ratio, determine the corresponding resolution adjustment ratio;

[0020] According to the resolution adjustment ratio, the resolution of the intermediate image data is increased to a fixed resolution supported by the image signal processor.

[0021] In one possible implementation, the step of scaling up the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio includes:

[0022] Based on the resolution adjustment ratio, a super-resolution image reconstruction algorithm is used to enlarge the resolution of the intermediate image data to a fixed resolution supported by the image signal processor.

[0023] In one implementation, the magnification is greater than or equal to a preset threshold.

[0024] In one possible implementation, the preset threshold is 2.

[0025] In one implementation, the method further includes:

[0026] The system receives a second instruction, which instructs the image sensor to operate in pixel binning mode; according to the second instruction, it performs binning processing on the acquired raw image data to obtain the output image data of the image sensor; and sends the output image data of the image sensor to the image signal processor.

[0027] Secondly, embodiments of this application provide another image processing method, the method being executed by a processor, the method comprising:

[0028] In response to receiving a zoom-in operation on a portion of the current screen of an electronic device, a first instruction is sent to an image sensor according to the zoom-in operation; the first instruction is used to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the zoom-in operation.

[0029] In this technical solution, when a selected area needs to be magnified, the processor in the camera software system can send instruction information to the image sensor to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the magnification operation. This enables the image sensor to support pixel rearrangement of the selected area, rather than pixel rearrangement of a fixed area, and allows the image sensor to output only to the selected area, thereby reducing the power consumption of the image sensor.

[0030] In one implementation, the first instruction is further configured to instruct the image sensor to adjust the resolution of the intermediate image data obtained after pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor, according to the magnification ratio corresponding to the magnification operation.

[0031] In this technical solution, the resolution of the intermediate image data obtained by the image sensor after pixel rearrangement of the selected area can be adjusted to a fixed resolution supported by the image signal processor. Thus, the resolution of the output image data is adjusted to this fixed resolution within the image sensor. For the ISP platform to receive the data, the resolution remains unchanged, ensuring that the ISP platform does not need to stop the flow and reconfigure.

[0032] In one implementation, the method further includes: in response to not receiving a zoom operation for a portion of the current image, sending a second instruction to the image sensor, the second instruction being used to instruct the image sensor to operate in pixel binning mode.

[0033] Thirdly, embodiments of this application provide yet another image processing method, the method being executed by an image signal processor, the method comprising:

[0034] The image sensor receives first output image data of a selected region; wherein the selected region is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected region.

[0035] In this technical solution, when a selected area needs to be magnified, the image signal processor can receive the output image data sent by the image sensor for the magnified selected area. In other words, it enables the image sensor to support pixel rearrangement of the selected area, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only the selected area. This reduces the power consumption of the image sensor and the system power consumption.

[0036] In one implementation, receiving first output image data of a selected area from an image sensor includes:

[0037] The image sensor receives a second output image data of a selected area; wherein the second output image data is obtained by adjusting the resolution of the intermediate image obtained after pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor according to the magnification ratio corresponding to the magnification operation.

[0038] In one implementation, the method further includes: receiving output image data sent by the image sensor; wherein the output image data is image data output by the image sensor operating in pixel binning mode.

[0039] Fourthly, embodiments of this application provide an image processing apparatus, comprising:

[0040] A receiving module is used to receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified.

[0041] The determining module is used to determine the selected area corresponding to the magnification operation according to the first instruction;

[0042] The processing module is used to perform pixel rearrangement processing on the original image data of the selected area to obtain the output image data of the image sensor for the selected area;

[0043] The transmitting module is used to transmit the output image data of the selected area to the image signal processor.

[0044] In one implementation, the processing module includes:

[0045] A pixel rearrangement unit is used to perform pixel rearrangement processing on the original image data of the selected region to obtain intermediate image data of the selected region.

[0046] The resolution processing unit is configured to determine the magnification ratio corresponding to the magnification operation according to the first instruction, adjust the resolution of the intermediate image data according to the magnification ratio, and use the image data obtained after resolution adjustment as the output image data of the image sensor for the selected area.

[0047] In one possible implementation, the pixel rearrangement unit is specifically used to: convert the original image data of the selected region into Bayer format using a re-mosaic algorithm to obtain intermediate image data of the selected region.

[0048] In one implementation, the resolution processing unit is specifically used to: magnify the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the magnification factor.

[0049] In one possible implementation, the resolution processing unit is specifically used to: determine a corresponding resolution adjustment ratio based on the magnification factor; and magnify the resolution of the intermediate image data to a fixed resolution supported by the image signal processor based on the resolution adjustment ratio.

[0050] In one possible implementation, the resolution processing unit is specifically used to: adjust the resolution ratio according to the resolution and use a super-resolution image reconstruction algorithm to enlarge the resolution of the intermediate image data to a fixed resolution supported by the image signal processor.

[0051] In one implementation, the magnification is greater than or equal to a preset threshold.

[0052] In one possible implementation, the preset threshold is 2.

[0053] In one implementation, the receiving module is further configured to receive a second instruction, the second instruction being used to instruct the image sensor to operate in pixel binning mode; the processing module is further configured to perform binning processing on the acquired raw image data according to the second instruction to obtain the output image data of the image sensor; and the sending module is further configured to send the output image data of the image sensor to the image signal processor.

[0054] Fifthly, embodiments of this application provide another image processing apparatus, comprising:

[0055] The sending module is configured to, in response to receiving a zoom-in operation on a portion of the current screen of an electronic device, send a first instruction to an image sensor according to the zoom-in operation; the first instruction is configured to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the zoom-in operation.

[0056] In one implementation, the first instruction is further configured to instruct the image sensor to adjust the resolution of the intermediate image data obtained after pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor, according to the magnification ratio corresponding to the magnification operation.

[0057] In one possible implementation, the sending module is further configured to send a second instruction to the image sensor in response to not receiving a portion of the current screen of the electronic device, the second instruction being configured to instruct the image sensor to operate in pixel binning mode.

[0058] Sixthly, embodiments of this application provide yet another image processing apparatus, comprising:

[0059] A receiving module is used to receive first output image data of a selected area sent by an image sensor; wherein the selected area is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected area.

[0060] In one implementation, the receiving module is specifically used to: receive second output image data of a selected area sent by an image sensor; wherein the second output image data is image data obtained by adjusting the resolution of an intermediate image obtained after pixel rearrangement processing of the selected area by the image sensor according to the magnification ratio corresponding to the magnification operation to a fixed resolution supported by the image signal processor.

[0061] In one possible implementation, the receiving module is further configured to receive output image data sent by the image sensor; wherein the output image data is an image output by the image sensor operating in pixel binning mode.

[0062] In a seventh aspect, embodiments of this application provide an electronic device, including:

[0063] processor;

[0064] Memory for storing the executable instructions of the processor;

[0065] The processor is configured to invoke and execute executable instructions stored in the memory to implement the method described in the first aspect above.

[0066] Eighthly, embodiments of this application provide another electronic device, including:

[0067] processor;

[0068] Memory for storing the executable instructions of the processor;

[0069] The processor is configured to invoke and execute executable instructions stored in the memory to implement the method described in the second aspect above.

[0070] Ninthly, embodiments of this application provide yet another electronic device, comprising:

[0071] processor;

[0072] Memory for storing the executable instructions of the processor;

[0073] The processor is configured to invoke and execute executable instructions stored in the memory to implement the method described in the third aspect above.

[0074] In a tenth aspect, embodiments of this application provide a computer-readable storage medium for storing instructions that, when executed, cause the method described in the first aspect to be implemented.

[0075] Eleventhly, embodiments of this application provide a computer-readable storage medium for storing instructions that, when executed, cause the method described in the second aspect above to be implemented.

[0076] In a twelfth aspect, embodiments of this application provide yet another computer-readable storage medium for storing instructions that, when executed, cause the method described in the third aspect above to be implemented. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0078] Figure 1A This is a schematic diagram illustrating the principle of Binning;

[0079] Figure 1B This is a schematic diagram illustrating the principle of remosaic.

[0080] Figure 2This is a flowchart of an image processing method provided in an embodiment of this application;

[0081] Figure 3 This is a flowchart of another image processing method provided in an embodiment of this application;

[0082] Figure 4 This is an example diagram of image processing methods in the prior art;

[0083] Figure 5 This is an example diagram of an image processing method according to an embodiment of this application;

[0084] Figure 6 This is a flowchart of yet another image processing method provided in the embodiments of this application;

[0085] Figure 7 This is a flowchart of yet another image processing method provided in an embodiment of this application;

[0086] Figure 8 A schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0087] Figure 9 A schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application;

[0088] Figure 10 A schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application;

[0089] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0090] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. In the description of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0091] It's worth noting that zooming, especially optical zoom, is a very common operation in mobile device (such as smartphone) cameras. For example, when it's necessary to magnify a certain area of ​​the preview image, the image sensor and ISP system need to work together, utilizing image sensor and ISP technology to magnify the area.

[0092] Most existing image sensors support a 2x2 pattern. During normal shooting, image sensors typically operate in pixel binning mode, meaning four pixels are combined and output together. When zooming or higher resolution is required, image sensors operate in remosaic mode, allowing for higher resolution output.

[0093] In related technologies, when zooming is required to enlarge a certain area in the preview image, a certain requirement is placed on the output resolution of the image sensor in order to prevent the ISP platform from interrupting the flow and reconfiguring. Even if the user only zooms in on a small area, the image sensor outputs a fixed resolution and area. Then, the ISP platform receives the data and performs several cropping and enlargement processes. However, this often leads to high power consumption of the image sensor.

[0094] To address the aforementioned problems, embodiments of this application provide an image processing method and apparatus that can reduce the power consumption of image sensors. The image processing method and apparatus of this application are described below with reference to the accompanying drawings.

[0095] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0096] 1) Binning: The pointer performs a binning operation on each pixel, merging four adjacent pixels in the initial image into one pixel block to generate a result image with a resolution of 12MP.

[0097] For example, Figure 1A This is a schematic diagram illustrating the principle of Binning. Squares filled with left diagonal lines represent pixels G, squares filled with right diagonal lines represent pixels R, and squares filled with intersecting diagonal lines represent pixels B. Figure 1A As shown, 4 G pixels are combined into one G pixel, and the same applies to R and B. The resolution is reduced to 1 / 4 of the original. The purpose of doing this is to reduce the output of the 48M image sensor to 12M, making subsequent algorithm processing more time- and memory-friendly.

[0098] 2) Pixel remosaic: The initial image is converted into Bayer format using a remosaic algorithm to obtain a result image with a resolution of 48M.

[0099] For example, Figure 1BThis is a schematic diagram illustrating the principle of remosaic. Squares filled with diagonal lines on the left represent green pixels (Gree), squares filled with diagonal lines on the right represent red pixels (Red), and squares filled with intersecting diagonal lines represent blue pixels (Blue). Figure 1B As shown, the original four identical G / R / B pixels are rearranged, and interpolation is performed at the corresponding positions.

[0100] 3) Super-resolution: Super-resolution reconstruction is the process of improving the resolution of an original image by using hardware or software methods. It involves obtaining a high-resolution image from a series of low-resolution images.

[0101] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application. It should be noted that the image processing method of this embodiment can be executed by an image sensor. That is, the image processing method of this embodiment can be described from the perspective of the image sensor. Figure 2 As shown, the image processing method may include, but is not limited to, the following steps.

[0102] Step 201: Receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified.

[0103] In one implementation, the first instruction can be a command sent by the processor in the camera software system to the image sensor after receiving a zoom-in operation for a portion of the current frame of the electronic device. As an example, the current frame of the electronic device can be a preview of the captured image.

[0104] Optionally, the above-mentioned magnification operation can be based on various gestures and other types of operations to magnify the image. For example, a user can magnify the preview image in the camera. When the camera software system detects that the user has magnified the preview image, it sends a first instruction to the image sensor to instruct the image sensor to perform pixel rearrangement processing on the magnified area selected by the user (i.e., the selected area corresponding to the magnification operation). This means that the image sensor is instructed to perform pixel rearrangement only on the area selected by the user.

[0105] Step 202: Determine the selected area corresponding to the zoom-in operation according to the first instruction.

[0106] As an example, the first instruction may include information about the selected area corresponding to the zoom-in operation. For instance, after receiving the first instruction, the image sensor may determine the selected area corresponding to the zoom-in operation based on the information contained in the first instruction, wherein the selected area can be understood as the area selected by the user when performing a zoom-in operation on the preview image.

[0107] Step 203: Perform pixel rearrangement processing on the original image data of the selected area to obtain the output image data of the image sensor for the selected area.

[0108] Optionally, after determining the selected area corresponding to the magnification operation, the image sensor can acquire the original image data of the selected area and convert the original image data of the selected area into Bayer format through a re-mosaic algorithm to realize pixel rearrangement of the original image data of the selected area, thereby obtaining the output image data of the image sensor for the selected area.

[0109] Step 204: Send the output image data of the selected area to the image signal processor.

[0110] Optionally, the image sensor may perform pixel rearrangement processing only on the original image data of the selected area, and may output only the output image data of the selected area to the image signal processor, so that the image signal processor can perform subsequent processing on the output image data of the selected area.

[0111] Therefore, when a selected area needs to be magnified, the image sensor of this application supports pixel rearrangement of any area selected by the user, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only to the selected area, which can reduce the power consumption of the image sensor.

[0112] To ensure the ISP platform doesn't need to stop streaming and reconfigure, the image sensor can adjust the resolution of the intermediate image data obtained after pixel rearrangement of the selected area to a fixed resolution supported by the image signal processor. This allows the image sensor to internally adjust the resolution of the output image data to this fixed resolution, ensuring the resolution remains constant for the ISP platform and preventing any need for streaming interruption and reconfiguration. Optionally, such as... Figure 3 As shown, Figure 3 This is a flowchart of another image processing method provided in an embodiment of this application. It should be noted that the image processing method of this application embodiment can be executed by an image sensor. That is, the image processing method of this application embodiment can be described from the perspective of the image sensor. Figure 3 As shown, the image processing method may include, but is not limited to, the following steps.

[0113] Step 301: Receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified.

[0114] In one implementation, the first instruction can be a command sent by the processor in the camera software system to the image sensor after receiving a zoom-in operation on a portion of the current screen of the electronic device. As an example, the portion of the current screen of the electronic device can be a preview image of the photograph. For instance, a user can zoom in on the preview image in the camera. When the camera software system detects this zoom-in operation, it sends a first instruction to the image sensor, instructing the image sensor to perform pixel rearrangement processing on the zoomed-in area (i.e., the selected area corresponding to the zoom-in operation). This means instructing the image sensor to perform pixel rearrangement only on the area selected by the user.

[0115] Step 302: Determine the selected area corresponding to the zoom-in operation according to the first instruction.

[0116] As an example, the first instruction may include information about the selected area corresponding to the zoom-in operation. For instance, after receiving the first instruction, the image sensor may determine the selected area corresponding to the zoom-in operation based on the information contained in the first instruction, wherein the selected area can be understood as the area selected by the user when performing a zoom-in operation on the preview image.

[0117] Step 303: Perform pixel rearrangement processing on the original image data of the selected area to obtain the intermediate image data of the selected area.

[0118] Optionally, after determining the selected area corresponding to the magnification operation, the image sensor can acquire the original image data of the selected area and convert the original image data of the selected area into Bayer format through a re-mosaic algorithm to obtain the intermediate image data of the selected area.

[0119] Step 304: Determine the magnification factor corresponding to the magnification operation according to the first instruction.

[0120] In one implementation, the first instruction may further include information about the magnification factor corresponding to the magnification operation. This magnification factor may be greater than or equal to a preset threshold; for example, the preset threshold may be 2. It should be noted that the above example is merely illustrative and should not be construed as limiting the preset threshold, etc., in the embodiments of this application.

[0121] For example, when a user zooms in on the preview image in the camera, the camera software system, upon detecting this zoom operation and determining that the zoom level is greater than or equal to 2, can send a first instruction to the image sensor. This first instruction can include information about the selected area and the zoom level corresponding to the zoom operation. This allows the image sensor to perform pixel rearrangement and resolution adjustment on the selected area based on the first instruction, enabling the image sensor to support pixel rearrangement of any user-selected area and output only to the user-selected area.

[0122] Step 305: Adjust the resolution of the intermediate image data according to the magnification, and use the image data obtained after resolution adjustment as the output image data of the image sensor for the selected area.

[0123] Optionally, the resolution of the intermediate image data of the selected area is magnified to a fixed resolution supported by the image signal processor, depending on the magnification, to obtain the output image data of the image sensor for the selected area.

[0124] In one implementation, the image sensor can magnify the resolution of intermediate image data in a selected area to a fixed resolution supported by the image signal processor, depending on the magnification factor. In this way, the image resolution remains unchanged for the ISP platform, thus ensuring that the ISP platform does not need to stop streaming and reconfigure.

[0125] It should be noted that different magnification levels correspond to different degrees of resolution adjustment. In one possible implementation, the corresponding resolution adjustment ratio can be determined based on the magnification level, and then the resolution of the intermediate image data of the selected area can be magnified to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio.

[0126] It should also be noted that there are many ways to upscale the intermediate image data of a selected region to a fixed resolution supported by the image signal processor. For example, a super-resolution image reconstruction algorithm can be used. In one implementation, the resolution can be adjusted proportionally, and a super-resolution image reconstruction algorithm can be used to upscale the intermediate image data of the selected region to a fixed resolution supported by the image signal processor. This super-resolution image reconstruction algorithm can be an existing technology, such as an interpolation-based super-resolution image reconstruction algorithm, a degradation model-based super-resolution image reconstruction algorithm, or a learning-based super-resolution image reconstruction algorithm, which will not be elaborated further here. This approach ensures that the image resolution is upscaled to a fixed resolution while maintaining image clarity.

[0127] Step 306: Send the output image data of the selected area to the image signal processor.

[0128] Optionally, the image sensor may perform pixel rearrangement processing only on the original image data of the selected area, and may output only the output image of the selected area to the image signal processor so that the image signal processor can perform subsequent processing on the output image of the selected area.

[0129] In some embodiments of this application, a second instruction may be received, which instructs the image sensor to operate in pixel binning mode; according to the second instruction, the acquired raw image data is subjected to binning processing to obtain the output image data of the image sensor; and the output image data of the image sensor is sent to the image signal processor.

[0130] For example, a user can choose not to zoom in on the preview image in the camera; they can simply click the shutter button to take a picture without any further action on the preview. When the camera software detects that the user hasn't zoomed in on the preview, it can send a second command to the image sensor, instructing it to operate in pixel binning mode. Upon receiving this second command, the image sensor can perform binning processing on the acquired raw image data, obtaining the image sensor's output image data. In other words, the image sensor will output an image at the binning size and send the output image data to the ISP for further processing.

[0131] To help those skilled in the art understand the advantages of this application, the Qualcomm platform will be used as an example below.

[0132] For example, taking the Qualcomm platform as an example, Qualcomm platforms currently do not support changes in the resolution of the image sensor output without continuous reconfiguration. Therefore, during magnification, if the magnification is equal to or greater than 2x, the image sensor will switch from Binning mode to full-resolution remosaic mode. However, even if the magnification is further increased and the FOV (Field of View) shrinks, the image sensor can only perform pixel remosaic on the central 1 / 4 area. Figure 4 As shown, after rearranging the pixels in the central 1 / 4 region, the image sensor still outputs a full-resolution image. After the ISP receives the image sensor data, the platform then crops it to the required area 40. It can be seen that the power consumption of the image sensor is very high in this way.

[0133] The image processing scheme of this application embodiment is as follows: When the camera software system detects that the user is zooming in on the preview image, it sends a first instruction to the image sensor. The image sensor can determine the selected area corresponding to the zoom operation according to the first instruction, perform pixel rearrangement processing on the original image data of the selected area to obtain an intermediate image of the selected area, and determine the magnification ratio corresponding to the zoom operation according to the first instruction. Based on the magnification ratio, the resolution of the intermediate image of the selected area is magnified to a fixed resolution supported by the image signal processor, thus obtaining the output image data of the image sensor for the selected area. Figure 5 As shown, the image sensor outputs the selected area's output image 50 to the image signal processor (ISP), allowing the ISP to directly perform subsequent processing on this selected area's output image 50. Therefore, the image sensor in this embodiment can support pixel remosaic for a selected area, rather than a fixed area. This allows the image sensor to output only the selected area as the magnification increases, reducing power consumption. Furthermore, the image sensor can internally reconstruct the selected area's image resolution to a fixed resolution. This ensures that the resolution remains unchanged for the receiving platform, eliminating the need for platform downtime and reconfiguration.

[0134] It is understood that the above embodiments describe the implementation of the image processing method of this application from the image sensor side. This application also proposes an image processing method, the implementation of which will be described below from the processor side. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating another image processing method provided in an embodiment of this application. It should be noted that the image processing method of this application embodiment can be applied to a processor, wherein the processor can be a processor in a camera software system. Figure 6 As shown, the image processing method may include, but is not limited to, the following steps.

[0135] Step 601: In response to receiving a zoom-in operation on a portion of the current screen of the electronic device, a first instruction is sent to the image sensor according to the zoom-in operation; the first instruction is used to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the zoom-in operation.

[0136] For example, a user can zoom in on the preview image in the camera. When the processor of the camera software system detects that the user has zoomed in on the preview image, it sends a first instruction to the image sensor to instruct the image sensor to perform pixel rearrangement processing on the zoomed-in area (i.e. the selected area corresponding to the zoom-in operation). In other words, the image sensor is instructed to perform pixel rearrangement only on the area selected by the user.

[0137] In this embodiment, after receiving the first instruction, the image sensor determines the selected region corresponding to the magnification operation according to the first instruction, and converts the original image data of the selected region into Bayer format through a re-mosaic algorithm to achieve pixel rearrangement of the original image data of the selected region, thereby obtaining the output image data of the image sensor for the selected region. It can be seen that the image sensor can perform pixel rearrangement processing only on the original image data of the selected region, and can output only the output image data of the selected region to the image signal processor, so that the image signal processor can perform subsequent processing on the output image data of the selected region.

[0138] By implementing the embodiments of this application, when it is necessary to magnify a selected area, the processor in the camera software system can send instruction information to the image sensor to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the magnification operation. This enables the image sensor to support pixel rearrangement of the selected area, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only to the selected area, thereby reducing the power consumption of the image sensor.

[0139] To ensure that the ISP platform does not need to be interrupted and reconfigured, in one implementation, the first instruction can also be used to instruct the image sensor to adjust the resolution of the intermediate image data obtained after pixel rearrangement of the selected area to a fixed resolution supported by the image signal processor, according to the magnification ratio corresponding to the magnification operation.

[0140] For example, when a user zooms in on the preview image in the camera, the camera software system, upon detecting this zoom-in and determining that the zoom level is greater than or equal to 2, can send a first instruction to the image sensor. This first instruction can include information about the selected area corresponding to the zoom-in operation and the zoom level. Upon receiving the first instruction, the image sensor determines the selected area corresponding to the zoom-in operation and converts the original image data of that selected area to Bayer format using a re-mosaic algorithm. This re-arranges the pixels of the original image data of the selected area to obtain intermediate image data for that area. The image sensor can also determine the zoom level corresponding to the zoom-in operation based on the first instruction, determine the corresponding resolution adjustment ratio based on the zoom level, and then use a super-resolution image reconstruction algorithm to upscale the resolution of the intermediate image data of the selected area to a fixed resolution supported by the image signal processor, thereby obtaining the output image data of the selected area from the image sensor.

[0141] Therefore, it can be seen that the resolution of the intermediate image data obtained by the image sensor after pixel rearrangement of the selected area can be increased to the fixed resolution supported by the image signal processor. Thus, the resolution of the output image data can be adjusted to this fixed resolution inside the image sensor. In this way, the resolution remains unchanged for the ISP platform to receive the data, which can ensure that the ISP platform does not need to stop the flow and reconfigure.

[0142] In some embodiments of this application, in response to the absence of a magnification operation for a portion of the current image, a second instruction is sent to the image sensor. This second instruction instructs the image sensor to operate in pixel binning mode. For example, a user can take a picture of the preview image without magnifying it; for instance, they can directly click the shutter button without any operation on the preview. When the processor of the camera software system detects that the user has not magnified the preview image, it can send the second instruction to the image sensor to instruct it to operate in pixel binning mode. Upon receiving this second instruction, the image sensor can perform binning processing on the acquired raw image data to obtain the image sensor's output image data. That is, the image sensor outputs an image at the binning size and sends the output image data to the ISP for further processing.

[0143] It is understood that the above embodiments describe the image processing method of the embodiments of this application from the perspectives of the image sensor and the processor. This application also proposes another image processing method, the implementation of which will be described below from the perspective of the image signal processor (ISP). Please refer to... Figure 7 , Figure 7 This is a flowchart illustrating another image processing method provided in an embodiment of this application. It should be noted that the image processing method of this application embodiment can be applied to an image signal processor (ISP). Figure 7 As shown, the image processing method may include, but is not limited to, the following steps.

[0144] Step 701: Receive first output image data of a selected area sent by the image sensor; wherein, the selected area is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected area.

[0145] As an example, the current screen of the electronic device could be a preview image from a camera. For instance, a user could zoom in on the preview image. When the processor of the camera software system detects this zoom-in, it sends a first instruction to the image sensor, instructing the image sensor to perform pixel rearrangement on the zoomed-in area (i.e., the selected area corresponding to the zoom-in operation). This means the image sensor can be instructed to perform pixel rearrangement only on the area selected by the user.

[0146] In this embodiment, after receiving the first instruction, the image sensor determines the selected region corresponding to the magnification operation according to the first instruction, and converts the original image data of the selected region into Bayer format using a re-mosaic algorithm to achieve pixel rearrangement of the original image data of the selected region, thereby obtaining the output image data of the image sensor for the selected region. The image sensor can send the output image data of the selected region to the image signal processor (ISP), so that the ISP can receive the first output image data of the selected region sent by the image sensor. Therefore, the image sensor can perform pixel rearrangement processing only on the original image data of the selected region, and can output only the output image data of the selected region to the image signal processor, so that the image signal processor can perform subsequent processing on the output image data of the selected region.

[0147] In one implementation, a second output image data of a selected area sent by an image sensor is received; wherein the second output image data is obtained by adjusting the resolution of an intermediate image obtained by pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor according to the magnification ratio corresponding to the magnification operation.

[0148] For example, when a user zooms in on the preview image in the camera, the camera software system, upon detecting this zoom-in and determining that the zoom level is greater than or equal to 2, can send a first instruction to the image sensor. This first instruction can include information about the selected region corresponding to the zoom-in operation and the zoom level. Upon receiving the first instruction, the image sensor determines the selected region corresponding to the zoom-in operation and converts the original image data of that region to Bayer format using a re-mosaic algorithm. This re-arranges the pixels of the original image data of the selected region to obtain intermediate image data for that region. The image sensor can also determine the zoom level corresponding to the zoom-in operation based on the first instruction, determine the corresponding resolution adjustment ratio based on the zoom level, and use a super-resolution image reconstruction algorithm to upscale the resolution of the intermediate image data of the selected region to a fixed resolution supported by the image signal processor, thus obtaining the image sensor's output image data for the selected region. In this way, the image sensor can output the selected region's image data, allowing the image signal processor (ISP) to receive the second output image data of the selected region sent by the image sensor.

[0149] Therefore, it can be seen that the resolution of the intermediate image data obtained by the image sensor after pixel rearrangement of the selected area can be increased to the fixed resolution supported by the image signal processor. Thus, the resolution of the output image data can be adjusted to this fixed resolution inside the image sensor. In this way, the resolution remains unchanged for the ISP platform to receive the data, which can ensure that the ISP platform does not need to stop the flow and reconfigure.

[0150] In some embodiments of this application, output image data sent by an image sensor is received; wherein the output image is the image output by the image sensor operating in pixel binning mode. For example, a user can take a picture directly by clicking the shutter button without zooming in on the preview image. When the processor of the camera software system detects that the user has not zoomed in on the preview image, it can send a second instruction to the image sensor to instruct the image sensor to operate in pixel binning mode. Upon receiving the second instruction, the image sensor can perform binning processing on the acquired raw image data to obtain the output image data of the image sensor, that is, the image sensor will output an image at the binning size, so that the image signal processor can obtain the output image data sent by the image sensor and then perform subsequent processing on the output image data.

[0151] The methods provided in the embodiments of this application above are described from the perspective of image sensors, processors, and image signal processors. To implement the functions of the methods provided in the embodiments of this application above, the sensors, processors, and image signal processors may include hardware structures and software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. One of the above functions can be executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules.

[0152] Please see Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 8 As shown, the image processing device may include: a receiving module 810, a determining module 820, a processing module 830, and a sending module 840.

[0153] The receiving module 810 is used to receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified.

[0154] The determination module 820 is used to determine the selected area corresponding to the zoom operation according to the first instruction.

[0155] The processing module 830 is used to perform pixel rearrangement processing on the original image data of the selected area to obtain the output image data of the image sensor for the selected area.

[0156] The transmitting module 840 is used to transmit the output image data of the selected area to the image signal processor.

[0157] To ensure that the ISP platform does not require network interruption and reconfiguration, in one implementation, the processing module may include a pixel rearrangement unit and a resolution processing unit. The pixel rearrangement unit performs pixel rearrangement processing on the original image data of the selected area to obtain intermediate image data for the selected area. The resolution processing unit determines the magnification ratio corresponding to the magnification operation according to a first instruction, adjusts the resolution of the intermediate image data according to the magnification ratio, and uses the resolution-adjusted image data as the output image data of the image sensor for the selected area.

[0158] In one possible implementation, the pixel rearrangement unit converts the original image data of the selected region into Bayer format using a re-mosaic algorithm to obtain intermediate image data of the selected region.

[0159] In one implementation, the resolution processing unit is specifically used to magnify the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the magnification factor. The magnification factor is greater than or equal to a preset threshold. As an example, the preset threshold can be 2.

[0160] In one possible implementation, the resolution processing unit is specifically used to: determine the corresponding resolution adjustment ratio based on the magnification; and, based on the resolution adjustment ratio, magnify the resolution of the intermediate image data of the selected area to a fixed resolution supported by the image signal processor.

[0161] Optionally, the resolution processing unit can adjust the ratio according to the resolution and use a super-resolution image reconstruction algorithm to enlarge the resolution of the intermediate image data of the selected area to a fixed resolution supported by the image signal processor.

[0162] In some embodiments of this application, the receiving module may also be used to receive a second instruction, the second instruction being used to instruct the image sensor to operate in pixel binning mode; the processing module may also be used to perform binning processing on the acquired raw image data according to the second instruction to obtain the output image data of the image sensor; and the sending module may also be used to send the output image data of the image sensor to the image signal processor.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] Therefore, when a selected area needs to be magnified, the image sensor of this application supports pixel rearrangement of any area selected by the user, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only to the selected area, which can reduce the power consumption of the image sensor.

[0165] Please see Figure 9 This is a schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application. Figure 9 As shown, the image processing device may include a transmitting module 910.

[0166] The sending module 910 is configured to, in response to receiving a zoom-in operation on a portion of the current screen of the electronic device, send a first instruction to the image sensor according to the zoom-in operation; the first instruction is configured to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the zoom-in operation.

[0167] In one implementation, the first instruction is further configured to instruct the image sensor to adjust the resolution of the intermediate image data obtained after pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor, according to the magnification ratio corresponding to the magnification operation.

[0168] In some embodiments of this application, the sending module 910 may also be used to send a second instruction to the image sensor in response to not receiving a zoom operation for a portion of the current screen of the electronic device, the second instruction being used to instruct the image sensor to operate in pixel binning mode.

[0169] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0170] By implementing the embodiments of this application, when it is necessary to magnify a selected area, the processor in the camera software system can send instruction information to the image sensor to instruct the image sensor to perform pixel rearrangement processing on the selected area corresponding to the magnification operation. This enables the image sensor to support pixel rearrangement of the selected area, rather than pixel rearrangement of a fixed area, and enables the image sensor to output only to the selected area, thereby reducing the power consumption of the image sensor.

[0171] Please see Figure 10 This is a schematic diagram of the structure of another image processing apparatus provided in an embodiment of this application. Figure 10 As shown, the image processing device may include a receiving module 1010.

[0172] The receiving module 1010 is used to receive first output image data of a selected area sent by the image sensor; wherein the selected area is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected area.

[0173] In one implementation, the receiving module is specifically used to: receive second output image data of a selected area sent by the image sensor; wherein, the second output image data is an image obtained by adjusting the resolution of the intermediate image obtained after pixel rearrangement processing of the selected area by the image sensor according to the magnification ratio corresponding to the magnification operation to a fixed resolution supported by the image signal processor.

[0174] In some embodiments of this application, the receiving module can also be used to receive output image data sent by the image sensor; wherein the output image data is image data output by the image sensor when it is operating in pixel binning mode.

[0175] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0176] Figure 11 This is a block diagram illustrating an electronic device 1100 according to an exemplary embodiment. For example, the electronic device 1100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0177] Reference Figure 11 The electronic device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power supply component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0178] Processing component 1102 typically controls the overall operation of electronic device 1100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0179] Memory 1104 is configured to store various types of data to support the operation of electronic device 1100. Examples of such data include instructions for any application or method operating on electronic device 1100, contact data, phonebook data, messages, pictures, videos, etc. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0180] Power supply component 1106 provides power to various components of electronic device 1100. Power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1100.

[0181] Multimedia component 1108 includes a touch display screen that provides an output interface between the electronic device 1100 and the user. In some embodiments, the touch display screen may include a liquid crystal display (LCD) and a touch panel (TP). The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0182] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when electronic device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0183] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0184] Sensor assembly 1114 includes one or more sensors for providing state assessments of various aspects of electronic device 1100. For example, sensor assembly 1114 may detect the on / off state of electronic device 1100, the relative positioning of components such as the display and keypad of electronic device 1100, changes in position of electronic device 1100 or a component of electronic device 1100, the presence or absence of user contact with electronic device 1100, the orientation or acceleration / deceleration of electronic device 1100, and temperature changes of electronic device 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0185] Communication component 1116 is configured to facilitate wired or wireless communication between electronic device 1100 and other devices. Electronic device 1100 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1116 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0186] In an exemplary embodiment, the electronic device 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0187] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, which can be executed by a processor 1120 of an electronic device 1100 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0188] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device 1100, enable the electronic device 1100 to perform an image processing method.

[0189] A computer program product, when the instructions in the computer program product are executed by the processor of an electronic device 1100, enables the electronic device 1100 to perform the functions of any of the above method embodiments.

[0190] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0191] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0192] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., involved in this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application, nor do they indicate the order of sequence.

[0193] At least one in this application can also be described as one or more, and multiple can be two, three, four or more, and this application does not impose any limitation. In the embodiments of this application, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", and there is no order or size among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0194] The correspondences shown in the tables of this application can be configured or predefined. The values ​​of the information in each table are merely examples and can be configured to other values; this application is not limited to these values. When configuring the correspondences between information and parameters, it is not necessarily required to configure all the correspondences shown in each table. For example, the correspondences shown in some rows of the tables in this application may not be configured. Furthermore, appropriate modifications and adjustments can be made based on the above tables, such as splitting, merging, etc. The names of the parameters shown in the headings of the above tables can also use other names that the communication device can understand, and the values ​​or representations of the parameters can also be other values ​​or representations that the communication device can understand. In the implementation of the above tables, other data structures can also be used, such as arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables, or hash tables, etc.

[0195] The term "predefined" in this application can be understood as definition, pre-defined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.

[0196] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is performed by an image sensor, and the method includes: Receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified. Based on the first instruction, the selected area corresponding to the zoom-in operation is determined; The original image data of the selected region is subjected to pixel rearrangement processing to obtain the output image data of the image sensor for the selected region, including: The original image data of the selected region is converted into Bayer format using a re-mosaic algorithm to obtain the intermediate image data of the selected region; The output image data of the selected area is sent to the image signal processor; The step of performing pixel rearrangement processing on the original image data of the selected region to obtain the output image data of the image sensor for the selected region includes: Based on the first instruction, determine the magnification factor corresponding to the magnification operation; The intermediate image data is adjusted for resolution according to the magnification, and the image data obtained after resolution adjustment is used as the output image data of the image sensor for the selected area. The step of adjusting the resolution of the intermediate image data according to the magnification factor includes: The corresponding resolution adjustment ratio is determined based on the magnification ratio, and the resolution of the intermediate image data is magnified to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio.

2. The method according to claim 1, characterized in that, The step of scaling up the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio includes: Based on the resolution adjustment ratio, a super-resolution image reconstruction algorithm is used to enlarge the resolution of the intermediate image data to a fixed resolution supported by the image signal processor.

3. The method according to any one of claims 1 to 2, characterized in that, The magnification is greater than or equal to a preset threshold.

4. The method according to claim 3, characterized in that, The preset threshold is 2.

5. The method according to claim 1, characterized in that, Also includes: Receive a second instruction, which instructs the image sensor to operate in pixel binning mode; According to the second instruction, the acquired raw image data is subjected to Binning processing to obtain the output image data of the image sensor; The output image data of the image sensor is sent to the image signal processor.

6. An image processing method, characterized in that, The method is executed by a processor, and the method includes: In response to receiving a zoom-in operation on a portion of the current screen of an electronic device, a first instruction is sent to an image sensor according to the zoom-in operation. The first instruction instructs the image sensor to perform pixel rearrangement processing on a selected area corresponding to the zoom-in operation, including: performing pixel rearrangement processing on the original image data of the selected area to obtain intermediate image data of the selected area, wherein the processing includes: converting the original image data of the selected area to Bayer format using a re-mosaic algorithm to obtain the intermediate image data of the selected area; determining a magnification ratio corresponding to the zoom-in operation according to the first instruction; adjusting the resolution of the intermediate image data according to the magnification ratio; and using the resolution-adjusted image data as the output image data of the image sensor for the selected area. The first instruction is further configured to instruct the image sensor to determine a corresponding resolution adjustment ratio based on the magnification ratio corresponding to the magnification operation, and to adjust the resolution of the intermediate image data to a fixed resolution supported by the image signal processor based on the resolution adjustment ratio.

7. The method according to claim 6, characterized in that, Also includes: In response to not receiving a zoom-in operation for a portion of the current image, a second instruction is sent to the image sensor, the second instruction being used to instruct the image sensor to operate in pixel binning mode.

8. An image processing method, characterized in that, The method is executed by an image signal processor, and the method includes: The system receives first output image data of a selected region sent by an image sensor; wherein the selected region is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected region, including: converting the original image data of the selected region into Bayer format through a re-mosaic algorithm to obtain intermediate image data of the selected region. The first output image data of the selected area sent by the receiving image sensor includes: The image sensor receives a second output image data of a selected area; wherein the second output image data is the image data obtained by adjusting the resolution of an intermediate image obtained by the image sensor after determining a corresponding resolution adjustment ratio based on the magnification ratio corresponding to the magnification operation, and then adjusting the resolution of the selected area to a fixed resolution supported by the image signal processor.

9. The method according to claim 8, characterized in that, Also includes: Receive the output image data sent by the image sensor; wherein the output image data is the image data output by the image sensor when it is operating in pixel binning mode.

10. An image processing apparatus, characterized in that, include: A receiving module is used to receive a first instruction, which indicates that a portion of the current screen of the electronic device has been magnified. The determining module is used to determine the selected area corresponding to the magnification operation according to the first instruction; The processing module is used to perform pixel rearrangement processing on the original image data of the selected area to obtain the output image data of the image sensor for the selected area; The sending module is used to send the output image data of the selected area to the image signal processor; The processing module includes: The pixel rearrangement unit converts the original image data of the selected region into Bayer format using a re-mosaic algorithm to obtain the intermediate image data of the selected region. A resolution processing unit is configured to determine the magnification ratio corresponding to the magnification operation according to the first instruction, and to adjust the resolution of the intermediate image data according to the magnification ratio, and to use the image data obtained after resolution adjustment as the output image data of the image sensor for the selected area. The resolution processing unit is specifically used for: Based on the magnification factor, a corresponding resolution adjustment ratio is determined, and based on the resolution adjustment ratio, the resolution of the intermediate image data is magnified to a fixed resolution supported by the image signal processor.

11. The apparatus according to claim 10, characterized in that, The resolution processing unit is specifically used for: Based on the resolution adjustment ratio, a super-resolution image reconstruction algorithm is used to enlarge the resolution of the intermediate image data to a fixed resolution supported by the image signal processor.

12. The apparatus according to any one of claims 10 to 11, characterized in that, The magnification is greater than or equal to a preset threshold.

13. The apparatus according to claim 12, characterized in that, The preset threshold is 2.

14. The apparatus according to claim 10, characterized in that, The receiving module is also configured to receive a second instruction, which is used to instruct the image sensor to operate in pixel binning mode. The processing module is further configured to perform Binning processing on the acquired raw image data according to the second instruction to obtain the output image data of the image sensor; The transmitting module is also used to transmit the output image data of the image sensor to the image signal processor.

15. An image processing apparatus, characterized in that, include: The transmitting module is configured to, in response to receiving a zoom-in operation on a portion of the current screen of the electronic device, send a first instruction to the image sensor according to the zoom-in operation; The first instruction is used to instruct the image sensor to perform pixel rearrangement processing on a selected area corresponding to the magnification operation, including: performing pixel rearrangement processing on the original image data of the selected area to obtain intermediate image data of the selected area, including: converting the original image data of the selected area to Bayer format through a re-mosaic algorithm to obtain intermediate image data of the selected area; determining the magnification ratio corresponding to the magnification operation according to the first instruction; adjusting the resolution of the intermediate image data according to the magnification ratio, and using the image data obtained after resolution adjustment as the output image data of the image sensor for the selected area; the first instruction is also used to instruct the image sensor to determine a corresponding resolution adjustment ratio according to the magnification ratio corresponding to the magnification operation, and adjust the resolution of the intermediate image data to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio.

16. The apparatus according to claim 15, characterized in that, The sending module is further configured to send a second instruction to the image sensor in response to not receiving a zoom operation for a portion of the current screen of the electronic device, the second instruction being used to instruct the image sensor to operate in pixel binning mode.

17. An image processing apparatus, characterized in that, include: A receiving module is configured to receive first output image data of a selected area sent by an image sensor; wherein the selected area is a region determined by the image sensor based on a magnification operation of a portion of the current screen of the electronic device, and the first output image data is image data obtained by the image sensor after pixel rearrangement processing of the original image data of the selected area, including: converting the original image data of the selected area into Bayer format through a re-mosaic algorithm to obtain intermediate image data of the selected area; The receiving module is specifically used for: The image sensor receives a second output image data of a selected area; wherein the second output image data is obtained by the image sensor determining a corresponding resolution adjustment ratio based on the magnification ratio corresponding to the magnification operation, and adjusting the resolution of the intermediate image obtained after pixel rearrangement processing of the selected area to a fixed resolution supported by the image signal processor according to the resolution adjustment ratio.

18. The apparatus according to claim 17, characterized in that, The receiving module is further configured to receive output image data sent by the image sensor; wherein the output image data is image data output by the image sensor when it is operating in pixel binning mode.

19. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to invoke and execute executable instructions stored in the memory to implement the image processing method as described in any one of claims 1 to 5.

20. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to invoke and execute executable instructions stored in the memory to implement the image processing method as described in any one of claims 6 to 7.

21. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to invoke and execute executable instructions stored in the memory to implement the image processing method as described in any one of claims 8 to 9.

22. A computer-readable storage medium for storing instructions that, when executed, cause the method of any one of claims 1 to 5 to be implemented.

23. A computer-readable storage medium for storing instructions that, when executed, cause the method of any one of claims 6 to 7 to be implemented.

24. A computer-readable storage medium for storing instructions that, when executed, cause the method of any one of claims 8 to 9 to be implemented.

Images