Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The application relates to an image processing method and device, electronic equipment and a computer readable storage medium. The image processing method comprises the following steps: based on a condition that a focusing operation acting on the electronic equipment is received, controlling n image sensors to output data streams, wherein n is greater than or equal to 2 and n is an integer; and performing spatial alignment processing on the data streams output by the n image sensors. By monitoring the focusing operation of the user in real time, the n lens output data streams required can be pulled up in time, and spatial alignment processing is performed, so that the image processing speed and the lens switching speed are improved, and the use experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] With technological advancements, electronic devices have become increasingly powerful in image processing, and multi-lens cameras have become standard equipment. Furthermore, as the number of lenses and sensor resolution continue to rise, the amount of image data requiring processing during shooting is also increasing. Therefore, current image processing speeds are no longer sufficient to meet user needs. Summary of the Invention

[0003] Therefore, it is necessary to provide an image processing method and apparatus, electronic device and computer-readable storage medium that can improve processing speed in response to the above-mentioned technical problems.

[0004] In a first aspect, this application provides an image processing method, the method comprising:

[0005] Based on the received focusing operation applied to the electronic device, control n image sensors to output data streams, where n is greater than or equal to 2 and is an integer;

[0006] Spatial alignment processing is performed on the data streams output by the n image sensors.

[0007] Secondly, this application provides an electronic device including m image sensors, a memory, and a processor. Each of the image sensors is configured to operate in a different magnification range. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0008] Thirdly, this application provides an image processing apparatus, the apparatus comprising:

[0009] The operation receiving module is used to control the output data streams of n image sensors, where n is greater than or equal to 2 and is an integer, based on the conditions received from the focusing operation performed on the electronic device.

[0010] The alignment module is used to perform spatial alignment processing on the data streams output by the n image sensors.

[0011] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0012] The aforementioned image processing method, apparatus, electronic device, and computer-readable storage medium can promptly activate the required n lens output data streams and perform spatial alignment processing by real-time monitoring of the user's focusing operation. This reduces the start-up time when an actual switching occurs and shortens the processing time after determining that a lens switching has occurred through pre-processed spatial alignment. In other words, it improves the image processing speed and the speed of lens switching, thereby enhancing the user experience. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the magnification range of one embodiment;

[0014] Figure 2 One of the flowcharts for an image processing method according to an embodiment;

[0015] Figure 3 This is a schematic diagram of the focus control in its unfolded state according to one embodiment;

[0016] Figure 4 This is a schematic diagram of a focus control in a folded state according to one embodiment;

[0017] Figure 5 This is a second flowchart of an image processing method according to one embodiment;

[0018] Figure 6 This is the third flowchart of an image processing method according to one embodiment;

[0019] Figure 7 The fourth flowchart is an embodiment of an image processing method;

[0020] Figure 8 This is a schematic diagram of a transition frame when switching between an ultra-wide-angle lens and a wide-angle lens, as shown in one embodiment.

[0021] Figure 9 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment;

[0022] Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] This application provides an image processing method applied to an electronic device including m image sensors, where m is greater than or equal to 2 and is an integer. Correspondingly, the electronic device includes m lenses, each lens corresponding to one of the m image sensors, and each image sensor is used to sense the light beam transmitted through its corresponding lens. The electronic device in this application includes, but is not limited to, personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices with multi-camera functionality. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices may include smartwatches, smart bracelets, and head-mounted devices. The focal length of each lens may be different, and each image sensor is configured to operate within a different magnification range. The camera's display page will show magnifications such as 0.6X, 1X, 2X, or 5X, 10X, with smaller numbers indicating a wider field of view and larger numbers indicating a farther field of view.

[0025] This application uses three common types of lenses found in electronic devices as examples: wide-angle lenses, ultra-wide-angle lenses, and telephoto lenses. The main camera lens of most electronic devices is generally a wide-angle lens with an equivalent focal length of approximately 28mm. A 28mm focal length is close to the field of view of the human eye. When the main camera lens is in its initial preset imaging position, the display shows a 1X magnification, suitable for shooting portraits, architecture, landscapes, documentary, etc. Ultra-wide-angle lenses offer a wider field of view than the main camera. Compared to the main camera, ultra-wide-angle lenses can capture a much wider scene, suitable for landscapes and architecture, producing impactful images. Furthermore, ultra-wide-angle lenses have a very large field of view; from the same position, you can obtain a wider scene, greatly facilitating post-processing cropping. Using an ultra-wide-angle lens when shooting landscapes not only allows more landscape elements to be included in the frame, but with proper composition, it can also create a sense of depth and space, making the entire landscape photo look grand and imposing. Telephoto lenses can capture higher-quality photos from a distance. When the telephoto lens is in its initial preset imaging position, the display interface can show a 5x magnification. For example, in situations where movement is inconvenient, and you want to frame a shot in a cluttered building, it's difficult to capture the subject using only the phone's main camera. Using the telephoto lens can add depth and dimension to an otherwise ordinary photo. The telephoto lens "closes" the distance between the background and foreground, creating a sense of spatial compression that makes the overall image fuller. This "compression" is one of the characteristics of telephoto lenses. The minimal distortion and weak perspective of telephoto lenses further enhance the relationship between foreground and background, creating unique visual effects. This characteristic can also be used to guide the viewer's attention to the subject in the depth of the image using straight lines such as roads and railings.

[0026] Exemplary examples show that the electronic device in this application may include a dual-camera system consisting of a wide-angle lens and a telephoto lens, or a triple-camera system consisting of a wide-angle lens, an ultra-wide-angle lens, and a telephoto lens, or a camera system consisting of four or more lenses with different focal lengths. This application does not limit the scope of the application. For example, Figure 1 This is a schematic diagram of the magnification range of one embodiment, for reference. Figure 1 When an electronic device employs a triple-camera system, the magnification range of the image sensor corresponding to the ultra-wide-angle lens can be configured as 0.5X-1X, the range of 1X-5X as the range of the image sensor corresponding to the wide-angle lens, and the range of 5X and above as the range of the image sensor corresponding to the telephoto lens. Within the corresponding magnification range, the image sensor can output a data stream at a certain frame rate, for example, 30fps. For ease of explanation, the image sensor corresponding to the ultra-wide-angle lens will be referred to as the ultra-wide-angle sensor, the image sensor corresponding to the wide-angle lens as the wide-angle sensor, and the image sensor corresponding to the telephoto lens as the telephoto sensor.

[0027] Figure 2 One of the flowcharts for an image processing method according to an embodiment is shown below. Figure 2 The image processing method in this embodiment includes steps 202 to 204.

[0028] Step 202: Based on the condition of receiving a focusing operation performed on the electronic device, control the output data streams of n image sensors.

[0029] Where n is greater than or equal to 2, and n is an integer. The focusing operation is used to adjust the imaging magnification of the electronic device from an initial magnification to a target magnification. That is, the initial magnification is the imaging magnification before receiving the focusing operation, and the target magnification is the imaging magnification indicated by the focusing operation. Specifically, when the user opens the display interface of the camera program, a preview image can be displayed on the interface for the user to observe, so that the user can accurately adjust the imaging parameters. For example, the focusing operation can be performed by the user on the display screen of the electronic device. For instance, the display screen displays focusing controls, and the user can perform the focusing operation by touching the focusing controls, with the camera HAL detecting the user's operation on the focusing controls. Figure 3 This is a schematic diagram of the focus control in its unfolded state according to one embodiment. (Refer to...) Figure 3The focusing controls include, but are not limited to, magnification buttons (e.g., 0.6X, 1X, and 5X in the figure) and a focusing dial. Users can expand the focusing dial (also called the zoom dial) by clicking the magnification buttons in the focusing controls, and change the imaging magnification by clicking different magnification buttons, or by sliding the focusing dial. It is understood that the user's next operation after waking up the focusing dial is usually adjusting the imaging magnification. Therefore, this embodiment does not limit whether the initial magnification and the target magnification are the same; that is, even if the user only wakes up the focusing dial without actually changing the imaging magnification, it is considered that the user has performed a focusing operation. Another example is that the focusing operation can be performed by the user on other physical components of the electronic device. For example, when the camera app is open, the volume control buttons of the electronic device can be reused as focusing buttons. Users can increase the imaging magnification by using the volume up button or decrease the imaging magnification by using the volume down button. In this way, this embodiment controls the output data streams of n image sensors when a magnification change is about to occur or has already occurred, thereby avoiding the problem of untimely data processing caused by a sudden excessive adjustment of the imaging magnification.

[0030] Taking an image processing method applied to an electronic device including three image sensors as an example, before receiving a focusing operation, the imaging magnification of the electronic device is an initial magnification. Only one of the three lenses outputs a data stream; this lens can be any one of an ultra-wide-angle lens, a wide-angle lens, and a telephoto lens, specifically determined by the initial magnification. For example, if the initial magnification is 1.5X, then the initial lens corresponding to the initial magnification is a wide-angle lens. When a focusing operation is received, for example, if it is detected that the user has unfolded the focusing dial via touch or adjusted the imaging magnification from 1.5X to 1.8X, at least one of the remaining two image sensors can be controlled to also start outputting a data stream. That is, a total of two or three image sensors are also outputting data streams. Optionally, when no focusing operation is received, the image sensors that are not outputting data streams can be in a closed state or in a state of being open but not outputting data streams; this embodiment does not limit this. If the image sensors that are not outputting data streams can be in a closed state, then after receiving a focusing operation, the above-mentioned image sensors need to be initialized, and after the initialization is completed, the image sensors are controlled to output data streams. It is understandable that if all image sensors are turned on, the output data stream speed is faster, but the power consumption is also relatively high; if only the image sensors that need to output data streams are turned on, the power consumption is lower, but a certain initialization time is required. Therefore, the specific mode of image sensor activation can be selected based on factors such as the battery level of the electronic device, and this embodiment does not impose any limitations.

[0031] Step 204: Spatial alignment processing is performed on the data streams output by the n image sensors.

[0032] Spatial alignment processing, also known as Spatial Alignment Transform (SAT), is a smooth zoom solution, essentially a multi-lens hybrid zoom solution. In related technologies, zoom is divided into two types: optical zoom and digital zoom. Optical zoom refers to increasing the image size by moving the distance between the lens and the image sensor, thus equivalent to increasing the focal length. Digital zoom refers to changing the image size through cropping and scaling, thus equivalent to changing the focal length. Due to the thickness limitations of mobile phones, it is difficult to integrate complex optical structures into zoom lenses; therefore, fixed-focus lenses are generally used to achieve the zoom effect. For example, an electronic device includes a wide-angle lens with an equivalent focal length of 27mm, an ultra-wide-angle lens with an equivalent focal length of 16mm, and a telephoto lens with an equivalent focal length of 125mm. The focal lengths between 16mm and 27mm and between 27mm and 125mm are continuously varied through digital zoom fusion, and optical zoom allows switching between different magnification ranges, ultimately achieving continuous zoom from 16mm to 125mm. When switching lenses, if only the imaging lens is switched, the preview image display can easily change noticeably, such as sudden changes in brightness or content, resulting in an unnatural preview image display. Therefore, spatial alignment processing can be used to align the images corresponding to the data streams of different image sensors, thereby achieving smooth zoom and improving the user experience.

[0033] Especially in cross-cutting scenarios, which refer to switching scenarios requiring sequential switching to at least three lenses. For example, adjusting the magnification from 0.8X to 6X involves switching from an ultra-wide-angle lens to a wide-angle lens, and then from a wide-angle lens to a telephoto lens, requiring waiting for the wide-angle and telephoto lenses to start streaming respectively. However, the solution in this embodiment can control the wide-angle lens's output data stream and perform spatial alignment processing before determining that a cross-cutting is needed. Furthermore, with sufficient data processing capabilities, the telephoto lens's output data stream can be controlled simultaneously, and spatial alignment processing can be performed on all three data streams. This significantly reduces the actual streaming start-up waiting time and processing time during a cross-cutting, reducing zoom lag. It is understood that this embodiment does not limit the conditions for determining when an actual switch occurs; as long as at least partial spatial alignment processing is completed when an actual switch occurs, the desired effect can be achieved. Therefore, in this embodiment, by monitoring the user's focusing operation in real time, the required n lens output data streams can be started in a timely manner and spatial alignment processing can be performed. This can reduce the start-up time when an actual switch occurs, and shorten the processing time after the lens switch is determined through the pre-processed spatial alignment processing.

[0034] In one embodiment, controlling the output data streams of n image sensors includes the following steps: controlling each of the n image sensors to output a data stream at a corresponding target output frame rate. The target output frame rate is negatively correlated with the magnification adjustment range, which is the distance between the target magnification corresponding to the focusing operation and the magnification range corresponding to each image sensor. Specifically, the magnification adjustment range can be the distance between the target magnification and the boundary value of the magnification range. For example, if the focusing operation adjusts the imaging magnification from 0.8X to 1.2X, the distance between the target magnification 1.2X and the magnification range of the ultra-wide-angle sensor (e.g., 0.5X-1X) is 0.2X, the distance between the target magnification 1.2X and the magnification range of the wide-angle sensor (e.g., 1X-5X) is 0, and the distance between the target magnification 1.2X and the magnification range of the telephoto sensor (e.g., greater than 5X) is 3.8X, then the target output frame rate of the wide-angle sensor can be controlled to be greater than the target output frame rate of the ultra-wide-angle sensor, and the target output frame rate of the ultra-wide-angle sensor can be controlled to be greater than the target output frame rate of the telephoto sensor. Optionally, to reduce the amount of data processed, if the magnification adjustment range of a certain image sensor exceeds a threshold, that image sensor can be controlled to not output data, thereby improving image processing speed. In this embodiment, the target output frame rate of the lens can be dynamically adjusted according to changes in the target magnification, so that the target output frame rate of image sensors closer to the target magnification is increased accordingly, while the target output frame rate of other image sensors is decreased accordingly, to ensure that the overall target output frame rate is within an acceptable range, so that the total data flow does not exceed the upper limit of the transmission bandwidth and meets the effect requirements.

[0035] In one embodiment, before controlling the n image sensors to output data streams at their respective target output frame rates, the method further includes the following step: controlling the n image sensors to output data streams at the same initial output frame rate, where the initial output frame rate is the maximum output frame rate of the image sensors. The maximum output frame rate is an inherent performance parameter of the image sensors, and for example, it can be 30fps. In this embodiment, by controlling each image sensor to output at the initial output frame rate first, it can be ensured that the image sensors can output data in a timely manner, avoiding excessive analysis of the target output frame rate in the preceding steps, which could affect the output speed. Moreover, since the initial output frame rate is the maximum output frame rate of the image sensors, it can be ensured that the output data stream will provide sufficient data for spatial alignment processing, thereby improving the accuracy of spatial alignment processing. The initial output frame rate can last for one or more frames; this embodiment does not impose any limitation.

[0036] In one embodiment, m is greater than 2. Before controlling the output data streams of n image sensors, the following step is included: determining n based on the initial magnification and the target magnification. It is understood that the larger the difference between the initial magnification and the target magnification, the greater the probability of a cross-cut. Correspondingly, the requirement for pre-performing at least partial spatial alignment processing is greater, i.e., more image sensors need to be controlled to output data streams. Therefore, this embodiment, by analyzing the initial magnification and the target magnification, can determine the number of image sensors that need to output data streams to a certain extent, avoiding the situation where the image sensors affected by the cross-cut fail to output data streams in advance, thereby improving the reliability of the data processing process. Moreover, in scenarios with a low probability of cross-cutting, the number of image sensors outputting data streams can be reduced, thereby reducing the power consumption of the electronic device.

[0037] In one embodiment, determining n based on the initial magnification and the target magnification includes the following steps: determining n = m based on the condition that there is at least one magnification interval between the initial magnification and the target magnification. Specifically, taking an image processing method applied to an electronic device with three lenses as an example, based on the condition that at least one magnification interval is detected between the initial magnification and the target magnification, the probability of triggering a cross-cut is extremely high, therefore all lenses within the cross-cut range need to be activated. Based on the condition that the magnification interval where the initial magnification is located is adjacent to the magnification interval where the target magnification is located, n < m is determined. For scenarios where the magnification span is small and cross-cut will not occur, only some lenses need to be activated, and the remaining lenses do not need to output data streams, thereby saving power consumption during lens switching in non-cross-cut scenarios. Furthermore, the number of image sensors activated also needs to consider the algorithm capabilities and alignment strategies of spatial alignment processing to accurately control the number of image sensors activated. Furthermore, when only some lenses are activated to output data streams, the required data stream can be selected according to the magnification interval where the target magnification is located to improve the reliability of the image processing method. For example, you can select only the output data streams of two image sensors that are close to the target magnification and use them as input to the algorithm for spatial alignment.

[0038] In one embodiment, the display interface of the electronic device shows a focusing control. The focusing operation includes a control pull-up operation, which switches the focusing control from a folded state to an unfolded state. Specifically, the unfolded focusing control can be referred to... Figure 3 The magnification button and focus dial are both displayed on the screen. Users can precisely adjust the imaging magnification using the expanded focus control. The resolution that can be adjusted can be, for example, 0.2X or 0.5X. This embodiment does not limit this. Figure 4 This is a schematic diagram of a focus control in a folded state according to one embodiment. (Refer to...) Figure 4Based on the condition that the focusing control is in a collapsed state, only multiple magnification buttons are displayed on the screen, and the focusing dial is hidden. Users can click the magnification button to switch the imaging magnification, thereby avoiding the focusing dial from obscuring the preview image.

[0039] Figure 5 A second flowchart of an image processing method according to an embodiment, refer to... Figure 5 In one embodiment, the image processing method includes steps 502 to 512. Steps 504 to 510 of this embodiment can be referred to the foregoing embodiments, and will not be described again in this embodiment.

[0040] Step 502: Based on the condition of receiving a control pull-up operation applied to the focus control, obtain the initial magnification and target magnification of the focus operation.

[0041] Step 504: Based on the condition that there is at least one range between the initial magnification and the target magnification, determine n = m.

[0042] Step 506: Based on the condition that the range of the initial multiplier is adjacent to the range of the target multiplier, determine that n < m.

[0043] Step 508: Control the n image sensors to output data streams at their respective target output frame rates.

[0044] Step 510: Spatial alignment processing is performed on the data streams output by the n image sensors.

[0045] Step 512: Based on the condition that the focus control switches from the unfolded state to the folded state, control the n-1 image sensors (excluding the target sensor) out of the n image sensors to stop outputting data streams.

[0046] The target sensor is the image sensor corresponding to the magnification range of the target magnification. For example, a user can switch the focus control from an unfolded state to a collapsed state by touching the focus control. For another example, the electronic device can automatically retract the focus dial when it detects that the user has not operated the focus dial for an extended period. After retracting the focus dial, this application can immediately stop outputting multiple data streams, thereby reducing the power consumption generated by multiplexing and spatial alignment processing. It is understood that when the number of data streams decreases, related resources, such as buffer resources allocated during initialization, can be released to reduce resource consumption.

[0047] In one embodiment, the focusing operation includes an adjustment operation to change the imaging magnification. Based on the received focusing operation acting on the electronic device, the data streams output by n image sensors are controlled, including the following steps: based on the condition that the magnification range of the target magnification is the same as the magnification range of the initial magnification, the data streams output by the n image sensors are controlled. Specifically, if the target magnification and the initial magnification are in the same magnification range, it means that although the imaging magnification has changed, the change is not significant enough to require lens switching, so only the data streams output by the n image sensors need to be controlled. Since the change is not significant enough to require lens switching, lens switching can be performed directly, thereby improving image processing speed and lens switching speed, and enhancing the user experience.

[0048] Figure 6 The third flowchart of an image processing method according to an embodiment is shown below. Figure 6 In one embodiment, the image processing method includes steps 602 to 612. The steps of this embodiment can be referred to the foregoing embodiments, and will not be repeated in this embodiment.

[0049] Step 602: Based on the conditions of receiving the adjustment operation acting on the electronic device, obtain the initial magnification and target magnification of the adjustment operation.

[0050] Step 604: Based on the condition that there is at least one range between the initial magnification and the target magnification, determine n = m;

[0051] Step 606: Based on the condition that the range of the initial multiplier is adjacent to the range of the target multiplier, determine that n < m.

[0052] Step 608: Control the n image sensors to output data streams at their respective target output frame rates.

[0053] Step 610: Spatial alignment processing is performed on the data streams output by the n image sensors.

[0054] Step 612: Based on the condition that the focus control switches from the unfolded state to the folded state, control the n-1 image sensors (excluding the target sensor) out of the n image sensors to stop outputting data streams.

[0055] Figure 7 A flowchart of an image processing method according to an embodiment is shown below (see flowchart four). Figure 7 In one embodiment, the image processing method includes steps 702 to 706, wherein steps 702 and 704 can be referred to the foregoing embodiments, and will not be described again in this embodiment.

[0056] Step 702: Based on the condition of receiving a focusing operation performed on the electronic device, control the n image sensors to output data streams.

[0057] Step 704: Spatial alignment processing is performed on the data streams output by the n image sensors.

[0058] Step 706: Based on the condition that the magnification range of the target magnification is different from the magnification range of the initial magnification, obtain the target preview image according to the data stream output by the target sensor and the result of spatial alignment processing.

[0059] In this embodiment, the target sensor is the image sensor corresponding to the magnification range where the target magnification is located. Because the magnification range of the target magnification differs from that of the initial magnification, a lens switching operation is required. For example, before switching from a telephoto lens to a wide-angle lens, both the telephoto and wide-angle lenses have already acquired images, and their corresponding image sensors are outputting data streams, with spatial alignment processing performed based on the data streams. Therefore, when switching lenses and displaying the target preview image based on the new lens's data stream, spatial alignment processing may not be necessary, or only a small amount of spatial alignment processing may be required. For instance, a coarser spatial alignment process can be performed before determining that a lens switching is needed, and a finer process can be performed based on the result of the coarse spatial alignment process after determining that a lens switching is needed, thereby improving alignment accuracy and reducing the time required for fine processing. In this embodiment, a faster image processing method for lens switching is provided through the preceding image processing method.

[0060] In one embodiment, the image processing method further includes the following step: controlling n image sensors to continuously output data streams for a first preset time period after updating the preview image. Further, the image processing method further includes the following step: after the first preset time period, controlling n-1 image sensors (excluding the target sensor) among the n image sensors to stop outputting data streams. Optionally, the first preset time period can be 1 second to 3 seconds, for example, 1 second, 2 seconds, etc. It is understood that in some scenarios, users have a need for repeated zooming, which requires constant switching between different lenses. Therefore, considering the user's needs, several data streams can be restored to their normal preview configuration after a certain delay. For example, the normal preview configuration could be one output stream and the other two not outputting. Based on the above method, when the user zooms again within a certain time period, since most of the originally started data streams have not yet been closed, it can effectively avoid repeated lens initialization caused by repeated switching by the user, and it does not require restarting the data streams, thereby improving the timeliness of zooming.

[0061] In one embodiment, before obtaining the target preview image based on the data stream output by the target sensor and the result of spatial alignment processing, the method further includes the following step: aligning and fusing the data stream output by the initial sensor and the data stream output by the target sensor based on the result of spatial alignment processing to obtain at least one frame of transition preview image. Here, the initial sensor is the image sensor corresponding to the magnification range where the initial magnification is located. Specifically, Figure 8 This is a schematic diagram of the transition frame when switching between an ultra-wide-angle lens and a wide-angle lens, as shown in the following embodiment. Figure 8 Taking the switching between ultra-wide-angle and wide-angle lenses as an example, the data streams from the wide-angle and ultra-wide-angle sensors can be fused based on the spatial alignment processing results to form a fused transition preview image, which is then displayed. In the embodiment shown in the figure, the output frame rate of the ultra-wide-angle sensor is 30fps before the transition frame. During the transition frame, both the ultra-wide-angle and wide-angle sensors output data streams, with the ultra-wide-angle sensor outputting a frame rate of 7.5fps and the wide-angle sensor outputting a frame rate of 30fps. After several transition frames, the output frame rate of the wide-angle sensor remains at 30fps, while the output frame rate of the ultra-wide-angle sensor drops to 0 (not shown in the figure). In this embodiment, based on the length of the saved switching time, the number of transition frames can be set to make reasonable use of the saved time, ensuring that the preview image does not undergo obvious changes visible to the naked eye, thus achieving a smoother switching effect. It is understood that in some embodiments, the target preview image captured by the new lens can be displayed directly without setting a transition frame for displaying the transition preview image; this embodiment does not limit this.

[0062] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0063] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0064] Figure 9 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment, with reference to... Figure 9 The device includes an operation receiving module and an alignment module. The operation receiving module controls the output data streams of n image sensors based on received focusing operations performed on the electronic device. The focusing operation adjusts the imaging magnification of the electronic device from an initial magnification to a target magnification, where n is greater than or equal to 2 and is an integer. The alignment module performs spatial alignment processing on the data streams output by the n image sensors.

[0065] In one embodiment, the image processing apparatus further includes a quantity determination module, which determines n based on an initial magnification and a target magnification.

[0066] In one embodiment, the image processing apparatus further includes a data stopping module for controlling n-1 image sensors (excluding the target sensor) to stop outputting data streams.

[0067] In one embodiment, the image processing apparatus further includes a target image acquisition module, which is used to acquire a target preview image based on the condition that the magnification range where the target magnification is located is different from the magnification range where the initial magnification is located, according to the data stream output by the target sensor and the result of spatial alignment processing, wherein the target sensor is the image sensor corresponding to the magnification range where the target magnification is located.

[0068] In one embodiment, the image processing apparatus further includes a delay module for controlling the n image sensors to continuously output data streams for a first preset time period after updating the preview image.

[0069] In one embodiment, the delay module is further configured to control n-1 image sensors (excluding the target sensor) among the n image sensors to stop outputting data streams after a first preset duration.

[0070] In one embodiment, the image processing apparatus further includes a transition image acquisition module, which is used to align and fuse the data stream output by the initial sensor and the data stream output by the target sensor according to the result of spatial alignment processing, so as to obtain at least one frame of transition preview image, wherein the initial sensor is the image sensor corresponding to the magnification range where the initial magnification is located.

[0071] Each module in the aforementioned devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0072] This application also provides an electronic device, including m image sensors, a memory, and a processor. Each image sensor is configured to operate in a different magnification range. The memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0073] In one embodiment, an electronic device is provided, which may be a terminal. Figure 10 This is a schematic diagram of the internal structure of an electronic device according to an embodiment, with reference to... Figure 10 The electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0074] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0075] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0076] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, An electronic device includes m image sensors, where m is greater than 2 and m is an integer. The method includes: Based on the condition that a user has performed a focusing operation on the electronic device, and based on the condition that there is at least one magnification interval between the initial magnification and the target magnification, n = m is determined; the focusing operation is used to adjust the imaging magnification of the electronic device from the initial magnification to the target magnification; Based on the condition that the range of the initial magnification is adjacent to the range of the target magnification, n < m is determined; Control the output data streams of n image sensors, where n is greater than or equal to 2 and is an integer; Spatial alignment processing is performed on the data streams output by the n image sensors; Based on the condition that the magnification range where the target magnification is located is different from the magnification range where the initial magnification is located, a target preview image is obtained according to the data stream output by the target sensor and the result of the spatial alignment processing. The target sensor is the image sensor corresponding to the magnification range where the target magnification is located.

2. The image processing method according to claim 1, characterized in that, The control of the output data streams of n image sensors includes: The n image sensors are controlled to output data streams at corresponding target output frame rates. The target output frame rate is negatively correlated with the magnification adjustment range, which is the distance between the target magnification corresponding to the focusing operation and the magnification range corresponding to each image sensor.

3. The image processing method according to any one of claims 1 to 2, characterized in that, The display interface of the electronic device shows a focus control. The focus operation includes a control pull-up operation, which is used to switch the focus control from a folded state to an unfolded state.

4. The image processing method according to claim 3, characterized in that, Also includes: Based on the condition that the focusing control switches from the unfolded state to the folded state, the n-1 image sensors (excluding the target sensor) among the n image sensors are controlled to stop outputting data streams. The target sensor is the image sensor corresponding to the magnification range where the target magnification is located.

5. The image processing method according to any one of claims 1 to 2, characterized in that, The focusing operation includes an adjustment operation used to change the imaging magnification. The step of controlling the output data streams of n image sensors based on the received focusing operation acting on the electronic device includes: Based on the condition that the magnification range of the target magnification is the same as the magnification range of the initial magnification, the output data streams of the n image sensors are controlled.

6. The image processing method according to claim 1, characterized in that, Also includes: The n image sensors are controlled to continuously output data streams for a first preset time period after updating the preview image.

7. The image processing method according to claim 6, characterized in that, Also includes: After the first preset time period, control the n-1 image sensors (excluding the target sensor) to stop outputting data streams.

8. The image processing method according to claim 1, characterized in that, Before obtaining the target preview image based on the data stream output by the target sensor and the result of the spatial alignment processing, the method further includes: Based on the result of the spatial alignment process, the data stream output by the initial sensor and the data stream output by the target sensor are aligned and fused to obtain at least one frame of transition preview image, wherein the initial sensor is the image sensor corresponding to the magnification range in which the initial magnification is located.

9. An electronic device, characterized in that, The method includes m image sensors, a memory, and a processor. Each image sensor is configured to operate in a different magnification range. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. An image processing apparatus, characterized in that, An electronic device includes m image sensors, where m is greater than 2 and m is an integer. The device includes: The quantity determination module is used to determine n = m based on the condition that a user has performed a focusing operation on the electronic device, and based on the condition that there is at least one magnification interval between the initial magnification and the target magnification; the focusing operation is used to adjust the imaging magnification of the electronic device from the initial magnification to the target magnification; and based on the condition that the magnification interval of the initial magnification is adjacent to the magnification interval of the target magnification, it is determined that n < m; The operation receiving module is used to control the output data streams of n image sensors, where n is greater than or equal to 2 and is an integer; The alignment module is used to perform spatial alignment processing on the data streams output by the n image sensors; The target image acquisition module is used to acquire a target preview image based on the condition that the magnification range where the target magnification is located is different from the magnification range where the initial magnification is located, according to the data stream output by the target sensor and the result of spatial alignment processing. The target sensor is the image sensor corresponding to the magnification range where the target magnification is located.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images