Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps that in response to zooming operation, a current zooming magnification corresponding to a current image frame and a target zooming magnification issued by a camera application are acquired, the current zooming magnification is used for determining optical change point identification information, and the target zooming magnification is used for determining zooming speed information; determining a target frame insertion number according to the optical change point identification information and the zooming speed information; and processing a target image by using the zoom magnification of the target frame insertion number. By adopting the method, the image picture switching process in the zooming process can be ensured to be smoother.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of image processing technology, multiple cameras configured in electronic devices correspond to different focal lengths. Each focal length corresponds to a different focal length range, so the corresponding field of view angles of the focal lengths are also different. Through image zoom technology, clear images at different distances can be captured.

[0003] However, during the zooming process, the image smoothness may be low due to camera switching. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the smoothness of image switching during zooming.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] In response to the zoom operation, a current zoom ratio corresponding to the current image frame and a target zoom ratio issued by the camera application are obtained; wherein the current zoom ratio is used to determine the light change point identification information, and the target zoom ratio is used to determine the zoom speed information;

[0007] Determining a target number of interpolated frames according to the light change point identification information and the zoom speed information;

[0008] The image is processed using the zoom factor of the target interpolation frame number.

[0009] In a second aspect, the present application further provides an image processing device, comprising:

[0010] An acquisition module, used to obtain, in response to a zoom operation, a current zoom ratio corresponding to a current image frame and a target zoom ratio issued by a camera application; wherein the current zoom ratio is used to determine light change point identification information, and the target zoom ratio is used to determine zoom speed information;

[0011] A scene module, used to determine a target number of interpolated frames according to the light change point identification information and the zoom speed information;

[0012] The processing module is used to process the target image using the zoom factor of the target interpolation frame number.

[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the image processing method of the first aspect when executing the computer program.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method of the first aspect.

[0015] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the image processing method of the first aspect when executed by a processor.

[0016] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product, in response to a zoom operation, obtain the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application, so as to clarify the current zoom ratio of the zoom process and the target zoom ratio to be achieved; determine the light change point identification information according to the current zoom ratio, and determine the zoom speed information according to the target zoom ratio, and the light change point identification information indicates whether the zoom process passes through the light change point, and the zoom speed information indicates the interval where the zoom rate is located, and then indirectly regulate the interpolation mode of the zoom process through the light change point identification information and the zoom speed information, and thus determine the target interpolation number according to the light change point identification information and the zoom speed information, so that the target interpolation number is adaptively adjusted to adapt to different scenes; and then, the target image is processed using the zoom ratio of the target interpolation number, so that the field of view angle of the target image in the zoom scene gradually changes, avoiding the situation of sudden change of the field of view angle, and realizing smooth switching of the target image during the zoom process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0019] Figure 2 is a schematic diagram of capturing an image in one embodiment;

[0020] Figure 3 is a schematic diagram of an electronic device and an image captured by the electronic device in one embodiment;

[0021] Figure 4 is a schematic flow chart of an image processing method in one embodiment;

[0022] Figure 5 A schematic diagram of a process for determining a target number of interpolated frames in one embodiment;

[0023] Figure 6 A schematic diagram of an interpolation curve in an embodiment;

[0024] Figure 7 It is a schematic diagram of the overall process in a sliding zoom scenario in one embodiment;

[0025] Figure 8 is a schematic diagram of zoom multiples in a sliding zoom scene in one embodiment;

[0026] Fig. 9 is a schematic diagram of a frame insertion strategy in an embodiment;

[0027] Fig.10 is a schematic diagram of a tail frame insertion curve in one embodiment;

[0028] Fig.11 is a structural block diagram of an image processing device in one embodiment;

[0029] Fig.12 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] Some electronic devices may be equipped with 3-5 different types of lenses. For example, the camera operation page of a multi-camera mobile phone will display focal length options of 0.6X, 1X, 2X, or 5X, 10X. The smaller the number, the wider the corresponding viewing angle and the farther the subject is, which roughly corresponds to the focal lengths of ultra wide-angle (Ultra Wide, UW), wide-angle (Wide, W) and telephoto (Telephoto, T) lenses.

[0032] Take a mobile phone as an example. In some cases, the main camera of the mobile phone is a wide-angle lens with a focal length of about 28 mm. Because this focal length is close to the field of view seen by the human eye, the main camera is the most frequently used camera. The operation page of the mobile phone camera will display the value of 1X, such as Figure 1 As shown, 1X is the focal length corresponding to the main camera, and the image quality is relatively clear, which is suitable for shooting portraits, architecture, scenery and documentaries, etc.

[0033] An ultra-wide-angle lens can provide a wider viewing angle than a wide-angle lens, that is, it can capture a wider picture than a wide-angle lens. Some wide-angle lenses are marked with the zoom ratio in the operation page, which can be 0.5X or 0.6X; other wide-angle lenses are marked with images or text. An ultra-wide-angle lens can allow more landscape elements to be captured, and is suitable for photographing scenery, architecture, etc. The captured images can have an impactful sense of picture. An ultra-wide-angle lens has the characteristic of lens distortion. Objects at the edge of the captured image are elongated and magnified. This distortion can be fully utilized to shoot upward, creating a visual impact of "near is big and far is small". When photographing buildings, the distortion can be used to make the buildings more magnificent. Images captured by an ultra-wide-angle lens such as Figure 2 shown.

[0034] In addition to wide-angle lenses and ultra-wide-angle lenses, mobile phone cameras can also include telephoto lenses, which are used to obtain images with a zoom ratio greater than or equal to 1X. The larger the number in front of X, the narrower the shooting range, but the farther the shooting is, and the higher the clarity of the distant objects. When shooting farther objects or zooming in on objects in the picture with a telephoto lens, the collected image will not be degraded like digital zoom. When it is inconvenient to move around and want to take pictures in a messy building, it is difficult to capture the subject with the main camera of the mobile phone alone. At this time, using a mobile phone telephoto lens to shoot can make the image more layered. The telephoto lens can "pull in" the distance between the background and the foreground, thereby bringing a sense of compression of the spatial distance, making the overall picture more substantial. The telephoto lens has little deformation and weak perspective effect, which can shorten the distance between the foreground and the background, enhance the relationship between the foreground and the background, and create unique visual effects. Using this feature, straight-extending scenery such as roads and railings can be used as guide lines to draw the audience's attention back to the subject in the depth of the picture.

[0035] Lenses corresponding to different focal lengths have different characteristics. The camera application is designed with a spatial alignment transform (SAT) solution to achieve real-time lens switching according to user needs in different shooting scenarios. For example, the wide-angle lens used in a mobile phone has an equivalent focal length of 27 mm, a resolution of 40 million pixels, and an aperture of 1.6 focal ratio; the ultra-wide-angle lens has an equivalent focal length of 16 mm and an aperture of 2.2 focal ratio; the telephoto lens has an equivalent focal length of 125 mm and an aperture of 3.4 focal ratio, which can continuously change between 16 and 27 mm and 27 to 125 mm focal lengths.

[0036] like Figure 3 The electronic device shown in (a) includes three cameras, namely, an ultra-wide-angle lens, a wide-angle lens, and a telephoto lens. The zoom ratios of the three cameras are switched as optical change points, and the optical change point between the ultra-wide-angle lens and the wide-angle lens is 1X. Figure 3 As shown in (b) in FIG. 1 , the wide-angle lens acquires images at a zoom ratio of 2X. Figure 3 As shown in (c) in the figure; the light change point 3X between the wide-angle lens and the telephoto lens, as shown in Figure 3 As shown in (d) in FIG. 1 , the telephoto lens acquires images at a zoom ratio of 5X. Figure 3 As shown in (e) in .

[0037] During the zooming process, how to ensure the smoothness of lens switching and reduce image jumps have become the key issues that current cameras need to consider. Smoothness refers to the smoothness of the image frames displayed on the screen during the zooming process, that is, continuous flow; while reducing image jumps is achieved based on spatial alignment processing to ensure the consistency of image effects before and after switching.

[0038] There are many zooming methods. For example, the image smoothness of the sliding zoom scene is poor. The sliding zoom scene has the characteristics of random direction and speed, and has high requirements for the start and end hand tracking. In addition, it is necessary to take into account the impact of power consumption and platform resource limitations. Therefore, it is necessary to balance all aspects as much as possible and focus on the characteristics of different sliding zoom scenes. Among them, power consumption includes but is not limited to the three-open problem, and the limitation of platform resources includes but is not limited to the 3TFE problem. Among them, the three-open problem is how to turn on the three cameras and the corresponding sensors. And the 3TFE problem is how to handle different touch tasks with three independent touch feature engines.

[0039] After analysis, the inventor of this application found that it is necessary to make more detailed scene divisions for the zoom process, and then make differentiated interpolation control for the detailed scenes. For example, the zoom-in scene focuses on the stop tracking and the smoothness of the field of view (FOV) change, and the zoom-out scene with fast sliding and passing the light change point focuses on the number of interpolation frames to buy time for the target lens to open.

[0040] The image processing method provided in the embodiment of the present application can be applied to electronic devices. Among them, the electronic device can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The electronic device can be a terminal or a server.

[0041] In some exemplary embodiments, Figure 4As shown, an image processing method is provided, and the method is applied to an electronic device, and the method includes the following steps 402 to 406, wherein:

[0042] Step 402, in response to the zoom operation, obtain the current zoom ratio corresponding to the current image frame and the target zoom ratio sent by the camera application; wherein the current zoom ratio is used to determine the light change point identification information, and the target zoom ratio is used to determine the zoom speed information.

[0043] The zoom operation is an operation for changing the focal length. The zoom operation may refer to an uncertain continuous zoom operation, that is, a zoom ratio (zoom ratio) without a clear target during the zoom process, by adjusting the zoom ratio and providing the result of the zoom screen, so that the user can get a better preview or recording screen. The zoom operation may also be a zoom ratio that slides to a certain target. The zoom operation may be implemented by touching or sliding a zoom control, and the zoom control may also be implemented by a zoom control on a camera application interface, and the zoom control on the camera application interface may be a zoom control in the form of a disk, a strip, or other shapes. In some possible implementations, the zoom operation may also be implemented by pressing a button, sliding, or rotating a hardware device, such as configuring a rotatable device on an electronic device, and adjusting the zoom ratio by rotating the rotatable device; a rotatable device is an element that can rotate.

[0044] The current image frame is the image frame that the electronic device needs to display at the current moment. The current image frame may be the current image frame that needs to be previewed, photographed, or recorded; correspondingly, the zoom operation may be a zoom operation in a camera preview scene, a photo scene, or a recording scene. The current zoom ratio refers to the zoom ratio of the current image frame.

[0045] The target zoom ratio refers to the zoom ratio generated by the camera application after the camera application detects a zoom operation. The camera application obtains the zoom ratio obtained by the zoom operation in real time or at a fixed time, and uses the obtained zoom ratio as the target zoom ratio. In the case where the zoom operation is a slide operation, the target zoom ratio may be the target zoom ratio of the currently obtained slide operation.

[0046] Exemplarily, responding to a zoom operation may refer to the electronic device responding to a zoom operation generated by a user by sliding a disc; responding to a zoom operation may also refer to a zoom operation generated by a user by rotating a hardware device; responding to a zoom operation may also refer to a zoom operation generated by a user by touching a screen with two fingers or multiple points. The electronic device may obtain the target zoom ratio through a hardware abstraction layer (HAL).

[0047] The light change point identification information refers to the situation in which the zoom process triggered by the zoom operation passes through the light change point, and is used to indicate whether the zoom process passes through the light change point. The light change point identification information is a dimension of the zoom scene type. The light change point identification information can indicate whether the zoom process passes through the light change point. The light change point identification information can be represented by numbers, letters, or characters. The light change point identification information is a first identification, indicating that the zoom process passes through the light change point, and the light change point identification information is a second identification, indicating that the zoom process does not pass through the light change point. For example, the first identification is true, indicating that the zoom process passes through the light change point, and the second identification is false, indicating that the zoom process does not pass through the light change point; for another example, the first identification is 1, indicating that the zoom process passes through the light change point, and the second identification is 2, indicating that the zoom process does not pass through the light change point.

[0048] Zoom speed information refers to the type of zoom speed of the zoom process triggered by the zoom operation, and is used to indicate the interval in which the zoom rate is located. Zoom speed information is a dimension of the zoom scene type. Zoom speed information can indicate the type of the interval in which the zoom speed is located. Zoom speed information includes at least a first zoom speed type and a second zoom speed type. The zoom speed of the first zoom speed type is lower than the zoom speed of the second zoom speed type. In this case, the first zoom speed type can be a slow zoom scene, and the second zoom speed type can be a fast zoom scene.

[0049] Exemplarily, the zoom speed information may be identification data. The zoom speed information may be represented by numbers, letters, or characters. The zoom speed information is a first mark, indicating a first zoom speed type or a slow zoom scene, and the zoom speed information is a second mark, indicating a second zoom speed type or a fast zoom scene. For example, the first mark is K and the second mark is S; for another example, the first mark is 3 and the second mark is 4.

[0050] In the case of zooming in or out, the light change point identification information can be determined according to the current zoom ratio, and the zoom speed information can be determined according to the target zoom ratio, forming two dimensions of the zoom scene type. The zoom scene type of the zoom process can be analyzed more carefully through these two dimensions, so as to facilitate smooth switching of images. In addition, in the case of zooming out, the light change point identification information can be determined according to the current zoom ratio, and the zoom speed information can be determined according to the target zoom ratio; and in the case of zooming in, other steps can be performed to adapt to the field of view angle change rules of the zooming out and zooming in situations.

[0051] Step 404: determine the target number of interpolated frames according to the light change point identification information and the zoom speed information.

[0052] The target interpolation number is the number of zoom magnifications calculated by the zoom magnification interpolation method, and is a zoom processing method for adjusting the number of zoom magnifications to make the zoom process smoother. The target interpolation number may include the number of zoom magnifications between the current zoom magnification and the target zoom magnification.

[0053] In the two dimensions of light change point identification information and zoom speed information, data adjustment in any dimension can change the calculation method of the target interpolation number and obtain the corresponding zoom ratio interpolation method; and then obtain the zoom ratio of the target interpolation number through the corresponding zoom ratio interpolation method. Among them, the zoom ratio interpolation method can be implemented by a mapping table or by a variety of functions. The zoom curve can also be adjusted or selected according to the light change point identification information and the zoom speed information to obtain the target interpolation number.

[0054] Exemplarily, when the light change point identification information remains unchanged, the method for determining the target interpolation number is adjusted by the zoom speed information to obtain the corresponding target interpolation number. When the zoom speed information remains unchanged, the method for determining the target interpolation number is adjusted by the light change point identification information to obtain the corresponding target interpolation number.

[0055] Step 406: Process the target image using the zoom factor of the target interpolation frame number.

[0056] The zoom ratio of the target interpolation frame number includes a transition zoom ratio between the current zoom ratio and the target zoom ratio. The target zoom curve can be used to insert a zoom ratio between the current zoom ratio and the target zoom ratio, and the inserted zoom ratio is a transition zoom ratio. Based on at least one scene dimension in the light change point identification information and the zoom speed information, a target zoom curve corresponding to the scene dimension can be determined in each zoom curve; each zoom curve can be expressed by using various functions or mapping tables, and the function can be, for example, a linear function, a power function, an exponential function, etc.;

[0057] Exemplarily, the collected target image data is processed at each zoom ratio based on the target number of interpolated frames to obtain the target image data at each transitional zoom ratio; the field of view angle is determined based on the target image data at each transitional zoom ratio; and the electronic device is controlled to display the target image frame at the field of view angle.

[0058] In the above-mentioned image processing method, in response to the zoom operation, the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application are obtained to clarify the initial current zoom ratio of the zoom process and the target zoom ratio to be achieved; the light change point identification information is determined according to the current zoom ratio, and the zoom speed information is determined according to the target zoom ratio, and the light change point identification information is used to indicate whether the zoom process passes through the light change point, and the zoom speed information indicates the interval in which the zoom rate is located, and then the interpolation mode of the zoom process is indirectly regulated by the light change point identification information and the zoom speed information, and then the target interpolation frame number is determined according to the light change point identification information and the zoom speed information, so that the target interpolation frame number is adaptively adjusted to adapt to different scenes; and then, the target image is processed using the zoom ratio of the target interpolation frame number, so that the field of view angle of the target image in the zoom scene gradually changes, avoiding the sudden change of the field of view angle, and realizing smooth switching of the target image during the zoom process.

[0059] In some embodiments, the current zoom ratio is used to determine the light change point identification information, including: when there is a light change point between the current zoom ratio and the target zoom ratio, obtaining the light change point identification information used to indicate that the zoom process passes through the light change point; when there is no light change point between the current zoom ratio and the target zoom ratio, obtaining the light change point identification information used to indicate that the zoom process does not pass through the light change point.

[0060] The situation where there is a light change point between the current zoom ratio and the target zoom ratio refers to the situation where there is at least one light change point in the focal length between the current zoom ratio and the target zoom ratio. If there are multiple target zoom ratios, then in the case where there is a light change point between the current zoom ratio and the latest target zoom ratio, light change point identification information for indicating that the zoom process passes through the light change point is obtained; in the case where there is no light change point between the current zoom ratio and the latest target zoom ratio, light change point identification information for indicating that the zoom process does not pass through the light change point is obtained.

[0061] In this embodiment, the scene of the zoom process is accurately refined by the existence of the light change point between the current zoom ratio and the target zoom ratio. When passing through the light change point, the image change can be made smoother by using a multi-frame insertion method.

[0062] In some embodiments, the target zoom ratio is used to determine the zoom speed information, including: comparing the difference value between different target zoom ratios with a difference threshold; if the difference value is less than the difference threshold, determining that the zoom speed information is a first zoom speed type; if the difference value is greater than or equal to the difference threshold, determining that the zoom speed information is a second zoom speed type.

[0063] In the sliding zoom scenario, the camera application will detect the target zoom ratio during the user's sliding zoom operation in real time or at a fixed time, and send the target zoom ratio to the hardware abstraction layer. The hardware abstraction layer obtains the two adjacent target zoom ratios sent down, and calculates the difference value between the two target zoom ratios. The difference value between different target zoom ratios is used to measure the change range of the target zoom ratio. The difference value can be the difference, ratio or bit operation result between adjacent target zoom ratios. Correspondingly, the difference threshold can be the difference threshold, ratio threshold or bit operation result threshold between adjacent target zoom ratios. The difference threshold can be preset or dynamically adjusted based on the scene. The difference value is compared with the difference threshold. If the difference value is less than the difference threshold, it can be determined that the zoom speed information is the first zoom speed type. If the difference value is greater than or equal to the difference threshold, it is determined that the zoom speed information is the second zoom speed type.

[0064] In this embodiment, a difference value is determined based on the respectively acquired target zoom magnifications, and then the difference value is compared with the difference threshold value, so as to perform qualitative analysis more efficiently. That is, when the difference value of different target zoom magnifications is small, the camera of the electronic device is in a slow zoom type, and when the difference value of different target zoom magnifications is large, the camera of the electronic device is in a fast zoom type.

[0065] In some embodiments, the current zoom ratio is used to determine the light change point identification information, including: when the current zoom ratio is greater than the target zoom ratio, determining the light change point identification information according to the current zoom ratio.

[0066] The situation where the current zoom ratio is greater than the target zoom ratio refers to a process in which the size of the object in the current image frame is reduced and the field of view angle is increased through the zoom operation.

[0067] In some embodiments, when the current zoom ratio is greater than the target zoom ratio, light change point identification information is determined according to the current zoom ratio, including: when the current zoom ratio is greater than the target zoom ratio and there is a light change point between the current zoom ratio and the target zoom ratio, light change point identification information is obtained to indicate that the zoom process passes through the light change point; when the current zoom ratio is greater than the target zoom ratio and there is no light change point between the current zoom ratio and the target zoom ratio, light change point identification information is obtained to indicate that the zoom process does not pass through the light change point. Thus, the light change point identification information is accurately determined by the two comparison results between the current zoom ratio and the target zoom ratio.

[0068] In some embodiments, when the current zoom ratio is greater than the target zoom ratio, determining the light change point identification information according to the current zoom ratio includes: when the current zoom ratio is greater than the target zoom ratio, if there is a light change point between the current zoom ratio and the target zoom ratio, obtaining the light change point identification information for indicating that the zoom process passes through the light change point; if there is no light change point between the current zoom ratio and the target zoom ratio, obtaining the light change point identification information for indicating that the zoom process does not pass through the light change point. Thus, the two comparison results between the current zoom ratio and the target zoom ratio are determined in sequence to accurately and orderly determine the light change point identification information.

[0069] In this embodiment, when the current zoom ratio is greater than the target zoom ratio, the field of view angle gradually increases, and thus the current image frame reflects less information than the image frame of the target zoom ratio; in this case, it is necessary to determine the light change point identification information based on the current zoom ratio in order to accurately determine whether to increase the number of interpolated frames based on the light change point identification information, thereby reserving the start time for the camera to be switched.

[0070] In some embodiments, the method further includes: when the current zoom ratio is less than the target zoom ratio, obtaining an initial number of interpolated frames; and processing the image using the zoom ratio of the initial number of interpolated frames.

[0071] The current zoom ratio is less than the target zoom ratio, which means that the object size in the current image frame is enlarged and the field of view is reduced through the zoom operation. In this case, even if the secondary camera is not pulled up in time, the FOV is sufficient to support cropping, so even if the light change point is delayed or misjudged, the smoothness of the image can be guaranteed.

[0072] The initial interpolation frame number is the interpolation frame number related to the zoom speed type, and the initial interpolation frame number includes at least a first initial interpolation frame number and a second initial interpolation frame number. The first zoom speed type belongs to a slow zoom scene, and the second zoom speed type belongs to a fast zoom scene; the first initial interpolation frame number is the interpolation frame number under the first zoom speed type, and the second initial interpolation frame number is the interpolation frame number under the second zoom speed type.

[0073] Exemplarily, the collected image data is processed at each zoom ratio based on the initial number of interpolated frames to obtain image data at each transitional zoom ratio; the field of view angle is determined based on the image data at each transitional zoom ratio; and the electronic device is controlled to display the image frame at the field of view angle.

[0074] In this embodiment, when the current zoom ratio is less than the target zoom ratio, the field of view angle gradually decreases, so that the current image frame reflects more information than the image frame of the target zoom ratio; in this case, image acquisition can be performed by digital zoom to achieve the smoothness of the corresponding image zoom process.

[0075] In some embodiments, Figure 5 As shown, in step 406, the target number of interpolated frames is determined according to the light change point identification information and the zoom speed information, including:

[0076] Step 502 : when the light change point identification information indicates that the zoom process does not pass through the light change point, determine the target number of interpolation frames according to the number of interpolation frames corresponding to the zoom speed information.

[0077] The number of interpolated frames corresponding to the zoom speed information is the number of interpolated frames adjusted based on the zoom speed information; the number of interpolated frames corresponding to the zoom speed information may be an actual number or a number represented by the zoom speed information.

[0078] For example, the zoom speed information can be mapped to obtain the number of interpolated frames corresponding to the zoom speed information; the calculation method of the target number of interpolated frames can also be adjusted based on the zoom speed information to obtain the zoom ratio interpolation method corresponding to the zoom speed; and then the zoom ratio of the target number of interpolated frames can be obtained through the zoom ratio interpolation method corresponding to the zoom speed; in this case, the number of interpolated frames corresponding to the zoom speed information is represented by the zoom speed information, and is the number of interpolated frames adjusted based on the zoom ratio interpolation method corresponding to the zoom speed.

[0079] Step 504 , when the light change point identification information indicates that the zoom process passes through the light change point, obtain the interpolation frame increment corresponding to the light change point, and obtain the target interpolation frame number according to the interpolation frame increment and the interpolation frame number corresponding to the zoom speed information.

[0080] The interpolation frame increment corresponding to the light change point is data used to increase the number of interpolation frames. The interpolation frame increment can be an actual number, a parameter used to adjust the calculation method of the target interpolation frame number, or a preset calculation method.

[0081] Exemplarily, the electronic device may obtain the interpolation increment corresponding to the light change point according to the mapping table, and add the interpolation increment corresponding to the light change point and the number of interpolation frames corresponding to the zoom speed information to obtain the target number of interpolation frames. The electronic device may also adjust the zoom ratio interpolation mode corresponding to the zoom speed information based on the interpolation increment corresponding to the light change point to obtain the zoom ratio interpolation mode corresponding to the light change point; and then obtain the zoom ratio of the target number of interpolation frames through the zoom ratio interpolation mode corresponding to the light change point; in this case, the interpolation increment corresponding to the light change point is a parameter for adjusting the calculation method of the target number of interpolation frames.

[0082] In addition, when the interpolation frame increment corresponding to the light change point is a preset calculation method, the interpolation frame increment corresponding to the light change point and the number of interpolation frames corresponding to the zoom speed information are functions of the zoom curve under the corresponding scene type, and the target number of interpolation frames is obtained by the function of the zoom curve.

[0083] In this embodiment, when the zoom process does not pass through the light change point, the light change point identification information does not affect the target number of interpolated frames. The target number of interpolated frames is determined solely based on the number of interpolated frames corresponding to the zoom speed information to ensure processing efficiency and avoid excessive power consumption due to the secondary camera pulling up the focal length too long, so as to achieve a balance between smoothness and power consumption. When the zoom process passes through the light change point, the focal length needs to be switched. Therefore, the number of interpolated frames corresponding to the zoom speed information is increased through the interpolated frame increment corresponding to the light change point. On the one hand, the camera that is not turned on and to be switched has sufficient startup time to ensure that the switching process of different image frames is smoother; on the other hand, the image alignment and fusion of the images collected by different cameras have a better effect of dynamic effect changes to ensure smoother image frames.

[0084] In some embodiments, the method for determining the number of interpolation frames corresponding to the zoom speed information includes: when the zoom speed information represents a first zoom speed type, obtaining a first initial number of interpolation frames, and obtaining a target number of interpolation frames based on the first initial number of interpolation frames. When the zoom speed information represents a second zoom speed type, obtaining a second initial number of interpolation frames, and obtaining a target number of interpolation frames based on the second initial number of interpolation frames; wherein the first initial number of interpolation frames is less than the second initial number of interpolation frames; and the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

[0085] The first zoom speed type and the second zoom speed type are different zoom speed information. The zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type, so the first zoom speed type belongs to a slow zoom scene, and the second zoom speed type belongs to a fast zoom scene. Exemplarily, when the zoom operation is a sliding operation, the first zoom speed type may be a slow sliding zoom scene, and the second zoom speed type may be a fast sliding zoom scene.

[0086] The first initial interpolation number is the number of interpolation frames under the first zoom speed type, and the second initial interpolation number is the number of interpolation frames under the second zoom speed type. The first initial interpolation number can be determined based on a function of the first zoom curve, and the second initial interpolation number can be determined based on a function of the second zoom curve. The first initial interpolation number and the second initial interpolation number can also be determined based on a mapping table; the first initial interpolation number in the mapping table can fit the first zoom curve, and the second initial interpolation number can fit the second zoom curve. Exemplarily, the first zoom curve is a curve characterized by an exponential function, and the second zoom curve is a curve characterized by a linear function.

[0087] Since the first initial interpolation frame number is smaller than the second initial interpolation frame number, the interpolation frame number obtained based on the first initial interpolation frame number can be smaller than the interpolation frame number obtained based on the second initial interpolation frame number.

[0088] Wherein, when the zoom operation belongs to the first zoom speed type, the first initial interpolation frame number may be used as the interpolation frame number corresponding to the zoom speed information; or the first initial interpolation frame number and other parameters may be used to filter out the interpolation frame number corresponding to the zoom speed information. When the zoom operation belongs to the second zoom speed type, the second initial interpolation frame number may be used as the interpolation frame number corresponding to the zoom speed information; or the second initial interpolation frame number and other parameters may be used to filter out the interpolation frame number corresponding to the zoom speed information.

[0089] In this embodiment, the zoom speed information has at least two zoom speed types, and the zoom speed corresponding to each zoom speed type is positively correlated with the number of initial interpolation frames. Therefore, when the zoom speed is relatively slow, the number of initial interpolation frames is relatively small, which not only makes the image alignment and fusion of the images collected by different cameras better, but also avoids the waste of computing power to ensure smoother image frames. Therefore, when the zoom speed is relatively fast, the number of target interpolation frames is larger, so the startup time of the target camera to be switched is more sufficient to ensure a smoother switching process of different image frames.

[0090] In some embodiments, in the zoom ratio sequence corresponding to the second initial interpolation frame number, the difference between adjacent zoom ratios in the first N zoom ratios is smaller than the difference between adjacent zoom ratios in the zoom ratios after the Nth one.

[0091] The zoom ratio sequence is the zoom ratio arranged in time. During a zoom operation, the first zoom ratio in the zoom ratio sequence can be the current zoom ratio or the zoom ratio determined by interpolation after the current zoom ratio; the last zoom ratio in the zoom ratio sequence can be the target zoom ratio or the last zoom ratio determined by interpolation before the target zoom ratio.

[0092] N is a positive integer greater than or equal to 2; the first N zoom magnifications include at least two zoom magnifications, so there is a concept of adjacent zoom magnifications; adjacent zoom magnifications can be any two zoom magnifications at adjacent moments. The zoom magnifications after the Nth may include the Nth zoom magnification, and may also include the N+Mth zoom magnification, where M is a positive integer. The difference between adjacent zoom magnifications may be a numerical difference in the zoom magnifications, or may be a normalized difference.

[0093] For example, the first two frames are slowed down, i.e. the zoom ratio changes slowly, and then a large span mapping is performed two frames later. When the zoom ratio is reduced (zoomin), the interpolation curve composed of the zoom ratio and the number of image frames is as follows: Figure 6 As shown in (a) in FIG. 14 , the zoom ratio corresponding to the 14th image frame is 3X; when the zoom ratio is reduced (zoom out), the interpolation curve composed of the zoom ratio and the number of image frames is as follows: Figure 6 As shown in (b) in FIG. 1 , the zoom ratio corresponding to the 14th image frame is 1X.

[0094] In this embodiment, in the initial stage of rapid zooming, the zoom ratio changes relatively slowly, which allows the image field angle to change more smoothly. After the initial stage of rapid zooming, the zoom ratio is adjusted quickly so that the target zoom ratio can be reached more quickly, thereby ensuring that the zoom ratio adjustment efficiency is high.

[0095] In some embodiments, the method also includes: when the zoom speed information represents a second zoom speed type, determining a target light change point according to the current zoom ratio and the target zoom ratio; determining a camera to be switched according to the target light change point; and controlling the camera to be switched to be in an on state.

[0096] The camera to be switched includes an image sensor. The camera to be switched includes at least a target camera whose focal length includes a target zoom ratio; and when the zooming process passes through a light change point, the camera to be switched may also include an intermediate camera whose focal length is between the current camera and the target camera.

[0097] The target light change point is a light change point that needs to be passed during the zooming process. The target light change point can be determined based on a focal length to which the current zoom ratio belongs and a focal length between the focal lengths to which the target zoom ratio belongs; or based on a difference or ratio between the current zoom ratio and the target zoom ratio.

[0098] Controlling the camera to be switched to be in an on state at least includes turning on the camera to be switched that is in an unturned state; and may also include controlling the camera to be switched that is in an on state to remain in an on state.

[0099] For example, the camera to be switched can be determined based on the latest target zoom ratio and the current zoom ratio, and the camera passed by the zoom process and the target camera where the final target zoom ratio is located are used as the camera to be switched, while keeping these cameras in the turned-on state. In addition, if these cameras are already in the turned-on state, their states are maintained unchanged.

[0100] In this embodiment, since the fast zoom scene has a fast zoom rate and the target zoom ratio may change, the target light change point is determined in advance, and the camera to be switched is determined by the light change point, so that the start-up node of the camera to be switched is earlier than the current image frame display node of the electronic device, thereby reducing the startup time to ensure that the switching process of different image frames is smoother.

[0101] In some embodiments, obtaining an interpolation frame increment corresponding to a light change point includes: when the zoom speed information represents a first zoom speed type, obtaining a first interpolation frame increment corresponding to the light change point; when the zoom speed information represents a second zoom speed type, obtaining a second interpolation frame increment corresponding to the light change point; wherein the number of interpolation frames increased by the first interpolation frame increment is greater than the number of interpolation frames increased by the second interpolation frame increment.

[0102] The first interpolation frame increment is an interpolation frame increment under the first zoom speed type, and the target interpolation frame number under the first zoom speed type can be determined by the first interpolation frame increment and the first initial interpolation frame number. The second interpolation frame increment is an interpolation frame increment under the second zoom speed type, and the target interpolation frame number under the second zoom speed type can be determined by the second interpolation frame increment and the second initial interpolation frame number.

[0103] Since the number of interpolation frames adjusted higher by the first interpolation frame increment is greater than the number of interpolation frames adjusted higher by the second interpolation frame increment, when the zoom process passes through the light change point, the amplitude of adjusting the first initial interpolation frame number by the first interpolation frame increment is greater than the amplitude of adjusting the second initial interpolation frame number by the second interpolation frame increment.

[0104] For example, regardless of whether the zoom ratio is in a zooming-in or zooming-out scenario, when the slow zoom process passes through the light changing point, 3 frames are added through the first interpolation increment to increase the smoothing effect; regardless of whether the zoom ratio is in a zooming-in or zooming-out scenario, when the fast zoom process passes through the light changing point, 1 frame is added through the second interpolation increment to wait for the camera to be switched to enter the on state.

[0105] In this embodiment, when the zoom process passes through the light change point and the zoom is slow, the number of interpolated frames adjusted by the first interpolated frame increment is relatively large, so that the alignment and fusion animation effect change effect is better, so that the zoom process information of the image frame can be displayed in more detail; when the zoom process passes through the light change point and the zoom is slow, the number of interpolated frames adjusted by the second interpolated frame increment is relatively small, so that the reserved time for the camera to be switched to enter the on state is more sufficient.

[0106] In some embodiments, after processing the target image using the zoom ratio of the target interpolation number, the method also includes: determining the number of tail interpolation frames corresponding to the end stage of the zoom process when the continuous and matching target zoom ratios reach a preset number; processing the tail image using the zoom ratio of the tail interpolation number; and capturing the tail image later than the target image.

[0107] The end stage of the zoom process is the stage where the zoom process stops. At the end stage of the zoom process, there is no new target zoom ratio, so the number of interpolation frames corresponding to the additional strategy can be used for smoothing. For example, an interpolation curve can be used to perform interpolation processing at the end to obtain the number of interpolation frames at the end to achieve interpolation at the end.

[0108] The tail interpolation number is the number of interpolation frames at the end of the zoom process, and no new target zoom ratio is detected. Exemplarily, compared with the beginning or other non-starting stages of the zoom process, the zoom ratio of the tail interpolation number is eased out and smoothed according to the tail curve.

[0109] The tail image is acquired and processed after the target image. The target image and the tail image are processed with different zoom magnifications, so the zoom speeds of the target image and the tail image are different.

[0110] Reaching the preset number of target zoom ratios includes at least two features, one feature is continuous and the other feature is matching; continuous means that the electronic device detects at adjacent moments, and matching means that the difference is equal or less than the preset target zoom ratio difference. For example, the target zoom ratios sent by the camera application three times in a row are all 1X, 1X, and 1X.

[0111] In some embodiments, determining the number of tail interpolation frames corresponding to the end stage of the zoom process includes: when the light change point identification information indicates that the zoom process does not pass through the light change point, obtaining the initial number of tail interpolation frames, and obtaining the number of tail interpolation frames based on the initial number of tail interpolation frames.

[0112] Exemplarily, the collected tail image data is processed at each zoom ratio based on the number of tail interpolation frames to obtain the tail image data at each transition zoom ratio; the field of view angle is determined based on the tail image data at each transition zoom ratio; and the electronic device is controlled to display the tail image frame at the field of view angle.

[0113] In this embodiment, the number of continuous and matching target zoom ratios is counted, and the end stage of the zoom process is determined based on the comparison result between the statistical result and the preset number, so as to change the interpolation strategy of the zoom ratio, and ensure the processing efficiency through the multiplexing process of the target zoom ratio.

[0114] In some embodiments, after processing the target image using the zoom ratio of the target interpolation frame number, the method also includes: determining the number of tail interpolation frames corresponding to the end stage of the zoom process when a stop event mark is detected; processing the tail image using the zoom ratio of the tail interpolation frame number; and the acquisition time of the tail image is later than the acquisition time of the target image.

[0115] The stop event flag is data used to indicate the end stage of the zoom process. The stop event flag can be, but is not limited to, an identifier, a comparison result, or a processing result of an event.

[0116] The tail image is acquired and processed after the target image. The target image and the tail image are processed with different zoom magnifications, so the zoom speeds of the target image and the tail image are different.

[0117] Exemplarily, the collected tail image data is processed at each zoom ratio based on the number of tail interpolation frames to obtain the tail image data at each transition zoom ratio; the field of view angle is determined based on the tail image data at each transition zoom ratio; and the electronic device is controlled to display the tail image frame at the field of view angle.

[0118] In this embodiment, the end stage of the zoom process is accurately determined by the stop event mark, so as to change the interpolation strategy of the zoom ratio, and the accuracy of identifying the end stage of the zoom process is guaranteed by the stop event mark.

[0119] In some embodiments, when the light change point identification information represents that the zoom process passes through the light change point, the number of tail interpolation frames corresponding to the end stage of the zoom process is determined, including: obtaining the initial number of tail interpolation frames and the tail interpolation frame increment corresponding to the light change point, and obtaining the number of tail interpolation frames according to the tail interpolation frame increment corresponding to the light change point and the initial number of tail interpolation frames.

[0120] The initial tail interpolation frame number may be the number of interpolation frames corresponding to the zoom ratio at the end of the zoom process when the light change point identification information indicates that the zoom process does not pass through the light change point. The initial tail interpolation frame number may be used to make the tail image present a slow-out effect relative to the target image. The initial tail interpolation frame number may be determined by a function of the tail curve.

[0121] The tail interpolation frame increment is data used to increase the number of interpolation frames, and the tail interpolation frame increment and the interpolation frame increment can be the same data or different data. The tail interpolation frame increment can be an actual number, a parameter used to adjust the calculation method of the target interpolation frame number, or a preset calculation method.

[0122] Exemplarily, the electronic device may obtain the tail interpolation increment corresponding to the light change point according to the mapping table, and add the tail interpolation increment corresponding to the light change point and the initial tail interpolation number to obtain the target interpolation number. The electronic device may also adjust the original tail zoom ratio interpolation method based on the tail interpolation increment corresponding to the light change point to obtain the tail zoom ratio interpolation method corresponding to the light change point; and then obtain the zoom ratio of the tail interpolation number through the tail zoom ratio interpolation method corresponding to the light change point; in this case, the tail interpolation increment corresponding to the light change point is a parameter used to adjust the calculation method of the tail interpolation number.

[0123] In this embodiment, when passing through the light change point, the initial number of tail interpolation frames is increased by the tail interpolation frame increment to obtain the number of tail interpolation frames, so as to make the image smoother at the end stage of the zoom process.

[0124] In some embodiments, the difference between the zoom ratio of the electronic device at the end of the zoom process and the target zoom ratio is negatively correlated with the number of tail interpolation frames. Therefore, if the difference at the end of the zoom process is too large, the number of tail interpolation frames can be increased by using a difference method, an average method, etc., so that a slow-out effect can be achieved at the end of the zoom process.

[0125] In some embodiments, the method further includes: when the current zoom ratio reaches the target zoom ratio, turning off each camera except the camera to which the current zoom ratio belongs.

[0126] The case where the current zoom ratio reaches the target zoom ratio may be a case where the current zoom ratio is equal to the target zoom ratio, or a case where the current zoom ratio and the target zoom ratio belong to the same focal length.

[0127] In this embodiment, when the current zoom ratio reaches the target zoom ratio, each camera and corresponding sensor other than the camera to which the current zoom ratio belongs are turned off, so as to reduce power consumption.

[0128] In some embodiments, turning off all cameras except the camera to which the current zoom ratio belongs includes: keeping each camera turned on according to the duration of the zoom operation until the zoom operation ends, and then turning off all cameras except the camera to which the current zoom ratio belongs.

[0129] In this embodiment, since zoom operations such as two-finger zoom and hardware zoom may update the target zoom ratio, the secondary camera closing time should be delayed according to the time the dial is kept pulled up to obtain the image under the dual cameras, thereby obtaining richer information.

[0130] In an exemplary embodiment, this solution is used for a sliding zoom scene, i.e., a sliding cut scene. In conventional technology, there is no clear processing strategy for the number of interpolated frames and the zoom curve used to represent the trend of field angle changes in the sliding cut zoom solution and experience. Therefore, when sliding, there may be a problem of smoothness with abrupt changes in the field angle, and the field angle change may be stuck or retracted due to the failure to pull up the secondary camera in time. In addition, due to the limitation of hardware resources, even if the three cameras are kept open for a long time, there is still a need to switch cameras in fast sliding scenes for scenes with reduced zoom ratios. In this process, one camera and the corresponding image sensor are turned off, and then the released resources are used to turn on the new camera and the corresponding image sensor. The closing and opening of the camera generally requires about 7 frames of images, so it is also necessary to ensure that the switching time is sufficient to avoid the field angle changing to the corresponding interval while the relevant camera has not been turned on in time, which will also cause problems. Although there is an interpolation solution for the sliding cut scene, it is only performed based on the expansion angle of the dial; in fact, the difference in the magnification of different intervals and whether the light change point is passed are different. Therefore, a more sophisticated interpolation strategy is needed for sliding cut scenarios to balance power consumption and hand tracking, as well as the smoothness of field of view angle changes.

[0131] Based on this, this embodiment refines the scene and focal length based on the three dimensions of light change point identification information, zoom speed information, and the direction of zoom ratio change, so as to differentiate the target zoom ratio for the sliding zoom scene and optimize the image processing during the zoom process. Prioritize some explicit needs and experiences to achieve a balance between hand tracking, smoothness, and power consumption.

[0132] The calculation of the initial number of interpolated frames and the initial number of interpolated frames at the end can be determined by the interpolation curve, i.e., the zoom curve; the zoom curve can be adjusted by the first N zoom ratios and the number of interpolated frames at the end, and the interpolation curve is designed in accordance with the principle of rapid and uniform change of the intermediate state, slow-in (fewer slow-in frames) and slow-out (more slow-out frames); the target number of interpolated frames and the number of interpolated frames at the end are determined by factors such as the light change point identification information, the zoom speed information, and the change in the size of the zoom ratio. For example, whether it is necessary to wait for the target lens to be pulled up is determined based on the light change point identification information to determine whether more interpolated frames are needed, so as to comprehensively ensure the overall zoom experience.

[0133] In an exemplary embodiment, when the user triggers a zoom operation, the zoom event sent by the camera application first determines whether it is a point-to-point zoom scene or a sliding zoom scene; if it is a sliding zoom scene, the interpolation strategy described in this solution is entered, and steps 402-406 are executed to obtain the target number of interpolation frames. If it is a point-to-point zoom scene, another solution is executed according to the point-to-point strategy. When it is detected that it is a sliding zoom scene, it is necessary to further identify and distinguish the scene according to the zoom ratio of the current image frame, the zoom ratio of the previous frame, and the latest target zoom ratio, and the stop event mark; after determining the scene, the zoom ratio of the next frame is calculated according to the determined interpolation strategy, and the cropping of the field of view angle is processed accordingly according to the calculated zoom ratio value; finally, when it is detected that the user has finished zooming, that is, the stop event mark of no longer sliding is sent by the camera application, the tail is slowed out according to the tail curve, and the number of tail interpolation frames is calculated.

[0134] The zoom ratio of the current frame is used to determine whether the lens has reached the light change point. The zoom ratio of the previous frame is used for error correction. The difference between the two latest target zoom ratios is compared with the threshold, and the zoom speed type is determined according to the interval of the comparison result.

[0135] In an exemplary embodiment, Figure 7 As shown, the embodiment in the sliding zoom scenario includes:

[0136] Step 701: Open the camera and start image preview or recording.

[0137] Step 702: Detect that the user adjusts the magnification (e.g., 0, 6X, 1X, 2X, 5X, 20X, etc.) or pulls up the magnification switching dial (slide switch), thereby triggering zoom.

[0138] Step 703: Perform zoom mode identification to determine the zoom scene type.

[0139] Zoom mode recognition can obtain point-cut scenes or slide-cut scenes.

[0140] Step 704: If it is a sliding cut scene, a target number of interpolated frames is obtained based on an interpolation strategy that determines whether the sliding cut zoom scene matches the current scene.

[0141] Step 705: Based on the zoom ratio calculated in step 704 and the equal division method or other methods, determine the cropping parameters of the next frame, and crop the image to obtain the final field of view angle.

[0142] Step 706: When a user zoom end event is detected, the interpolation frame curve is updated to a tail interpolation frame curve, and tail interpolation processing is performed to obtain the number of tail interpolation frames.

[0143] Step 707, finally resume normal preview or recording. That is, after the current zoom ratio reaches the target zoom ratio, turn off the secondary camera; turn off the interpolation processing algorithm and the spatial alignment algorithm, complete the entire spatial alignment switching process; and then continue the preview or shooting process.

[0144] In one embodiment, Figure 8 As shown in the figure, the angle between the two dotted parts is the opening and closing angle corresponding to the sliding cut. It can be noticed that even with the same opening and closing angle in the high and low magnification ranges, the multiples of the magnification change at the starting point and the landing point are different. For the low magnification range, even a small opening and closing angle will cause a significant change in the field of view. Therefore, it is unreasonable to interpolate frames based only on the opening and closing angle for different focal lengths, which will lead to inconsistent smoothness of the zoom curve. Therefore, interpolation is needed for transition.

[0145] In an exemplary embodiment, the interpolation strategy of the three dimensions of light change point identification information, zoom speed information and zoom ratio size change is as follows: Fig. 9 The mask value can be used to indicate the type of fast sliding zoom. If you slide quickly, you can zoom in on the three cameras. If you slide slowly, you can take your time.

[0146] For zoom magnification scenes, since the change in the field of view angle is a process of gradually decreasing, even if the secondary camera cannot pull up its field of view angle in time, it is sufficient to crop. For zoom magnification scenes, we only need to consider the hand tracking and the smoothness of the zoom. Generally speaking, the camera and the corresponding image sensor can be turned on before stopping the zoom. For zoom magnification and zoom reduction scenes, the slow sliding across the light change point is done by inserting multiple frames before the light change point. In this embodiment, the first insert frame increment can be a multi-frame zoom ratio of 3 frames, 4 frames, 5 frames, etc. The first insert frame increment is used to provide a smooth sliding effect, thereby providing more double-open frames for alignment and fusion animation.

[0147] For fast sliding scenes, firstly, the target camera and the corresponding image sensor are judged based on the latest zoom ratio, and the intermediate cameras and the corresponding image sensors and the target camera and the corresponding image sensor are pulled up at the same time. At this time, if the cameras and the corresponding image sensors that the target passes through and the target camera and the corresponding image sensor are already in the on state, their states are maintained unchanged. At the same time, when it is detected that the new target zoom ratio of the sliding cut crosses the light change point compared with the zoom ratio of the current frame, the number of interpolated frames is increased in response; it should be noted that the added frames in this scene need to be before the light change point; in general, at least one more interpolated frame is added before each light change point, that is, the second interpolated frame increment can be 1 frame, 2 frames, 3 frames, etc., but is not limited to this.

[0148] For fast sliding and zooming out scenes, firstly, based on the latest zoom ratio, the target camera and the corresponding image sensor are judged, and the intermediate cameras and the corresponding image sensors and the target camera and the corresponding image sensor are pulled up at the same time. At this time, if the cameras and the corresponding image sensors that the target passes through and the target camera and the corresponding image sensor are already in the on state, their states are maintained unchanged. At the same time, when it is detected that the new target zoom ratio of the sliding cut crosses the light change point compared with the zoom ratio of the current frame, the number of interpolated frames is increased in response; it should be noted that the added frames in this scenario need to be before the light change point; in general, one more interpolated frame needs to be added before each light change point, that is, the second interpolated frame increment is 1 frame.

[0149] Although both are fast sliding scenes, the interpolation strategies of fast sliding with zoom magnification and fast sliding with zoom reduction are also different when the light change point is not passed. This is mainly because the judgment of fast sliding is relatively accurate for sliding cuts, but it is not accurate to judge whether it will pass the light change point. For the zoom magnification scene, even if the judgment is delayed or misjudged, even if the secondary camera is not pulled up in time, the field of view is sufficient to support cropping, so the impact is not significant; but for the zoom reduction scene, it is possible that the fast sliding will not pass the light change point, which means that the zoom reduction scene is an intermediate state where the light change point is quickly passed by the scene, that is, the subsequent sliding may still pass the light change point. Therefore, when inserting frames, the tail interpolation cannot be performed only according to the smoothness standard to achieve smooth interpolation. After determining the target number of interpolation frames for the zoom ratio, the slope of the interpolation curve in the rapid change process can be determined by the equal division method or other methods until the app sends 2 or 3 consistent zoom ratio values ​​in succession, and it is determined that the electronic device has entered the end stage of the zoom process, and then the tail smooth interpolation is performed. Through this solution, it is possible to determine whether the sliding scene passes through the light change point.

[0150] In an exemplary embodiment, the end interpolation curve of the fast sliding scene is as follows: Fig.10As shown in the figure, the interpolation frames in the front part of the interpolation curve are uniform interpolation frames. When the camera application sends the slide stop event, the zoom ratio is smoothly interpolated. Therefore, this requires that the slide stop event should also be sent in a timely manner based on the hardware abstraction layer (HAL) interface. In addition, the target zoom ratios in the figure are different, and the time to enter the end stage of the zoom process is different. In this case, the difference between the zoom ratio entering the end stage of the zoom process and the target zoom ratio is negatively correlated with the number of tail interpolation frames.

[0151] In addition, for the interpolation of sliding scenes, interpolation is not performed every time a new zoom ratio sent by the camera application is detected; in the case of the first zoom rate type, if the difference is less than 0.1, no interpolation is performed to determine the zoom ratio. At this time, the change in the field of view angle is slow enough, and the relationship of the multiple can be directly added to capture images according to the multiple provided by the user.

[0152] Based on this, and unlike the current scheme of spatial alignment and sliding cutting that only inserts frames based on angles, this embodiment performs targeted differentiated segmented and scene-by-scene optimization based on the characteristics of the complexity of the sliding cutting scene (random stop point, random sliding speed, and random sliding direction). First, it ensures the stopping tracking and smoothness of the field angle change in the zoom magnification scene and the zoom magnification reduction scene that does not pass the light change point. Secondly, for the slow sliding scene, multiple frames are inserted before the light change point to make the alignment and dynamic effects have more double frames, thereby obtaining better alignment and dynamic effects. Finally, for the complex sliding cutting scene where the zoom magnification is reduced and the light change point is quickly passed, multiple frames are inserted evenly before the light change point to buy time for the secondary camera to be turned on, so as to avoid excessive power consumption due to the secondary camera pulling up the focal length too long, thereby achieving a balance between smoothness and power consumption.

[0153] In this case, some virtual transition frames are inserted in the process of spatial alignment switching based on the characteristics of scene changes to make the switching effect smoother and improve the user experience. That is, by subdividing and sorting the sliding scenes, distinguishing scenes such as zoom magnification, zoom magnification reduction, fast sliding, slow sliding, and light transition points, and then implementing differentiated interpolation strategies to achieve a balance between smoothness and hand tracking; by considering the stop hand tracking and the smoothness of the field of view change, differentiated interpolation strategies are implemented for different sliding scenes, and the implementation of the strategy takes into account the adjustment of the curve.

[0154] Exemplarily, based on the comparison between the zoom ratio of the current frame and the zoom ratio of the previous frame, combined with the change of the target zoom ratio and the stop sliding event sent by the app, different sliding scenes (zoom ratio enlargement, zoom ratio reduction, fast sliding, slow sliding, light change point, etc.) are comprehensively judged as the basis for subsequent frame insertion and curve adjustment. At the same time, the differentiated frame insertion strategy, that is, the zoom ratio mapping strategy, needs to be adjusted with emphasis on the points of different scenes; for example, for ordinary zoom ratio enlargement scenes, since there is no situation of insufficient field of view, the tracking of the stop and the smoothness of the field of view change can be focused on; and for zoom ratio reduction scenes, especially fast sliding zoom ratio reduction across light change points, it is necessary to focus on the number of frames inserted to buy time for the secondary camera to open, so as to avoid insufficient field of view cutting due to the untimely pulling up of the secondary camera, so as to avoid image twitching or pauses.

[0155] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0156] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0157] In an exemplary embodiment, Fig.11 As shown, an image processing device is provided, comprising:

[0158] The acquisition module 1102 is used to obtain the current zoom ratio corresponding to the current image frame and the target zoom ratio issued by the camera application in response to the zoom operation; wherein the current zoom ratio is used to determine the light change point identification information, and the target zoom ratio is used to determine the zoom speed information;

[0159] A scene module 1104 is used to determine a target number of interpolated frames according to the light change point identification information and the zoom speed information;

[0160] The processing module 1106 is used to process the target image using the zoom factor of the target interpolation frame number.

[0161] In one embodiment, the scenario module 1104 is used to:

[0162] When the light change point identification information indicates that the zoom process does not pass through the light change point, determining a target number of interpolation frames according to the number of interpolation frames corresponding to the zoom speed information;

[0163] When the light change point identification information indicates that the zoom process passes through the light change point, an interpolation frame increment corresponding to the light change point is obtained, and a target interpolation frame number is obtained according to the interpolation frame increment and the interpolation frame number corresponding to the zoom speed information.

[0164] In one embodiment, the scenario module 1104 is used to:

[0165] In a case where the zoom speed information represents a first zoom speed type, obtaining a first initial interpolation frame number, and obtaining an interpolation frame number corresponding to the zoom speed information based on the first initial interpolation frame number;

[0166] In a case where the zoom speed information represents a second zoom speed type, obtaining a second initial interpolation frame number, and obtaining an interpolation frame number corresponding to the zoom speed information based on the second initial interpolation frame number;

[0167] Among them, the first initial interpolation frame number is smaller than the second initial interpolation frame number; the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

[0168] In one of the embodiments, in the zoom ratio sequence corresponding to the second initial interpolation frame number, the difference between adjacent zoom ratios in the first N zoom ratios is smaller than the difference between adjacent zoom ratios in the zoom ratios after the Nth one.

[0169] In one embodiment, the processing module 1106 is used to:

[0170] In a case where the zoom speed information represents a second zoom speed type, determining a target light changing point according to the current zoom magnification and the target zoom magnification;

[0171] Determining a camera to be switched according to the target light change point;

[0172] Control the camera to be switched to be in an on state.

[0173] In one embodiment, the scenario module 1104 is used to:

[0174] When the zoom speed information represents the first zoom speed type, obtaining a first interpolation frame increment corresponding to the light change point;

[0175] When the zoom speed information represents the second zoom speed type, obtaining a second interpolation frame increment corresponding to the light change point;

[0176] The number of interpolation frames increased by the first interpolation frame increment is greater than the number of interpolation frames increased by the second interpolation frame increment.

[0177] In one embodiment, the acquisition module 1102 is used to:

[0178] In the case where there is a light change point between the current zoom magnification and the target zoom magnification, obtaining light change point identification information for indicating that the zoom process passes through the light change point;

[0179] In the case that there is no light changing point between the current zoom magnification and the target zoom magnification, light changing point identification information is obtained for indicating that the zooming process does not pass through the light changing point.

[0180] In one embodiment, the acquisition module 1102 is used to:

[0181] Comparing the difference between the target zoom factors with a difference threshold;

[0182] If the difference value is less than the difference threshold, determining that the zoom speed information is a first zoom speed type;

[0183] If the difference value is greater than or equal to the difference threshold, the zoom speed information is determined to be a second zoom speed type.

[0184] In one embodiment, the acquisition module 1102 is used to:

[0185] When the current zoom magnification is greater than the target zoom magnification, light changing point identification information is determined according to the current zoom magnification.

[0186] In one embodiment, the processing module 1106 is used to:

[0187] When the current zoom ratio is less than the target zoom ratio, obtaining the initial number of interpolated frames;

[0188] The image is processed using the zoom factor of the initial interpolation frame number.

[0189] In one embodiment, after processing the target image using the zoom factor of the target interpolation number, the acquisition module 1102 is used to:

[0190] When the target zoom ratios that are continuous and matched reach a preset number, determining the number of tail interpolation frames corresponding to the end stage of the zoom process;

[0191] The tail image is processed using the zoom ratio of the tail interpolation frame number; the acquisition time of the tail image is later than the acquisition time of the target image.

[0192] In one embodiment, after processing the target image using the zoom factor of the target interpolation number, the acquisition module 1102 is used to:

[0193] When the stop event flag is detected, determining the number of tail interpolation frames corresponding to the end stage of the zoom process;

[0194] The tail image is processed using the zoom ratio of the tail interpolation frame number; the acquisition time of the tail image is later than the acquisition time of the target image.

[0195] In one embodiment, when the light change point identification information indicates that the zoom process passes through the light change point, the acquisition module 1102 is used to:

[0196] The initial number of tail interpolation frames and the tail interpolation frame increment corresponding to the light change point are obtained, and the number of tail interpolation frames is obtained according to the tail interpolation frame increment corresponding to the light change point and the initial number of tail interpolation frames.

[0197] In one embodiment, the processing module 1106 is used to:

[0198] When the current zoom ratio reaches the target zoom ratio, each camera except the camera to which the current zoom ratio belongs is turned off.

[0199] In one embodiment, the processing module 1106 is used to:

[0200] The cameras are kept turned on according to the duration of the zoom operation until the zoom operation ends, and then the cameras except the camera to which the current zoom ratio belongs are turned off.

[0201] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0202] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.12As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0203] Those skilled in the art will understand that Fig.12 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0204] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0205] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0206] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0207] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0208] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0209] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0210] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: In response to the zoom operation, obtaining a current zoom ratio corresponding to the current image frame and a target zoom ratio issued by the camera application, wherein the current zoom ratio is used to determine the light change point identification information, and the target zoom ratio is used to determine the zoom speed information; Determining a target number of interpolated frames according to the light change point identification information and the zoom speed information; The target image is processed using the zoom factor of the target interpolation frame number.

2. The method according to claim 1, characterized in that: The determining a target number of interpolated frames according to the light change point identification information and the zoom speed information includes: When the light change point identification information indicates that the zoom process does not pass through the light change point, determining a target number of interpolation frames according to the number of interpolation frames corresponding to the zoom speed information; When the light change point identification information indicates that the zoom process passes through the light change point, an interpolation frame increment corresponding to the light change point is obtained, and a target interpolation frame number is obtained according to the interpolation frame increment and the interpolation frame number corresponding to the zoom speed information.

3. The method according to claim 2, characterized in that The method for determining the number of interpolated frames corresponding to the zoom speed information includes: In a case where the zoom speed information represents a first zoom speed type, obtaining a first initial interpolation frame number, and obtaining an interpolation frame number corresponding to the zoom speed information based on the first initial interpolation frame number; When the zoom speed information represents a second zoom speed type, obtaining a second initial interpolation frame number, and obtaining an interpolation frame number corresponding to the zoom speed information based on the second initial interpolation frame number; Among them, the first initial interpolation frame number is smaller than the second initial interpolation frame number; the zoom speed corresponding to the first zoom speed type is lower than the zoom speed corresponding to the second zoom speed type.

4. The method according to claim 3, characterized in that: In the zoom ratio sequence corresponding to the second initial interpolation frame number, the difference between adjacent zoom ratios in the first N zoom ratios is smaller than the difference between adjacent zoom ratios in the zoom ratios after the Nth one.

5. The method according to claim 3, characterized in that: The method further comprises: In a case where the zoom speed information represents a second zoom speed type, determining a target light changing point according to the current zoom magnification and the target zoom magnification; Determining a camera to be switched according to the target light change point; Control the camera to be switched to be in an on state.

6. The method according to claim 3, characterized in that The step of obtaining the interpolation frame increment corresponding to the light change point includes: When the zoom speed information represents the first zoom speed type, obtaining a first interpolation frame increment corresponding to the light change point; When the zoom speed information represents the second zoom speed type, obtaining a second interpolation frame increment corresponding to the light change point; The number of interpolation frames increased by the first interpolation frame increment is greater than the number of interpolation frames increased by the second interpolation frame increment.

7. The method according to claim 1, characterized in that The current zoom factor is used to determine the light change point identification information, including: In the case where there is a light change point between the current zoom magnification and the target zoom magnification, obtaining light change point identification information for indicating that the zoom process passes through the light change point; In the case that there is no light changing point between the current zoom magnification and the target zoom magnification, light changing point identification information is obtained for indicating that the zooming process does not pass through the light changing point.

8. The method according to claim 1, characterized in that: The target zoom ratio is used to determine the zoom speed information, including: Comparing the difference between the target zoom factors with a difference threshold; If the difference value is less than the difference threshold, determining that the zoom speed information is a first zoom speed type; If the difference value is greater than or equal to the difference threshold, it is determined that the zoom speed information is a second zoom speed type.

9. The method according to claim 1, characterized in that: The current zoom factor is used to determine the light change point identification information, including: When the current zoom magnification is greater than the target zoom magnification, light changing point identification information is determined according to the current zoom magnification.

10. The method according to claim 9, characterized in that The method further comprises: When the current zoom ratio is less than the target zoom ratio, obtaining the initial number of interpolated frames; The image is processed using the zoom factor of the initial interpolation frame number.

11. The method according to claim 1, characterized in that After processing the target image using the zoom factor of the target interpolation frame number, the method further includes: When the target zoom ratios that are continuous and matched reach a preset number, determining the number of tail interpolation frames corresponding to the end stage of the zoom process; The tail image is processed using the zoom ratio of the tail interpolation frame number; the acquisition time of the tail image is later than the acquisition time of the target image.

12. The method according to claim 1, characterized in that After processing the target image using the zoom factor of the target interpolation frame number, the method further includes: When the stop event flag is detected, determining the number of tail interpolation frames corresponding to the end stage of the zoom process; The tail image is processed using the zoom ratio of the tail interpolation frame number; the acquisition time of the tail image is later than the acquisition time of the target image.

13. The method according to claim 11 or 12, characterized in that: In a case where the light change point identification information indicates that the zoom process passes through the light change point, determining the number of tail interpolation frames corresponding to the end stage of the zoom process includes: The initial number of tail interpolation frames and the tail interpolation frame increment corresponding to the light change point are obtained, and the number of tail interpolation frames is obtained according to the tail interpolation frame increment corresponding to the light change point and the initial number of tail interpolation frames.

14. The method according to claim 1, characterized in that The method further comprises: When the current zoom ratio reaches the target zoom ratio, each camera except the camera to which the current zoom ratio belongs is turned off.

15. The method according to claim 14, characterized in that The step of shutting down each camera except the camera to which the current zoom ratio belongs includes: The cameras are kept turned on according to the duration of the zoom operation until the zoom operation ends, and then the cameras except the camera to which the current zoom ratio belongs are turned off.

16. An image processing device, characterized in that: The device comprises: An acquisition module, used to obtain, in response to a zoom operation, a current zoom ratio corresponding to a current image frame and a target zoom ratio issued by a camera application; wherein the current zoom ratio is used to determine light change point identification information, and the target zoom ratio is used to determine zoom speed information; A scene module, used to determine a target number of interpolated frames according to the light change point identification information and the zoom speed information; The processing module is used to process the target image using the zoom factor of the target interpolation frame number.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images