Image display method, device, apparatus, storage medium, and program product

Abstract

The present disclosure relates to an image display method, device, equipment, storage medium and program product. According to an example of the present disclosure, the method comprises: in response to detecting that a camera switching event occurs, acquiring a first image collected by a first camera and a second image collected by a second camera, the first camera being a camera before switching, and the second camera being a camera after switching; determining a target optical flow deformation field based on the first image and the second image; determining a transition image required for switching the first image to the second image based on the target optical flow deformation field and the first image; and switching the first image to the second image through the transition image in an image preview area. The present disclosure can improve the fluency of the image switching process, realize smooth zooming, and thus can improve the user's shooting experience.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image display method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of smartphone and other electronic device technologies, configuring multiple cameras to improve image quality has become a mainstream trend. For example, electronic devices can be equipped with one or more cameras such as ultra-wide, wide, telephoto, and ultra-telephoto.

[0003] In related technologies, when electronic devices zoom using multiple cameras, switching between two cameras with adjacent focal lengths can be achieved by switching between them. For example, when a user needs to switch from a close-up to a distant view, they can control the electronic device to switch the camera from wide-angle to telephoto. However, during the zoom process described above, there may be abrupt changes in the image content or size between different cameras, resulting in less smooth zooming between multiple cameras and thus affecting the user's shooting experience. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides an image display method, apparatus, device, storage medium, and program product to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, an image display method is provided, the method comprising:

[0006] In response to detecting a camera switching event, a first image captured by a first camera and a second image captured by a second camera are acquired, wherein the first camera is the camera before the switch and the second camera is the camera after the switch.

[0007] Determine the target optical flow deformation field based on the first image and the second image;

[0008] Based on the target optical flow deformation field and the first image, determine the transition image required to switch from the first image to the second image;

[0009] In the image preview area, the first image is switched to the second image via the transition image.

[0010] In some embodiments, determining the target optical flow deformation field based on the first image and the second image includes:

[0011] The fourth image is preprocessed based on the third image to obtain a preprocessed image. The third image is the image with a smaller field of view between the first image and the second image, and the fourth image is the image with a larger field of view between the first image and the second image. The preprocessed image is aligned with the field of view of the third image and has the same size as the fourth image.

[0012] The preprocessed image and the third image are input into a pre-constructed recursive cascaded network to obtain the target optical flow deformation field, which is used to describe the pixel position transformation information from the preprocessed image to the third image.

[0013] In some embodiments, the preprocessing of the fourth image based on the third image includes:

[0014] A cropped image aligned with the field of view of the third image is extracted from the fourth image;

[0015] The cropped image is restored to the size of the fourth image to obtain the preprocessed image.

[0016] In some embodiments, the recursive cascaded network comprises an N-level recursive network, where N is a positive integer greater than or equal to 3;

[0017] The step of inputting the preprocessed image and the third image into a pre-constructed recursive cascade network to obtain the target optical flow deformation field includes:

[0018] The preprocessed image and the third image are input into the first-level recursive network for processing to obtain the first optical flow field. Based on the first optical flow field, the preprocessed image is warped to obtain the first-level warped image.

[0019] The first-level deformed image and the third image are input into the second-level recursive network for processing to obtain the second optical flow field. Based on the second optical flow field, the first-level deformed image is deformed to obtain the second-level deformed image.

[0020] This process continues until the Nth level deformed image is obtained, at which point the similarity between the Nth level deformed image and the third image is determined.

[0021] In response to the similarity being less than a preset similarity threshold, the parameters of the N-level recursive network are updated through backpropagation;

[0022] The above process is repeated based on the updated N-level recursive network until the similarity between the N-level deformed image and the third image is greater than or equal to the preset similarity threshold. Then, the N-level optical flow field output by the N-level recursive network is determined as the target optical flow deformed field.

[0023] In some embodiments,

[0024] The step of determining the transition image required to switch from the first image to the second image based on the target optical flow deformation field and the first image includes:

[0025] Based on the current zoom ratio, the target optical flow deformation field is interpolated to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio. The interpolated optical flow deformation field is used to describe the pixel position transformation information of the transition image from the first image to the current zoom ratio.

[0026] Based on the first image and the interpolated optical flow deformation field, a transition image for the current zoom magnification is determined.

[0027] In some embodiments, determining the transition image for the current zoom magnification based on the first image and the interpolated optical flow deformation field includes:

[0028] According to the offset of each pixel in the interpolated optical flow deformation field, the position of the corresponding pixel in the first image is adjusted to obtain the transition image of the current zoom magnification.

[0029] In some embodiments, the type of the camera switching event includes at least one of the following:

[0030] Camera switching events triggered by shooting distance;

[0031] Camera switching events triggered by zoom adjustment operations.

[0032] According to a second aspect of the present disclosure, an image display apparatus is provided, the apparatus comprising:

[0033] The image acquisition module is used to acquire a first image captured by a first camera and a second image captured by a second camera in response to the detection of a camera switching event, wherein the first camera is the camera before the switch and the second camera is the camera after the switch;

[0034] A deformation field determination module is used to determine the target optical flow deformation field based on the first image and the second image;

[0035] A transition image determination module is used to determine, based on the target optical flow deformation field and the first image, the transition image required to switch from the first image to the second image;

[0036] An image switching module is used to switch the first image to the second image in the image preview area via the transition image.

[0037] In some embodiments, the deformation field determination module includes:

[0038] An image preprocessing unit is used to preprocess a fourth image based on a third image to obtain a preprocessed image. The third image is the image with a smaller field of view between the first image and the second image, and the fourth image is the image with a larger field of view between the first image and the second image. The preprocessed image is aligned with the field of view of the third image and has the same size as the fourth image.

[0039] The deformation field acquisition unit is used to input the preprocessed image and the third image into a pre-constructed recursive cascaded network to obtain the target optical flow deformation field, which is used to describe the pixel position transformation information from the preprocessed image to the third image.

[0040] In some embodiments, the image preprocessing unit is further configured to:

[0041] A cropped image aligned with the field of view of the third image is extracted from the fourth image;

[0042] The cropped image is restored to the size of the fourth image to obtain the preprocessed image.

[0043] In some embodiments, the recursive cascaded network comprises an N-level recursive network, where N is a positive integer greater than or equal to 3;

[0044] The deformation field acquisition unit is further used for:

[0045] The preprocessed image and the third image are input into the first-level recursive network for processing to obtain the first optical flow field. Based on the first optical flow field, the preprocessed image is warped to obtain the first-level warped image.

[0046] The first-level deformed image and the third image are input into the second-level recursive network for processing to obtain the second optical flow field. Based on the second optical flow field, the first-level deformed image is deformed to obtain the second-level deformed image.

[0047] This process continues until the Nth level deformed image is obtained, at which point the similarity between the Nth level deformed image and the third image is determined.

[0048] In response to the similarity being less than a preset similarity threshold, the parameters of the N-level recursive network are updated through backpropagation;

[0049] The above process is repeated based on the updated N-level recursive network until the similarity between the N-level deformed image and the third image is greater than or equal to the preset similarity threshold. Then, the N-level optical flow field output by the N-level recursive network is determined as the target optical flow deformed field.

[0050] In some embodiments,

[0051] The transition graph determination module includes:

[0052] The deformation field interpolation unit is used to interpolate the target optical flow deformation field based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio. The interpolated optical flow deformation field is used to describe the pixel position transformation information of the transition image from the first image to the current zoom ratio.

[0053] A deformation field interpolation unit is used to determine a transition image for the current zoom magnification based on the first image and the interpolated optical flow deformation field.

[0054] In some embodiments, the deformation field interpolation unit is further configured to adjust the position of the corresponding pixel in the first image according to the pixel offset in the interpolated optical flow deformation field, so as to obtain the transition image of the current zoom magnification.

[0055] In some embodiments, the type of the camera switching event includes at least one of the following:

[0056] Camera switching events triggered by shooting distance;

[0057] Camera switching events triggered by zoom adjustment operations.

[0058] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0059] Processor and memory used to store computer programs;

[0060] The processor is configured to implement the image display method described above when executing the computer program.

[0061] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the image display method described in any of the preceding claims.

[0062] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the image display method described in any of the preceding claims.

[0063] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0064] This disclosure improves the smoothness of the image switching process and achieves smooth zoom by responding to the detection of a camera switching event, acquiring a first image captured by a first camera and a second image captured by a second camera, determining a target optical flow deformation field based on the first image and the second image, and determining a transition image required to switch the first image to the second image based on the target optical flow deformation field and the first image, and then switching the first image to the second image through the transition image in the image preview area. This improves the user's shooting experience.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0067] Figure 1A This is a flowchart illustrating an image display method according to an exemplary embodiment of the present disclosure;

[0068] Figure 1B This is a schematic diagram illustrating an application scenario of a user performing zoom adjustment operation according to an exemplary embodiment of the present disclosure;

[0069] Figure 2 This is a flowchart illustrating how to determine a target optical flow deformation field based on the first image and the second image, according to an exemplary embodiment of this disclosure;

[0070] Figure 3A This is a flowchart illustrating an image display method according to an exemplary embodiment of the present disclosure;

[0071] Figure 3B This is a schematic diagram of a recursive cascaded network according to an exemplary embodiment of the present disclosure;

[0072] Figure 4 This is a flowchart illustrating how to determine the transition image required to switch from the first image to the second image, according to an exemplary embodiment of this disclosure;

[0073] Figure 5 This is a block diagram illustrating an image display device according to an exemplary embodiment of the present disclosure;

[0074] Figure 6 This is a block diagram illustrating yet another image display device according to an exemplary embodiment of the present disclosure;

[0075] Figure 7This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] Figure 1A This is a flowchart illustrating an image display method according to an exemplary embodiment. The method of this embodiment can be executed by an image display device, which can be configured in an electronic device with multiple cameras, such as a mobile terminal (e.g., mobile phone, tablet computer, etc.), wearable device (e.g., glasses, watch, etc.), camera, video camera, etc.

[0078] It is worth noting that the aforementioned electronic device may have 3 to 4 cameras, and each pair of cameras with adjacent focal lengths can be considered a dual-camera system, repeating the same processing logic. Therefore, the various embodiments of this disclosure are mainly described using a dual-camera system (e.g., a system composed of a first camera and a second camera) as an example.

[0079] like Figure 1A As shown, the method includes the following steps S101-S103:

[0080] In step S101, in response to detecting a camera switching event, a first image captured by the first camera and a second image captured by the second camera are acquired.

[0081] In this embodiment, the electronic device can, in response to detecting a camera switching event, acquire a first image captured by a first camera and a second image captured by a second camera.

[0082] In some embodiments, the camera switching events described above may include different types, and different types of camera switching events may be triggered by different entities. For example, the types of camera switching events described above may include at least one of the following:

[0083] Camera switching events triggered by shooting distance; camera switching events triggered by zoom adjustment operations; at least one of the following.

[0084] For example, when taking a photo using an electronic device, the device can trigger a camera switch based on the shooting distance. In this case, the electronic device is the subject that triggers the camera switch event.

[0085] For another example, Figure 1B This is a schematic diagram illustrating an application scenario of a user performing zoom adjustment operation according to an exemplary embodiment of this disclosure; such as Figure 1B As shown, when a user clicks the camera app icon on the main page of the electronic device to open the shooting interface, the screen of the electronic device displays the shooting interface, which includes a photo-taking function area, a zoom data area, and an image preview area. Then, the user can use two fingers to spread or pinch in the image preview area to form a zoom command for multiple cameras, and the screen will display a magnified or reduced preview image. Optionally, spreading two fingers triggers a zoom-in command, and pinching two fingers triggers a zoom-out command. Based on this, the electronic device can respond to the zoom adjustment operation by triggering a camera switching event, in which case the user is the subject triggering the camera switching event.

[0086] The first camera mentioned above is the camera before the switch, and the second camera is the camera after the switch.

[0087] In step S102, the target optical flow deformation field is determined based on the first image and the second image.

[0088] In this embodiment, after acquiring the first image captured by the first camera and the second image captured by the second camera, the electronic device can determine the target optical flow deformation field based on the first image and the second image.

[0089] In some embodiments, the target optical flow deformation field may include the optical flow deformation fields from the first image to the second image. For example, after obtaining the first image and the second image, an optical flow deformation field from the first image to the second image can be generated. This optical flow deformation field can be used to describe the pixel position transformation information from the first image to the second image. The method for generating this optical flow deformation field can be found in related art descriptions, and this embodiment does not limit it.

[0090] In step S103, a transition image required to switch from the first image to the second image is determined based on the target optical flow deformation field and the first image.

[0091] In this embodiment, after the target optical flow deformation field is determined based on the first image and the second image, a transition image required to switch from the first image to the second image can be determined based on the target optical flow deformation field and the first image, so that the subsequent first image can be smoothly switched to the second image through the aforementioned transition image.

[0092] In some embodiments, different camera switching event types can correspond to different transition image frame numbers. For example, when the camera switching event type is a camera switching event triggered based on shooting distance, the corresponding transition image frame number can be a preset value. When the camera switching event type is a camera switching event triggered based on zoom adjustment operation, the corresponding transition image frame number can be determined based on the speed at which the user performs the zoom adjustment operation.

[0093] For example, when the user zooms in quickly, fewer transition image frames are needed, meaning a smooth transition from the first image to the second image can be achieved with fewer transition image frames. Conversely, when the user zooms in slowly, more transition image frames are needed, meaning a smooth transition from the first image to the second image can be achieved with more transition image frames.

[0094] In other embodiments, the method for determining the transition image required to switch from the first image to the second image may also be described below. Figure 4 The embodiments shown will not be described in detail here.

[0095] In step S103, in the image preview area, the first image is switched to the second image via the transition image.

[0096] In this embodiment, after determining the transition image required to switch from the first image to the second image based on the target optical flow deformation field and the first image, the first image can be switched to the second image through the transition image in the image preview area.

[0097] For example, after obtaining the transition image through the above steps, the first image can be displayed in the image preview area, followed by the sequential display of each transition image until the second image is displayed in the image preview area. This can reduce the jump phenomenon when the camera zooms, making the transition from the first image to the second image smoother and more seamless.

[0098] As described above, the method of this embodiment, in response to detecting a camera switching event, acquires a first image captured by a first camera and a second image captured by a second camera, determines a target optical flow deformation field based on the first image and the second image, and determines a transition image required to switch the first image to the second image based on the target optical flow deformation field and the first image. Then, in the image preview area, the first image is switched to the second image through the transition image, which can improve the smoothness of the image switching process, achieve smooth zoom, and thus improve the user's shooting experience.

[0099] Building upon the above embodiments, to improve the smoothness of subsequent image switching processes and better achieve smooth zooming, this embodiment can first register the first image and the second image, and then determine the optical flow deformation field of the two registered images. Specifically, considering the different positions and imaging environments of each camera, images captured by multiple cameras often exhibit geometric deformation and positional errors, thus requiring image registration. Related technologies include image registration algorithms such as feature matching algorithms or deep learning. However, while these solutions can achieve registration to a certain extent, they rely heavily on experience and domain knowledge due to the need for manual feature design and extraction, resulting in problems such as low accuracy, computational complexity, and difficulty in real-time application. Specifically, the registration accuracy of feature matching algorithms depends on the quality and quantity of feature points; when key features are lost or insufficient, their registration performance degrades. Feature matching methods may fail or become unstable when there are significant changes in illumination, scale, and viewing angle. Deep learning-based registration networks using feature matching typically include a feature extraction network, a feature matching module, and subsequent calculation of the target optical flow deformation field. These modules need to work together, requiring the design and training of multi-stage deep networks. Furthermore, incompatibility or optimization issues may exist between different modules. The multi-stage processing from feature extraction to feature matching and then to target optical flow deformation field calculation is time-consuming, and the computational complexity of feature extraction and matching is high, making them unsuitable for real-time applications.

[0100] Therefore, this disclosure proposes a scheme to determine the target optical flow deformation field from the preprocessed image to the third image based on a recursive cascaded network, thereby achieving end-to-end registration.

[0101] For example, Figure 2 This is a flowchart illustrating how to determine a target optical flow deformation field based on the first image and the second image according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking how to determine a target optical flow deformation field based on the first image and the second image as an example.

[0102] like Figure 2 As shown, the determination of the target optical flow deformation field based on the first image and the second image in step S102 above may include the following steps S201-S202:

[0103] In step S201, the fourth image is preprocessed based on the third image to obtain a preprocessed image.

[0104] In this embodiment, when the target optical flow deformation field is determined based on the first image and the second image, the fourth image can be preprocessed based on the third image to obtain a preprocessed image.

[0105] The third image can be the image with a smaller field of view between the first image and the second image, while the fourth image can be the image with a larger field of view between the first image and the second image.

[0106] The preprocessed image described above is aligned with the field of view of the third image and has the same size as the fourth image. This embodiment, by aligning the field of view of the first and second images, can achieve a more accurate acquisition of the optical flow deformation field between them, thereby enabling smoother subsequent zooming.

[0107] It is worth noting that in this embodiment, the field of view alignment between the preprocessed image and the third image can mean that the field of view (FOV) of the preprocessed image is similar to that of the third image (for example, the difference in the field of view size between the two frames is less than or equal to 20%).

[0108] For example, during zooming in, the first image has a larger field of view (therefore, the first image is considered the fourth image), while the second image has a smaller field of view (therefore, the second image is considered the third image). Therefore, the first image can be preprocessed based on the second image to obtain a preprocessed image. Similarly, during zooming out, the first image has a smaller field of view (therefore, the first image is considered the third image), while the second image has a larger field of view (therefore, the second image is considered the fourth image). Therefore, the second image can be preprocessed based on the first image to obtain a preprocessed image.

[0109] In some embodiments, when preprocessing a fourth image based on a third image, a cropped image aligned with the field of view of the third image can be extracted from the fourth image. The size of the cropped image is then resized to match the size of the fourth image to obtain the preprocessed image. For example, assuming the native magnification of the first camera is 1x and the native magnification of the second camera is 5x, the FOV ratio between these two cameras is 5. Therefore, a cropping factor approximately 5 (e.g., 4.95 or 4.9, not necessarily 5) can be selected to crop the first image so that the FOV of the first image is similar to that of the second image.

[0110] In step S202, the preprocessed image and the third image are input into a pre-constructed recursive cascade network to obtain the target optical flow deformation field.

[0111] In this embodiment, after preprocessing the fourth image based on the third image to obtain the preprocessed image, the preprocessed image and the third image can be input into a pre-constructed recursive cascade network to obtain the target optical flow deformation field.

[0112] The aforementioned target optical flow deformation field can be used to describe the pixel position transformation information from the preprocessed image to the third image.

[0113] In some embodiments, the type of the above-mentioned recursive cascaded network can be set according to requirements, such as setting it as a recursive cascaded network with shared weights, etc., and this embodiment does not limit this.

[0114] In the above Figure 2 Based on the illustrated embodiment, Figure 3A This is a flowchart illustrating an image display method according to an exemplary embodiment of the present disclosure; as shown below. Figure 3A As shown, when an electronic device with multiple cameras performs dual-camera zoom, the first and second cameras can be activated to acquire a first image captured by the first camera and a second image captured by the second camera. Then, image preprocessing can be performed on the first and / or second images to make their field of view (FOV) similar (including the same case). Next, the preprocessed dual-camera image data stream can be input into a recursive cascaded network to obtain the target optical flow deformation field of the dual-camera images. Then, deformation field interpolation is performed based on the target optical flow deformation field and the current zoom ratio to obtain a transition image through the interpolated deformation field. Then, in the image preview area displayed on the electronic device screen, the first image can be switched to the second image through the transition image (for example, the transition image can be inserted between the first and second images by frame interpolation).

[0115] The following explanation uses a recursive cascaded network with shared weights as an example.

[0116] For example, Figure 3B This is a schematic diagram of a recursive cascaded network illustrated according to an exemplary embodiment of the present disclosure; as shown below. Figure 3B As shown, this recursive cascade network can contain N levels of recursive networks, i.e. Figure 3B The network consists of a first-level recursive network, a second-level recursive network, ..., an Nth-level recursive network (not shown in the diagram). Here, N is a positive integer greater than or equal to 3. It's worth noting that the size of N can be determined based on a trade-off between the performance of the electronic device and the registration effect of the two frames. A larger N results in more network layers and requires more computational power from the electronic device; therefore, an excessively large N is unsuitable for implementation in mobile terminals like smartphones. Conversely, an excessively small N may not achieve the desired registration effect. As an example, in this embodiment, N = 5.

[0117] Based on this, once the target optical flow deformation field is determined, the preprocessed image and the third image can be used as the secondary image and the primary image, respectively, and input into the system. Figure 3BThe first-level recursive network in the image is processed to obtain the first optical flow field, and the preprocessed image is warped based on the first optical flow field to obtain the first-level warped image.

[0118] Then, the first-level deformed image and the third image are input into the second-level recursive network for processing to obtain the second optical flow field, and the first-level deformed image is deformed based on the second optical flow field to obtain the second-level deformed image;

[0119] This process continues until the Nth level deformed image is obtained. Then, the similarity between the Nth level deformed image and the third image is determined and compared with a preset similarity threshold.

[0120] If the similarity is greater than or equal to the preset similarity threshold, then the Nth optical flow field output by the Nth level recursive network can be determined as the target optical flow deformation field.

[0121] If the similarity is less than a preset similarity threshold, the parameters of the N-level recursive network can be updated through backpropagation. The above process is repeated based on the updated N-level recursive network until the similarity between the N-level deformed image and the third image is greater than or equal to the preset similarity threshold. At this point, the Nth optical flow field output by the N-level recursive network can be determined as the target optical flow deformed field. Backpropagation is a commonly used algorithm in deep learning and neural networks; for detailed explanations and descriptions, please refer to relevant technologies. This embodiment will not elaborate on it further.

[0122] This embodiment determines the target optical flow deformation field from the preprocessed image to the third image based on a recursive cascaded network, enabling end-to-end registration of the dual-camera images. The entire process from image input to deformation field output is completed within a single network, eliminating the need for manual feature design and extraction. This reduces reliance on experience and domain knowledge, resulting in more accurate registration. Furthermore, by omitting intermediate steps such as feature matching, anomaly removal, and transformation calculations, computational complexity is reduced, computational efficiency is improved, and real-time applicability is enhanced. This allows for better handling of challenges such as noise, local occlusion, and lighting variations, making it suitable for registration in a wider range of scenarios (including weak texture, textureless scenes, and blurred images). Consequently, it improves the smoothness of subsequent zooming based on the registration results.

[0123] Figure 4 This is a flowchart illustrating how to determine the transition image required for switching from the first image to the second image according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking the determination of the transition image required for switching from the first image to the second image as an example.

[0124] like Figure 4 As shown, the step S103 above, which involves determining the transition image required to switch from the first image to the second image based on the target optical flow deformation field and the first image, may include the following steps S401-S402:

[0125] In step S401, the target optical flow deformation field is interpolated based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio.

[0126] In this embodiment, after the target optical flow deformation field is determined, the target optical flow deformation field can be interpolated based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio.

[0127] The interpolated optical flow deformation field is used to describe the pixel position transformation information of the transition image from the first image to the current zoom magnification.

[0128] It's worth noting that the aforementioned current zoom ratio can be the zoom ratio corresponding to the current frame image (e.g., a zoom ratio determined autonomously by the electronic device based on camera switching algorithms, or a zoom ratio indicated by a user-triggered zoom adjustment operation). This current frame image can refer to the image data currently displayed in the electronic device's image preview during smooth zooming, and its size can lie between the first zoom ratio corresponding to the first image and the second zoom ratio corresponding to the second image. For example, if the native zoom ratio of the first camera is 1x and the native zoom ratio of the second camera is 5x, and the user zooms to 3x through a zoom adjustment operation, then the current zoom ratio is 3x.

[0129] In some embodiments, the target optical flow deformation field can be interpolated based on a preset interpolation method. This preset interpolation method can be selected from relevant technologies based on requirements, such as linear interpolation, bilinear interpolation, or spline interpolation, etc., and this embodiment does not limit it in this regard.

[0130] In step S402, a transition image for the current zoom magnification is determined based on the first image and the interpolated optical flow deformation field.

[0131] In this embodiment, after interpolating the target optical flow deformation field based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio, the transition image of the current zoom ratio can be determined based on the first image and the interpolated optical flow deformation field.

[0132] For example, when determining the transition image of the current zoom magnification based on the first image and the interpolated optical flow deformation field, the position of the corresponding pixel in the first image can be adjusted according to the offset in the interpolated optical flow deformation field corresponding to each transition image magnification to obtain the transition image of each transition image magnification.

[0133] As described above, this embodiment interpolates the target optical flow deformation field based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio. Based on the first image and the interpolated optical flow deformation field, a transition image for the current zoom ratio is determined. This allows for accurate determination of the transition image required to switch from the first image to the second image. Consequently, the first image can be switched to the second image in the image preview area using the transition image, thus improving the smoothness of the image switching process and enabling smooth switching between lenses with different focal lengths, thereby enhancing the user's shooting experience.

[0134] Figure 5 This is a block diagram illustrating an image display device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be configured in an electronic device having multiple cameras, such as a mobile terminal (e.g., a mobile phone, tablet computer, etc.), a wearable device (e.g., glasses, watch, etc.), a camera, a camcorder, etc. Figure 5 As shown, the device may include: an image acquisition module 110, a deformation field determination module 120, a transition map determination module 130, and an image switching module 140, wherein:

[0135] Image acquisition module 110 is used to acquire a first image captured by a first camera and a second image captured by a second camera in response to detecting a camera switching event, wherein the first camera is the camera before the switch and the second camera is the camera after the switch;

[0136] The deformation field determination module 120 is used to determine the target optical flow deformation field based on the first image and the second image;

[0137] Transition image determination module 130 is used to determine the transition image required for switching from the first image to the second image based on the target optical flow deformation field and the first image;

[0138] The image switching module 140 is used to switch the first image to the second image through the transition image in the image preview area.

[0139] As described above, the device in this embodiment, in response to detecting a camera switching event, acquires a first image captured by a first camera and a second image captured by a second camera, determines a target optical flow deformation field based on the first image and the second image, and determines a transition image required to switch the first image to the second image based on the target optical flow deformation field and the first image. Then, in the image preview area, the first image is switched to the second image through the transition image, which can improve the smoothness of the image switching process, achieve smooth zoom, and thus improve the user's shooting experience.

[0140] Figure 6 This is a block diagram illustrating another image display device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be configured in an electronic device with multiple cameras, such as a mobile terminal (e.g., mobile phone, tablet computer, etc.), wearable device (e.g., glasses, watch, etc.), camera, camcorder, etc. The image acquisition module 210, deformation field determination module 220, transition map determination module 230, and image switching module 240 are integrated with... Figure 5 The image acquisition module 110, deformation field determination module 120, transition map determination module 130, and image switching module 140 in the illustrated embodiment have the same functions, which will not be described in detail here.

[0141] like Figure 6 As shown, the deformation field determination module 220 may include:

[0142] Image preprocessing unit 221 is used to preprocess a fourth image based on a third image to obtain a preprocessed image. The third image is the image with a smaller field of view between the first image and the second image, and the fourth image is the image with a larger field of view between the first image and the second image. The preprocessed image is aligned with the field of view of the third image and has the same size as the fourth image.

[0143] The deformation field acquisition unit 222 is used to input the preprocessed image and the third image into a pre-constructed recursive cascade network to obtain the target optical flow deformation field, which is used to describe the pixel position transformation information from the preprocessed image to the third image.

[0144] In some embodiments, the image preprocessing unit 221 can also be used for:

[0145] A cropped image aligned with the field of view of the third image is extracted from the fourth image;

[0146] The cropped image is restored to the size of the fourth image to obtain the preprocessed image.

[0147] In some embodiments, the recursive cascade network comprises an N-level recursive network, where N is a positive integer greater than or equal to 3;

[0148] Furthermore, the deformation field acquisition unit 222 can also be used for:

[0149] The preprocessed image and the third image are input into the first-level recursive network for processing to obtain the first optical flow field. Based on the first optical flow field, the preprocessed image is warped to obtain the first-level warped image.

[0150] The first-level deformed image and the third image are input into the second-level recursive network for processing to obtain the second optical flow field. Based on the second optical flow field, the first-level deformed image is deformed to obtain the second-level deformed image.

[0151] This process continues until the Nth level deformed image is obtained, at which point the similarity between the Nth level deformed image and the third image is determined.

[0152] In response to the similarity being less than a preset similarity threshold, the parameters of the N-level recursive network are updated through backpropagation;

[0153] The above process is repeated based on the updated N-level recursive network until the similarity between the N-level deformed image and the third image is greater than or equal to the preset similarity threshold. Then, the N-level optical flow field output by the N-level recursive network is determined as the target optical flow deformed field.

[0154] In some embodiments, the transition map determination module 230 may include:

[0155] The deformation field interpolation unit 231 is used to interpolate the target optical flow deformation field based on the current zoom ratio to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio. The interpolated optical flow deformation field is used to describe the pixel position transformation information of the transition image from the first image to the current zoom ratio.

[0156] The deformation field interpolation unit 232 is a unit used to determine the transition image of the current zoom magnification based on the first image and the interpolated optical flow deformation field.

[0157] In some embodiments, the deformation field interpolation unit 232 can also be used to adjust the position of the corresponding pixel in the first image according to the pixel offset in the interpolated optical flow deformation field to obtain the transition image of the current zoom magnification.

[0158] In some embodiments, the type of the above-mentioned camera switching event may include at least one of the following:

[0159] Camera switching events triggered by shooting distance;

[0160] Camera switching events triggered by zoom adjustment operations.

[0161] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0162] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0163] Reference Figure 7 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0164] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the image display method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0165] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0166] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0167] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0168] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0169] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0170] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0171] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0172] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image display method described above.

[0173] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to complete the image display method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0174] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 920 of the device 900 to perform the image display method described above.

[0175] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0176] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image display method, characterized in that, The method includes: In response to detecting a camera switching event, a first image captured by a first camera and a second image captured by a second camera are acquired, wherein the first camera is the camera before the switch and the second camera is the camera after the switch. Determine the target optical flow deformation field based on the first image and the second image; Based on the target optical flow deformation field and the first image, determine the transition image required to switch from the first image to the second image; In the image preview area, the first image is switched to the second image via the transition image.

2. The method according to claim 1, characterized in that, Determining the target optical flow deformation field based on the first image and the second image includes: The fourth image is preprocessed based on the third image to obtain a preprocessed image. The third image is the image with a smaller field of view between the first image and the second image, and the fourth image is the image with a larger field of view between the first image and the second image. The preprocessed image is aligned with the field of view of the third image and has the same size as the fourth image. The preprocessed image and the third image are input into a pre-constructed recursive cascaded network to obtain the target optical flow deformation field, which is used to describe the pixel position transformation information from the preprocessed image to the third image.

3. The method according to claim 2, characterized in that, The preprocessing of the fourth image based on the third image includes: A cropped image aligned with the field of view of the third image is extracted from the fourth image; The cropped image is restored to the size of the fourth image to obtain the preprocessed image.

4. The method according to claim 2, characterized in that, The recursive cascaded network contains N levels of recursive networks, where N is a positive integer greater than or equal to 3; The step of inputting the preprocessed image and the third image into a pre-constructed recursive cascade network to obtain the target optical flow deformation field includes: The preprocessed image and the third image are input into the first-level recursive network for processing to obtain the first optical flow field. Based on the first optical flow field, the preprocessed image is warped to obtain the first-level warped image. The first-level deformed image and the third image are input into the second-level recursive network for processing to obtain the second optical flow field. Based on the second optical flow field, the first-level deformed image is deformed to obtain the second-level deformed image. This process continues until the Nth level deformed image is obtained, at which point the similarity between the Nth level deformed image and the third image is determined. In response to the similarity being less than a preset similarity threshold, the parameters of the N-level recursive network are updated through backpropagation; The above process is repeated based on the updated N-level recursive network until the similarity between the N-level deformed image and the third image is greater than or equal to the preset similarity threshold. Then, the N-level optical flow field output by the N-level recursive network is determined as the target optical flow deformed field.

5. The method according to claim 1, characterized in that, The step of determining the transition image required to switch from the first image to the second image based on the target optical flow deformation field and the first image includes: Based on the current zoom ratio, the target optical flow deformation field is interpolated to obtain the interpolated optical flow deformation field corresponding to the current zoom ratio. The interpolated optical flow deformation field is used to describe the pixel position transformation information of the transition image from the first image to the current zoom ratio. Based on the first image and the interpolated optical flow deformation field, a transition image for the current zoom magnification is determined.

6. The method according to claim 5, characterized in that, The step of determining the transition image for the current zoom magnification based on the first image and the interpolated optical flow deformation field includes: According to the offset of each pixel in the interpolated optical flow deformation field, the position of the corresponding pixel in the first image is adjusted to obtain the transition image of the current zoom magnification.

7. The method according to claim 1, characterized in that, The types of camera switching events include at least one of the following: Camera switching events triggered by shooting distance; Camera switching events triggered by zoom adjustment operations.

8. An image display device, characterized in that, The device includes: The image acquisition module is used to acquire a first image captured by a first camera and a second image captured by a second camera in response to the detection of a camera switching event, wherein the first camera is the camera before the switch and the second camera is the camera after the switch; A deformation field determination module is used to determine the target optical flow deformation field based on the first image and the second image; A transition image determination module is used to determine, based on the target optical flow deformation field and the first image, the transition image required to switch from the first image to the second image; An image switching module is used to switch the first image to the second image in the image preview area via the transition image.

9. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the image display method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image display method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image display method according to any one of claims 1 to 7.

Images