Image Processing Method, Apparatus, Terminal, and Computer-Readable Storage Medium

Abstract

An embodiment of the present application discloses an image processing method, apparatus, terminal, and computer-readable storage medium, which can improve the smoothness of the preview image change during the zooming process. The method includes: in the case of zooming from the first camera to the second camera, performing zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment to obtain a first image frame and a second image frame; determining the current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame; determining the estimated image offset at the next moment based on the current image offset and the zoom data at the current moment; and determining the preview image of the next frame based on the estimated image offset.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to an image processing method, apparatus, terminal, and computer-readable storage medium. Background Art

[0002] Currently, terminals such as mobile phones are usually equipped with multiple cameras with multiple focal lengths. When a user takes a picture, zooming can be performed by zooming in the preview interface during shooting. However, currently, when switching between two cameras with different focal lengths during the zooming process, for example, during the process of switching from a 1.0x zoom ratio to a 2.0x zoom ratio, the preview image displayed on the preview interface is based on fixed data calibrated at the factory to determine the correspondence between the three-dimensional geometric position of the surface of the spatial object and the corresponding points in the image, and the camera lens picture is cropped and displayed. In actual shooting, at different zoom ratios, the position of the actual object relative to the lens is very difficult to be the same as the fixed object position in the calibration data. Therefore, using fixed calibration data to determine the preview interface display during the zooming process will cause the picture change on the preview interface to shift or mutate, that is, the smoothness of the preview image change is poor. Summary of the Invention

[0003] Embodiments of this application are expected to provide an image processing method, apparatus, terminal, and computer-readable storage medium, which can improve the smoothness of the preview image change during the zooming process.

[0004] The technical solution of this application is implemented as follows:

[0005] Embodiments of this application provide an image processing method, including:

[0006] In the case of zooming from a first camera to a second camera, perform zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment to obtain a first image frame and a second image frame;

[0007] Determine the current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame;

[0008] Based on the current image offset and the zoom data at the current moment, determine the estimated image offset at the next moment;

[0009] Based on the estimated image offset, determine the next frame of preview image.

[0010] Embodiments of this application provide an image processing apparatus, including:

[0011] A zoom module, configured to perform zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment when zooming from the first camera to the second camera, so as to obtain a first image frame and a second image frame;

[0012] A matching module, configured to determine the current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame;

[0013] A determination module, configured to determine the estimated image offset at the next moment based on the current image offset and the zoom data at the current moment; and determine the next-frame preview image based on the estimated image offset.

[0014] An embodiment of the present application provides a terminal, including:

[0015] A memory, configured to store executable instructions;

[0016] A processor, configured to implement the image processing method provided by the embodiment of the present application when executing the executable instructions stored in the memory.

[0017] An embodiment of the present application provides a chip, including:

[0018] A memory, configured to store executable instructions;

[0019] A processor, configured to implement the image processing method provided by the embodiment of the present application when executing the executable instructions stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium, storing executable instructions, which are used to cause a processor to implement the image processing method provided by the embodiment of the present application when executed.

[0021] An embodiment of the present application provides a computer program product, including a computer program or instruction, where the computer program or instruction implements the image processing method provided by the embodiment of the present application when executed by a processor.

[0022] The embodiments of the present application provide an image processing method, apparatus, terminal, and computer-readable storage medium. During the process of zooming from the first camera to the second camera, by performing feature matching on the first image frame and the second image frame obtained by zooming the two cameras under the zoom data at the current moment, the current image offset between the first image frame and the second image frame is determined. Furthermore, by combining the zoom data at the current moment and the current image offset corresponding to the zoom data at the current moment, the estimated image offset at the next moment is determined. In this way, the estimated image offset at the next moment is applied to the next frame of the image during the zooming process to determine the next frame of the preview image. The embodiments of the present application achieve, during the process of zooming between cameras, calculating the current image offset through real-time feature matching, and using the zoom data at the current moment and the current image offset to predict the image offset at the next moment. Thus, the accuracy of determining the image offset between different cameras under the continuously changing zoom data during the zooming process is improved. Furthermore, based on the image offset at the next moment, the next frame of the preview image is obtained, which can improve the smoothness of the change of the preview image during the zooming process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. is an optional flowchart of the image processing method provided by the embodiments of the present application;

[0024] Figure 2 FIG. is a schematic diagram of the effect of performing a focusing operation on the preview interface provided by the embodiments of the present application;

[0025] Figure 3 FIG. is a schematic diagram of the focal length ranges of different cameras configured on the terminal provided by the embodiments of the present application;

[0026] Figure 4 FIG. is an optional flowchart of the image processing method provided by the embodiments of the present application;

[0027] Figure 5 FIG. is a schematic diagram of the effect of the image correspondence relationship represented by the initial matching feature point pairs provided by the embodiments of the present application;

[0028] Figure 6 FIG. is an optional flowchart of the image processing method provided by the embodiments of the present application;

[0029] Figure 7 FIG. is a fitting curve graph of the zoom data corresponding to each image acquisition moment during the zooming process;

[0030] Figure 8 FIG. is an optional flowchart of the image processing method provided by the embodiments of the present application;

[0031] Figure 9An optional flowchart of applying the image processing method provided by the embodiments of the present application to an actual scenario;

[0032] Figure 10 An optional structural schematic diagram of the image processing device provided by the embodiments of the present application;

[0033] Figure 11 An optional structural schematic diagram of the chip provided by the embodiments of the present application. Detailed implementation manners

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be construed as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0035] In the following descriptions, reference is made to "some embodiments", which describe subsets of all possible embodiments. However, it can be understood that "some embodiments" can be the same subsets or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] In the following descriptions, the terms "first / second / third" are only used to distinguish similar objects, and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0038] On a traditional camera, zooming is achieved by operating the zoom ring on the camera lens. However, on most terminals with image acquisition capabilities such as mobile phones, it is obviously impossible to carry such a complex mechanical mechanism. Therefore, optical zoom on terminals such as mobile phones is usually achieved by switching between cameras with different focal lengths. Different cameras have different focal lengths. Thus, when switching between different cameras, the zoom function can naturally be achieved. However, if the user directly switches between two cameras with different focal lengths on the mobile phone interface, such as directly clicking to switch from a camera with a 1.0x zoom ratio to a camera with a 2.0x zoom ratio, since the position of the subject to be photographed is different in the two cameras, there will be a sudden change in the preview screen at the moment of switching.

[0039] Currently, optical zoom on mobile phones mimics the smooth zoom effect of traditional cameras through computational photography algorithms such as Spatial Alignment Transform (SAT). The current SAT switching smoothness scheme uses fixed sensor calibration data set at the factory to calculate the field-of-view translation amount between cameras with different focal lengths, and then determines the cropping and display of the lens image during the SAT process based on the field-of-view translation amount to achieve smooth switching.

[0040] However, in actual shooting, due to different zoom ratios, the fields of view (FOVs) of each camera are different, the positions of different cameras on the mobile phone are different, and the positions of the actual objects to be photographed relative to the lens and the calibration object are also very difficult to be the same; and from a spatial perspective, there is not only translation but also slight rotation between different cameras. Therefore, simply translating the image cannot align every pixel on the screen. The above factors will cause great changes in the size and position of the object to be photographed in the image during the zoom process. Therefore, it is difficult to achieve good results by using the offset calculated from fixed calibration data to determine the displayed image during the zoom process, and the smoothness of the image change during the zoom process is poor.

[0041] Embodiments of the present application provide an image processing method, apparatus, terminal, and computer-readable storage medium, which can improve the smoothness of image changes during zooming. The following describes an exemplary application of the terminal provided by the embodiments of the present application. The terminal provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, and mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable game devices).

[0042] See Figure 1 , Figure 1 which is an optional flowchart of the image processing method provided by the embodiments of the present application, and will be described in conjunction with Figure 1 the steps shown.

[0043] S101. In the case of zooming from the first camera to the second camera, perform zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment to obtain a first image frame and a second image frame.

[0044] In the embodiments of the present application, at least two cameras with different focal lengths are configured on the terminal. When zooming from the first camera to the second camera on the terminal, the first camera and the second camera are in an on state. The terminal captures image frames through the first camera and the second camera respectively at the current moment, and performs zoom processing on the two image frames captured by the first camera and the second camera at the current moment according to the zoom data at the current moment, to obtain a first image frame and a second image frame.

[0045] In some embodiments, the zoom data may be data representing the focal length. Exemplarily, the zoom data may be a zoom ratio. The terminal performs cropping or enlarging processing on the image frames captured by the first camera and the second camera respectively according to the zoom ratio at the current moment, to obtain a first image frame and a second image frame. Or the zoom data may also include other data forms representing the focal length, which is specifically selected according to the actual situation, and the embodiments of the present application do not make a limitation.

[0046] In some embodiments, the terminal may determine the zoom data corresponding to the current moment by receiving an operation on a preset preview interface; and determine to zoom from the first camera to the second camera when the zoom data at the current moment is within a preset focal length range between the first camera and the second camera. Exemplarily, the preset preview interface may be a preview interface after starting an application with an image capture function such as a camera on the terminal. As Figure 2 shown, Figure 2 FIG. is a schematic diagram of the effect of performing a zoom operation (focus adjustment operation) on the preview interface of the camera. When the user performs a zoom operation by sliding the zoom control 21 or pinching with fingers on the preview interface as Figure 2 shown, the terminal may determine the zoom data adjusted at the current moment by receiving the user's operation, such as Figure 2 5.1X shown, and correspondingly display preview images under different zoom data through the preview interface as Figure 2 shown during the zoom process.

[0047] In some embodiments, a schematic diagram of the focal length range between multiple cameras on the terminal may be as Figure 3 shown. Referring to Figure 3 , a wide-angle camera, a main camera, and a telephoto camera are configured on the terminal. Among them. The focal length range of the wide-angle camera is as shown in the wide-angle focal length range, and the focal length range of the main camera is as shown in the main camera focal length range. The above preset focal length range may be as Figure 3as described by the focal length ranges corresponding to 31 and 32 therein. When the zoom data enters the preset focal length range 31 from the wide-angle focal length range, it is determined that the zoom is from the wide-angle camera to the main camera; or, when the zoom data enters the preset focal length range 31 from the main camera focal length range, it is determined that the zoom is from the main camera to the wide-angle camera. When the zoom data enters the preset focal length range 32 from the main camera focal length range, it is determined that the zoom is from the main camera to the telephoto camera; or, when the zoom data enters the preset focal length range 32 from the telephoto focal length range, it is determined that the zoom is from the telephoto camera to the main camera. Here, the preset focal length range in the embodiments of the present application can be set with reference to the intersection of the focal length ranges between the first camera and the second camera. Specific selection is made according to the actual situation, and the embodiments of the present application do not make limitations.

[0048] In some embodiments, the zoom processing can crop and magnify the original image frames collected by the camera according to the zoom data, and fill the missing pixels after magnification through a preset image processing algorithm. The specific processing method can be implemented according to the actual situation, and the embodiments of the present application do not make limitations.

[0049] S102. Determine the current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame.

[0050] In the embodiments of the present application, the terminal performs feature matching on the first image frame and the second image frame, and determines the correspondence between the first image frame and the second image frame, that is, the mapping relationship of the spatial transformation, according to the obtained feature point set. Furthermore, the current image offset between the first image frame and the second image frame can be determined according to the correspondence between the first image frame and the second image frame, and the different pixel positions of the matched feature points in the first image frame and the second image frame respectively for offset calculation. Here, the current image offset corresponds to the zoom data at the current moment.

[0051] S103. Estimate the image offset at the next moment based on the current image offset and the zoom data at the current moment.

[0052] In the embodiments of the present application, the terminal can predict the image offset between the image frames collected by the first camera and the second camera at the next moment according to the zoom data at the current moment and the current image offset determined under the zoom data at the current moment, so as to obtain the image offset at the next moment.

[0053] In some embodiments, the terminal can predict the zoom data at the next moment based on the current zoom speed, such as the sliding speed of pinching and zooming on the screen or the sliding speed of a preset zoom control, and then determine the image offset at the next moment based on the current image offset corresponding to the zoom data at the current moment and the predicted zoom data at the next moment.

[0054] In some embodiments, the terminal may, by means of data fitting, fit based on the current zoom data at the current moment and the zoom data at the historical moment to predict the zoom data at the next moment. Furthermore, based on the current image offset corresponding to the zoom data at the current moment, predict the image offset at the next moment corresponding to the zoom data at the next moment to obtain the predicted image offset at the next moment.

[0055] S104. Determine the next preview image based on the predicted image offset.

[0056] In the embodiments of the present application, at the next moment during the zooming process, the terminal continues to collect image frames through the first camera and the second camera respectively, and performs zoom processing on the image frames collected by the first camera and the second camera at the next moment according to the actually obtained zoom data at the next moment to obtain the next first image frame and the next second image frame. Furthermore, apply the image offset at the next moment to the next first image frame or the next second image frame to determine the next preview image.

[0057] In some embodiments, the terminal may display the next preview image on the preview interface, and continue to perform the methods in S102 - S103 above based on the next first image frame, the next second image frame, and the zoom data at the next moment to determine the image offset at the next - next moment, and determine the preview image at the next - next moment according to the image offset at the next - next moment, and perform cyclic processing in the above - mentioned manner until the zooming from the first camera to the second camera is completed. Exemplarily, until the zoom data at a certain moment exceeds the preset focal length range, the terminal exits the zooming between the two cameras and uses a single camera for image acquisition.

[0058] It can be understood that during the zooming from the first camera to the second camera, by performing feature matching on the first image frame and the second image frame obtained by zooming the two cameras at the zoom data at the current moment, determine the current image offset between the first image frame and the second image frame. Furthermore, combine the zoom data at the current moment and the current image offset corresponding to the zoom data at the current moment to determine the predicted image offset at the next moment. In this way, apply the predicted image offset at the next moment to the next frame of the image during the zooming process to determine the next preview image. The embodiments of the present application achieve, during the process of zooming between cameras, calculating the current image offset through real - time feature matching, and using the zoom data at the current moment and the current image offset to predict the image offset at the next moment. Thus, it improves the accuracy of determining the image offset between different cameras under the continuously changing zoom data during the zooming process, and furthermore, based on the image offset at the next moment, obtain the next preview image, which can improve the smoothness of the change of the preview image during the zooming process.

[0059] In some embodiments, based on Figure 1 , as Figure 4 shown, the process of S102 above can be implemented by executing S1021 - S1023, as follows:

[0060] S1021. Perform feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame.

[0061] In the embodiments of the present application, the terminal can extract feature points from the first image frame and the second image frame respectively, and perform feature matching on the first set of feature points extracted from the first image frame and the second set of feature points extracted from the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame.

[0062] In some embodiments, the terminal can use a hardened operator module to perform feature matching on the first image frame and the second image frame to determine an initial set of matching feature points; the hardened operator module includes a hardware-accelerated feature matching algorithm.

[0063] Exemplarily, the hardened operator module can be an Intelligent Video Engine (IVE) on the terminal. The IVE includes hardware-accelerated feature extraction operators and feature extraction operators, such as hardware-accelerated (Oriented Fast and Rotated Brief, ORB) operators. ORB can be used to quickly create feature vectors for key points in an image and is, to a certain extent, unaffected by noise and image transformations, such as rotation and scaling transformations. Of course, it can also be other hardware-accelerated feature extraction algorithms, such as a feature extraction algorithm based on corner detection (Features from Accelerated Segment Test, FAST), a key point detection algorithm for scale-invariant feature transform (Scale-Invariant Feature Transform, SIFT), and so on. Similarly, feature matching algorithms, such as brute-force matching and K-Nearest Neighbor classification (KNN), can also be implemented in the hardened operator module, and specific selection is made according to the actual situation, which is not limited in the embodiments of the present application.

[0064] It can be understood that by performing feature matching through the hardened operator module, a video analysis algorithm with high resource consumption can be executed by a hardware-accelerated chip, reducing the occupancy of the Central Processing Unit (CPU) during the image processing process, reducing CPU resource consumption, and accelerating the execution speed.

[0065] In some embodiments, among the initial matching feature point sets obtained by matching between the first image frame and the second image frame, the corresponding relationships between the initial matching feature point pairs can be as Figure 5 shown. It can be seen that through feature matching, the corresponding relationship between the first pixel point in the first image frame and the second pixel point in the second image frame can be determined, that is, the spatial position relationship corresponding to the pixel points representing the same object in different image frames.

[0066] S1022. Screen the initial matching feature point set to determine the matching feature point set.

[0067] In the embodiments of the present application, since there are usually many matching feature point pairs generated by the feature matching algorithm, and due to the complexity of the actual image, there may be certain mismatches in feature matching. Here, the terminal can screen the initial matching feature point set, remove the mismatched point pairs from the initial feature point set, such as the initial matching point pairs with the feature vector distance greater than the preset distance threshold, and determine the matching feature point set. It can be understood that among the matching feature point sets obtained through screening, the matching degree between the matching feature points is higher and the corresponding relationship is more accurate.

[0068] In some embodiments, the terminal can use a digital signal processor (DSP) to run a preset screening algorithm to determine the matching feature point set; the preset screening algorithm is used to remove the mismatched point pairs from the initial matching feature point set.

[0069] Here, the preset screening algorithm can be the random sample consensus (Ransac) algorithm, or other screening algorithms for matching feature points, which is specifically selected according to the actual situation and is not limited in the embodiments of the present application.

[0070] It can be understood that by running the feature matching algorithm in the hardening operator module for feature matching between image frames, and running the preset screening algorithm in the digital signal processor for screening mismatched points, the heterogeneous operation of the image processing algorithm is realized, the working burden of the CPU is reduced, the computing power of the CPU is released, and the image processing efficiency is improved.

[0071] In some embodiments, since the transformation of the image is a rigid transformation, the rigid transformation can only ensure the alignment of points on the same plane. In the actual scenario, the object distances of different points are different, and parallax needs to be considered for alignment. When performing feature matching and screening of the feature point set in the embodiments of the present application, the first main region in the first image frame and the second main region in the second image frame can also be determined; feature matching is performed on the first main region and the second main region to determine an initial matching feature point set. Furthermore, the initial matching feature point set is screened to determine the matching feature point set. In this way, by using the main regions in the same plane in the image frames for feature point matching and selection, the change of the main region can be ensured to be minimized, the accuracy of the matching feature point set can be further improved, and then the accuracy of the image correspondence relationship determined based on the matching feature point set and the accuracy of the estimated image offset determined based on the image correspondence relationship can be improved.

[0072] In some embodiments, the region of interest (ROI) in the image frame can be determined as the main region, or the main region in the image frame can be determined by methods such as image segmentation, and specific selection is made according to the actual situation, which is not limited in the embodiments of the present application.

[0073] S1023. Determine the current image offset between the first image frame and the second image frame based on the matching feature point set.

[0074] In the embodiments of the present application, the matching point pairs in the matching feature point set represent the different positions of the points on the photographed object in the first image frame and the second image frame. Since different cameras are in different positions, the offsets in translation and rotation between different positions of the pixel points can be determined through the corresponding position relationship represented by the matching point pairs, and thus the current image offset between the first image frame and the second image frame can be determined according to the offset of the pixel points.

[0075] It can be understood that in the embodiments of the present application, the correspondence relationship between the image frames captured by cameras with different focal lengths is determined through the matching feature point set, and the current image offset is determined accordingly. Compared with setting a fixed image offset in the current related technologies, the accuracy of the image offset is greatly improved, and then the smoothness of the preview image display based on the image offset during the zooming process is improved.

[0076] In some embodiments, based on Figure 1 or Figure 4 , as Figure 6 shown, the above S103 can be implemented by executing the process of S1031 - S1032 as follows:

[0077] S1031. Determine the estimated zoom data for the next moment according to at least one historical zoom data at at least one historical moment during the zooming process and the zoom data at the current moment.

[0078] In the embodiments of the present application, the terminal can obtain and record the zoom data corresponding to each moment during the zooming process. In this way, the terminal can determine the zooming speed during the zooming process, such as the sliding speed of the focusing interface and / or the change trend of the zooming speed, based on at least one piece of historical zoom data at at least one historical moment during the zooming process and the zoom data at the current moment, and predict the zoom data at the next moment based on the zooming speed and / or the change trend of the zooming speed to obtain the estimated zoom data at the next moment.

[0079] In some embodiments, the terminal can perform data fitting on at least one piece of historical zoom data and the zoom data at the current moment, and make a prediction for the next moment based on the data fitting result to determine the estimated zoom data at the next moment. Of course, other algorithms can also be used for prediction, which is specifically selected according to the actual situation, and the embodiments of the present application do not make limitations.

[0080] Exemplarily, the current moment t and at least one piece of historical zoom data corresponding to at least one historical moment from t - 6 to t - 1 can be as Figure 7 . It can be seen that the frame rate of the camera for collecting image frames is generally fixed, and the change curve of the zoom data between adjacent image acquisition moments can be generated by performing cubic spline interpolation between adjacent image acquisition moments. Exemplarily, Figure 7 For each curve segment of the zoom data in, such as the generation function of curve segments 1 to m (m > 1), it can be represented by the following formula:

[0081] y = a i + b i x + c i x 2 + d i x 3 (1)

[0082] Wherein, x represents the moment corresponding to each interpolation between adjacent image acquisition moments, and y represents the zoom data corresponding to the interpolation moment x. a i , b i , c i and d i Are the coefficients corresponding to the cubic spline interpolation. Through formula (1), the curve of the zoom data between adjacent image acquisition moments can be fitted, and through formula (1), using the segment m corresponding to the current moment t, the zoom data at the next frame t + 1 moment, that is, the estimated zoom data at the next moment, can be fitted, as shown by the hollow dots in Figure 7 .

[0083] S1032. Determine the predicted image offset based on the zoom data at the current moment, the current image offset, and the predicted zoom data at the next moment.

[0084] In the embodiments of the present application, the terminal can predict the image offset corresponding to the predicted zoom data at the next moment based on the current image offset corresponding to the zoom data at the current moment, and determine the predicted image offset at the next moment.

[0085] In some embodiments, the terminal can also use the method of data fitting to combine the zoom data at the current moment and the current image offset to determine the predicted image offset at the next moment corresponding to the predicted zoom data at the next moment.

[0086] It can be understood that by using the zoom data at the historical moment and the current moment, the zoom data at the next moment is obtained through data fitting, and the predicted image offset corresponding to the zoom data at the next moment is predicted based on the zoom data at the current moment and its corresponding current image offset, which improves the accuracy of determining the image offset, and further improves the smoothness of the change of the preview image displayed during the zooming process based on the image offset.

[0087] In some embodiments, based on Figure 1 、 Figure 4 and Figure 6 any one of Figure 8 as shown in

[0088] S1041. Process the image frames captured by the first camera and the second camera at the next moment according to the zoom data at the next moment to obtain the next first image frame and the next second image frame.

[0089] In the embodiments of the present application, the terminal can determine the corresponding relationship between the image frame captured by the first camera and the image frame captured by the second camera according to the set of matching feature points obtained at the current moment. When reaching the next moment, the terminal controls the first camera and the second camera to capture image frames respectively, and obtains the actual zoom data at the next moment, such as the zoom data actually adjusted by the user on the focus adjustment interface. According to the actual zoom data obtained at the next moment, the image frames captured by the first camera and the second camera at the next moment are processed to obtain the next first image frame and the next second image frame.

[0090] S1042. Determine the target image frame from the next first image frame and the next second image frame by comparing the viewing angles of the first camera and the second camera.

[0091] In the embodiments of the present application, the terminal compares the viewing angles of the first camera and the second camera, determines the viewing angle of the camera suitable for the current zoom trend therefrom, and determines the image frame corresponding to the camera as the target image frame.

[0092] In some embodiments, the terminal compares the viewing angles of the first camera and the second camera, and determines the image frame corresponding to the camera with the larger viewing angle between the first camera and the second camera as the target image frame.

[0093] S1043. Process the target image frame according to the estimated image offset to obtain the next frame of preview image.

[0094] In the embodiments of the present application, the terminal adjusts the target image frame according to the estimated image offset to obtain the next frame of preview image.

[0095] In some embodiments, since the estimated image offset is obtained by feature point matching, compared with the fixed translation value in the current related technologies, the estimated image offset in the embodiments of the present application can include offset values such as rotation, translation, and scaling. The terminal can perform at least one of the operations of translation, zooming in, cropping, and rotating on the target image frame according to the estimated image offset to determine the next frame of preview image.

[0096] It can be understood that performing at least one of the operations of translation, zooming in, cropping, and rotating on the target image frame according to the estimated image offset can make the spatial adjustment of the target image frame more accurate and further improve the smoothness of the change of the obtained next frame of preview image.

[0097] Next, reference will be made to Figure 9 , to illustrate the exemplary application of the embodiments of the present application in an actual application scenario.

[0098] S901. The camera is turned on and enters the preview interface.

[0099] S902. Receive the user's sliding operation and start zooming.

[0100] In S902, the terminal receives the user's sliding operation on the preview interface. When the zoom ratio corresponding to the sliding operation enters the preset focal length range between camera A and camera B, the dual cameras are turned on for zooming.

[0101] S903. Determine the currently displayed camera A and the target displayed camera B according to the sliding direction.

[0102] In S903, according to the direction of the sliding operation, the camera at the start of the zoom process is determined, that is, the currently displayed camera A, and the camera corresponding to the zoom target, that is, the target displayed camera B. Exemplarily, when the user slides the zoom from 0.6X to 1X, the wide-angle camera corresponding to 0.6X is the currently displayed camera A, and the main camera corresponding to 1X is the target displayed camera B.

[0103] Here, the processes of S901 - S902 are the same as those described in S101 and will not be elaborated here.

[0104] S904. Use the ORB algorithm to perform feature point matching on the two cameras to obtain an initial set of matching point pairs.

[0105] In S904, the terminal sets each of the currently displayed camera A and the target displayed camera B to output a small-sized image frame. The image frames output by the two cameras are used for feature point matching using the hardened ORB algorithm to obtain an initial set of matching point pairs. Exemplarily, the initial set of matching point pairs can include feature point pair (initial matching point pair) 1, feature point pair 2 up to feature point pair n, where n is greater than or equal to 3. Feature point pair 1 represents the correspondence between point 1 in the first image frame corresponding to camera A and point 1' in the second image frame corresponding to camera B, and so on.

[0106] S905. Use the Ransac algorithm to remove mismatched point pairs and obtain a set of matching point pairs.

[0107] In S905, the terminal uses the Ransac algorithm to remove mismatched point pairs from the initial set of matching feature point pairs to obtain a set of matching point pairs.

[0108] Here, the processes of S904 - S905 are the same as those described in S1021 - S1022 and will not be elaborated here.

[0109] S906. Calculate the correspondence between the image frames according to the set of matching point pairs.

[0110] In S906, the terminal calculates the correspondence between the first image frame and the second image frame according to the set of matching point pairs, that is, the spatial transformation mapping relationship.

[0111] S907. Predict the zoom ratio of the next frame according to the current sliding speed and the current zoom ratio.

[0112] In S907, the terminal can perform data fitting based on at least one historical zoom ratio (at least one historical zoom data) at at least one historical moment and the current zoom ratio (equivalent to the zoom data at the current moment) to predict the zoom data at the next moment, that is, the zoom ratio of the next frame.

[0113] Here, the process of S907 is the same as the process described in S1031 above, and will not be elaborated here.

[0114] S908. Calculate the offset data amount of the next frame and crop the image with a larger viewing angle to obtain the preview image of the next frame.

[0115] In S908, the terminal determines the current frame offset data amount (current image offset) according to the correspondence between the first image frame and the second image frame, and predicts the offset data amount of the next frame (estimated image offset) according to the current zoom ratio, the current image offset, and the zoom ratio of the next frame.

[0116] In S908, when the time of the next frame arrives, the terminal performs a cropping process on the image frame with a larger viewing angle (target image frame) in the next frame image frame according to the actually obtained zoom ratio of the next frame and the offset data amount of the next frame. Exemplarily, the offset data amount of the next frame is applied to the crop region parameter of the camera with a larger viewing angle to obtain the preview image frame of the next frame.

[0117] Here, the process of S908 is the same as the processes described in S1023, S1032, and S1041 - S1043 above, and will not be elaborated here.

[0118] S909. Display the preview image of the next frame.

[0119] In S909, the terminal displays the preview image of the next frame on the preview interface.

[0120] During the process of zooming from camera A to camera B by the terminal, Figure 9 the process is used to process the image frames output by the dual cameras at each moment, and determine the corresponding preview image at each moment, so as to achieve the effect of smooth change of the image during the zooming process.

[0121] It can be understood that in the embodiments of the present application, the ORB algorithm is used for image matching, and the image matching relationship between the two cameras is calculated in real time, which can more accurately obtain the corresponding relationship between the images of different cameras. According to the image corresponding relationship, the image offset of the current frame is determined, and then the matching relationship between the zoom data of the current frame and the image offset is used to predict the image offset of the next frame in combination with the zoom speed, so as to apply the calculation result of the current frame image to the next frame to predict the viewing angle offset between different cameras, and improve the smoothness of the change of the preview image during the zooming process.

[0122] The embodiments of the present application also provide a sensor control device. Figure 10 For the structural schematic diagram of the image processing device provided by the embodiments of the present application; as Figure 10As shown in the figure, the image processing device 1 includes: a zoom module 11, a matching module 12, and a determination module 13, where:

[0123] The zoom module 11 is configured to, when zooming from the first camera to the second camera, perform zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment, to obtain a first image frame and a second image frame;

[0124] The matching module 12 is configured to determine the current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame;

[0125] The determination module 13 is configured to determine the estimated image offset at the next moment based on the current image offset and the zoom data at the current moment; and determine the next frame of preview image based on the estimated image offset.

[0126] In some embodiments, the determination module 13 is further configured to determine the estimated zoom data at the next moment according to at least one historical zoom data at at least one historical moment during the zoom process and the zoom data at the current moment; and determine the estimated image offset according to the zoom data at the current moment, the current image offset, and the estimated zoom data at the next moment.

[0127] In some embodiments, the matching module 12 is further configured to perform feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame; screen the initial set of matching feature points to determine the set of matching feature points; and determine the current image offset between the first image frame and the second image frame based on the set of matching feature points.

[0128] In some embodiments, the matching module 12 is further configured to use a hardening operator module to perform feature matching on the first image frame and the second image frame to determine the initial set of matching feature points; the hardening operator module includes a hardware-implemented feature matching algorithm; use a digital signal processor to run a preset screening algorithm to screen the initial set of matching feature points to determine the set of matching feature points; the preset screening algorithm is used to remove mismatched point pairs from the initial set of matching feature points.

[0129] In some embodiments, the determination module 13 is further configured to perform data fitting on the at least one historical zoom data and the zoom data at the current moment, and perform prediction at the next moment based on the data fitting result to determine the estimated zoom data at the next moment.

[0130] In some embodiments, the matching module 12 is further configured to determine a first subject area in the first image frame and a second subject area in the second image frame; perform feature matching on the first subject area and the second subject area to determine the initial matching feature point set.

[0131] In some embodiments, the determining module 13 is further configured to process the image frames collected by the first camera and the second camera at the next moment according to the zoom data at the next moment, to obtain a next first image frame and a next second image frame; determine a target image frame from the next first image frame and the next second image frame by comparing the viewing angles of the first camera and the second camera; and process the target image frame according to the estimated image offset to obtain the next preview image.

[0132] In some embodiments, the determining module 13 is further configured to perform at least one of operations of translation, zooming in, cropping, and rotation on the target image frame according to the estimated image offset to determine the next preview image.

[0133] In some embodiments, the zooming module 11 is further configured to determine the zoom data at the current moment by receiving an operation on a preset preview interface; and determine to zoom from the first camera to the second camera when the zoom data at the current moment is within a preset focal length range between the first camera and the second camera.

[0134] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments and has similar beneficial effects to the method embodiments. For the technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0135] The embodiments of the present application further provide a chip. Figure 11 FIG. is an optional structural schematic diagram of the chip provided by the embodiments of the present application. As Figure 11 shown, the chip 2 includes: a memory 22 and a processor 23. Among them, the memory 22 and the processor 23 are connected through a communication bus 24; the memory 22 is used for storing executable instructions; the processor 23 is used for implementing the image processing method provided by the embodiments of the present application when executing the executable instructions stored in the memory 22.

[0136] The embodiments of the present application further provide a terminal, and the above-mentioned chip provided by the above embodiments can be integrated in the terminal. Refer to Figure 11As shown, the chip 2 may include a processor 23 and a memory 22 storing executable instructions executable by the processor; the processor 23 and the memory 22 communicate via a communication bus 24; the processor 23 may call and run the executable instructions from the memory 22 to implement the image processing method provided by the embodiments of the present application.

[0137] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0138] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0139] As an example, the executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file storing other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program under discussion, or stored in multiple cooperating files (such as files storing one or more modules, subroutines, or code portions).

[0140] As an example, the executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected via a communication network.

[0141] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, system, or computer program product. Therefore, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0142] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0143] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operating steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0145] As described above, only the preferred embodiments of the present application are given, and they are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, comprising: When zooming from the first camera to the second camera, performing zoom processing on the image frames collected by the first camera and the second camera according to the zoom data at the current moment to obtain a first image frame and a second image frame; Determining a current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame; Determining a predicted image offset for the next moment based on the current image offset and the zoom data at the current moment; Determining a preview image for the next frame based on the predicted image offset; Wherein, the determining a current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame includes: Performing feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame; Filtering the initial set of matching feature points to determine the set of matching feature points; Determining a current image offset between the first image frame and the second image frame based on the set of matching feature points.

2. The method according to claim 1, characterized in that, The determining a predicted image offset for the next moment based on the current image offset and the zoom data at the current moment includes: Determining predicted zoom data for the next moment according to at least one historical zoom data at at least one historical moment during the zoom process and the zoom data at the current moment; Determining the predicted image offset according to the zoom data at the current moment, the current image offset and the predicted zoom data for the next moment.

3. The method according to claim 1, characterized in that, The performing feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame includes: Using a hardened operator module to perform feature matching on the first image frame and the second image frame to determine the initial set of matching feature points; the hardened operator module includes a hardware-implemented feature matching algorithm; The filtering the initial set of matching feature points to determine the set of matching feature points includes: Using a digital signal processor to run a preset filtering algorithm to filter the initial set of matching feature points to determine the set of matching feature points; the preset filtering algorithm is used to remove mismatched point pairs from the initial set of matching feature points.

4. The method according to claim 2, characterized in that, The determining predicted zoom data for the next moment according to at least one historical zoom data at at least one historical moment during the zoom process and the zoom data at the current moment includes: Performing data fitting on the at least one historical zoom data and the zoom data at the current moment, and making a prediction for the next moment based on the data fitting result to determine the predicted zoom data for the next moment.

5. The method according to claim 1, characterized in that, Performing feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame includes: Determining a first main region in the first image frame and a second main region in the second image frame; Performing feature matching on the first main region and the second main region to determine the initial set of matching feature points.

6. The method according to any one of claims 1-5, wherein, Determining the next frame of preview image based on the estimated image offset includes: Processing the image frames captured by the first camera and the second camera at the next moment according to the zoom data at the next moment to obtain a next first image frame and a next second image frame; Determining a target image frame from the next first image frame and the next second image frame by comparing the viewing angles of the first camera and the second camera; Processing the target image frame according to the estimated image offset to obtain the next frame of preview image.

7. The method according to claim 6, wherein, Processing the target image frame according to the estimated image offset to obtain the next frame of preview image includes: Performing at least one of translation, zooming in, cropping, and rotation operations on the target image frame according to the estimated image offset to determine the next frame of preview image.

8. The method according to any one of claims 1-5 or 7, wherein, The method further includes: Determining the zoom data at the current moment by receiving an operation on a preset preview interface; Determining to zoom from the first camera to the second camera when the zoom data at the current moment is within a preset focal length range between the first camera and the second camera.

9. An image processing apparatus, wherein, Comprising: A zoom module for performing zoom processing on the image frames captured by the first camera and the second camera according to the zoom data at the current moment when zooming from the first camera to the second camera to obtain a first image frame and a second image frame; A matching module for determining a current image offset between the first image frame and the second image frame by performing feature matching on the first image frame and the second image frame; A determining module for determining an estimated image offset at the next moment based on the current image offset and the zoom data at the current moment; determining the next frame of preview image based on the estimated image offset; wherein, the matching module is further configured to perform feature matching on the first image frame and the second image frame to determine an initial set of matching feature points between the first image frame and the second image frame; screening the initial set of matching feature points to determine the set of matching feature points; Determining the current image offset between the first image frame and the second image frame based on the set of matching feature points.

10. A terminal, wherein, Comprising: A memory for storing executable instructions; A processor, when executing the executable instructions stored in the memory, implements the method according to any one of claims 1 to 8.

11. A chip, characterized in that it comprises: a memory for storing executable instructions; a processor, when executing the executable instructions stored in the memory, executes the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that it stores executable instructions for causing a processor to implement the method according to any one of claims 1 to 8 when executed.

Images