Image processing method, electronic device, chip system and computer readable storage medium

Abstract

The embodiment of the present application provides an image processing method, which relates to the field of terminal technology. The method includes: by matching the feature points of the previous frame image and the current frame image during shooting, a vector pointing from the feature point of the previous frame to the feature point of the next frame is obtained, and then the ratio of the vector and the distance from the feature point of the previous frame to the center of the feature point is calculated, and the first adjustment length is further determined. Then the second adjustment length is obtained by smoothing the first adjustment length including the current frame and the previous multiple frames. Finally, the compensation adjustment length is determined based on the difference between the first adjustment length and the second adjustment length, and the size of the cropping window of the displayed image is adjusted. In this way, when the user uses a mobile phone to shoot, he can always see a stable image, and the objects in the picture will not appear to be large or small due to random jitter on the Z axis, further improving the user's shooting experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, an electronic device, a chip system, and a computer-readable storage medium. Background Art

[0002] Nowadays, more and more users are accustomed to using mobile phones to take photos or videos anytime and anywhere, and hope that the pictures during the shooting process are stable and will not shake with random slight changes in the user's posture.

[0003] One existing image stabilization technology is EIS, which uses software algorithms to reduce image blur caused by camera movement. EIS can analyze the motion data provided by the gyroscope and accelerometer, and then make appropriate adjustments to the image to compensate for the camera movement.

[0004] However, the prior art generally does not consider the translation anti-shake processing in the direction perpendicular to the mobile phone screen, which may cause the objects in the picture to appear larger or smaller. Summary of the invention

[0005] An embodiment of the present application provides an image processing method, which is applied to the field of terminal technology. The displacement vectors of feature points in two adjacent frames of images are calculated to obtain the movement ratio, and then the compensation adjustment length is calculated through smoothing and other steps to adjust the cropping window in the original image. This can avoid abrupt changes in the image caused by jitter in the direction perpendicular to the terminal device interface.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Determine a first adjustment length according to a first feature point set in the first original image and a second feature point set in the second original image, the first adjustment length being used to adjust a size of a cropping window corresponding to the first original image, the cropping window being used to crop the first original image to obtain a first image frame, and the first original image is acquired at a time after the second original image is acquired;

[0008] updating a first curve according to a first adjustment length corresponding to the first original image, wherein the first curve includes adjustment lengths corresponding to the plurality of original images respectively;

[0009] Smoothing the first curve to obtain a second curve, and obtaining a second adjustment length corresponding to the first original image in the second curve;

[0010] Determining a compensation adjustment length according to a difference between the first adjustment length and the second adjustment length;

[0011] The cropping window corresponding to the first original image is adjusted according to the compensation adjustment length to determine a first image frame to be displayed in the first original image.

[0012] In this implementation, the first original image can be understood as the original image of the current frame, which is a complete image captured by the camera. Similarly, the second original image can be understood as the original image of the previous frame. In the original image, multiple feature points can be obtained by a feature point detection algorithm, and the feature points in the first original image and the second original image can be matched to obtain multiple feature point pairs. Based on multiple feature point pairs, the first adjustment length can be determined. The first adjustment length can be understood as a length of image enlargement or reduction caused by jitter in the Z-axis direction, or jitter perpendicular to the terminal device interface direction, which reflects the degree to which the image is enlarged or reduced. Further, the first adjustment length sequence corresponding to the current frame and the previous multiple frames is smoothed to obtain a compensation adjustment length. Finally, the size of the cropping window is determined according to the compensation adjustment length, and corresponding adjustments are made so that the picture seen in the display interface of the user terminal device remains stable, and even if there is jitter on the Z axis, the size of the person or object in the image will not be large or small.

[0013] In a possible implementation, determining the first adjustment length according to the first feature point set in the first original image and the second feature point set in the second original image includes:

[0014] Performing matching processing on the first feature point set and the second feature point set to determine a plurality of feature point pairs;

[0015] For any one of the plurality of feature point pairs, determining a movement ratio corresponding to the feature point pair according to a first coordinate corresponding to a first feature point in the feature point pair and a second coordinate corresponding to a second feature point in the feature point pair;

[0016] The first adjustment length is determined according to the movement ratios corresponding to the plurality of feature point pairs.

[0017] In a possible implementation, determining the movement ratio corresponding to the feature point pair according to a first coordinate corresponding to a first feature point in the feature point pair and a second coordinate corresponding to a second feature point in the feature point pair includes:

[0018] Determine a displacement vector corresponding to the feature point pair according to the first coordinate and the second coordinate;

[0019] According to the ratio of the first length to the second length, the movement ratio corresponding to the feature point pair is determined, where the first length is the distance between the second feature point in the feature point pair and the center position of the feature point, the second length is the vector length of the displacement vector, and the center position of the feature point is the position corresponding to the mean of the coordinates of the multiple second feature points in the second feature point set.

[0020] In this implementation, the displacement vector from the second coordinate to the first coordinate can represent the position change of the matched feature point pair in two adjacent frames due to the scaling of the image. The moving distances of different feature point pairs are different. It can be understood that the closer the feature point is to the center of the feature point or the scaling center, the shorter the moving distance, and vice versa. The farther the feature point is from the center of the feature point or the scaling center, the longer the moving distance. However, the ratio of the first length to the second length, that is, the movement ratio obtained by dividing the modulus of the displacement vector by the length from the second coordinate to the center of the feature point, is theoretically equal for different feature point pairs when errors are ignored. In other words, the degree of image enlargement or reduction caused by Z-axis jitter can be accurately described by the movement ratio, providing a basis for subsequent further processing.

[0021] In a possible implementation, determining the first adjustment length according to the movement ratios corresponding to the plurality of feature point pairs includes:

[0022] Obtaining a third length, where the third length is the distance between the first vertex of the cropping window corresponding to the second original image and the center position of the feature point;

[0023] For any one of the plurality of feature point pairs, determining a sub-adjustment length of the feature point pair corresponding to the first vertex according to a product of the third length and the movement ratio corresponding to the feature point pair;

[0024] The first adjustment length is determined according to the average values ​​of the sub-adjustment lengths corresponding to the plurality of feature point pairs.

[0025] In this implementation, the sub-adjustment length is obtained by combining the calculated movement ratio and the product of the third length, which can be used to describe the pixel length that needs to be adjusted from the first fixed point of the image to the center of the feature point, that is, the proportional relationship is converted into the actual length. Furthermore, by calculating the mean of the sub-adjustment lengths of multiple feature point pairs, the first adjustment length can be made more accurate and the possible error of a single feature point pair can be reduced. After the first adjustment length is obtained, it can be more convenient to perform smoothing based on the first adjustment length and calculate the compensation adjustment length.

[0026] In a possible implementation, adjusting the cropping window corresponding to the first original image according to the compensation adjustment length to determine a first image frame to be displayed in the first original image includes:

[0027] adjusting the size of the cropping window in the first original image according to the compensated adjustment length;

[0028] The first image frame is obtained by cropping the first original image according to the cropping window after the size is adjusted.

[0029] In a possible implementation, adjusting the size of the cropping window in the first original image according to the compensation adjustment length includes:

[0030] Determine an adjustment ratio according to a ratio of the compensation adjustment length to a third length, where the third length is a distance between a first vertex of the cropping window corresponding to the second original image and a center position of the feature point;

[0031] Determining a first distance to be adjusted in the length direction of the cropping window according to the adjustment ratio and the length of the first original image;

[0032] Determining a second distance to be adjusted in the height direction of the cropping window according to the adjustment ratio and the height of the first original image;

[0033] In the first original image, the cropping window is adjusted according to the first distance and the second distance to obtain a size of the adjusted cropping window.

[0034] In this implementation, based on the compensation adjustment length and the size of the original image, the distance that needs to be adjusted in the length and height of the cropping window is calculated. In the embodiment of the present application, the length of the cropping window can also be understood as the width of the cropping window. It is necessary to adjust the first distance in the length direction and the second distance in the height direction respectively. And the first distance and the second distance themselves can also have a positive and negative relationship. For example, when the first distance is positive, it can be indicated that the length of the cropping window is reduced, and vice versa. Through the above steps, the cropping window is adjusted to a suitable size, which can just compensate for the image reduction or enlargement caused by the jitter displacement of the Z axis. For example, when the camera is away from the object being photographed, causing the content in the original image to shrink, the cropping window is also correspondingly reduced. In this way, when the picture in the cropping window is presented to the display interface of the user terminal device, it can still maintain the same size to offset the impact of the Z-axis jitter.

[0035] In a possible implementation, in the first original image, adjusting the cropping window according to the first distance and the second distance to obtain a size of the adjusted cropping window includes:

[0036] Taking the center position of the feature point in the first original image as the scaling center, adjusting the initial length of the cropping window by a first distance, and adjusting the initial height of the cropping window by a second distance, to obtain an adjusted size of the cropping window.

[0037] In this implementation, it is emphasized that the size of the cropping window is adjusted with the center position of the feature point as the scaling center. After such adjustment, on the one hand, the size of the person or object in the image can be kept stable, and on the other hand, the relative position of the photographed object in the image displayed by the user terminal device can be ensured to remain unchanged. Otherwise, in some cases, especially when the center position of the feature point does not coincide with or is close to the center position of the image, the relative position of the person or object seen in the image displayed by the user terminal device will change after the size of the cropping window is scaled. Therefore, it is necessary not only to determine the size of the cropping window, but also to make reasonable adjustments to achieve the best effect.

[0038] In a possible implementation, adjusting the initial length of the cropping window by a first distance includes:

[0039] Adjusting the initial length of the cropping window by a first length distance from the left side, and adjusting the initial length of the cropping window by a second length distance from the right side, wherein the sum of the first length distance and the second length distance is equal to the first distance;

[0040] The method also includes:

[0041] In the case where the first length distance is greater than the first remaining length, the first remaining length is determined as the first length distance; the first remaining length is the horizontal distance between the left edge of the cropping window and the left edge of the first original image;

[0042] When the second length distance is greater than the second remaining length, the second remaining length is determined as the second length distance; the second remaining length is the horizontal distance between the right edge of the cropping window and the right edge of the first original image.

[0043] In a possible implementation, adjusting the initial height of the cropping window by a second distance includes:

[0044] Adjusting the initial height of the cropping window by a first height distance from the upper side, and adjusting the initial height of the cropping window by a second height distance from the lower side, wherein the sum of the first height distance and the second height distance is equal to the second distance;

[0045] The method also includes:

[0046] In the case where the first height distance is greater than the first remaining height, the first remaining height is determined as the first height distance; the first remaining height is the vertical distance between the upper edge of the cropping window and the upper edge of the first original image;

[0047] When the second height distance is greater than the second remaining height, the second remaining height is determined as the second height distance; the second remaining height is the vertical distance between the lower edge of the cropping window and the lower edge of the first original image.

[0048] In this implementation, the size of the calculated cropping window is further restricted. It is understandable that, for example, when the original cropping window needs to be enlarged, the enlarged cropping window may exceed the boundary of the original image. In this way, if the image in the cropping window is directly displayed without excessive processing, the image seen by the user in the display interface of the terminal device may have an area where the image is missing, which will significantly affect the user's experience. After limiting the size of the cropping window, it is possible to avoid the situation where the boundary of the cropping window exceeds the boundary of the original image in a few cases, so that the image processing method provided in the embodiment of the present application can always operate normally.

[0049] In a second aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0050] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0051] In a fourth aspect, the present application provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0052] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0053] It should be understood that the second to fourth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the rotation axis and translation axis of a terminal device provided in an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the implementation of the EIS provided in the embodiment of the present application Figure 1 ;

[0056] Figure 3 Schematic diagram of the implementation of the EIS provided in the embodiment of the present application Figure 2 ;

[0057] Figure 4 Schematic diagram of the implementation of the EIS provided in the embodiment of the present application Figure 3 ;

[0058] Figure 5 Schematic diagram of a user shooting scene provided in an embodiment of the present application Figure 1 ;

[0059] Figure 6 Schematic diagram of a terminal device displaying an image provided in an embodiment of the present application Figure 1 ;

[0060] Figure 7 Schematic diagram of a user shooting scene provided in an embodiment of the present application Figure 2 ;

[0061] Figure 8 Schematic diagram of a user shooting scene provided in an embodiment of the present application Figure 3 ;

[0062] Fig. 9 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0064] Fig.11 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 1 ;

[0065] Fig.12 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 2 ;

[0066] Fig.13 A schematic diagram of implementing translation compensation provided in an embodiment of the present application;

[0067] Fig.14 Vector diagram of feature point pairs provided in the embodiment of the present application Figure 1 ;

[0068] Fig.15 Vector diagram of feature point pairs provided in the embodiment of the present application Figure 2 ;

[0069] Fig.16A schematic diagram of an implementation of calculating a first adjustment length provided in an embodiment of the present application;

[0070] Fig.17 Schematic diagram of the implementation of the smoothing process provided in the embodiment of the present application Figure 1 ;

[0071] Fig.18 Schematic diagram of the implementation of the smoothing process provided in the embodiment of the present application Figure 2 ;

[0072] Fig.19 A schematic diagram of implementing the calculation of cropping window size adjustment provided in an embodiment of the present application;

[0073] Fig. 20 Schematic diagram of adjusting the cropping window provided in the embodiment of the present application Figure 1 ;

[0074] Fig.21 Schematic diagram of adjusting the cropping window provided in the embodiment of the present application Figure 2 ;

[0075] Fig. 22 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 3 ;

[0076] Fig.23 A schematic diagram of an implementation of the optical flow method provided in an embodiment of the present application;

[0077] Fig.24 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 4 ;

[0078] Fig.25 A schematic diagram of an implementation of the accelerometer method provided in an embodiment of the present application;

[0079] Fig.26 A schematic diagram of a terminal device functional interface provided in an embodiment of the present application;

[0080] Fig. 27 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 1 ;

[0081] Fig.28 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 2 ;

[0082] Fig.29 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 3 ;

[0083] Fig.30 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 5 ;

[0084] Fig.31 A table schematic diagram of the characteristics of the first adjustment length acquisition method provided in an embodiment of the present application;

[0085] Fig.32 A schematic diagram of a usage strategy of the first adjustment length acquisition method provided in an embodiment of the present application;

[0086] Fig.33 A schematic diagram of smoothing based on a model method provided in an embodiment of the present application;

[0087] Fig.34 A schematic diagram of an anti-cropping window exceeding the original image boundary provided in an embodiment of the present application;

[0088] Fig.35 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0090] 1. Gyroscope

[0091] Gyroscopes can be used as a sensor in terminal devices. Their working principle is based on the law of conservation of angular momentum and the Coriolis effect in physics. Gyroscopes in terminal devices usually refer to micro-electromechanical system gyroscopes. Inside the gyroscope, there is a very small oscillating mass block. When the terminal device rotates as a whole, this oscillating mass block will feel an additional force due to the Coriolis effect, causing the oscillation mode to change. By measuring this change, the speed and direction of rotation can be calculated.

[0092] The gyroscope can be used as a three-dimensional input device, allowing users to perform certain operations through the terminal device. The gyroscope can detect changes in the direction of the terminal device and notify the operating system to automatically adjust the screen display direction to suit the user's viewing habits. The gyroscope can also be used in conjunction with the camera on the terminal device to achieve anti-shake function when taking photos or recording videos.

[0093] 2. Accelerometer

[0094] Accelerometer is another sensor widely used in modern intelligent terminal devices. It is mainly used to measure the linear acceleration of the device along the X-axis, Y-axis and Z-axis. Most accelerometers in modern terminal devices are manufactured using micro-electromechanical system technology. Its core component is a miniature cantilever beam or mass block. When the device is stationary, gravity will cause the cantilever beam or mass block to produce a certain displacement. When the device accelerates or decelerates, the cantilever beam or mass block will be displaced due to inertia. By detecting this displacement and converting it into an electrical signal, the acceleration of the device can be measured.

[0095] Accelerometers can detect vibrations when users walk or run, thereby counting steps. Many health and fitness applications rely on accelerometers to record user activity. In applications such as games, users can rely on accelerometers to trigger certain actions by shaking the terminal device. Similarly, accelerometers can also be used for processing in the camera anti-shake function of terminal devices.

[0096] 3. EIS

[0097] Electronic Image Stabilization (EIS) is a technology used to reduce image blur caused by hand shaking or other irregular movements during shooting. EIS can first capture information such as the rotation and linear acceleration of the device through the built-in sensor, and the processor analyzes this data through an algorithm to calculate the motion trend. The processor then crops and realigns each frame of the image based on the motion trend to keep the center of the image stable.

[0098] EIS is an effective image stabilization technology that mainly relies on software algorithms, so it is less expensive and is particularly suitable for devices that want to improve shooting stability without adding additional hardware costs. It can also be optimized according to different application scenarios and requirements, and is highly flexible.

[0099] 4. Feature point detection / matching

[0100] In the field of image processing and computer vision, feature point detection and matching is a very important technology, which is used to extract useful feature information from images and match these features between different images, thereby realizing a variety of applications such as image recognition, scene reconstruction, and target tracking. Examples include Harris corner detection, SIFT algorithm, SURF algorithm, and ORB algorithm.

[0101] After feature point detection, it is usually necessary to perform a feature point matching step, that is, to find the corresponding feature points in two or more images. This step can be achieved by calculating the distance between feature point descriptors and selecting the pair with the smallest distance as the matching pair. In order to improve the accuracy of matching, strategies such as the nearest neighbor ratio test can also be used to screen out more reliable matching results.

[0102] 5. Optical flow method

[0103] Optical flow is a technique used in computer vision to estimate the motion of pixels in an image sequence. It is mainly used to analyze the displacement of pixels between consecutive frames to infer the motion of objects in the scene. The optical flow method is based on several basic assumptions: constant brightness assumption, small motion assumption, and spatial consistency assumption.

[0104] Optical flow is a powerful tool that helps us understand motion patterns in dynamic scenes and has a wide range of applications in video analysis, motion capture, autonomous driving, and other fields. For example, in an autonomous driving assistance system, by performing optical flow analysis on the video sequence captured by the front-view camera, the moving direction and speed of obstacles such as vehicles and pedestrians can be detected in real time, providing the vehicle with the necessary obstacle avoidance decision support.

[0105] 6. Low-pass filter

[0106] Low-pass filter (LPF) is a commonly used signal processing technology. The basic idea is to allow low-frequency signals to pass through while attenuating high-frequency signals. It can be used to remove high-frequency noise and retain low-frequency signals. In digital signal processing, common low-pass filters include infinite impulse response (IIR) filters and finite impulse response (FIR) filters.

[0107] 7. Other terms

[0108] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0109] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0110] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0111] 8. Electronic devices

[0112] The electronic device of the embodiment of the present application may include a handheld device with a shooting function, a vehicle-mounted device, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (public landmobile networks). The embodiments of the present application do not limit this.

[0113] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0114] In addition, in the embodiments of the present application, the electronic device may also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0115] The electronic device in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0116] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0117] Based on the above introduction, the relevant technologies involved in this application are further introduced in detail below.

[0118] With the continuous advancement of smartphone photography technology, more and more users tend to use their phones to capture wonderful moments in life or record important moments. Modern mobile phones are not only equipped with high-performance cameras, but also integrate a series of advanced image stabilization technologies to ensure that users can take clear and stable photos and videos even in moving or unstable situations.

[0119] Among them, Optical Image Stabilization (OIS) technology is widely praised for its efficiency and reliability. OIS senses the shake of the phone through the built-in gyroscope, quickly calculates the compensation angle, and uses the voice coil motor to adjust the position of the lens to offset the shake, thereby reducing image blur caused by hand shake. In addition, Electronic Image Stabilization (EIS) is another common anti-shake technology, which mainly uses software algorithms to crop and adjust video frames to achieve picture stability. Although this method may cause a slight loss in image quality, in most cases it can effectively improve the video viewing experience. These technologies not only provide excellent anti-shake effects when shooting still pictures, but also significantly improve the stability of the picture when recording videos, allowing users to easily shoot high-quality videos while walking, running, and even exercising.

[0120] Nowadays, whether it is daily family gatherings, travel adventures, or professional sports event coverage, the image stabilization function of mobile phones has become an indispensable part, greatly enriching the user's multimedia creation experience. With the continuous development of technology, the future mobile phone anti-shake technology is expected to become more intelligent and efficient, meeting the more diverse and high-standard shooting needs of users. Figures 1 to 4 , a brief introduction to the existing EIS anti-shake technology is given.

[0121] First, if Figure 1 As shown, it is assumed here that the terminal device is a mobile phone, which contains 3 directional axes and 6 corresponding posture change modes. Among them, the X-axis and Y-axis can be understood as two axes parallel to the mobile phone screen or the back of the mobile phone, and the two are perpendicular to each other. The Z-axis is perpendicular to the plane formed by the X-axis and the Y-axis, which can also be understood as the Z-axis is perpendicular to the plane where the mobile phone screen is located. Along the X-axis direction, translation movement can be performed, and its components can be expressed as ; Similarly, the translational motion along the Y axis can be expressed as ; The translational motion along the X-axis direction can be expressed as In addition to translation in the directions of the three axes, it also includes rotation around the three axes. The corresponding three rotation components can be expressed as , and , respectively, represent the rotation around the X-axis, Y-axis, and Z-axis, and the unit can be degrees or radians. The three rotation modes of the mobile phone posture can also be compared to the rotation modes of an airplane, which are called pitch, roll / tumble, and yaw, and the corresponding rotation angles can also be called pitch angle, roll / tumble angle, and yaw angle.

[0122] After understanding the coordinate axis of the mobile phone itself and the 6 different posture changes around the coordinate axis, you can better understand the existing EIS anti-shake technology. Considering that the shaking or posture change of the mobile phone will also lead to the change of the camera posture, which in turn causes certain changes or shaking in the picture captured by the camera. Then from the perspective of software algorithms, the main idea of ​​image stabilization or anti-shake processing is to adjust or restore the image in various ways to restore it to the image that should be taken without shaking.

[0123] like Figure 2 As shown, in Figure 2 The original image 201 in the figure represents the image actually captured by the camera in the mobile phone in a certain frame, and the cropping window 202 is to display the sub-image contained therein on the user display interface. It can be understood that the image actually seen by the user on the display interface corresponds to the image in the cropping window 202, rather than the complete original image 201 originally captured by the camera. The cropping window can also be called a display frame, a sub-image frame, etc., which does not affect its actual meaning. Assume that the size of the original image 201 is W in length and H in height, and the size of the cropping window 202 is w in length and h in height. For example, there is a hexagonal object 203 in the original image 201. The object 203 is located in the center of the original image 201, and from the sub-image included in the cropping window 202, the object 203 is located in the lower right corner of the sub-image.

[0124] In the above manner, when the mobile phone shakes and the original image taken by the camera shakes, the image stabilization function can be achieved by adjusting the position of the cropping window and performing image processing on the original image. Specifically, the existing EIS technology generally first performs perspective transformation on various rotation axes of the X-axis, Y-axis and Z-axis, and then performs translation transformation on the X-axis and Y-axis.

[0125] refer to Figure 3 The contents in Figure 3 As shown in (a) of FIG. 1 , assuming that the image 301 of the previous frame, that is, the N-1th frame, is used as a reference, there is a rectangular object 302 placed on a plane in the picture. Assuming that the camera rotates on the X axis, or the pitch angle changes, for example, the camera is raised to an angle upward, then the following will appear: Figure 3 The situation shown in (b) in Figure 3 In (b), that is, in the current frame where the camera changes its pitch angle, or in the image 303 of the Nth frame, due to the change in angle, based on the perspective principle that objects far away are small and near are large, the rectangular object 302 appears to be transformed into a trapezoidal object 304. In EIS, the image processing can be reversed by obtaining the components of the gyroscope rotating in three directions, that is, perspective transformation processing. Figure 3As shown in (c), it can be understood as being similar to stretching or compressing the Nth frame image 303, for example, stretching the image in the upper extension direction and compressing the image in the lower extension direction, thereby obtaining an image 305 after perspective transformation, in which the object is restored to a rectangular object 306 as in the N-1th frame which has not been rotated.

[0126] After the perspective transformation, a translation transformation is also performed, such as Figure 4 As shown, in Figure 4 In (a), it is assumed that there is a hexagonal object 403 in the original image 401, and the cropping window 402 also includes the object 403. For the sake of convenience, it is assumed that the object 403 is located at the center of the cropping window 402. Next, due to the shaking of the mobile phone, the image captured by the camera may also be offset to a certain extent, such as Figure 3 As shown in (b), assuming that the object 403 itself is stationary and the camera is displaced toward the upper left corner, then in the original image 401 of the current frame, the object 403 will be relatively displaced toward the lower right corner. In the existing EIS technology, the acceleration in the X-axis and Y-axis directions after stripping the gravity acceleration can be obtained in the accelerometer, and then the displacement in the X-axis and Y-axis directions can be obtained by integrating the acceleration. Finally, Figure 4 As shown in (c), by moving the cropping window 402 in the opposite direction of the compensation vector 404, the object 403 can still be located at the center of the cropping window 402. From the user's perspective, although the handheld mobile phone has a certain degree of shaking, the objects in the picture can still remain stable and will not shake randomly.

[0127] In addition to the sensor-based translation transformation, it can also be performed based on image feature points. The compensation vector can be obtained by calculating the coordinate distance of the feature point before and after. The calculation formula can be as follows:

[0128] Formula 1

[0129] Formula 2

[0130] Among them, m represents the number of feature point pairs, and Respectively represent the displacement in the X-axis direction and the Y-axis direction, in pixels. Represents the horizontal coordinate of the i-th feature point in the N-th frame. The remaining variables can be understood similarly.

[0131] However, there are some deficiencies or problems in the implementation of the above-mentioned image stabilization or anti-shake function. In order to better explain this problem, the following first combines some actual scenarios for explanation.

[0132] Scenario 1: The user's handheld mobile phone moves on the Z axis, causing jitter in the captured image.

[0133] This scenario is more likely to occur when users actually use their mobile phones to take pictures. Figure 5 and Figure 6 to explain.

[0134] like Figure 5 As shown, in Figure 5 In (a), assume that user 501 holds mobile phone 502 and starts to shoot a cylindrical object 503, and user 501 stands still. The distance between mobile phone 502 and object 503 can be regarded as the object distance, or can be understood as the distance on the Z axis of the mobile phone. Assume that the distance between object 503 and mobile phone 502 on the Z axis is D1. It is understandable that when the user holds the mobile phone, some deviation will inevitably occur, such as Figure 5 As shown in (b) of FIG. 5 , it is assumed that the user accidentally moves the mobile phone 502 forward for a certain distance. At this time, the distance between the object 503 and the mobile phone 502 on the Z axis is shortened, and it is assumed that it becomes D2 at this time. Similarly, in Figure 5 In (c), assuming that the user holds the mobile phone and deviates backward, or in other words, deviates in a direction away from the object 503, then the corresponding Z-axis distance D3 at this time will be greater than the above-mentioned Z-axis distances D1 and D2.

[0135] It is understandable that when the camera is translated on the Z axis, causing the object distance to change, the size of the object in the captured image will change accordingly, and this phenomenon will be more obvious when the object distance is shorter. Figure 6 As shown, Figure 6 Corresponds to Figure 5 The content in Figure 6 (a) in corresponds to Figure 5 The picture seen by the user at the moment (a) is that there is a cylindrical object 602 in the image interface of the mobile phone 601. Figure 6 (b) in the above corresponds to Figure 5 In the picture at time (b), the cylindrical object 602 in the image interface of the mobile phone 601 becomes larger due to the camera shaking and the shortened object distance. Figure 6 (c) in corresponds to Figure 5 In the picture at time (c), the cylindrical object 602 in the image interface of the mobile phone 601 is shrunk due to the increase in object distance.

[0136] Scenario 2: When tracking and shooting a moving object, the size of the object on the phone screen changes as it approaches or moves away.

[0137] Sometimes, users want to focus on tracking a person, animal or object. In this case, the camera will generally automatically focus on the object being tracked, and as the object moves, it will try to keep it close to the center of the image. When the displacement of the object being tracked is large, for example, it exceeds the original screen boundary, if the camera is not equipped with an automatic steering function, the user needs to adjust the camera's shooting angle to include the object being tracked in the camera's shooting field of view.

[0138] Here we focus on the situation when the tracked object moves along the Z axis of the mobile phone, such as Figure 7 As shown, in Figure 7 In (a), it is assumed that user 701 holds mobile phone 702 and remains still, tracking and photographing a kitten 703 in front of him. The distance between kitten 703 and mobile phone 702 on the Z axis is assumed to be Z axis distance D1. Kitten 703 is quite lively. Figure 7 As shown in (b), the cat 703 runs forward, shortening the object distance, and the distance between the two is the Z-axis distance D2. Figure 7 As shown in (c), the kitten 703 may suddenly run backwards to increase the object distance. At this time, the object distance between the two is the Z-axis distance D3.

[0139] In this scenario, the sudden movement of the tracked object away from or towards the camera will cause the image presented in the camera to suddenly become smaller or larger. In some cases, the user may prefer that the tracked object maintain an appropriate size in the picture, or change smoothly, rather than suddenly changing size, which would affect the viewing experience.

[0140] Scenario 3: When a user is photographing an object while actively approaching or moving away from the object, the size of the photographed object may change undesirably in the mobile phone screen.

[0141] This scenario is also a scenario that often occurs in real life, such as Figure 8 As shown, in Figure 8 In (a), assume that there is a fixed cylindrical object 803. At the beginning, user 801 is far away from object 803, assuming that the distance is Z-axis distance D1. User 801 holds mobile phone 802 and uses mobile phone 802 to shoot object 803. Assume that user 801 wants to see the details of object 803 more clearly, so he gradually approaches object 803. Figure 8 As shown in (b), user 801 walks forward, and the object distance is shortened, assuming that the Z-axis distance is D2. Similarly, Figure 8 As shown in (c), the user 801 continues to walk forward, and at this time, the Z-axis distance is assumed to be D3, which is shorter than the previous distances D1 and D2.

[0142] It is understandable that when a user is moving and shooting an object, the movement of the person and the shaking of the hand will make the phone unstable, which will inevitably cause the captured object to shake in the image. In this scenario, an ideal situation is that as the user moves, the captured object changes smoothly in the phone's screen. For example, when the user gradually approaches the captured object, the user hopes that the object seen on the phone will grow larger smoothly, and does not want the object to grow larger and smaller in the process of gradually growing larger.

[0143] The above scenarios all involve the problem that the objects in the captured image are sometimes larger and sometimes smaller and unstable due to the displacement of the Z axis during the shooting process of the terminal device. Since the existing image stabilization technology generally lacks the function of anti-shake for Z axis translation, the image stabilization or anti-shake function in the existing terminal devices cannot solve the problems in the above scenarios.

[0144] In a small number of terminal devices, there is a Z-axis translation anti-shake function, which is mainly based on the acceleration data in the Z-axis direction obtained by sensors such as accelerometers, and the movement distance of the terminal device on the Z-axis is calculated, and then the image is scaled and compensated according to the Z-axis movement distance to achieve Z-axis anti-shake. However, such a Z-axis anti-shake function also has some problems. First, the scheme has various errors: 1. There is a certain measurement error in sensors such as accelerometers; 2. The process of gradual integration calculation will also accumulate some errors; 3. The Z-axis of the accelerometer is always consistent with the Z-axis of the terminal device. As the terminal device shakes, its Z-axis will also shift, and such errors are difficult to avoid; 4. The conversion from the Z-axis movement distance to the image scaling compensation will also have errors due to the complexity of the imaging system. Second, the scheme defaults to treating all displacements on the Z-axis as jitters, while ignoring the possible directional movement of the user. Third, relying solely on sensors such as accelerometers to achieve Z-axis anti-shake has certain deficiencies in the completeness of the function.

[0145] For example, in the above scenario 3, if the displacement caused by the user's movement is also regarded as random jitter and corresponding scaling compensation is performed, then it may be seen that the size of the object being photographed in the mobile phone remains unchanged as the user moves, but this does not meet the user's expectations. For another example, in the above scenario 2, since the terminal device remains stationary and the object being photographed is moving, the displacement on the Z axis obtained from the accelerometer data is zero in this case, and no corresponding compensation will be performed.

[0146] Based on this, the embodiments of the present application propose the following technical concepts: First, after the existing EIS anti-shake processing, multiple vectors are obtained based on the matching multiple feature point pairs on the two adjacent frames of images before and after shooting, and each feature point pair constitutes a vector. The vector represents the change of the feature point on the latter frame of the image relative to the corresponding feature point on the previous frame of the image. Furthermore, the modulus of the vector is used to compare the distance between the feature point and the center of the feature point in the previous frame. The ratio can be understood as the ratio of the enlargement or reduction of the objects in the image after the Z-axis translational jitter. By multiplying this ratio with the distance between a vertex of the cropping window and the center of the feature point, a pixel distance can be obtained, which can also be called the first adjustment length, corresponding to the compensation amount of the translation length of the camera on the Z axis of the current frame compared to the previous frame.

[0147] Then, a sequence consisting of the first adjustment lengths of the current frame and the previous multiple frames is smoothed, and the sequence can also be called a first curve. After smoothing, a second adjustment length can be obtained, and the difference between the first adjustment length and the second adjustment length is the compensation adjustment length. Finally, based on the above compensation adjustment length, the size of the cropping window is adjusted to obtain the image of the current frame on the display interface of the terminal device.

[0148] In this way, since the change of the distance on the Z axis corresponds to the change of the feature point pair, the image scaling information can be obtained based on the change information of the feature point pair. On this basis, the directional motion and random jitter are separated through smoothing processing, achieving a good Z-axis translation anti-shake function.

[0149] The technical solution provided in this application can be applied to terminal devices. The following first briefly introduces the terminal devices.

[0150] For example, Fig. 9 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application is shown.

[0151] Fig. 9A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 900 may include a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charging management module 940, a power management module 941, a battery 942, an antenna 1, an antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, an earphone interface 970D, a sensor module 980, a button 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0152] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 900. In other embodiments of the present application, the terminal device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0153] The processor 910 may include one or more processing units, for example, the processor 910 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0154] The processor 910 may be used to control the execution logic of the image processing method and be responsible for part of the numerical calculation functions thereof.

[0155] The terminal device 900 implements the display function through a GPU, a display screen 994, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 910 may include one or more GPUs, which execute program instructions to generate or change display information.

[0156] In the embodiments of the present application, the GPU can also be used to implement specific computing functions for image processing. For example, for algorithms involving many pixels such as optical flow, the powerful parallel capabilities of the GPU can significantly speed up the calculation process and shorten the processing time.

[0157] The display screen 994 is used to display images, videos, etc. The display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 900 may include 1 or N display screens 994, where N is a positive integer greater than 1.

[0158] The terminal device 900 can realize the shooting function through an ISP, a camera 993, a video codec, a GPU, a display screen 994, and an application processor.

[0159] ISP is used to process the data fed back by camera 993. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 993.

[0160] For users, the captured images are mainly viewed through the display screen of the terminal device. Therefore, the main purpose of Z-axis stabilization is to ensure that the images viewed by users on the display screen are free of jitter and sudden changes.

[0161] The camera 993 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 900 may include 1 or N cameras 993, where N is a positive integer greater than 1.

[0162] In the embodiment of the present application, the picture taken by the camera can be called the original image. Different from the picture seen by the user on the display screen, the range of the original image can be larger, and many image processing steps are also implemented based on the original image.

[0163] The gyroscope sensor in the sensor module 980 can be used to determine the motion posture of the terminal device 900. In some embodiments, the angular velocity of the terminal device 900 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor detects the angle of the terminal device 900 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 900 through reverse movement to achieve anti-shake. The gyroscope sensor can also be used for navigation and somatosensory game scenes.

[0164] The acceleration sensor in the sensor module 980 can detect the magnitude of the acceleration of the terminal device 900 in various directions (generally three axes). When the terminal device 900 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers. It can be understood that the displacement distance on the Z axis can be calculated using the acceleration sensor or accelerometer, which can then be used for anti-shake processing on the Z axis.

[0165] The software system of the terminal device 900 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 900.

[0166] For example, Fig.10 A software structure diagram of a terminal device provided in an embodiment of the present application. Fig.10As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, the system may include an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, a hardware abstraction layer (HAL) and a kernel layer. It should be noted that the embodiment of the present application is illustrated by taking the Android system as an example. In other operating systems (such as Hongmeng system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the solution of the present application can also be implemented.

[0167] Among them, the application layer can include a series of application packages.

[0168] like Fig.10 As shown, the application layer may include applications such as camera, gallery, calendar, call, map, navigation, wireless local area network (WLAN), Bluetooth, music, video, short message, lock screen application, setting application, etc. Of course, the application layer may also include other application packages, such as payment application, shopping application, banking application, social application and other third-party applications, which are not limited in this application.

[0169] Among them, third-party applications can have functions such as face recognition, video calls, scanning, photo taking and video recording.

[0170] The Z-axis anti-shake image processing method proposed in the embodiment of the present application can be applied to various apps that include shooting functions, including cameras, videos, etc.

[0171] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, it may include an image stabilizer, an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, etc., and the embodiments of the present application do not impose any restrictions on this.

[0172] Among them, the function of the image stabilizer corresponds to the image processing function of Z-axis anti-shake proposed in the embodiment of the present application. When this function is called, multiple frames of images and cropping window data processed by EIS can be used as input to obtain the size data of the cropping window processed by Z-axis anti-shake.

[0173] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0174] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0175] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0176] OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0177] SGL is a graphics engine for 2D graphics.

[0178] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0179] The HAL layer is an encapsulation of the Linux kernel driver, providing an interface to the upper layer and shielding the implementation details of the low-level hardware.

[0180] The HAL layer can include sensor HAL, Wi Fi HAL, audio HAL, camera HAL Server unit of the HAL layer and software code library, etc.

[0181] Through the sensor HAL, upper-level applications can interact with sensors such as accelerometers in real time to obtain the latest data and implement fast Z-axis stabilization processing.

[0182] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, charging driver, etc.

[0183] Among them, the camera driver is the driver layer of the camera device or camera device, which is mainly responsible for the interaction with the hardware module.

[0184] The following, in conjunction with the accompanying drawings, describes in detail the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above-mentioned technical problems with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0185] First combine Figures 11 to 21 An image processing method provided in an embodiment of the present application is described. Fig.11 Schematic diagram of the Z-axis anti-shake process Figure 1 , Fig.12 Schematic diagram of the Z-axis anti-shake process Figure 2 , Fig.13 Schematic diagram of the implementation of translation compensation. Fig.14 Vector diagram of feature point pairs Figure 1 , Fig.15 Vector diagram of feature point pairs Figure 2 , Fig.16 Schematic diagram for calculating the first adjustment length. Fig.17 Schematic diagram of the smoothing process Figure 1 , Fig.18 Schematic diagram of the smoothing process Figure 2 , Fig.19 This is a schematic diagram of calculating the size of the cropping window. Fig. 20 Implementation diagram for adjusting the cropping window Figure 1 , Fig.21 Implementation diagram for adjusting the cropping window Figure 2 .

[0186] It is understandable that the image processing method provided in the embodiment of the present application can be applied after the existing EIS anti-shake method. Fig.11 As shown in FIG. 1 , the overall process is shown: S1101, perspective transformation; S1102, translation transformation; S1103, Z-axis translation jitter compensation. Among them, S1101 and S1102 are the main steps in the existing EIS anti-shake method. Before explaining S1103, Fig.13 A brief introduction to S1102 translation transformation is given.

[0187] like Fig.13 As shown in (a) in FIG. 1 , it is assumed that at the N-1th frame, the original image 1301 captured by the camera has a relatively smaller cropping window 1302, and there is a hexagonal object 1303 in the cropping window 1302, and it is assumed that the hexagonal object 1303 is at the center of the cropping window 1302. At the Nth frame, the user shakes the phone, causing the image captured by the camera to change, as shown in FIG. Fig.13As shown in (b) of FIG. 1 , the hexagonal object 1304 appears smaller than in the Nth frame, and its position in the original image 1301 also shifts to the lower right. After the translation transformation, we can get: Fig.13 As shown in (c) in the figure, the cropping window 1305 is also translated to the lower right direction for a distance corresponding to the translation distance of the hexagonal object 1304. Thus, after the translation transformation, the hexagonal object 1304 is still located at the center of the cropping window 1305. Fig.13 (c) and (a) in the figure, that is, comparing the Nth frame and the N-1th frame after the translation transformation, the relative position of the object in the cropping window has not changed significantly, but due to the Z-axis jitter of the camera and no compensation for it, the size of the object may change significantly.

[0188] It should be noted that in the actual image processing process, the number of feature points may reach hundreds or thousands. For the sake of convenience, the embodiments of the present application use some simplified examples for illustration, such as the hexagonal object mentioned above, and assume that its six fixed points are all feature points. For the actual more complex objects and more feature points, the image processing method provided in the embodiments of the present application is also applicable.

[0189] Next, S1103 will be described in detail. Fig.12 Steps from S1201 to S1208 shown in:

[0190] S1201: Perform matching processing on the first feature point set and the second feature point set to determine a plurality of feature point pairs.

[0191] The first feature point set in this step refers to the feature point set obtained by the feature point detection algorithm on the original image of the current frame, such as the Harris corner point detection algorithm, the SIFT algorithm, etc. The second feature point set corresponds to the feature point set in the original image of the previous frame. Matching processing is performed on the two feature point sets, for example, by comparing the distance between the feature point descriptors, the two feature points with the smallest distance are taken as a pair, or the nearest neighbor ratio test and other strategies can be used. The specific matching processing method can refer to the existing related technology, and the embodiments of the present application do not impose specific restrictions on this.

[0192] S1202: Determine a displacement vector corresponding to the feature point pair based on a first coordinate of a first feature point and a second coordinate of a second feature point in any feature point pair.

[0193] The first feature point here can be understood as a feature point belonging to the first feature point set. Similarly, the second feature point belongs to the feature point in the second feature point set, and the two are a feature point pair. Among them, the first coordinate and the second coordinate can be understood as the coordinates of the feature point in the original image, for example, the upper left corner of the original image is the coordinate origin, the right direction is the X-axis square, and the downward direction is the positive direction of the Y-axis, and the unit can be pixels. Of course, the embodiments of the present application do not limit the specific coordinate axes and coordinates, and can also be calculated based on other coordinate forms after transformation, such as using the upper left corner of the cropping window as the coordinate axis origin for coordinate calculation. The following will be uniformly described with the upper left corner of the original image as the coordinate axis origin.

[0194] It should be noted that when determining the displacement vector corresponding to the feature point, the first coordinate of the first feature point minus the second coordinate of the second feature point can be used to obtain the vector, that is, a vector containing direction and length information is determined by two points. And the second coordinate of the second feature point here is not its original coordinate, but corresponds to the coordinate after translation transformation. Fig.14 In the example, the hexagonal object in the image is still used for illustration. It is assumed that the hexagonal object 1401 belongs to the previous frame image, the hexagonal object 1402 belongs to the current frame, and their six corners are detected as feature points, and they are successfully matched in pairs: feature point A1 and feature point A2, feature point B1 and feature point B2, feature point C1 and feature point C2, feature point D1 and feature point D2, feature point E1 and feature point E2, feature point F1 and feature point F2. In the actual processing process, there may be hundreds or thousands of feature points. Here, for the convenience of understanding and explanation, some simplification is made, and only the six corners of the hexagon are used as feature points for explanation.

[0195] As mentioned above, the hexagonal object 1401 in the previous frame and the hexagonal object 1402 in the current frame have not only a size change due to the displacement of the Z axis, but also a position change due to the translation of the X axis and the Y axis. The hexagonal object 1401 here can be understood as having its coordinate points also undergone translation compensation. Now the centers of the feature points of the hexagonal object 1401 and the hexagonal object 1402 coincide. Compared with the hexagonal object 1401, the hexagonal object 1402 can be regarded as being reduced with the feature points as the center, and the vectors corresponding to the six feature point pairs contain the necessary information for this reduction.

[0196] For the calculation method of the center coordinates of the feature points, you can refer to the following formula:

[0197] Formula 3

[0198] Formula 4

[0199] The calculation formula of the displacement vector can be expressed as:

[0200] Formula 5

[0201] in, and Respectively represent the horizontal and vertical coordinates of the center of the feature point, represents the displacement vector, represents the x-coordinate of the i-th feature point pair in the N-1th frame after translation transformation, It represents the x-coordinate of the ith feature point pair of the Nth frame, and the other variables can be understood accordingly. The coordinates of the feature point center can also be calculated by the feature points of the Nth frame, which is theoretically consistent with the coordinates of the feature center point calculated by the translated feature points of the N-1th frame.

[0202] S1203: Taking the ratio of the distance between the second feature point and the center position of the feature point to the vector length of the displacement vector as the movement ratio.

[0203] The center position in this step refers to the position of the center of the feature point, and the second length corresponds to the modulus length of the above displacement vector.

[0204] To better understand the meaning of this step, you can refer to Fig.15 To understand, Fig.15 In the example, a four-pointed star shape is used to illustrate. Fig.15 As shown, it is assumed that the four-pointed star 1501 belongs to the previous frame image, and the four-pointed star 1502 belongs to the current frame image, and there are 8 feature point pairs between the two, such as feature points A1 and A2 at the convex part, and feature points B1 and B2 at the concave part. The centers of the feature points of the four-pointed star 1501 and the four-pointed star 1502 coincide, and the 8 feature points of each can belong to two concentric circles. Taking the four-pointed star 1501 as an example, the four feature points including the feature point A2 at the convex part can belong to the circle 1507 with the center of the feature point as the center of the circle, and correspondingly, the four feature points including the feature point B2 at the concave part can belong to the circle 1508 with the center of the feature point as the center of the circle. In this way, looking at the two groups of displacement vectors corresponding to the two groups of feature points separately, it can be found that: 1. The modulus length of the displacement vectors belonging to a concentric circle is consistent; 2. The closer to the center of the feature point, the smaller the modulus length of the displacement vector, and vice versa.

[0205] Furthermore, when an image is enlarged or reduced, each part of the image is enlarged or reduced in the same proportion. Fig.15Take the four-pointed star in 1501 as an example, assuming that the size is reduced by half from four-pointed star 1501 to four-pointed star 1502, then the distance from the corresponding feature point A2 to the center of the feature point should be twice the distance from the feature point A1 to the center of the feature point. Similarly, the distance from the feature point B2 to the center of the feature point should be twice the distance from the feature point B1 to the center of the feature point. The ratio of the vector modulus corresponding to feature point A2 and feature point A1 to the distance from feature point A2 to the center of the feature point should be equal to the ratio of the vector modulus corresponding to feature point B2 and feature point B1 to the distance from feature point B2 to the center of the feature point.

[0206] That is to say, the direction of the displacement vector represents the direction of change of the feature point at that location, and the modulus of the displacement vector represents the magnitude of change of the feature point at that location. The ratio of the modulus of the displacement vector to the distance from the coordinates of the second feature point to the center of the feature point can be expressed as the ratio of image reduction and enlargement, and for each pair of feature points, the calculated movement ratio should be equal in theory.

[0207] S1204: Determine a sub-adjustment length based on the product of the movement ratio of each feature point pair and the third length, and calculate an average of the multiple sub-adjustment lengths as the first adjustment length.

[0208] It is understandable that in order to ultimately achieve a stable image for the user on the terminal device, the cropping window needs to be enlarged or reduced. For example, if the object in the original image becomes relatively smaller due to the camera's shaking on the Z axis, the cropping window can be correspondingly reduced. Conversely, if the object in the original image becomes relatively larger, the cropping window can be correspondingly increased. Now that the movement ratio of each feature point pair is known, it is necessary to know how much the size of the cropping window should be adjusted.

[0209] For ease of understanding, the four vertices of the cropping window can also be regarded as feature points, and the vertices of the cropping window of the previous frame and the vertices of the cropping window to be calculated and determined in the current frame are matched one by one. In this way, according to the previous description, we can have such a proportional relationship equation:

[0210] Formula 6

[0211] In Formula 6, the first vertex can be understood as one of the four vertices of the cropping window of the previous frame, such as the vertex in the upper left corner. The distance from the first vertex to the center of the feature point can also be called the third length. Correspondingly, the adjustment length in Formula 6 can be understood as the moving distance of the first vertex toward the center of the feature point, which can also be called the sub-adjustment length calculated for the corresponding feature point pair.

[0212] Here you can refer to the content in 16 for an example. Fig.16In the figure, it is assumed that cropping window 1601 is the cropping window corresponding to the previous frame, and cropping window 1602 is the cropping window corresponding to the current frame that needs to be determined. It is also assumed that hexagonal object 1603 and hexagonal object 1604 belong to the images of the previous frame and the current frame respectively, and the feature points of the two are matched in pairs, for example, feature point A1 and feature point A2 belong to a feature point pair. A displacement vector is obtained from feature point A2 pointing to feature point A1, and the length of the vector is D1. At the same time, the distance from feature point A2 to feature point center 1605 is D2, so the movement ratio is D1 / D2.

[0213] Furthermore, the distance from the vertex of the upper left corner of the cropping window 1601 to the center of the feature point 1605 is D3, and assuming that the distance from the vertex of the upper left corner of the cropping window 1601 to the vertex of the upper left corner of the cropping window 1602 is D4, the corresponding relationship can be obtained:

[0214] Formula 7

[0215] By slightly changing formula 7, we can get:

[0216] Formula 8

[0217] D4 can also be called a sub-adjustment length, which corresponds to a compensation length of the Z-axis translation. For example, if the object captured in the image is reduced due to the Z-axis displacement, the size of the cropping window can be adjusted accordingly to keep the size of the object seen by the user in the display interface of the terminal device unchanged. The value of the cropping window vertex adjustment can correspond to the sub-adjustment length calculated above.

[0218] Although theoretically the sub-adjustment lengths calculated for each feature point should be the same, considering the errors in the actual image processing process, in order to further improve the accuracy, the average of the sub-adjustment lengths calculated for each feature point pair is calculated as the first adjustment length of the current frame. The following is a calculation formula consistent with the above content:

[0219] Formula 9

[0220] in, represents the first adjustment length, m represents the number of feature points, and Indicates the coordinates of the feature point after translation in the previous frame. and Indicates the coordinates of the first vertex of the clipping window.

[0221] In some implementations, the distance from the feature point of the current frame to the center of the feature point may also be considered to calculate the movement ratio, which is equivalent to Formula 9. The specific formula is as follows:

[0222] Formula 10

[0223] In some implementations, it is considered that the direction of the displacement vector may have two situations, away from the center of the feature point or toward the center of the feature point, which respectively correspond to the enlargement or reduction of the object in the original image. Therefore, this information can also be included in the calculation. For example, when the direction of the displacement vector is away from the center of the feature point, it is negative, and vice versa. The specific judgment method can be determined based on the displacement vector and the vector between the feature point and the center of the feature point of the previous frame. Similar to the displacement vector, the vector between the feature point and the center of the feature point of the previous frame can be expressed as follows:

[0224] Formula 11

[0225] in, Represents the vector of the feature point and the center of the feature point of the previous frame. Further, the symbol judgment formula can be as follows:

[0226] Formula 12

[0227] That is, the vector modulus length after the superposition of two vectors can be If the modulus becomes larger, it means that the object is enlarged, otherwise it means that the object is reduced in the image. Combined with formula 12, the above formula 9 can be further written as follows:

[0228] Formula 13

[0229] In some special cases, or approximate cases where a certain error is allowed, for example, when the center of the feature point coincides with or is very close to the center of the cropping window, the size of the cropping window can be used directly to simplify the calculation:

[0230] Formula 14

[0231] Wherein, w and h represent the width and height of the cropping window of the previous frame, which can be equivalent to calculating the third length distance from the first vertex to the center of the feature point.

[0232] The calculation formula provided in the embodiment of the present application may have different variations, as long as they are substantially consistent, and will not be described in detail here. The following description will be continued based on Formula 13.

[0233] S1205 , smoothing the first sequence including the first adjustment length of the current image frame to obtain a second adjustment length.

[0234] It is understandable that in real usage scenarios, the movement of the Z axis may include two parts, one part comes from the user's own trending movement, and the other part is random jitter. As for the random jitter part, it is what the image processing method provided in the embodiment of the present application really wants to compensate for, so how to separate the random jitter part from the existing calculated data is very important.

[0235] Therefore, the embodiment of the present application will further perform smoothing on the basis of the first adjustment length calculated above. The current frame corresponds to a first adjustment length, and similarly, each previous frame also corresponds to its own first adjustment length. In this way, a sequence of first adjustment lengths can be obtained by arranging them in time sequence, which can be called a first sequence. Since it is always changing, it can also be called a first curve.

[0236] In some implementations, a low-pass filter-based method can be used for smoothing. Specifically, the first adjustment lengths corresponding to the current frame and the previous frames can be superimposed by weighted addition to obtain a smoothed value, that is, the second adjustment length. It should be noted that in order to ensure the effectiveness of the smoothing result, the sum of the weight coefficients used for superposition needs to be 1. An example is given below to illustrate:

[0237] Formula 15

[0238] in, Indicates the second adjustment length, Indicates the first adjustment length of the current frame, Indicates the first adjustment length of the previous frame, and the rest can be understood by analogy.

[0239] It can be seen that the coefficient closer to the current frame is larger, and the coefficient farther away from the current frame is smaller, that is to say, when superimposing, more emphasis will be placed on data closer to the current frame in terms of timing, while also taking into account previous data. The second adjusted length obtained after smoothing can be understood as the part of the current frame with trending motion.

[0240] In some implementations, such as in the case of video recording, in addition to using the current frame and the sequence of previous multi-frame data for smoothing, the "future" multi-frame data can also be combined. For example, the data of the next frame of the current frame, that is, the N+1 frame, and the N+2 frame, etc. can be included and calculated together to obtain the second adjustment length of the current frame. In this case, a more accurate smoothing result can be obtained, which can also bring a better Z-axis anti-shake effect.

[0241] S1206: Determine a compensation adjustment length according to a difference between the first adjustment length and the second adjustment length.

[0242] As mentioned above, obtaining the second adjustment length through smoothing is not the ultimate goal, but to extract the random jitter part from the first adjustment length through smoothing, or to separate the noise data information from the first adjustment length. Therefore, after obtaining the second adjustment length, the difference between the first adjustment length and the second adjustment length is calculated to obtain the compensation adjustment length. The calculation formula for the compensation adjustment length is as follows:

[0243] Formula 16

[0244] in, Indicates the compensation adjustment length.

[0245] In order to better illustrate the role of smoothing and compensating the length adjustment, the following will combine Fig.17 and Fig.18 Let's take an example. First, let's look at Fig.17 (a) includes 3 columns of data, which can be understood as simulated experimental data to demonstrate the effect of the smoothing method. Among them, the adjustment length column corresponds to the first adjustment length mentioned above, and is arranged in chronological order. It is assumed here that the adjustment length mainly includes random jitter. The smoothing length can be understood as calculated using Formula 15, which corresponds to the second adjustment length. Because the calculation of the smoothing length involves 5 frames of data, the smoothing length of the first 4 frames is directly set to 0 here, which will not affect the experimental results. In the actual smoothing process, different methods and variants can be used, and the embodiments of the present application do not impose specific restrictions. The compensation length of the third column is calculated by subtracting the smoothing length of the second column from the adjustment length of the first column, which corresponds to the above-mentioned compensation adjustment length. Fig.17 In (b), Fig.17 The three columns of data in (a) are plotted as curves for display. It can be seen that the adjustment length curve fluctuates around the value of 0. The smoothed length curve also fluctuates around the value of 0 starting from the 5th image frame, but because it is the result of smoothing, the amplitude of the fluctuation is small. The compensation length curve is very close to the adjustment length curve starting from the 5th frame, and as the image frames increase, the compensation length curve is closer to the adjustment length curve, which shows that the compensation adjustment length can accurately represent the information of random jitter. The compensation adjustment length obtained by such processing can keep the size of the content in the image of the display interface of the user terminal device unchanged when there is only random jitter.

[0246] Let's see next Fig.18 ,exist Fig.18 In (a), it is assumed that in addition to the random jitter on the Z axis, the user also has an active displacement in the Z axis direction. The directional displacement in the first column of the table represents the value of the user's active displacement in each frame. This can be understood as in the simulation experiment, the physical distance the user moves on the Z axis is equivalent to the adjustment distance on the image. The unit can be pixels, rather than the actual physical distance the user moves on the Z axis. Similarly, the random jitter in the second column of the table corresponds to the value of the random jitter part of the user during the shooting process. The adjustment length in the third column is the result of adding the directional displacement in the first column and the random jitter in the second column, indicating the result of the superposition of the user's trend movement and random jitter, which is equivalent to the first adjustment length mentioned above. The smoothing length in the fourth column is Fig.17 The smoothing length in (a) has the same meaning, and can also be calculated using Formula 15, which corresponds to the second adjustment length, and the fifth column compensation length is also the same as Fig.17 The meaning of the compensation length in (a) is consistent with that of the compensation length, which corresponds to the compensation adjustment length. Fig.18 In the curve graph in (b), the adjustment length curve still fluctuates up and down, but because it contains both the user's directional displacement and random jitter, and it is assumed here that each frame contains a fixed directional displacement value of 5, the adjustment length fluctuates around 5. For the smooth length curve, there is also a small fluctuation around 5 after the 5th frame, but it is relatively flat overall. For the compensation length curve, it can be seen that after the 5th frame, its curve shape or trend is almost the same as the adjustment length curve, but it does not coincide with the adjustment length curve, but fluctuates around 0. This well illustrates that the compensation adjustment length calculated after smoothing can exclude the information of directional movement and only represents random jitter information. Using the compensation adjustment length for anti-shake processing can achieve a better effect. In this way, when the user carries the terminal device and moves to shoot at the same time, it will not affect the normal gradual change of the size of the object in the captured image caused by the change in the distance between the terminal device and the subject, but eliminates the uncoordinated sudden increase or decrease that may exist in the process, making the entire shooting process smoother.

[0247] S1207: Determine an adjustment ratio according to a ratio of the compensation adjustment length to the third length, and determine a length distance and a height distance of the cropping window according to the adjustment ratio.

[0248] It is understandable that after the object distance on the Z axis changes, the size of the person or object in the original image captured by the camera will also change accordingly. In order to offset this change and prevent the image viewed by the user in the terminal device from changing abruptly, the size of the cropping window needs to be adjusted accordingly. For example, if the object distance on the Z axis increases and the object in the original image shrinks, the cropping window also needs to be reduced accordingly. Conversely, the cropping window needs to be expanded accordingly.

[0249] This step is mainly used to determine the actual width and height distance of the cropping window that needs to be adjusted, or the width and height of the cropping window after adjustment. In the embodiment of the present application, the width of the cropping window can also be called the length of the cropping window. The calculation formula is as follows:

[0250] Formula 17

[0251] Formula 18

[0252] in, and They represent the width and height of the cropping window that need to be adjusted, and w and h represent the width and height of the cropping window of the previous frame.

[0253] The above formula can be further referenced Fig.19 To understand, Fig.19 As shown, the width of the cropping window 1901 of the previous frame is w, the height is h, the four vertices are vertex A, vertex B, vertex D and vertex E, and the center of the feature point is C. Correspondingly, assuming that the width of the cropping window 1902 to be calculated in the current frame is , Gao Wei , the four vertices are ,vertex ,vertex and vertices . Vertex A and vertex The distance between can correspond to , which is known, the width of the cropping window 1902 is He Gaowei It is unknown, but can be calculated through the relationship between them.

[0254] Understandably, in Fig.19 In the triangle AEC and triangle Construct similar triangles, similar, triangle ABC and triangle Construct similar triangles, triangle BDC and triangle Construct similar triangles, triangle DEC and triangle They all have the same proportional relationship, such as triangle ABC and triangle For example, their side lengths satisfy the following equation:

[0255] Formula 19

[0256] in, The value is equal to , that is, the compensation adjustment length, The value is equal to the width of the new cropping window. .

[0257] In the actual processing flow, compared with calculating the overall size of the clipping window, it is generally more desirable to obtain the vertex coordinates of the clipping window or the adjustment distance of each edge. For example, there is the following equation:

[0258] Formula 20

[0259] Where D1 represents the distance from the feature point center C to the AB edge, and D2 represents the distance from the AB edge of the cropping window 1901 to the AB edge of the cropping window 1902. distance. This means the total length that needs to be adjusted in height of the cropping window 1902 relative to the cropping window 1901. The total length that needs to be adjusted in width can also be calculated similarly, so that the meanings of the above formulas 17 and 18 can be understood. In addition, similarly, for example, the distance D3 from the center of the feature point to the BD edge can be calculated to obtain D4, as well as the distances that need to be adjusted for the other edges.

[0260] The size of the cropping window after adjustment can also be calculated by the distance the cropping window needs to be adjusted. The formula is as follows:

[0261] Formula 21

[0262] Formula 22

[0263] in, and Respectively represent the new width and height of the cropping window after adjustment.

[0264] For w and h, they can also be calculated by the scaling factor of the original image:

[0265] Formula 23

[0266] Formula 24

[0267] Among them, r represents the scaling factor or proportional coefficient of the original image and the cropping window.

[0268] S1208: Taking the center position of the feature point in the first original image as the scaling center, the size of the cropping window is adjusted according to the length distance and the height distance.

[0269] In this step, the cropping window will be adjusted accordingly based on the size of the new cropping window. Fig. 20 and Fig.21 For explanation. Fig. 20 As shown, in Fig. 20 In (a), it is assumed that in the N-1th frame, the original image 2001 contains a cropping window 2002, there is a hexagonal object 2003 in the cropping window, and it is assumed that the hexagonal object 2003 is located at the center of the cropping window. After the Nth frame is processed by translation transformation, as shown in Fig. 20 As shown in (b), the hexagonal object 2003 is still located at the center of the cropping window, but it becomes smaller due to the translation of the Z axis. After the above steps, including calculating the first adjustment length, smoothing, and calculating the size of the new cropping window, the original cropping window needs to be reduced. Fig. 20 As shown in (c), the cropping window is adjusted according to the width and height calculated by Formula 21 and Formula 22. Finally, in the display interface of the user terminal device, the following can be obtained: Fig. 20 In the image shown in (d), the hexagonal object 2005 in the image 2004 of the display interface still maintains the same size as in the N-1th frame and is not affected by random jitter.

[0270] exist Fig. 20 In the figure, a relatively simple and special case is shown, that is, the center of the feature point coincides with the center of the cropping window. In this case, it is only necessary to adjust the width and height of the cropping window in the same way, that is, D1 equals D3, and D2 equals D4. In daily shooting, due to the large number of feature points, the center of the feature point is very close to the center of the cropping window in most cases. In order to ensure the universality of the method, we will then combine Fig.21 The specific steps are explained when the center of the feature point does not coincide with the center of the cropping window.

[0271] like Fig.21 As shown, in Fig.21 In (a), it is assumed that the original image 2101 at the N-1th frame contains a cropping window 2102, there is a hexagonal object 2103 in the cropping window 2102, and it is assumed that the hexagonal object 2103 is located at the right midpoint of the cropping window 2102. Similarly, after the translation transformation, Fig.21 As shown in (b), the hexagonal object 2103 in the image is reduced in size and is still located at the right midpoint of the cropping window 2102. and After that, the cropping window needs to be adjusted based on the center of the feature point. Fig.21 As shown in (c), the four sides of the cropping window 2102 need to be adjusted by different distances. In this example, since the hexagonal object 2103 is located at the center of the cropping window 2102 in the height direction, the upper and lower sides of the cropping window 2102 are moved inward according to the above formula. , corresponding to D1 and D3. Since the hexagonal object 2103 is located at the right half of the cropping window 2102 in the width direction, the left side of the cropping window 2102 moves to the right. , while the right side moves to the left , corresponding to D4 and D2 respectively, and then a new cropping window 2104 is obtained. In other words, the distance that each edge of the cropping window needs to move is proportional to the relative position of the center of the feature point in the width and height directions of the cropping window. In some implementations, the coordinates of the four corners of the cropping window can also be directly calculated according to this method to determine the new cropping window. The embodiment of the present application does not limit the specific method. Finally, Fig.21 As shown in (d), on the display interface of the user's terminal device, it can be seen that the hexagonal object 2106 in the image 2105 is still located on the right side, and its size remains consistent with that of the N-1th frame.

[0272] Based on the multiple steps introduced above, the image processing method is further summarized and described below. First, a displacement vector is obtained by a feature point pair formed by feature points on two adjacent frames of images, and the displacement ratio is obtained by combining the center of the feature point, and then the first adjustment length is calculated. Then, the first adjustment length sequence including the current frame and the previous continuous multiple frames is smoothed to obtain the second adjustment length of the current frame, and the difference between the two is used as the compensation adjustment length. Then, according to the compensation adjustment length, the distance that the size of the new cropping window needs to be adjusted is calculated, and the scaling adjustment is performed based on the center of the feature point.

[0273] Through the above method, on the one hand, compensation processing for Z-axis translation jitter is realized, so that when users use terminal devices to shoot, people or objects in the picture will not appear to suddenly increase or decrease due to some random jitter, so that users have a better shooting experience. On the other hand, the image processing method proposed in the embodiment of the present application has good universality and can be adapted to a variety of shooting situations. For example, processing is performed based on feature points, so the Z-axis jitter caused by the photographed object can still be compensated. And for the directional movement of the user himself during shooting, the real Z-axis jitter can be separated by smoothing processing without affecting the normal changes of the image. The people or objects in the picture can be gradually enlarged or reduced to avoid the situation of sudden enlargement and reduction, so that users can more easily shoot excellent works.

[0274] Based on the above embodiments, Fig. 22 and Fig.23 Another image processing method provided in the embodiment of the present application is described in detail. Fig. 22 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 3 , Fig.23 A schematic diagram of an implementation of the optical flow method provided in an embodiment of the present application.

[0275] refer to Fig. 22 The method flow in:

[0276] S2201. Obtain displacement data of each pixel in the first original image compared with the second original image according to an optical flow method.

[0277] The optical flow method used in the embodiment of the present application is similar to the method of feature points used in the previous embodiment, and can be compared to using more pixels as feature points for calculation in this step. For example, the pixels in the previous frame image can be regarded as the second feature point set, and the pixels in the current frame image can be regarded as the first feature point set. The displacement corresponding to each pixel is calculated by the algorithm through the two feature point sets, and then the corresponding subsequent steps are processed.

[0278] For ease of understanding, a specific optical flow method is briefly described here. The optical flow method assumes that the pixels of the previous frame will move to the adjacent area in the current frame, and the formula is as follows:

[0279] Formula 25

[0280] in, Represents the grayscale value of the pixel at the x and y positions at frame t, corresponding to, Represents the grayscale value of the pixel in the current frame. Taylor expansion is performed on its left side:

[0281] Formula 26

[0282] Due to the grayscale invariance assumption, we can get:

[0283] Formula 27

[0284] Further transformation yields:

[0285] Formula 28

[0286] In formula 28, and It can be understood that the gradient value of the pixel gray value along the X direction and the Y direction can be directly calculated from the image. The change in the gray value of a pixel between two frames can also be directly calculated from the image difference. and They represent the moving speed of the pixel in the X direction and the Y direction respectively, which need to be solved.

[0287] For example, a specific solution method may be used to treat the neighboring area of ​​a pixel as a sub-area, and assuming that the pixels in the sub-area have the same moving speed in the X direction and the Y direction, a pixel corresponds to an equation of Formula 28, and the pixels in the sub-area may constitute a set of equations, and by solving the set of equations, the changes of each pixel are obtained. For more methods, reference may be made to the relevant prior art, which will not be described in detail here.

[0288] In order to better understand the optical flow method, let's combine it with Fig.23 For explanation. Fig.23 In the figure, the horizontal axis of the coordinate axis is a pixel sequence. An image has two directions, X and Y. Here, it is assumed that only one of the directions is explained, and the vertical axis of the coordinate axis is the gray value. The curve in the image represents the change of the pixel gray value in one direction from a certain point on the image, wherein curve 2301 represents the distribution of the pixel gray value of the previous frame, and for each pixel therein, the gradient of the pixel along this direction can be calculated. For example, for pixel point A, the gray value near it will increase to the right, so the gradient of pixel point A is also to the right. Curve 2302 represents the distribution of the pixel gray value of the current frame. At this time, if we look at the gray value corresponding to pixel point A again, we will find that it has become smaller than that in the previous frame. Then, according to the gradient of pixel point A, we can know that pixel point A has moved a distance to the right. In this way, the changes of each pixel in two adjacent frames can be quantitatively calculated to obtain the new pixel coordinates after the change.

[0289] S2202: Calculate a first adjustment length according to the displacement data of each pixel.

[0290] Similar to the process of calculating the first adjustment length based on the feature point method in the previous embodiment, the pixel points in this step can be compared to the feature points, the displacement vectors between the paired pixel points are calculated, and then the displacement ratio and the first adjustment length are calculated in combination with the center coordinates. For more information, please refer to the previous Fig.12 S1203 and S1204 in the above will not be described in detail here.

[0291] S2203: Smoothing the first sequence of the first adjustment length to obtain a second adjustment length, and determining a compensation adjustment length according to a difference between the first adjustment length and the second adjustment length.

[0292] This step can refer to the previous Fig.12 The second part, that is, S1205 and S1206, will not be described in detail here.

[0293] S2204: Adjust the cropping window corresponding to the first original image according to the compensation adjustment length to determine a first image frame to be displayed in the first original image.

[0294] This step can refer to the previous Fig.12 The third part, that is, S1207 and S1208, will not be described here.

[0295] In summary, the image processing method proposed in the embodiment of the present application is summarized below. By introducing the optical flow method to calculate the first adjustment length, and then performing smoothing to obtain the compensation adjustment length, and adjusting the size of the cropping window, the picture seen in the user terminal device interface can remain stable, avoiding the problems caused by Z-axis displacement jitter. If the performance of the user terminal device allows, the optical flow method can bring better anti-shake effect and further enhance the user experience.

[0296] Based on the above embodiments, Fig.24 and Fig.25 Another image processing method provided in the embodiment of the present application is described in detail. Fig.24 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 4 , Fig.25 A schematic diagram of an implementation of the accelerometer method provided in an embodiment of the present application.

[0297] refer to Fig.24 The method flow in:

[0298] S2401. Integrate and calculate the displacement length based on the data of the accelerometer on the Z axis.

[0299] It can be understood that the accelerometer in the terminal device can measure the acceleration values ​​on the three axes, so the acceleration on the Z axis can be used for integral calculation to obtain or approximately obtain the displacement of the terminal device on the Z axis. The following will be explained with examples in combination with the formula. Here, it is assumed that at the initial moment, or in some scenarios, it can also be understood as the moment when the camera is turned on. At this time, it is considered that the camera remains stationary at the moment, that is, the initial velocity is 0, and the cumulative displacement is also 0, then the displacement formula on the Z axis can be as follows:

[0300] Formula 29

[0301] in, It represents the acceleration measured in the Z-axis direction. It indicates the accumulated displacement on the Z axis from the initial moment obtained by double integration, and the unit is physical distance, such as millimeter, centimeter, etc. In the actual calculation process, it can also be calculated in a discrete way, that is, the current acceleration can be obtained in each frame, and then the displacement between the current frame and the previous frame can be calculated with the time difference between the two frames.

[0302] S2402. Convert the displacement length into a first adjustment length in combination with the depth information.

[0303] After obtaining the displacement of the Z axis through the accelerometer, it is necessary to further convert it into a first adjustment length to finally determine how the size of the cropping window should be adjusted. Fig.25 How to convert the physical displacement distance of the Z axis into the first adjustment length is described.

[0304] like Fig.25 As shown in (a), a simplified imaging system is shown, which includes a lens 2501, represented by a convex lens, an imaging sensor 2502 and an object 2503 to be photographed. It is further assumed that the object 2503 to be photographed is a plane, and at this time, the object distance, image distance and focal length satisfy the optical imaging relationship. When the camera shakes, as shown in Fig.25 As shown in (b), assuming that the camera moves along the Z axis and approaches the object 2503, the object distance changes, and the corresponding photographed area changes compared to Fig.25 The photographed area in (a) becomes smaller, while the image presented on the imaging sensor 2502 becomes relatively larger.

[0305] That is to say, when the object distance changes in the Z-axis direction, the person or object in the captured image will also become larger or smaller, and the proportion of the person or object in the image becoming larger or smaller is related to the displacement in the Z-axis direction and the depth information, including the focal length, image distance and object distance at this time. For example, under certain depth information conditions, if the camera on the Z-axis is close to the object being photographed, the content in the image will be magnified, otherwise it will be reduced. For another example, when the image distance and focal length are relatively small, that is, when the camera itself is close to the object being photographed, the movement of the camera on the Z-axis will cause the image to be enlarged or reduced more obviously. It can be expressed as follows using the formula:

[0306] Formula 30

[0307] Among them, f, u and v represent focal length, object distance and image distance respectively. It can be understood as the movement ratio in the previous embodiment. The specific relationship F generally needs to be calibrated and determined by simulation according to the actual scene, and will not be elaborated here.

[0308] After the movement ratio is calculated through accelerometer conversion, the first adjustment length can be calculated by following the method in the previous embodiment, and then a series of subsequent processing steps can be performed.

[0309] S2403: Smoothing the first sequence of the first adjustment length to obtain a second adjustment length, and determining a compensation adjustment length according to a difference between the first adjustment length and the second adjustment length.

[0310] This step can refer to the previous Fig.12 The second part, that is, S1205 and S1206, will not be described in detail here.

[0311] S2404: Adjust the cropping window corresponding to the first original image according to the compensation adjustment length to determine a first image frame to be displayed in the first original image.

[0312] This step can refer to the previous Fig.12 The third part, that is, S1207 and S1208, will not be described here.

[0313] Here is a summary of the above description. In the embodiment of the present application, an accelerometer is combined to perform anti-shake compensation processing on the Z axis. By integrating the data of the accelerometer on the Z axis, the physical displacement of the terminal device or camera on the Z axis can be obtained. Combined with the depth information, the physical displacement on the Z axis is converted into the movement ratio of the image, and then the first adjustment length is calculated, and subsequent processing steps can be performed. In general, the accelerometer has a fast response speed and low power consumption, and can be applied to some terminal devices with energy consumption restrictions. In addition, when using an accelerometer, the requirements for image quality will be reduced, and it can be adapted to scenes with low-quality images, such as at night or in rainy and snowy weather.

[0314] Next, based on the above-mentioned embodiments, Figure 26 to Figure 29 Another image processing method provided in the embodiment of the present application is described. Fig.26 A schematic diagram of a terminal device functional interface provided in an embodiment of the present application, Fig. 27 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 1 , Fig.28 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 2 , Fig.29 Schematic diagram of the Z-axis anti-shake function of the terminal device provided in the embodiment of the present application Figure 3 .

[0315] In some implementations, after the Z-axis anti-shake processing function provided in the above embodiment is used in the terminal device, relevant settings can be made in the terminal device. Fig.26 As shown, in Fig.26 In (a) of FIG. 26 , the user can find and click the anti-shake control 2603 below the camera preview interface 2602 of the terminal device 2601. After clicking, the user can see the following Fig.26 In the interface shown in (b), in the interface for setting the anti-shake function, it can include whether to turn on the anti-shake function, which is used to control whether the overall anti-shake function is turned on. Furthermore, some detailed settings can be made, such as whether to turn on the Z-axis displacement anti-shake function and set the anti-shake accuracy. Among them, different anti-shake accuracy can correspond to different anti-shake methods, which will be explained in detail later and will not be repeated here.

[0316] Assume that the user has turned on the anti-shake function and the Z-axis displacement anti-shake function, for example, in a selfie scene, Fig. 27 As shown. Fig. 27In the example, user 2701 uses mobile phone 2702 to shoot his own face. During the shooting process, shaking may occur, for example, mobile phone 2702 shakes forward or backward in the Z-axis direction. However, since the Z-axis anti-shake image processing method provided by the above embodiment is activated, the face seen by user 2701 in the image preview interface 2703 of his mobile phone 2702 will not suddenly become larger or smaller due to shaking, and can always maintain a certain size, making shooting easier and simpler.

[0317] In another scenario, Fig.28 As shown, in Fig.28 In (a), it is assumed that user 2801 uses a terminal device to shoot a distant person 2803. User 2801 approaches person 2803 while shooting, corresponding to Fig.28 (b) and Fig.28 In (c), the distance between the user 2801 and the person 2803 gradually shortens from distance D1 to D2, and further to D1. Assuming that the user 2801 uses the image processing method for Z-axis anti-shake provided in the above embodiment, it can be seen that Fig.29 The screen shown. Fig.29 (a), (b) and (c) correspond to Fig.28 (a), (b) and (c) in . For example, Fig.29 In (a), the image 2902 of the person captured by the terminal device 2901 is relatively small, while in Fig.29 In (b) and (c), the image 2902 of the person captured by the terminal device 2901 gradually becomes larger, and the image 2903 and the image 2904 are obtained, wherein the image 2903 is larger than the image 2902, and the image 2904 is also larger than the image 2903. And this change process is relatively smooth, and there will be no sudden changes in size due to shaking.

[0318] In summary, after using the Z-axis anti-shake function in the terminal device, some related settings can be made for it, including but not limited to choosing whether to turn it on, and precision control, etc. When the terminal device runs the Z-axis anti-shake solution, the user can avoid image mutations caused by shaking when shooting with the terminal device, and the changes of people or objects in the captured picture can look more natural, bringing a better viewing experience.

[0319] Based on the above embodiments, Figure 30 to Figure 34 Another image processing method provided in the embodiment of the present application is described. Fig.30 Schematic diagram of the Z-axis anti-shake process provided in the embodiment of the present application Figure 5 , Fig.31 A table schematic diagram of the characteristics of the first adjustment length acquisition method provided in an embodiment of the present application, Fig.32 A schematic diagram of a usage strategy of the first adjustment length acquisition method provided in an embodiment of the present application, Fig.33 A schematic diagram of smoothing based on a model method provided in an embodiment of the present application, Fig.34 A schematic diagram of an anti-cropping window exceeding the original image boundary provided in an embodiment of the present application.

[0320] refer to Fig.30 Steps in the process:

[0321] S3001. Determine a first adjustment length according to a first feature point set in a first original image and a second feature point set in a second original image.

[0322] In this step, the main purpose is to determine the first adjustment length, which is used to guide the subsequent adjustment of the cropping window size. For different scenarios, different methods can be used according to needs. Fig.31 The features of the methods introduced in the above embodiments are sorted out.

[0323] like Fig.31 As shown in the table, the three columns of the table correspond to three different methods, including optical flow method, feature point method and accelerometer method. The optical flow method and feature point method are relatively close and can also be regarded as a class of methods. The three rows of the table correspond to different conditions or requirements, including requirements on accuracy, power consumption and usage environment conditions. Fig.31 As can be seen from the table, the optical flow method has the highest accuracy, but the power consumption is also high, and it is related to the actual use environment. For example, at night or in heavy fog, the effect of image processing based on the optical flow method will be affected. Similarly, for the feature point method, its accuracy is also relatively high, and the power consumption is relatively low, which is also affected by the use environment. For the accelerometer method, although its accuracy is relatively low, it also has low power consumption and is generally not affected by the use environment.

[0324] Therefore, in actual use, several methods can be combined according to their characteristics. Fig.32 A process for determining how to use or combine the two is listed in the following. Fig.32The process in is explained as follows: 1. Start; 2. Determine whether the accuracy requirement is high. If not, you can directly use the accelerometer method, otherwise continue the process; 3. Determine whether the power consumption requirement is low. If you want relatively low power consumption, you can also use the accelerometer method, otherwise continue the process; 5. Determine whether the environmental conditions are such as image analysis and processing. If it is not conducive to image analysis and processing, you can still use the acceleration method, otherwise you can use the feature point method or the optical flow method; 7. Determine whether the accuracy needs to be further improved. If the accuracy needs to be further improved, you can combine the feature point method and the accelerometer method, or combine the optical flow method and the accelerometer method to achieve it; 9. End.

[0325] Combining the feature point method with the accelerometer method, or combining the optical flow method with the accelerometer method, can be achieved through weighted calculation. For example, the error of the feature point method and the accelerometer method can be measured through experiments. The larger the error, the smaller the weight coefficient. Conversely, the smaller the error, the larger the weight coefficient. By combining them in this way, the accuracy can be further improved.

[0326] For more details of this step, please refer to the part of obtaining the first adjustment length in the previous embodiment, which will not be repeated here.

[0327] S3002: Smoothing the first sequence of the first adjustment length to obtain a second adjustment length, and determining a compensation adjustment length according to a difference between the first adjustment length and the second adjustment length.

[0328] The method for smoothing the first adjustment length including the current frame and the previous frames can be selected according to the actual situation and effect. In addition to the low-pass filter for smoothing as described above, other methods can also be used, such as setting an evaluation function for the degree of smoothing, and then solving the optimal solution of the smoothness through the optimization method, and then determining the smoothing parameters. Smoothing can also be performed through a model, such as Fig.33 As shown, assuming that a smoothing model is trained through manually calibrated data, the input end of the model can receive multiple frames of data of a window including the current frame, such as the first adjustment length of the Nth frame, the first adjustment length of the N-1th frame, etc. After the smoothing model receives the input data and processes it, it can output the second adjustment length of the Nth frame, corresponding to the length of the separated random jitter part. In some implementations, the output of the balance model can also be directly the compensation adjustment length. More smoothing processing methods are not specifically limited in the embodiments of the present application, as long as the effect can be achieved.

[0329] For more details of this step, please refer to the part related to the smoothing process in the previous embodiment, which will not be described again here.

[0330] S3003: Adjust the cropping window corresponding to the first original image according to the compensation adjustment length to determine a first image frame to be displayed in the first original image.

[0331] After the above steps, the compensation adjustment length is obtained, and the size of the cropping window can be further determined by the compensation adjustment length and adjusted accordingly. In actual use, in order to ensure the correctness of image display, some reasonable restrictions can also be added to the size of the cropping window.

[0332] like Fig.34 As shown, the original image 3401 includes a cropping window 3402, and there is a certain amount of free space between the cropping window 3402 and the four sides of the original image 3401, and the unit can be pixels. For example, the upper border of the cropping window 3402 and the upper border of the original image 3401 are at a distance 3403, the lower border of the cropping window 3402 and the lower border of the original image 3401 are at a distance 3405, the left border of the cropping window 3402 and the left border of the original image 3401 are at a distance 3406, and the right border of the cropping window 3402 and the right border of the original image 3401 are at a distance 3404. That is, when adjusting the size of the cropping window 3402, it cannot exceed the free space between it and the original image, which can be expressed by the formula:

[0333] Formula 31

[0334] Formula 32

[0335] in, and Respectively represent the free space between the cropping window and the left and right sides of the original image, and Respectively represent the free distance between the cropping window and the upper and lower sides of the original image. Through such restrictions, the adjusted cropping window can be made not to exceed the boundary of the original image. For more information about adjusting the size of the cropping window, please refer to the relevant content in the above embodiment, which will not be repeated here.

[0336] The following is a summary based on the above description: The image processing method for Z-axis anti-shake provided in the embodiment of the present application mainly calculates the moving ratio of image enlargement or reduction by means of feature points, and obtains the first adjustment length by conversion calculation. Then the sequence of the first adjustment length is smoothed by methods such as filters or models to obtain the second adjustment length and the compensation adjustment length. Finally, the cropping window is adjusted according to the compensation adjustment length, so that the size of the cropping window is adapted to the image after the Z-axis shake, and the size of the image in the display interface of the user terminal device is maintained without mutation. On this basis, the selection of specific methods for different scenarios is proposed, and if they are used in combination, and the smoothing method is expanded, so that the image processing method for Z-axis anti-shake provided in the embodiment of the present application can have better versatility in various scenarios, and can be flexibly adapted according to different conditions.

[0337] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0338] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0339] The image processing method of the embodiment of the present application has been described above, and the device for executing the above method is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device can execute the steps in the above image processing method.

[0340] The above image processing method can be applied to electronic devices with a shooting function. The electronic devices include terminal devices, and the specific device form of the terminal device can refer to the above related descriptions, which will not be repeated here.

[0341] In one implementation, an embodiment of the present application provides an electronic device, Fig.35 Schematic diagram of the hardware structure of the electronic device.

[0342] like Fig.35 As shown, the electronic device 3500 includes: a processor 3501 and a memory 3502; the memory 3502 stores computer-executable instructions; the processor 3501 executes the computer-executable instructions stored in the memory 3502, so that the electronic device 3500 executes the above method.

[0343] When the memory 3502 is independently provided, the electronic device further includes a bus 3503 for connecting the memory 3502 and the processor 3501 .

[0344] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0345] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0346] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage device, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile discs (DVD), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0347] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Determining a first adjustment length according to a first feature point set in a first original image and a second feature point set in a second original image, wherein the first adjustment length is used to adjust a size of a cropping window corresponding to the first original image, wherein the cropping window is used to crop the first original image to obtain a first image frame, and a capture time of the first original image is after a capture time of the second original image; updating a first curve according to a first adjustment length corresponding to the first original image, wherein the first curve includes adjustment lengths corresponding to respective ones of the plurality of original images; Smoothing the first curve to obtain a second curve, and obtaining a second adjustment length corresponding to the first original image in the second curve; Determining a compensating adjustment length according to a difference between the first adjustment length and the second adjustment length; adjusting a cropping window corresponding to the first original image according to the compensation adjustment length, so as to determine a first image frame to be displayed in the first original image; The determining the first adjustment length according to the first feature point set in the first original image and the second feature point set in the second original image includes: Performing matching processing on the first feature point set and the second feature point set to determine a plurality of feature point pairs; For any one of the plurality of feature point pairs, determining a movement ratio corresponding to the feature point pair according to a first coordinate corresponding to a first feature point in the feature point pair and a second coordinate corresponding to a second feature point in the feature point pair; The first adjustment length is determined according to the movement ratios corresponding to the plurality of feature point pairs.

2. The method according to claim 1, characterized in that The determining the movement ratio corresponding to the feature point pair according to the first coordinate corresponding to the first feature point in the feature point pair and the second coordinate corresponding to the second feature point in the feature point pair includes: Determining a displacement vector corresponding to the pair of feature points according to the first coordinate and the second coordinate; The movement ratio corresponding to the feature point pair is determined according to the ratio of the first length to the second length, wherein the first length is the distance between the second feature point in the feature point pair and the center position of the feature point, the second length is the vector length of the displacement vector, and the center position of the feature point is the position corresponding to the mean of the coordinates of each of the plurality of second feature points in the second feature point set.

3. The method according to claim 2, characterized in that The determining the first adjustment length according to the movement ratios corresponding to the plurality of feature point pairs respectively includes: Acquire a third length, where the third length is the distance between a first vertex of the cropping window corresponding to the second original image and a center position of the feature point; For any one of the plurality of feature point pairs, determining a sub-adjustment length of the feature point pair corresponding to the first vertex according to a product of the third length and the movement ratio corresponding to the feature point pair; The first adjustment length is determined according to the average values ​​of the sub-adjustment lengths corresponding to the plurality of feature point pairs.

4. The method according to claim 2 or 3, characterized in that: The step of adjusting the cropping window corresponding to the first original image according to the compensation adjustment length to determine a first image frame to be displayed in the first original image includes: adjusting the size of the cropping window in the first original image according to the compensation adjustment length; The first image frame is obtained by cropping the first original image according to the cropping window after adjusting the size.

5. The method according to claim 4, characterized in that The adjusting the size of the cropping window in the first original image according to the compensation adjustment length includes: Determining an adjustment ratio according to a ratio of the compensation adjustment length to a third length, wherein the third length is a distance between a first vertex of a cropping window corresponding to the second original image and a center position of the feature point; Determining a first distance to be adjusted in the length direction of the cropping window according to the adjustment ratio and the length of the first original image; Determining a second distance to be adjusted in a height direction of the cropping window according to the adjustment ratio and the height of the first original image; In the first original image, the cropping window is adjusted according to the first distance and the second distance to obtain a size of the adjusted cropping window.

6. The method according to claim 5, characterized in that The step of adjusting the cropping window in the first original image according to the first distance and the second distance to obtain a size of the cropping window after adjustment includes: Taking the center position of the feature point in the first original image as the scaling center, adjusting the initial length of the cropping window by the first distance, and adjusting the initial height of the cropping window by the second distance, so as to obtain the adjusted size of the cropping window.

7. The method according to claim 6, characterized in that The step of adjusting the initial length of the cropping window by the first distance includes: Adjusting the initial length of the cropping window by a first length distance from the left side, and adjusting the initial length of the cropping window by a second length distance from the right side, wherein the sum of the first length distance and the second length distance is equal to the first distance; The method further comprises: In the case where the first length distance is greater than the first remaining length, the first remaining length is determined as the first length distance; the first remaining length is the horizontal distance between the left edge of the cropping window and the left edge of the first original image; When the second length distance is greater than the second remaining length, the second remaining length is determined as the second length distance; the second remaining length is the horizontal distance between the right edge of the cropping window and the right edge of the first original image.

8. The method according to claim 6, characterized in that The step of adjusting the initial height of the cropping window by the second distance includes: Adjusting the initial height of the cropping window by a first height distance from the upper side, and adjusting the initial height of the cropping window by a second height distance from the lower side, wherein the sum of the first height distance and the second height distance is equal to the second distance; The method further comprises: In the case where the first height distance is greater than a first remaining height, determining the first remaining height as the first height distance; the first remaining height is a vertical distance between an upper edge of the cropping window and an upper edge of the first original image; When the second height distance is greater than the second remaining height, the second remaining height is determined as the second height distance; the second remaining height is the vertical distance between the lower edge of the cropping window and the lower edge of the first original image.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 8.

Images