Image processing method, terminal equipment and computer readable storage medium

Abstract

The invention provides an image processing method, terminal equipment and a computer readable storage medium, and relates to the technical field of image processing, the method is applied to the terminal equipment comprising a first camera and a second camera, and the method comprises the following steps: in response to a first operation of opening a camera application by a user, displaying a preview interface of the camera application, the preview interface comprises a shooting control, and the camera application starts a background blurring function and an anti-shake function; in response to a second operation of the user on the shooting control, storing the target image; wherein the target image is an image obtained by performing background blurring processing on the first image based on depth information of the first image, the depth information is determined based on the first calibration data, the first image and a second image, the first image is an image shot by the first camera at a first moment, and the second image is an image shot by the second camera at a second moment; the second image is an image shot by the second camera at the first moment, and the first calibration data is used for indicating a pose relationship between the first camera and the second camera at the first moment.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image processing method, a terminal device, and a computer-readable storage medium. Background Art

[0002] With the development of technologies, portable intelligent devices such as smart phones and tablet computers are used in more and more scenarios in people's daily lives. For example, people are more accustomed to using convenient and portable intelligent devices to take pictures to record wonderful moments in life.

[0003] Currently, the background blurring function and the anti-shake function of intelligent devices cannot be enabled normally at the same time. Otherwise, due to anti-shake processing, the lens will move, and the depth information cannot be accurately calculated, thus affecting the blurring effect. Summary of the Invention

[0004] In view of this, this application provides an image processing method, a terminal device, and a computer-readable storage medium, which can achieve an anti-shake effect while ensuring the blurring effect of the photo in the scene captured by the user.

[0005] In a first aspect, this application provides an image processing method, which is applied to a terminal device including a first camera and a second camera. The method includes: in response to a first operation of the user to open the camera application, displaying a preview interface of the camera application, the preview interface including a shooting control, and the camera application enabling the background blurring function and the anti-shake function; in response to a second operation of the user on the shooting control, storing a target image; wherein, the target image is an image obtained by performing background blurring processing on a first image based on the depth information of the first image, the depth information is determined based on first calibration data, a first image, and a second image, the first image is an image captured by the first camera at a first moment, the second image is an image captured by the second camera at the first moment, and the first calibration data is used to indicate the pose relationship between the first camera and the second camera at the first moment.

[0006] Wherein, the second operation of the user on the shooting control is an operation of the user pressing the shutter.

[0007] Based on the above process, in the case where the background blurring function and the anti-shake function are enabled, the terminal device can, in response to the operation of the user pressing the shutter, determine the target image based on the anti-shake data, the first image, and the second image, ensure that the terminal device can perform anti-shake when capturing the target image, and determine the depth information for blurring according to the anti-shake data, thereby ensuring the blurring effect of the target image.

[0008] In a possible implementation of the first aspect, the method further includes: obtaining first anti-shake data corresponding to a first moment, where the first anti-shake data is a movement parameter for moving a lens of a target camera at the first moment, and the target camera includes a first camera and / or a second camera; determining first calibration data based on the first anti-shake data and second calibration data, where the second calibration data is used to indicate an initial pose relationship between the first camera and the second camera.

[0009] Since the first anti-shake data is a movement parameter for moving the lens of the target camera at the first moment, it indicates that the lens of the target camera has moved. Then, based on the second calibration data indicating the initial pose relationship between the first camera and the second camera and the first anti-shake data, the pose relationship between the first camera and the second camera after the lens of the target camera moves at the first moment can be accurately calculated. In this way, the accuracy of the first calibration data can be ensured, thereby improving the accuracy of calculating depth information based on the first calibration data, and then improving the defocus effect.

[0010] In a possible implementation of the first aspect, obtaining the first anti-shake data at the first moment includes:

[0011] determining a target dual-camera image, where the target dual-camera image is a dual-camera image collected by the terminal device at the first moment, and the target dual-camera image includes a first image and a second image;

[0012] obtaining the first anti-shake data based on a time stamp of the target dual-camera image; a time difference between a time stamp of the first anti-shake data and a time stamp of the target dual-camera image is within a preset time range.

[0013] Among them, the first anti-shake data is used to represent the direction and distance of moving the lens in the camera of the terminal device, and the movement control process is continuous. That is to say, within a certain time, the anti-shake data will not show a cliff-like change. Thus, if the time difference between the time stamp of the first anti-shake data and the time stamp of the target dual-camera image is within the preset time range, and within this preset time range, the change of the anti-shake data is small, it means that the time difference between the time stamp of the first anti-shake data and the time stamp of the target dual-camera image is acceptable. In this case, it can be considered that the anti-shake data obtained at the time stamp of the target dual-camera image is the first anti-shake data.

[0014] In addition, the terminal device can determine a defocus image based on the target dual-camera image. Therefore, the first anti-shake data can be accurately determined based on the time stamp of the target dual-camera image, ensuring the correlation between the first anti-shake data and the target dual-camera image, that is, the time difference between the time stamp of the first anti-shake data and the time stamp of the target dual-camera image is within the preset time range.

[0015] In the above implementation process, after determining the target dual-camera image, the first anti-shake data can be accurately determined based on the timestamp of the target dual-camera image to ensure the accuracy of the first anti-shake data, thereby ensuring the accuracy of the first calibration data and improving the defocusing effect.

[0016] In a possible implementation manner of the first aspect, determining the target dual-camera image includes: obtaining multiple groups of dual-camera images, where each group of dual-camera images in the multiple groups of dual-camera images includes an image captured by the first camera and an image captured by the second camera at the same moment, and the time difference between the timestamp corresponding to each group of dual-camera images in the multiple groups of dual-camera images and the first moment is within a preset time range; determining the target dual-camera image from the multiple groups of dual-camera images.

[0017] In the scenarios of snapshot or ordinary shooting, due to jitter at the shooting moment or misalignment with the object to be photographed, that is, the dual-camera images obtained at the first moment have poor quality such as being unclear, therefore, when determining the target dual-camera image, multiple groups of dual-camera images with the time difference between their timestamps and the first moment within the preset time range can be obtained first, and then, the target dual-camera image that meets the requirements can be determined from the multiple groups of dual-camera images.

[0018] Among them, the target dual-camera image that meets the requirements can be a dual-camera image with a clarity greater than the preset clarity, or can be a dual-camera image with a similarity between the first image and the second image greater than the preset similarity, etc.

[0019] The same moment can mean that the moment when the first camera captures the image is exactly the same as the moment when the second camera captures the image, or can also mean that the difference between the moment when the first camera captures the image and the moment when the second camera captures the image is within a preset range.

[0020] Among them, in the case where the difference between the moment when the first camera captures the image and the moment when the second camera captures the image is within the preset range, it can indicate that although the moment when the first camera captures the image is not exactly the same as the moment when the second camera captures the image, but the difference is small, and at this time, the similarity between the image captured by the first camera and the image captured by the second camera is relatively high, so it can be considered as the same moment.

[0021] In the above implementation process, it can be ensured that a target dual-camera image is selected from the dual-camera images with timestamps closer to the first moment to ensure the quality of the images.

[0022] In a possible implementation manner of the first aspect, based on the timestamp of the target dual-camera image, obtaining the first anti-shake data includes: obtaining the second anti-shake data corresponding to each group of dual-camera images in the multiple groups of dual-camera images; determining the first anti-shake data based on the timestamp of the target dual-camera image and multiple second anti-shake data.

[0023] After determining multiple sets of dual-camera images, multiple second anti-shake data can be determined. To ensure the accuracy of the finally obtained first anti-shake data, the first anti-shake data can be further determined by screening, interpolation calculation, etc. based on the timestamp of the target dual-camera image and the multiple second anti-shake data, so as to improve the defocusing effect based on the dual-camera image in the case of anti-shake.

[0024] In a possible implementation manner of the first aspect, determining the first anti-shake data based on the timestamp of the target dual-camera image and the multiple second anti-shake data includes: determining the third anti-shake data corresponding to the timestamp of the target dual-camera image from the multiple second anti-shake data based on the timestamp of the target dual-camera image; in the case where there are multiple third anti-shake data, performing interpolation processing on the multiple third anti-shake data to obtain the first anti-shake data.

[0025] Since there may be no anti-shake data at the timestamp of the target dual-camera image, the terminal device can first determine the third anti-shake data corresponding to the timestamp of the target dual-camera image from the multiple second anti-shake data. There is a time difference between the timestamp of the third anti-shake data and the timestamp of the target dual-camera image, and the time difference is within a preset range. Then, the terminal device performs interpolation processing on the third anti-shake data to obtain the anti-shake data at the timestamp of the target dual-camera image, so as to ensure that the depth information can be accurately calculated based on the anti-shake data to improve the defocusing effect of the dual-camera image.

[0026] In a possible implementation manner of the first aspect, the multiple sets of dual-camera images are the valid dual-camera images among the dual-camera images whose time difference between the corresponding timestamp and the first moment is within a preset time range; the multiple second anti-shake data are the valid anti-shake data among the anti-shake data corresponding to each set of dual-camera images in the multiple sets of dual-camera images.

[0027] The valid dual-camera images can be dual-camera images with a clarity greater than a preset clarity, or can be dual-camera images with a similarity greater than a preset similarity between the first image and the second image, etc. It should be noted that the valid dual-camera images are similar to the determination process of the above-mentioned target dual-camera images that meet the conditions, but are not exactly the same. The preset clarity and preset similarity involved in the two determination processes can be the same or different, and specific values are configured according to the actual application.

[0028] The valid anti-shake data can be anti-shake data with an anti-shake parameter less than a preset anti-shake parameter.

[0029] By determining the validity of multiple sets of dual-camera images and multiple second anti-shake data in the above manner, the quality of the target image can be ensured.

[0030] In a possible implementation of the first aspect, the terminal device includes a memory, and the memory includes at least two storage areas; obtaining second anti-shake data corresponding to each group of dual-camera images in multiple groups of dual-camera images includes: reading the second anti-shake data from the first storage area of the memory; wherein, when reading the anti-shake data from the first storage area, the terminal device stores the anti-shake data to be stored in the second storage area, the first storage area is any one of the at least two storage areas, and the second storage area is the storage area other than the first storage area among the at least two storage areas.

[0031] The terminal device can read the second anti-shake data from the first storage area. During the process of the terminal device reading the second anti-shake data, the terminal device can continue to store the anti-shake data to be stored in the second storage area. In this way, it is possible to avoid the problem of missing anti-shake data due to insufficient storage space or the reading process of the terminal device, ensure accurate acquisition of the anti-shake data, and thus improve the defocus effect.

[0032] In a second aspect, the present application provides a terminal device, the terminal device includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processor are coupled; the display screen is used to display the image generated by the processor, the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the terminal device is caused to execute the method as described in the first aspect and any of its possible design manners.

[0033] In a third aspect, the present application provides a computer-readable storage medium, including computer instructions, when the computer instructions run on the terminal device, the terminal device is caused to execute the method as described in the first aspect and any of its possible design manners above.

[0034] In a fourth aspect, the present application provides a computer program product, including a computer program / instructions, when the computer program / instructions are executed by a processor, the method as described in the first aspect and any of its possible design manners above is implemented.

[0035] In a fifth aspect, the present application provides a device, the device is included in the terminal device, and the device has the function of implementing the behavior of the terminal device in any of the methods in the above aspects and possible implementation manners. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, an allocation module or unit, a scanning module or unit, a recycling module or unit, a moving module or unit, and a storage module or unit, etc.

[0036] In a sixth aspect, an embodiment of the present application provides a chip system. The chip system includes a processor and may further include a memory, and is used to implement any one of the methods provided in the first aspect above. The chip system may be composed of chips or may include chips and other discrete devices.

[0037] It can be understood that the terminal device described in the second aspect and any possible design manner thereof provided above, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of a virtual photo shooting process provided by an embodiment of the present application;

[0039] Figure 2 Schematic diagram of a virtual photo provided by an embodiment of the present application;

[0040] Figure 3 Schematic diagram of a photo anti-shake shooting process provided by an embodiment of the present application;

[0041] Figure 4 Schematic diagram of an image processing process provided by an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the structure of a terminal device provided by an embodiment of the present application;

[0043] Figure 6 Schematic block diagram of the software structure of a terminal device provided by an embodiment of the present application;

[0044] Figure 7 Schematic flow chart of an image processing method provided by an embodiment of the present application Figure 1 ;

[0045] Figure 8 Schematic flow chart of an image processing method provided by an embodiment of the present application Figure 2 ;

[0046] Figure 9 Schematic diagram of a dual-camera image storage process provided by an embodiment of the present application;

[0047] Figure 10 Schematic diagram of an anti-shake data storage process provided by an embodiment of the present application;

[0048] Figure 11 Schematic diagram of a dual-camera image and anti-shake data provided by an embodiment of the present application;

[0049] Figure 12 A schematic diagram corresponding to a dual-camera image and anti-shake data provided by an embodiment of the present application;

[0050] Figure 13 A schematic diagram of smoothing filtering of anti-shake data provided by an embodiment of the present application;

[0051] Figure 14 A schematic diagram of interpolation of anti-shake data provided by an embodiment of the present application;

[0052] Figure 15 A schematic diagram of depth calculation provided by an embodiment of the present application. Detailed implementation manners

[0053] Hereinafter, the terms "first" and "second" are only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this embodiment, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the associated relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or similar expressions thereof refer to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c may be single or multiple.

[0054] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0055] Before introducing the embodiments of the present application, the technologies related to the embodiments of the present application will be introduced in detail first.

[0056] 1. The shooting anti-shake technology is a technology proposed for the situation that when a user takes a photo with a handheld terminal device, the photo taken is unclear due to hand shaking.

[0057] In the scenario where a user takes a photo, before pressing the shutter, the light reflected by the object to be photographed forms a clear image on the plane where the photosensitive sensor is located through the lens in the camera. If the shutter is pressed at this time and it is ensured that the terminal device does not shake, then a clear photo can be taken. However, at the moment of pressing the shutter, the terminal device shakes due to the shaking of the user's hand, resulting in the displacement of the lens in the camera. Then, the position between the lens and the object to be photographed changes, and the light reflected by the object to be photographed forms a blurred image on the plane where the photosensitive sensor is located, so only a blurred photo can be obtained.

[0058] To solve the above problems, a related technology proposes an optical image stabilizer (OIS). The OIS can monitor the displacement information of the shaking of the body of the terminal device. Then, the terminal device calculates the displacement amount that needs to be compensated according to the displacement information of the shaking. After that, the terminal device moves the lens in the camera based on the displacement amount that needs to be compensated to offset the displacement caused by the shaking, thereby avoiding the blurring of the photographed photo.

[0059] For example, in a plane rectangular coordinate system with the x-axis as the horizontal axis and y as the vertical axis, if the center position of the lens was originally at (0, 0). The OIS monitors that the body shakes, and the displacement information of the shaking is that the body moves 1 unit length in the positive direction of the x-axis. It can be understood that the body moves 1 unit length in the positive direction of the x-axis, then correspondingly, the center position of the lens reaches (1, 0). Then, the terminal device can determine based on the displacement information of the body shaking that it is necessary to move the lens 1 unit length in the negative direction of the x-axis from the current position after the shaking. In the case of using capital letters and symbols to represent the moving direction, for example, X+ represents the positive direction of the x-axis, Y- represents the negative direction of the y-axis, and the number represents the moving length. According to the detection result of the OIS, the terminal device can record the displacement information as X+1, indicating that when the lens shakes, it moves 1 unit length in the positive direction of the x-axis. Then, the displacement amount that needs to be compensated calculated by the terminal device is X-1, indicating that the OIS can control the center position of the lens in the camera to move 1 unit length in the negative direction of the x-axis from (1, 0), so that the center position of the lens returns to (0, 0). In this way, the problem of the photographed photo being unclear due to shaking can be avoided.

[0060] In the above example, the displacement information of the shaking is used to represent the moving amount of the terminal device or the lens in the camera of the terminal device, which includes the moving direction and the moving distance. The displacement amount that needs to be compensated calculated by the terminal device is used to represent the moving amount that needs to control the movement of the lens in the camera of the terminal device. For the sake of easy description, the embodiments of the present application call it anti-shake data, or OIS data. The anti-shake data also correspondingly includes the moving direction and the moving distance.

[0061] In subsequent embodiments of the present application, the moving direction and the moving distance may be moving parameters in the present application.

[0062] It can be understood that the moving direction in the displacement information of the jitter is opposite to the moving direction in the anti-shake data, and the moving distance in the displacement information of the jitter is equal in value to the moving distance in the anti-shake data.

[0063] 2. Blur is a photographing technique. In the photograph obtained by using this technique, the foreground is clear while the background (or called the backdrop) is blurred.

[0064] Generally, the photographer adjusts the photographing parameters of the lens to adjust the propagation path of light in the lens, making the shape or contour of the background other than the object being photographed become blurred or unclear, thereby producing a blur effect.

[0065] Due to reasons such as cost or technology, the portable terminal device cannot be equipped with a lens whose photographing parameters can be adjusted, so it cannot take blurred photos in the above-mentioned manner. In order to meet the user's need to take blurred photos, generally, the algorithm blur method is currently used to take blurred photos, that is, the blur effect is achieved by using image processing technology. Specifically, the terminal device first takes an unblurred photo, and then uses an algorithm to process it to ensure that the clarity of the main part in the unblurred photo remains unchanged while the background part becomes blurred, thus achieving the blur effect.

[0066] In the case of taking a blurred photo by using the algorithm blur method, the terminal device needs to first determine the area in the image that needs to be kept clear and the area that needs to be blurred, so that the blur effect can be achieved. The terminal device can determine the foreground and the background, that is, the area that needs to be kept clear and the area that needs to be blurred, through the distance between the object being photographed in this area and the camera. That is to say, the terminal device needs to use the depth information of the image to perform blur processing on the image.

[0067] Currently, the principle of human eye triangulation can be used to calculate the distance between the object being photographed and the camera. For this reason, a dual-camera algorithm is proposed to achieve blur. As Figure 1As shown in the figure, during the process of taking a blurred photo using a dual-camera algorithm, the terminal device simultaneously acquires images using two cameras in the dual-camera module. Since the positions of the main camera and the auxiliary camera are different, when the same object to be photographed is captured from the positions of the main camera and the auxiliary camera respectively, the position of the object to be photographed in the field of view captured by the main camera is different from the position of the object to be photographed in the field of view captured by the auxiliary camera. There is an angle between the line connecting the position of the main camera and the object to be photographed and the line connecting the position of the auxiliary camera and the object to be photographed. This angle is called the parallax angle. Then, the object to be photographed, the main camera, and the auxiliary camera can form a triangle. In addition, given the positions of the main camera and the auxiliary camera, and the optical parameters such as the positions and focal lengths of the inner lenses of the main camera and the auxiliary camera, the distance between the object to be photographed and the camera can be calculated using the principle of similar triangles. Specific calculation formulas and the like can be referred to in related technologies and will not be elaborated here.

[0068] That is to say, when the pose relationship between the two cameras in the dual-camera module is determined, the terminal device can obtain the distance between the object corresponding to each pixel point in each image and the camera based on the two images captured by the dual-camera module, which is also called the depth information of the image.

[0069] Generally, the terminal device blurs the main image captured by the main camera based on the depth information to obtain a blurred photo. Specifically, the terminal device can use the depth information of the image to determine the main part to be highlighted and the background part to be blurred, and perform a blurring process on the background part to be blurred, and a blurred photo can be obtained. As Figure 2 shown, it is a schematic diagram of a blurred photo. The main part in area 1 of this blurred photo is clear, while the part in area 2 is blurred.

[0070] Currently, when users take photos using portable terminal devices, there is still a problem that the image is unclear due to jitter. For example, when the user's hand shakes during shooting, driving the terminal device to move may cause the photosensitive sensor of the terminal device to receive different lights at the same position, and the different lights received at the same position are superimposed together, resulting in a blurred image. For this reason, the OIS mentioned in the above related technologies can be used to solve the problem of unclear images caused by jitter. As Figure 3 shown, configure OIS in the terminal device to control the movement of the camera and avoid the problem of blurred photos caused by jitter.

[0071] However, currently, the terminal device cannot enable the background blurring function and the anti-shake function implemented by the dual-camera algorithm simultaneously. If the background blurring function and the anti-shake function implemented by the dual-camera algorithm are enabled simultaneously, and then, when the user presses the shutter, the user's hands shake. In order to avoid the problem of image blurring caused by the shake, the terminal device will move the lens of at least one camera in the dual-camera module according to the detected shake, so as to ensure that at least one camera in the dual-camera module can take clear photos. However, due to the change of the lens, the terminal device cannot determine the pose relationship between the two cameras after the change of the lens, and then cannot accurately calculate the depth information based on the two images obtained by the dual-camera module, resulting in a poor blurring effect of the photo, or even unable to blur the photo.

[0072] Therefore, the present application provides an image processing method, which is applied to a terminal device. The terminal device can detect the displacement information of the terminal device body caused by the shake, and perform anti-shake control on the lens according to the detected displacement information to ensure that clear dual-camera images can be obtained. Then, the terminal device compensates the parameters of the lens for taking the dual-camera images according to the anti-shake control information to determine the position of the lens when taking the dual-camera images, so as to ensure that the depth information of the image can be accurately calculated based on the position of the lens taking the dual-camera images and the dual-camera images, and then obtain a blurred photo and ensure the blurring effect of the blurred photo.

[0073] In some examples, as Figure 4 shown, the anti-shake module of the terminal device can detect the displacement information of the terminal device caused by the shake, and control the lens in the camera in the dual-camera module to move in the opposite direction according to the displacement information to offset the displacement caused by the shake and ensure that two clear images are output by the dual-camera module. Then, the dual-camera compensation module obtains the anti-shake data of the anti-shake module controlling the lens movement in the dual-camera module, and compensates the calibrated position of the lens in the camera in the dual-camera module based on the anti-shake data to accurately obtain the current position of the lens, that is, the compensated position of the lens. Finally, the depth blurring module of the terminal device can accurately determine the depth information in the image based on the compensated position of the lens and the two images, and perform accurate blurring processing on the image based on the depth information to obtain a blurred photo.

[0074] To better understand the embodiments of the present application, the terminal device provided by the embodiments of the present application will be introduced first.

[0075] The image processing method provided by the embodiments of the present application can be applied to a terminal device. The terminal device can specifically be a mobile phone, a tablet computer, a smart screen, a laptop computer, a vehicle-mounted device, a wearable device (such as a smart watch), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence device, or other terminal devices with a photographing function. The embodiments of the present application do not limit the specific type of the terminal device or the operating system installed thereon.

[0076] The following introduces the hardware structure of the terminal device.

[0077] Figure 5 FIG. shows a schematic structural diagram of a terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, an anti-shake module 196, etc.

[0078] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0080] Among them, the controller may be the nerve center and command center of the terminal device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0081] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0082] The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function.

[0083] The internal memory 121 may be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store the data created during the use of the terminal device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0084] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0085] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. While charging the battery 142, the charging management module 140 may also supply power to the terminal device through the power management module 141. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110.

[0086] The wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0087] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0088] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through the antenna 1 and radiate it out.

[0089] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays images or videos through the display screen 194.

[0090] The wireless communication module 160 can provide solutions for wireless communications such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the terminal device 100.

[0091] The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be transmitted from the processor 110, perform frequency modulation and amplification on it, and convert it into electromagnetic waves through the antenna 2 and radiate it out.

[0092] In some embodiments, antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, enabling the terminal device 100 to communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc.

[0093] The terminal device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0094] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0095] The terminal device 100 can implement the shooting function through the ISP, the camera 193, the video image codec, the GPU, the display screen 194, and the application processor, etc.

[0096] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The optical signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

[0097] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0098] In some embodiments, the terminal device includes two cameras. One of the cameras can be referred to as the main camera, and the other camera can be referred to as the auxiliary camera or the secondary camera. In this embodiment, it can be considered that these two cameras form a module, such as a dual-camera module.

[0099] The anti-shake module 196 is used to detect the displacement information of the body of the terminal device shaking, and move the lens in the camera 193 according to the displacement amount that needs to be compensated calculated from the displacement information of the shake, so as to avoid the photos taken being blurred due to the shaking of the terminal device.

[0100] The digital signal processor is used to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0101] The video image codec is used to compress or decompress digital videos or images. The terminal device 100 can support one or more video image codecs. In this way, the terminal device 100 can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc. Or, the terminal device 100 can display or capture images in multiple encoding formats, such as: Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), etc.

[0102] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0103] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, speakers, receivers, microphones, headphone interfaces, and application processors.

[0104] The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module can also be used to encode and decode audio signals. The speaker is used to convert audio electrical signals into sound signals. The receiver is used to convert audio electrical signals into sound signals. The microphone is used to convert sound signals into electrical signals. The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface 130, or it can be other interfaces such as a 3.5mm open mobile terminal platform (OMTP) standard interface.

[0105] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0106] The gyroscope sensor can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. Gyroscope sensors can also be used for navigation and somatosensory game scenes.

[0107] The button 190 includes a power-on button, volume buttons, etc. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, battery level change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the terminal device 100. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can be compatible with an external memory card. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communication.

[0108] Next, the software structure of the terminal device will be introduced.

[0109] The software system of the terminal device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of the present invention, Figure 6 taking the software structure block diagram of the terminal device 100 shown as an example, the software structure of the terminal device 100 will be exemplarily described.

[0110] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system is divided into four layers, from top to bottom, namely the application layer, application framework layer, hardware abstraction layer (HAL), and kernel layer.

[0111] For ease of description, Figure 6 the hardware layer connected to the software architecture is also shown in

[0112] It should be noted that the embodiments of the present application only take a common operating system as an example for description. In other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the solution of the present application can also be implemented.

[0113] The application layer may include a series of application packages. As Figure 6 shown, the application packages may include applications such as a camera, gallery, etc.

[0114] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0115] As Figure 6As shown, the application framework layer may include a camera service. The camera service is used to connect the camera application in the application layer and the relevant hardware in the hardware layer. In some embodiments, the camera service may implement the connection between the camera application and the hardware layer by calling the hardware interface.

[0116] Understandably, the application framework layer may also include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc. Among them, the window manager is used to manage window programs. The content provider is used to store and obtain data, and make this data accessible to application programs. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. The telephone manager is used to provide the communication function of the terminal device 100. The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables application programs to display notification information in the status bar, can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.

[0117] The HAL layer is located above the kernel layer. The HAL layer can abstract the hardware in the hardware layer, hide the differences between different hardware, and provide a unified hardware interface for the upper layer.

[0118] As Figure 6 shown, the HAL layer includes a camera device, an image processing module, and a storage module.

[0119] The camera device is used to connect the camera service to the underlying camera driver and the camera.

[0120] The image processing module is used to obtain images and related data, and process them based on the images and related data. In some embodiments, the image processing module can perform operations such as format conversion, content processing, and encoding on the images.

[0121] In the embodiments of the present application, the image processing module can determine the blurred image according to the two images obtained by the dual-camera module and the anti-shake data of the anti-shake module according to the image processing method provided by the embodiments of the present application.

[0122] The kernel layer is the layer between the hardware layer and the software. As Figure 6 shown, the kernel layer contains a camera driver, and the camera driver is used to control the camera in the hardware layer to perform the shooting action.

[0123] Understandably, the kernel layer may also contain a display driver, an audio driver, a sensor driver, etc., and no more details about these drivers will be described here.

[0124] The working processes of the software and hardware of the terminal device 100 will be exemplarily described below in combination with the shooting scenario.

[0125] When the camera application located in the application layer displays a shooting interface in the terminal device and the shooting interface includes shooting controls, the user can click the shooting controls in the shooting interface. When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The framework layer obtains the raw input event from the kernel layer and can identify that the control corresponding to the input event is the shooting control. Then, the camera calls the interface provided by the camera service in the framework layer. After the interface provided by the camera service is called, the camera device in the HAL layer can call the camera driver in the kernel layer, and the camera driver can drive the dual-camera module to capture two images. After that, the dual-camera module sends the two images to the image processing module in the HAL layer through the camera driver. The image processing module can also obtain anti-shake data from the anti-shake module, and the anti-shake data is used to record the direction and distance of the lens movement controlled by the anti-shake module in the dual-camera module. Finally, the image processing module can process the anti-shake data and the two images to obtain a blurred photo.

[0126] In the embodiments of the present application, the operation of the user to open the camera application is the first operation. In response to the first operation, the terminal device can display a shooting interface, which is also referred to as the preview interface mentioned above.

[0127] Next, an image processing method provided by the embodiments of the present application will be introduced.

[0128] In a scenario where both the anti-shake function and the virtual shooting function are enabled, the terminal device can take a blurred photo through the image processing method provided by the present application. Exemplarily, the shooting configuration page of the terminal device may include an anti-shake option and a virtual shooting option. When both the anti-shake option and the virtual shooting option are selected, it means that the terminal device enables both the anti-shake function and the virtual shooting function.

[0129] In some scenarios, when the terminal device is in a shooting mode that requires both the anti-shake function and the virtual shooting function to be enabled, it can also take a blurred photo through the image processing method provided by the present application. For example, the portrait shooting mode, the sports shooting mode, etc. Shooting in the above example modes are all possible application scenarios, and the shooting modes involved in the actual application scenarios can be determined according to the actual shooting needs, and the shooting modes are not limited here.

[0130] Next, in combination with Figure 7 the flowchart of the image processing method shown Figure 1, introduce an image processing method provided by an embodiment of the present application. This method is executed by a terminal device and includes the following steps:

[0131] S701. The terminal device acquires a dual-camera image captured by the dual-camera module and anti-shake data at the moment when the dual-camera image is captured.

[0132] As Figure 4 shown, the dual-camera module includes a main camera and a secondary camera. When the user presses the shutter, both the main camera and the secondary camera capture images, thereby obtaining a dual-camera image. The dual-camera image includes a main image captured by the main camera and a secondary image captured by the secondary camera.

[0133] Among them, in the embodiment of the present application, the operation of the user pressing the shutter, that is, the operation of the user clicking the shooting control, is referred to as the second operation, and the moment when the user presses the shutter can be the first moment in the present application.

[0134] In some embodiments, due to technical problems such as instruction transmission delay, there may be a difference between the moment when the main camera acquires the main image and the moment when the secondary camera acquires the secondary image. Therefore, the situation where the time difference between the moment when the main camera acquires the main image and the moment when the secondary camera acquires the secondary image is within an acceptable error range can be regarded as the main camera and the secondary camera acquiring images simultaneously. For example, when the acceptable error range is ±0.15 milliseconds of the moment when the main camera acquires the main image, if the moment when the main camera acquires the main image is the T0 moment, and the moment when the secondary camera acquires the secondary image is the T0 + 0.1 millisecond moment, it can be considered that the main image acquired by the main camera and the secondary image acquired by the secondary camera are acquired simultaneously. At this time, it can also be considered that the main image acquired by the main camera and the secondary image acquired by the secondary camera are corresponding. The main image acquired by the main camera and the secondary image acquired by the secondary camera are referred to as a set of dual-camera images.

[0135] As Figure 4 shown, since the anti-shake module in the embodiment of the present application will perform movement control on the lenses of the cameras in the dual-camera module according to the displacement information detected due to the jitter of the terminal device. That is to say, at the moment when the dual-camera image is captured, if the position of the lens in the dual-camera module moves, then the terminal device needs to acquire anti-shake data to determine the position of the lens after movement based on the anti-shake data and the original position of the lens. In this way, the depth information of the image can be accurately calculated based on the position of the lens after movement and the dual-camera image.

[0136] Among them, the anti-shake module can control only the movement of the lens of the main camera in the dual-camera module, or can control the movement of the lens of the main camera and the lens of the secondary camera in the dual-camera module.

[0137] Among them, the anti-shake data obtained at the moment of shooting the dual-camera image in the embodiments of the present application may be the first anti-shake data in the present application.

[0138] In some embodiments, the hardware structure corresponding to the anti-shake module may be determined according to the actual situation. For example, in the case where OIS can detect the displacement information of the terminal device due to jitter and control the movement of the lens according to the displacement information, the anti-shake module may only include OIS. Another example is that in the case where OIS cannot detect the displacement information of the terminal device due to jitter and can only control the movement of the lens according to the displacement information, the anti-shake module may include OIS and a sensor for detecting the displacement information of the terminal device due to jitter. Still another example is that in the case where OIS can detect the displacement information of the terminal device due to jitter and can control the movement of the lens, but cannot determine how to control the movement of the lens according to the displacement information, the anti-shake module may include OIS and a processor for determining how to control the movement of the lens according to the displacement information. Or, the anti-shake module may only include OIS, and the anti-shake module may cooperate with other processors for determining how to control the movement of the lens according to the displacement information to jointly implement anti-shake control.

[0139] S702. The terminal device compensates the calibration data of the dual-camera module based on the anti-shake data to obtain the compensated calibration data.

[0140] The calibration data of the dual-camera module is used to record the initial position of the lens in the dual-camera module. Specifically, if only the lens of the main camera in the dual-camera module can move, the calibration data may only be used to record the initial position of the lens in the main camera. If the lenses of both the main camera and the auxiliary camera in the dual-camera module can move, the calibration data may only be used to record the initial position of the lens in the main camera.

[0141] Among them, in the embodiments of the present application, the compensated calibration data may be the first calibration data in the present application, and the calibration data for recording the initial position of the lens in the dual-camera module in the dual-camera module may be the second calibration data in the present application.

[0142] In some examples, in the case where the lens in the main camera can move and the lens in the auxiliary camera cannot move, the anti-shake module can only control the movement of the lens of the main camera in the dual-camera module. Then, the anti-shake data records the direction and distance of the movement of the lens of the main camera. The terminal device compensates the calibration data of the dual-camera module based on the anti-shake data to obtain the compensated calibration data. That is to say, the terminal device can calculate the position of the lens in the main camera after movement based on the anti-shake data and the calibration data for recording the initial position of the lens in the main camera.

[0143] In some other examples, when the lenses in the main camera can move and the lenses in the secondary camera can also move, the anti-shake module can control only the movement of the lenses in the main camera of the dual-camera module, or only control the movement of the lenses in the secondary camera, or can control the movement of the lenses in the main camera and the lenses in the secondary camera of the dual-camera module. The anti-shake data records the direction and distance of the movement of the lenses in the main camera, and the direction and distance of the movement of the lenses in the secondary camera. The terminal device compensates the calibration data of the dual-camera module based on the anti-shake data, and obtains the compensated calibration data. That is to say, the terminal device can calculate the position of the lenses in the main camera after movement and the position of the lenses in the secondary camera after movement based on the anti-shake data and the calibration data used to record the initial positions of the lenses in the main camera and the initial positions of the lenses in the secondary camera.

[0144] S703. The terminal device calculates the depth information of the dual-camera image based on the compensated calibration data and the dual-camera image.

[0145] S704. The terminal device obtains a blurred photo based on the depth information and the dual-camera image.

[0146] Based on the above embodiments, it can be known that the compensated calibration data represents the position of the lenses in the dual-camera module after movement. Therefore, the terminal device calculates the depth information of the dual-camera image based on the compensated calibration data and the dual-camera image. Then, the terminal device can blur the image captured by the dual-camera module according to the depth information. The above specific calculations can refer to the related technologies for realizing blurring in the dual-camera algorithm above, which will not be elaborated here.

[0147] In some embodiments, the terminal device blurs the main image captured by the main camera in the dual-camera module. For the specific blurring, reference can be made to the introduction part of the blurring technology above, which will not be elaborated here.

[0148] Furthermore, one of the two images captured in the dual-camera image is used as the finally obtained photo, and the other is used to assist in calculating the depth information of the image. Therefore, generally, the main image captured by the main camera is used as the finally obtained photo, and it is ensured that the clarity of the main image captured by the main camera is higher than the clarity of the secondary image captured by the secondary camera. In this way, a clear photo can be captured.

[0149] Among them, the main camera in the embodiments of the present application can be the first camera in the present application, the main image captured by the main camera is the first image in the present application, the secondary camera can be the second camera in the present application, and the secondary image captured by the secondary camera is the second image in the present application.

[0150] In some embodiments, after the terminal device executes S704 to obtain the blurred photo, it can store the blurred photo for the user to view the blurred photo in the terminal device, and the terminal device can store, transmit, edit, etc. the blurred photo in response to the user's operation.

[0151] Among them, the blurred photo in the embodiment of the present application may refer to the target image in the present application, and the dual-camera image in the embodiment of the present application may be the target dual-camera image in the present application.

[0152] Combine the following Figure 8 The process diagram of the image processing method shown is as follows Figure 2 , an image processing method provided in an embodiment of the present application is introduced, the method is executed by a terminal device, and includes the following steps:

[0153] S801: The terminal device stores multiple groups of dual-camera images A output by the dual-camera module and the timestamp of each group of dual-camera images A in an image buffer memory.

[0154] When the user opens the camera application installed in the terminal device and turns on the background blur function, and the terminal device implements blur based on the dual-camera algorithm, the dual-camera module starts to work, and the main camera and auxiliary camera in the dual-camera module start to continuously capture images and output these images in the order of capturing images. We call these images output in the capture order as image sequences. The image sequence output by the dual-camera module is sent to the preview stream processing module and the data stream processing module respectively. The image sequence processed by the preview stream processing module is displayed in sequence on the display screen of the terminal device to provide the user with a preview screen during the shooting process. When the user presses the shutter, the terminal device selects one group of dual-camera images from the image sequence processed by the data stream processing module, and determines the photos taken by the user based on the group of dual-camera images. The timestamp of the group of dual-camera images corresponds to the time when the user presses the shutter. That is to say, after the shooting action is completed, a group of dual-camera images processed by the preview stream processing module are processed and saved in the terminal device in the format of image files for the user to view. It can be understood that in the above process, only the processing of the preview stream and the data stream during the shooting process is briefly introduced. For a more detailed processing process, please refer to the relevant technology.

[0155] In the snapshot scenario, if the terminal device determines that the image sequence of the input data stream processing module and the image acquired at that moment are taken as the photographed photo when the user presses the shutter, the acquired photo will be unclear. Therefore, after the user opens the camera application, the terminal device can store multiple groups of dual-camera images A and the timestamps of the dual-camera images A in the image sequence output by the dual-camera module in the image buffer memory, so that the terminal device can select a suitable dual-camera image from the multiple dual-camera images A.

[0156] In some embodiments, the terminal device may include an image buffer memory. A set of dual-camera images output by the dual-camera module includes a main image captured by the main camera and a secondary image captured by the secondary camera. Then, the terminal device may store the main image captured by the main camera, the timestamp of the main image, the secondary image captured by the secondary camera, and the timestamp of the secondary image in the image buffer memory. Alternatively, the terminal device may only store the main image captured by the main camera, the timestamp of the main image, and the secondary image captured by the secondary camera in the image buffer memory.

[0157] In some examples, the timestamp of each set of dual-camera images A may be two timestamps, that is, the terminal device stores the timestamp of each image in each set of dual-camera images A in the image buffer memory. For example, when the acquisition time of the main image is 0.05 ms and the acquisition time of the secondary image is 0.09 ms, the timestamps of this set of dual-camera images A may be 0.05 ms and 0.09 ms. Then, the terminal device stores the dual-camera images and the timestamps 0.05 ms and 0.09 ms of the dual-camera images in the image buffer memory.

[0158] The terminal device stores the dual-camera images A output by the dual-camera module in real time and the timestamps of the dual-camera images A. Since the storage space in the image buffer memory is limited, the terminal device may delete the earliest stored dual-camera images A and the timestamps of the dual-camera images A in the image buffer memory, and overwrite and store the dual-camera images A and the timestamps of the dual-camera images A recently acquired by the dual-camera module in the image buffer memory.

[0159] In some other examples, the timestamp of each set of dual-camera images A may be one timestamp. Specifically, when the time difference between the acquisition times of the main image and the secondary image in a set of dual-camera images A can be ignored, for example, when the time difference between the acquisition times of the main image and the secondary image in each set of dual-camera images is within 0.05 ms, the terminal device can obtain a blurred photo with a better blurring effect based on this set of dual-camera images. Then, the acquisition time of the main image can be used as the timestamp of this set of dual-camera images, so as to facilitate subsequent determination of corresponding anti-shake data based on this timestamp.

[0160] In some other embodiments, the image buffer memory of the terminal device may include a main image buffer memory and a secondary image buffer memory, as Figure 9 shown. In this embodiment, the image sequence output by the main camera may be stored in the main image buffer memory, and the timestamp of each image output by the main camera may also be stored in the main image buffer memory. The image sequence output by the secondary camera may be stored in the secondary image buffer memory, and the timestamp of each image output by the secondary camera may also be stored in the secondary image buffer memory.

[0161] In some examples, the terminal device may first obtain a main image and the timestamp of the main image from the main image buffer memory. Then, based on the timestamp of the main image, the terminal device may determine the auxiliary image corresponding to the main image from the auxiliary image buffer memory, and use the main image and the auxiliary image as a set of dual-camera images.

[0162] S802. The terminal device stores multiple anti-shake data A output by the anti-shake module and the timestamp of each anti-shake data A in the anti-shake data buffer memory.

[0163] The anti-shake data is used to record the direction and distance of the lens movement in the dual-camera module controlled by the anti-shake module. When the camera application is running and the anti-shake function is enabled, the anti-shake module starts to output anti-shake data. To ensure that the depth information of the image can be accurately calculated based on the dual-camera images output by the dual-camera module, the terminal device may first determine the position of the lens in each camera of the dual-camera module when the dual-camera module outputs the dual-camera images. Therefore, the terminal device may store multiple anti-shake data A output by the anti-shake module and the timestamp of each anti-shake data A in the anti-shake data buffer memory.

[0164] In a snapshot scenario, there may be a situation where the user frequently presses the shutter. Each time the user presses the shutter, the terminal device can obtain a set of dual-camera images and anti-shake data. As described in the above steps, the terminal device stores multiple dual-camera images A and the timestamp of each image in the image buffer memory, and stores multiple anti-shake data A and the timestamp of each anti-shake data A in the anti-shake data buffer memory. This can ensure that each time the user presses the shutter, the terminal device can obtain the corresponding dual-camera image from the image buffer memory and obtain the anti-shake data from the anti-shake data buffer memory, avoiding the situation where the dual-camera images and the anti-shake data become invalid and the image cannot be blurred.

[0165] In some embodiments, the anti-shake data buffer memory may include multiple buffer areas, each of which is independent. When the user does not press the shutter, the terminal device stores multiple anti-shake data in multiple buffer areas simultaneously, so that in the case where one buffer area has a problem, another buffer area can store the anti-shake data, ensuring the accuracy of the anti-shake data.

[0166] In some examples, when the user presses the shutter, the terminal device stops storing anti-shake data in one of the buffer areas and obtains the corresponding anti-shake data from that buffer area. In addition, the terminal device continues to store anti-shake data in other buffer areas. For example, when the user presses the shutter, the terminal device may select one of the multiple buffer areas as the read object and configure that buffer area as the read mode, allowing only the terminal device to read data from that buffer area, while other buffer areas can continue to store anti-shake data.

[0167] In some examples, taking the anti-shake data buffer memory including buffer 1 and buffer 2 as an example, as Figure 10 shown, when the user does not press the shutter, the anti-shake data output by the anti-shake module is stored in both buffer 1 and buffer 2 at the same time. At this time, if both buffers can store anti-shake data normally. The anti-shake data stored in buffer 1 and buffer 2 should be the same.

[0168] Among them, the storage space in the buffer is limited. The terminal device can overwrite the anti-shake data stored earliest in the anti-shake module with the anti-shake data recently output by the anti-shake module to ensure that the data in the buffer is updated in real time.

[0169] Further, when the user presses the shutter for the first time, the corresponding dual-camera compensation module in the terminal device responds to the operation of the user pressing the shutter and starts to read the anti-shake data from buffer 1, and the anti-shake module continues to store the anti-shake data in buffer 2. And, in order to prevent the anti-shake data in buffer 1 from being overwritten by the anti-shake data recently output by the anti-shake module, at the moment when the dual-camera compensation module reads the anti-shake data from buffer 1, the anti-shake module stops storing new anti-shake data in buffer 1.

[0170] Further, when the user presses the shutter again within a certain period of time, and at this time the dual-camera compensation module has read the anti-shake data from buffer 1, and the anti-shake module continues to store new anti-shake data in buffer 1, the corresponding dual-camera compensation module in the terminal device responds to the operation of the user pressing the shutter and starts to read the anti-shake data from buffer 2, and the anti-shake module continues to store the anti-shake data in buffer 1. Thus, it is avoided that because buffer 1 did not store the recently output anti-shake data in time before, the terminal device cannot read the anti-shake data corresponding to the current shutter press moment from buffer 1. And, in order to prevent the anti-shake data in buffer 2 from being overwritten by new anti-shake data, the anti-shake module stops storing new anti-shake data in buffer 2.

[0171] It can be understood that if after the above-mentioned embodiments, the user presses the shutter again within a certain period of time, then the dual-camera compensation module can read the anti-shake data from buffer 1, and the anti-shake module continues to store the anti-shake data in buffer 2. Through the above ping-pong mechanism, it can be ensured that the anti-shake module continuously and stably provides anti-shake data to the subsequent image processing process.

[0172] It should be noted that here the dual-camera compensation module selects the buffer to read the anti-shake data according to the order of buffer arrangement. Actually, it can also select a buffer to read the anti-shake data in other ways.

[0173] In the above S801 and S802, since the dual-camera images, the timestamps of the dual-camera images, the anti-shake data, and the timestamps of each anti-shake data are all stored in the corresponding buffer memories, it is possible to relieve the reading pressure of the dual-camera images and the anti-shake data in the scenario where the user presses the shutter multiple times within a short period, so as to ensure that the subsequent image processing process can obtain the corresponding dual-camera images and anti-shake data each time the shutter is pressed.

[0174] After storing the dual-camera images and the anti-shake data in the corresponding buffer memories through the above S801 and S802, the terminal device can execute the following S803 in response to the user's operation of pressing the shutter:

[0175] S803. The terminal device obtains multiple groups of dual-camera images B within a time period corresponding to the moment when the user presses the shutter from the image buffer memory and the anti-shake data buffer memory, and multiple anti-shake data B corresponding to each group of dual-camera images B.

[0176] Corresponding to the moment when the user presses the shutter means that the time difference between the timestamps of the multiple groups of dual-camera images B and the moment when the user presses the shutter is within a preset range. In other words, it is multiple groups of dual-camera images B and multiple anti-shake data B within a time period corresponding to the moment when the user presses the shutter. For example, if the time period corresponding to the moment when the user presses the shutter is within 0.15 milliseconds before and 0.15 milliseconds after the moment when the user presses the shutter, as Figure 11 shown, there is dual-camera image 1 and anti-shake data 1 corresponding to time T0, anti-shake data 2 corresponding to time T1, dual-camera image 2 corresponding to time T2, dual-camera image 3 and anti-shake data 3 corresponding to time T3, no dual-camera image and anti-shake data corresponding to time T4, and dual-camera image 4 and anti-shake data 4 corresponding to time T5. If the user presses the shutter at time T3, T3 - 0.15 ms is a certain moment between T0 and T1, and T3 + 0.15 ms is a certain moment after T5, then the terminal device can select the anti-shake data 2 corresponding to time T1, the dual-camera image 2 corresponding to time T2, the dual-camera image 3 and anti-shake data 3 corresponding to time T3, and the dual-camera image 4 and anti-shake data 4 corresponding to time T5 as multiple groups of dual-camera images B and multiple anti-shake data B. It can be understood that the moment when the user presses the shutter is not necessarily the same as the timestamps of obtaining the dual-camera images and the anti-shake data. As in the above example of Figure 11 , the moment when the user presses the shutter can also be T3 + 0.05 milliseconds.

[0177] Understandably, the above embodiments only illustrate one situation. The time period corresponding to the moment when the user presses the shutter can also be a period of time after the moment when the user presses the shutter, or a period of time before the moment when the user presses the shutter. Or, it can also be a relatively long period of time after the moment when the user presses the shutter and a relatively short period of time before the moment when the user presses the shutter.

[0178] It should be noted that the maximum acquisition frequency of the anti-shake data is greater than the maximum acquisition frequency of the dual-camera images. Therefore, the amount of anti-shake data that can be acquired at most within one unit time is greater than the amount of dual-camera images that can be acquired at most within one unit time.

[0179] In some embodiments, a group of dual-camera images can correspond to multiple anti-shake data. Specifically, the terminal device can first obtain multiple dual-camera images B. For each of the dual-camera images B, it then obtains multiple anti-shake data B corresponding to the dual-camera image B. For example, the terminal device obtains five groups of dual-camera images B, and the timestamps of the five groups of dual-camera images B are: Timestamp1, Timestamp2, Timestamp3, Timestamp4, Timestamp5. Then, based on the five timestamps, the terminal device respectively finds the timestamps Timestamp_OIS1, Timestamp_OIS2, Timestamp_OIS3, Timestamp_OIS4, Timestamp_OIS5 that are closest to the above timestamps among the timestamps of the multiple anti-shake data A. Then, based on the exposure time of the dual-camera module or a preset time length, the terminal device determines the timestamp range where the anti-shake data B corresponding to the dual-camera image within the exposure time is located. When the exposure time is 2t ms and t is a known number, the timestamp ranges of the anti-shake data B corresponding to each group of dual-camera images are respectively: [Timestamp_OIS1 - t ms, Timestamp_OIS1 + t ms], [Timestamp_OIS2 - t ms, Timestamp_OIS2 + t ms], [Timestamp_OIS3 - t ms, Timestamp_OIS3 + t ms], [Timestamp_OIS4 - t ms, Timestamp_OIS4 + t ms], [Timestamp_OIS5 - t ms, Timestamp_OIS5 + t ms]. That is to say, the anti-shake data B within the range of [Timestamp_OIS1 - t ms, Timestamp_OIS1 + t ms] corresponds to a group of dual-camera images B with the timestamp of Timestamp1.

[0180] In some examples, a group of dual-camera images has multiple anti-shake data. Such asFigure 12 As shown, dual-camera image B1 corresponds to a total of 17 anti-shake data B from anti-shake data B1 to anti-shake data B17, and dual-camera image B2 corresponds to a total of 17 anti-shake data B from anti-shake data B18 to anti-shake data B34.

[0181] In some other embodiments, since the anti-shake module only outputs anti-shake data when jitter occurs, there is a situation where the amount of anti-shake data B corresponding to different dual-camera images B is different. For example, as Figure 12 shown, both dual-camera image B1 and dual-camera image B2 correspond to 17 anti-shake data B, while dual-camera image B3 corresponds to a total of 6 anti-shake data B from anti-shake data B35 to anti-shake data B40, and dual-camera image B4 corresponds to a total of 3 anti-shake data B from anti-shake data B41 to anti-shake data B43.

[0182] In still some other embodiments, since the anti-shake module only outputs anti-shake data when jitter occurs, and there is a situation where the time stamps between two dual-camera images are quite different or slightly different, there is a situation where the anti-shake data B corresponding to two consecutively acquired dual-camera images B is discontinuous. For example, as Figure 12 shown, dual-camera image B4 corresponds to a total of 3 anti-shake data B from anti-shake data B41 to anti-shake data B43, while dual-camera image B5 corresponds to anti-shake data B60 and anti-shake data B61.

[0183] Through S803, the terminal device can obtain multiple sets of dual-camera images B and multiple anti-shake data B, as well as the time stamp of each dual-camera image B and the time stamp of each anti-shake data B. Optionally, the terminal device can execute S804 to filter out invalid data from the dual-camera images B and multiple anti-shake data B to ensure data accuracy.

[0184] In some embodiments, among the multiple sets of dual-camera images B and multiple anti-shake data B selected by the terminal device in S803, there may be invalid dual-camera images B and / or anti-shake data B. Therefore, the terminal device can determine valid data through the following S804.

[0185] S804: The terminal device determines multiple valid sets of dual-camera images C and multiple anti-shake data C from multiple sets of dual-camera images B and multiple anti-shake data B.

[0186] The terminal device can determine whether a dual-camera image C is valid based on the image content in the dual-camera image B.

[0187] In some embodiments, the terminal device may identify and compare the main image and the auxiliary image in the dual-camera image B, calculate the similarity between the main image and the auxiliary image. If the similarity between the main image and the auxiliary image is less than the preset similarity, it indicates that the main image and the auxiliary image may not be taken at the same moment or at a similar moment. Then, the dual-camera image B can be regarded as an invalid dual-camera image B.

[0188] In some other embodiments, the terminal device may identify the clarity of the main image and the auxiliary image in the dual-camera image B. If the clarity of both the main image and the auxiliary image is less than the preset clarity, it indicates that there may be a problem with the clarity of the main image and the auxiliary image. Then, the dual-camera image B can be regarded as an invalid dual-camera image B.

[0189] The terminal device may determine whether the anti-shake data C is valid based on the data content in the anti-shake data B.

[0190] In some embodiments, the terminal device may determine whether the anti-shake data B is valid by judging the moving direction and the moving distance in the anti-shake data B.

[0191] In some examples, the terminal device may regard the anti-shake data B with a moving direction different from the movable direction of the lens as invalid anti-shake data B. For example, in a plane rectangular coordinate system where the x-axis is the horizontal axis and y is the vertical axis, the lens can only move within this plane rectangular coordinate system. If the data in the z direction appears in the anti-shake data B, it indicates that the anti-shake data B is invalid.

[0192] In some examples, the terminal device may regard the anti-shake data B with a moving distance greater than the maximum movable distance of the lens as invalid anti-shake data B. For example, in a plane rectangular coordinate system where the x-axis is the horizontal axis and y is the vertical axis, the lens can only move along the x-axis direction and the y-axis direction, and the maximum moving distance in each direction is 5 unit lengths. If the anti-shake data B shows a movement of 6 unit lengths along a certain direction, such as the x-axis direction, it indicates that the anti-shake data B is invalid anti-shake data B.

[0193] In some other embodiments, the terminal device may also use the method of filtering and noise reduction to determine the valid anti-shake data C.

[0194] In some examples, within the exposure time of a group of dual-camera images or within the exposure time of the main image in a group of dual-camera images, one method is selected from various window function filters for filtering, such as Gaussian filtering.

[0195] In some other examples, within the exposure time of a group of dual-camera images or within the exposure time of the main image in a group of dual-camera images, the filtering window is smoothly moved in the time domain of the anti-shake data B to select the anti-shake data B corresponding to each filtering window. Figure 13As shown, there are a total of five anti-shake data B, namely: B1, B2, B3, B4, B5. One filtering window can have three anti-shake data B. Then, during the smoothing process in the time domain, the terminal device can determine B1' based on B1, B2, B3, determine B2' based on B2, B3, B4, and determine B3' based on B3, B4, B5. Among them, the terminal device can calculate the average value of B1, B2, B3 as B1', or calculate the maximum value of B1, B2, B3 as B1'. The specific calculation method is not limited here. Finally, the terminal device determines B1', B2', B3' as the filtered anti-shake data B, that is, determines the effective anti-shake data C.

[0196] In some other examples, the terminal device can use multiple filtering methods to filter the anti-shake data B multiple times to determine the effective anti-shake data C. For example, the terminal device can first use the Gaussian filtering method to determine the anti-shake data B' from the anti-shake data B. Then, the filtering window is smoothly moved in the time domain of the anti-shake data B' to select the anti-shake data B" corresponding to each filtering window, that is, determine the effective anti-shake data C.

[0197] It can be understood that other methods can also be used here to filter the anti-shake data B to select the anti-shake data C. In this way, through filtering, the influence of noise on the anti-shake data can be reduced, and the accuracy of calculating depth information based on the anti-shake data subsequently can be improved.

[0198] After the terminal device determines the effective multiple sets of dual-camera images C and multiple anti-shake data C from multiple sets of dual-camera images B and multiple anti-shake data B, the terminal device can configure the invalid dual-camera images B and invalid anti-shake data B with invalid flag bits, or assign values representing invalidity to their corresponding flag bits. For example, assign a value of 0 to the flag bit corresponding to the invalid anti-shake data B to indicate that the anti-shake data is invalid, and assign a value of 1 to the flag bit corresponding to the effective anti-shake data B to indicate that the anti-shake data is effective.

[0199] Furthermore, the terminal device can identify the effective anti-shake data C based on whether the anti-shake data B has a flag bit representing invalidity.

[0200] In the above embodiment, when the terminal device executes S803 but does not execute S804, the dual-camera image B can be the multiple sets of dual-camera images in this application, and the anti-shake data B can be the second anti-shake data in this application. When the terminal device executes S803 and S804, the anti-shake data C can be the second anti-shake data in this application.

[0201] S805: The terminal device determines the target dual-camera image D from multiple sets of dual-camera images C and determines the timestamp a of the target dual-camera image D.

[0202] When the user presses the shutter, multiple sets of dual-camera images B corresponding to the moment when the user presses the shutter can be selected from multiple sets of dual-camera images A stored in the image buffer memory according to the time stamp. Herein, corresponding to the moment when the user presses the shutter means that the time difference between the time stamp of the selected multiple sets of dual-camera images B and the moment when the user presses the shutter is within a preset range. In this way, the terminal device can determine the effective dual-camera images C from the selected multiple sets of dual-camera images B, and then select one set of dual-camera images from the effective dual-camera images C as the target dual-camera image D to ensure the clarity of the finally captured photo.

[0203] The target dual-camera image D in the embodiment of the present application can be the target dual-camera image in the present application.

[0204] After being processed, the target dual-camera image D is the photo that is finally determined to be presented to the user and stored in the terminal device. In some embodiments, the terminal device can determine the clearest image from multiple sets of dual-camera images C as the target dual-camera image D, or the terminal device can also determine the most wonderful image from multiple sets of dual-camera images C as the target dual-camera image D. Herein, the most wonderful image can be defined as the dual-camera image determined by the terminal device with a clarity higher than a certain value and including the largest number of faces in the image. Specifically, the most wonderful image can be determined according to the user's needs or the actual shooting scene, and no excessive limitation is made here.

[0205] In addition, after the terminal device determines the target dual-camera image D from multiple sets of dual-camera images C, it can determine the time stamp of the target dual-camera image D, or the time stamp of the main image in the target dual-camera image D as the time stamp a to ensure the accuracy of the time stamp a.

[0206] Optionally, the terminal device can execute S806 to ensure accurate determination of the anti-shake data corresponding to the time stamp.

[0207] S806: The terminal device determines multiple anti-shake data D corresponding to the time stamp a from multiple anti-shake data C.

[0208] As known in the above embodiments related to S803, the terminal device can determine the corresponding anti-shake data B based on the time stamp of the dual-camera image B. In some scenarios, the data volume of the anti-shake data B determined by the terminal device based on the time stamp of the dual-camera image B in S803 is greater than the preset data volume, or after the terminal device determines the anti-shake data B in S803, the data volume of the effective anti-shake data C determined by the terminal device when executing S804 is greater than the preset data volume. Then, the terminal device can execute S806 to ensure accurate determination of multiple anti-shake data D corresponding to the time stamp a.

[0209] In S803, if the data volume of the corresponding anti-shake data B determined by the terminal device based on the timestamp of the dual-camera image B is less than the preset data volume, or, after the terminal device determines the anti-shake data B in S803, the data volume of the effective anti-shake data C determined by the terminal device in S804 is less than the preset data volume, then the terminal device may or may not execute S806.

[0210] In the above embodiment, when the terminal device executes S806, the multiple anti-shake data D corresponding to the timestamp a may be the third anti-shake data in this application.

[0211] Optionally, when the terminal device executes S806 to obtain multiple anti-shake data D, it may execute S807:

[0212] S807: The terminal device interpolates based on the multiple anti-shake data D to obtain the anti-shake data E corresponding to the timestamp a.

[0213] Since the sampling frequency of the anti-shake data is different from that of the dual-camera image, generally, the sampling frequency of the anti-shake data is greater than that of the dual-camera image. Therefore, the generation time of the anti-shake data cannot be exactly the same as that of the dual-camera image. That is to say, it is more likely that there is no anti-shake data D for the timestamp a. Based on this, the terminal device can obtain the anti-shake data E corresponding to the timestamp a by interpolating the anti-shake data D whose time difference from the timestamp a is within the preset time difference range. Among them, the anti-shake data E corresponding to the timestamp a means that the acquisition time of the anti-shake data E is the timestamp a.

[0214] As Figure 14 shown, the terminal device obtains the dual-camera image 1 and the anti-shake data 1 at the moment T0, obtains the anti-shake data 2 at the moment T1, obtains the dual-camera image 2 at the moment T2, and obtains the anti-shake data 3 at the moment T3. The terminal device can interpolate based on the anti-shake data 2 and the anti-shake data 3 to obtain the anti-shake data 4, and the anti-shake data 4 corresponds to the moment T2. In this way, the terminal device can accurately obtain the anti-shake data 4 corresponding to the acquisition moment T2 of the dual-camera image 2, and then perform blurring on the dual-camera image 2 based on the anti-shake data 4, which can effectively improve the blurring effect.

[0215] In some embodiments, as Figure 14 the anti-shake data 2 and the anti-shake data 3 are two anti-shake data D obtained by the terminal device executing S806, and the time interval between the timestamps of the anti-shake data 2 and the anti-shake data 3 and the timestamp of the dual-camera image 2 is less than the preset time interval, then the terminal device can use a function in the form of interpolation(anti-shake data 2, T1, anti-shake data 3, T3, T2) to obtain the anti-shake data 4 at the moment T2.

[0216] Among them, the interpolation method adopted by the terminal device can also be linear interpolation or other methods. For example, the terminal device can also use a larger number of anti-shake data for interpolation. For example, it can use the two anti-shake data closest to the time before T2 and the two anti-shake data closest to the time after T2, and use the Bicubic interpolation algorithm to obtain the anti-shake data at T2. Regarding this, the present application will not elaborate, and reference can be made to related technologies.

[0217] S808. The terminal device compensates the dual-camera calibration data of the dual-camera module based on the anti-shake data E to obtain new dual-camera calibration data.

[0218] The dual-camera calibration data is used to record the initial position of the movable lens in the dual-camera module, and the anti-shake data is used to record the direction and distance of the lens movement in the dual-camera module controlled by the anti-shake module. Therefore, the terminal device compensates the dual-camera calibration data of the dual-camera module based on the anti-shake data to obtain new dual-camera calibration data, that is, the position of the lens in the dual-camera module after movement.

[0219] In some embodiments, the anti-shake module pushes the lens of the main camera in the dual-camera module to perform translational movements in the X and Y directions perpendicular to the optical axis. The corresponding anti-shake data is (x_offset_ois, y_offset_ois), where x_offset_ois represents the movement direction and distance of the lens of the main camera in the X-axis direction, and y_offset_ois represents the movement direction and distance of the lens of the main camera in the Y-axis direction. The dual-camera calibration data of the dual-camera module is (dual_cam_x_offset, dual_cam_y_offset). Then, after the terminal device executes S808, the new dual-camera calibration data obtained is (dual_cam_x_offset_new, dual_cam_y_offset_new). Among them, dual_cam_x_offset_new = dual_cam_x_offset + x_offset_ois, and dual_cam_y_offset_new = dual_cam_y_offset + y_offset_ois.

[0220] In the above embodiments, when the terminal device executes S806 and does not execute S807 and S808, the anti-shake data D in the embodiments of the present application can be the first anti-shake data in the present application. When the terminal device executes S807 and S808, or executes S806 - S808, the anti-shake data E in the embodiments of the present application can be the first anti-shake data in the present application.

[0221] S809. The terminal device calculates depth information based on the new dual-camera calibration data and the target dual-camera image D.

[0222] As shown Figure 15 in the figure, the depth calculation module of the terminal device can calculate depth information based on the new dual-camera calibration data and the main image and the auxiliary image in the target dual-camera image D. Specifically, it can be calculated by referring to the binocular depth disparity and depth calculation methods in the related art, which will not be elaborated here.

[0223] After the terminal device calculates the depth information, it can execute S810 to blur the main image:

[0224] S810. The terminal device blurs the main image in the target dual-camera image D based on the depth information.

[0225] The terminal device can set the area in the main image of the target dual-camera image D where the depth is within the preset depth range to be clear, and the area not within the preset depth range to be blurred, so as to obtain a blurred image as shown Figure 2 in the figure.

[0226] The embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions run on the above terminal device, the terminal device is enabled to execute each function or step in the above method embodiment.

[0227] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program runs on the terminal device, the terminal device is enabled to execute each function or step in the above method embodiment.

[0228] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used for illustration. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0229] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0230] The unit described as a separation component may or may not be physically separated. The component shown as a unit may be a single physical unit or multiple physical units, that is, it may be located in one place or distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, can also exist separately as individual physical units, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks, or optical discs and other various media that can store program codes.

[0233] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to a terminal device including a first camera and a second camera, the method includes: In response to a first operation by a user to open a camera application, display a preview interface of the camera application, the preview interface including shooting controls, and the camera application having enabled a background blurring function and an anti-shake function; In response to a second operation by the user on the shooting controls, store a target image; Wherein, the target image is an image obtained by performing background blurring processing on a first image based on depth information of the first image, the depth information being based on first calibration data, the first image and a second image being determined, the first image being an image captured by the first camera at a first moment, the second image being an image captured by the second camera at the first moment, and the first calibration data being used to indicate a pose relationship between the first camera and the second camera at the first moment.

2. The method according to claim 1, wherein The method further includes: Obtain first anti-shake data corresponding to the first moment, the first anti-shake data being movement parameters for moving a lens of a target camera at the first moment, the target camera including the first camera and / or the second camera; Based on the first anti-shake data and second calibration data, determine the first calibration data; the second calibration data is used to indicate an initial pose relationship between the first camera and the second camera.

3. The method according to claim 2, characterized in that, The obtaining of the first anti-shake data at the first moment includes: Determine a target dual-camera image, the target dual-camera image being a dual-camera image collected by the terminal device at the first moment, the target dual-camera image including the first image and the second image; Based on the time stamp of the target dual-camera image, obtain the first anti-shake data; the time difference between the time stamp of the first anti-shake data and the time stamp of the target dual-camera image is within a preset time range.

4. The method according to claim 3, wherein The determining of the target dual-camera image includes: Obtain multiple sets of dual-camera images, each set of dual-camera images in the multiple sets of dual-camera images including an image captured by the first camera and an image captured by the second camera at the same moment, and the time difference between the time stamp corresponding to each set of dual-camera images in the multiple sets of dual-camera images and the first moment is within a preset time range; Determine the target dual-camera image from the multiple sets of dual-camera images.

5. The method according to claim 4, wherein The obtaining of the first anti-shake data based on the time stamp of the target dual-camera image includes: Obtain second anti-shake data corresponding to each set of dual-camera images in the multiple sets of dual-camera images; Based on the time stamp of the target dual-camera image and multiple pieces of the second anti-shake data, determine the first anti-shake data.

6. The method according to claim 5, wherein The determining of the first anti-shake data based on the time stamp of the target dual-camera image and multiple pieces of the second anti-shake data includes: Based on the time stamp of the target dual-camera image, determine third anti-shake data corresponding to the time stamp of the target dual-camera image from the multiple pieces of the second anti-shake data; In the case where there are multiple pieces of the third anti-shake data, perform interpolation processing on the multiple pieces of the third anti-shake data to obtain the first anti-shake data.

7. The method according to claim 5 or 6, characterized in that The multiple sets of dual-camera images are valid dual-camera images among the dual-camera images whose time difference between the corresponding timestamp and the first moment is within a preset time range; the multiple second anti-shake data are valid anti-shake data corresponding to each set of dual-camera images among the multiple sets of dual-camera images.

8. The method according to any one of claims 5-7, characterized in that, The terminal device includes a memory, and the memory includes at least two storage areas; The obtaining of the second anti-shake data corresponding to each set of dual-camera images among the multiple sets of dual-camera images includes: Reading the second anti-shake data from the first storage area of the memory; Wherein, when reading the anti-shake data from the first storage area, the terminal device stores the anti-shake data to be stored in the second storage area. The first storage area is any one of the at least two storage areas, and the second storage area is the storage area other than the first storage area among the at least two storage areas.

9. A terminal device, characterized in that, The terminal device includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processors are coupled; the display screen is used for displaying the images generated by the processors, the memory is used for storing computer program codes, and the computer program codes include computer instructions; when the processors execute the computer instructions, the terminal device executes the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It includes computer instructions, and when the computer instructions run on the terminal device, the terminal device executes the method according to any one of claims 1-8.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the method according to any one of claims 1-8 is implemented.

Images