Image processing method and device, terminal equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, terminal equipment, a storage medium and a program product. The image processing method comprises the steps of performing primary focusing according to a determined focusing target; acquiring an initial image distance and an initial temperature; wherein the initial image distance is the distance from a lens assembly to an image sensor after the initial focusing is completed, and the initial temperature is the temperature when the initial focusing is carried out; according to the current temperature, the initial image distance and the initial temperature, performing at least one time of real-time focusing on the focusing target; and performing imaging after the at least one time of real-time focusing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to an image processing method and device, a terminal device, a storage medium, and a computer program product. BACKGROUND

[0002] With the development of technology, more and more terminal devices appear, and the functions of the terminal devices are also more and more rich. Each terminal device can be applied in a corresponding use scenario, and the use scenarios of different terminal devices can be different, and the functions thereof can also be different.

[0003] In some terminal devices, an image acquisition module is included, and the terminal devices have the functions of image acquisition and video acquisition. The image acquisition module can be used to acquire images and videos. SUMMARY

[0004] The present disclosure provides an image processing method and device, a terminal device, a storage medium, and a computer program product.

[0005] In a first aspect of the embodiments of the present disclosure, an image processing method is provided, including: performing primary focusing on a focusing target according to a determination; obtaining an initial image distance and an initial temperature; wherein the initial image distance is a distance from a lens assembly to an image sensor after the primary focusing is completed, and the initial temperature is a temperature when the primary focusing is performed; performing at least one real-time focusing on the focusing target according to a current temperature, the initial image distance, and the initial temperature; and performing imaging after the at least one real-time focusing.

[0006] In one embodiment, the performing at least one real-time focusing on the focusing target according to a current temperature, the initial image distance, and the initial temperature includes: determining a temperature variation amount of the current temperature and the initial temperature; determining an image distance difference corresponding to the temperature variation amount; determining a target image distance of the current temperature according to the image distance difference and the initial image distance; and performing the at least one real-time focusing on the focusing target according to the target image distance.

[0007] In one embodiment, the obtaining an initial image distance includes: determining the initial image distance according to a preset mapping relationship and the initial temperature; wherein the preset mapping relationship is a relationship between a distance from the lens assembly to the image sensor and a temperature in a case where the focusing target is unchanged.

[0008] In an embodiment, the determining the distance difference corresponding to the temperature change amount comprises: determining a first reference distance and a second reference distance according to the preset mapping relationship; wherein the first reference distance is a distance from the lens assembly to the image sensor after focusing on a reference target at the initial temperature; the second reference distance is a distance from the lens assembly to the image sensor after focusing on the reference target at the current temperature; and the distance difference is determined according to the first reference distance and the second reference distance.

[0009] In an embodiment, the preset mapping relationship comprises: a linear relationship; a nonlinear relationship.

[0010] In an embodiment, the target distance includes first distance information; the determining the target distance at the current temperature comprises: converting the first distance information into the second distance information; converting the second distance information into the first distance information; wherein, within a preset distance range, a one-to-one first mapping relationship exists between the first distance information and the second distance information, and outside the preset distance range, a many-to-one second mapping relationship exists between the first distance information and the second distance information; determining the first distance information corresponding to each frame of image in the mth frame to the nth frame of image; and if third distance information in the first distance information is out of the preset distance range, determining the target distance when the ith frame of image is captured at the current temperature according to the first distance information.

[0011] In an embodiment, the state of the initial focusing on the focusing target is a preview state.

[0012] In an embodiment, the method further comprises: detecting a focusing operation, and determining the focusing target according to the focusing operation; or determining the focusing target according to a preset focusing parameter.

[0013] In an embodiment, the imaging mode comprises at least any one of: a time-lapse shooting mode; a movie mode; a slow-motion mode; a live mode; a video mode; a photo mode; a portrait mode; and a preview mode.

[0014] A second aspect of the embodiments of the present disclosure provides an image processing apparatus, comprising: a first focusing module configured to perform initial focusing according to a determined focusing target; an acquisition module configured to acquire an initial distance and an initial temperature; wherein the initial distance is a distance from a lens assembly to an image sensor after the initial focusing is completed, and the initial temperature is a temperature when the initial focusing is performed; a second focusing module configured to perform at least one real-time focusing on the focusing target according to a current temperature, the initial distance, and the initial temperature; and an imaging module configured to perform imaging after the at least one real-time focusing.

[0015] In a third aspect, the present disclosure provides a terminal device, comprising a processor and a memory for storing executable instructions capable of running on the processor, wherein when the processor runs the executable instructions, the executable instructions perform the method in any of the above embodiments.

[0016] In a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium, wherein the computer-readable storage medium stores computer executable instructions, and the computer executable instructions, when executed by a processor, perform the method in any of the above embodiments.

[0017] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program or executable instructions, and when the computer program or executable instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0018] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0019] In the scheme of the embodiments of the present disclosure, after initial focusing, the initial image distance and the initial temperature corresponding to the initial focusing are taken as references, and according to the current temperature, the initial temperature and the initial image distance, the focusing of the focusing target can be re-performed when the temperature changes. In this way, the focusing effect of the focusing target can be adjusted according to the change of the temperature, so that the focusing of the same focusing target can be realized when the temperature changes, and the focusing target does not change with the change of the temperature. The temperature compensation is performed on the focusing condition. The embodiments improve the focusing effect of focusing the focusing target, reduce the situation that the optical reference of the lens assembly changes due to the change of the temperature, so that the focusing target changes, that is, the problem of defocusing caused by the change of the temperature is reduced, the imaging quality of the image is improved, and the user's use experience is improved.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0022] Figure 1 is a schematic diagram of an image processing method according to an exemplary embodiment;

[0023] Figure 2 is a schematic diagram of an image according to an exemplary embodiment;

[0024] Figure 3is a schematic diagram of another image according to an example embodiment;

[0025] Figure 4 is a schematic diagram of performing at least one real-time focusing according to an example embodiment;

[0026] Figure 5 is a schematic diagram of a preset mapping relationship according to an example embodiment;

[0027] Figure 6 is a schematic diagram of determining an image distance difference according to an example embodiment;

[0028] Figure 7 is a schematic diagram of an image processing apparatus according to an example embodiment;

[0029] Figure 8 is a schematic diagram of another image processing method according to an example embodiment;

[0030] Figure 9 is a block diagram of a terminal device according to an example embodiment. DETAILED DESCRIPTION

[0031] The example embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to represent the same elements in different drawings. The embodiments described in the following example embodiments do not represent all the implementations consistent with the present disclosure. Instead, they only represent examples consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] Reference Figure 1 is a schematic diagram of an image processing method, the image processing method comprising:

[0033] S100, performing primary focusing according to the determined focusing target.

[0034] S200, obtaining an initial image distance and an initial temperature; wherein the initial image distance is a distance from a lens assembly to an image sensor after the primary focusing is completed at an initial temperature, and the initial temperature is a temperature when the primary focusing is performed.

[0035] S300, performing at least one real-time focusing on the focusing target according to a current temperature, the initial image distance, and the initial temperature.

[0036] S400, performing imaging after the at least one real-time focusing.

[0037] The scheme of this embodiment can be applied to a terminal device, and the execution subject can be a terminal device with an image acquisition function, which can include mobile terminal devices and fixed terminal devices, etc. The mobile terminal device can include a mobile phone, a tablet computer, a vehicle-mounted device, a wearable device, a smart device, and a flying device, etc. The smart device can further include a smart office device, a smart home device, and a robot, etc.

[0038] The terminal device can acquire images. The terminal device has an image acquisition module, and the number, specifications, and other parameters of the image acquisition module can be determined according to the use requirements. The configurations of the image acquisition modules in different terminal devices can be different.

[0039] For example, the image acquisition module can include a camera, such as a wide-angle camera, a long-focus camera, a macro camera, and an ultra-wide-angle camera, etc. The long-focus camera can include a periscope long-focus camera.

[0040] For example, the image acquisition module includes a lens assembly, which can include at least one lens. The image acquisition module can also include a photosensitive assembly, which can include an image sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) image sensor and a Charged Coupled Device (CCD) sensor.

[0041] The position of the photosensitive assembly in the image acquisition module is fixed. The image acquisition module is assembled in the terminal device, and the position of the image acquisition module in the terminal device is fixed. The position of the photosensitive assembly in the terminal device is also relatively fixed. The position of the lens assembly in the image acquisition module can change, for example, the lens assembly can move along the optical axis in the image acquisition module. The lens assembly in the image acquisition module can move within a preset movement range. That is, the position of the lens assembly relative to the photosensitive assembly is changeable and adjustable. When the position of the lens assembly relative to the photosensitive assembly changes, the relative position of the lens assembly in the image acquisition module also changes.

[0042] The position of the lens assembly in the image acquisition module can be adjusted according to the focusing information, thereby achieving focusing.

[0043] For example, the preset movement range of the lens assembly in different image acquisition modules can be different.

[0044] The terminal device can also have a display module, such as a display screen, which can display the captured image and can also perform image preview, display a preview image, and the like. The display screen can be a touch display screen that can detect touch operations on the display screen, such as focusing operations and image capture operations on an image capture control, which can include a shutter control.

[0045] The terminal device has an application program installed for capturing images, such as a camera application program. When the application program is started, the image can be captured.

[0046] For S100, during the image capturing process, focusing is needed, and the image is captured after focusing. The captured image has a focusing subject, the blurring degree of the focusing subject is lower than that of other regions, and the clarity of the focusing subject is higher than that of other regions. The focusing subject can also be a focusing target.

[0047] During focusing, the focusing operation can be used to focus. For example, after starting the image capture application program, a preview interface can be displayed on the display screen, and a preview image can be displayed in the preview interface. During focusing, the terminal device can detect a focusing operation on the preview image through the display screen, determine the focusing position according to the focusing operation, and thus determine the focusing target to achieve focusing.

[0048] For example, the focusing target can be determined in the following ways:

[0049] Detecting a focusing operation and determining a focusing target according to the focusing operation.

[0050] Alternatively,

[0051] Determining a focusing target according to a preset focusing parameter.

[0052] This embodiment provides two ways to determine the focusing target. The focusing target can be determined by a manual focusing operation, or the terminal device can automatically determine the focusing target, such as driving the focusing target according to a preset focusing parameter.

[0053] For example, the focusing target can be a photographed object, including but not limited to: a person, a plant, a mountain, a river, the sky, a road, an animal, a vehicle, a building, an electronic and electrical device, and the like.

[0054] For example, each focusing operation corresponds to a respective focusing target, and focusing operations on different positions correspond to different focusing targets.

[0055] After the focusing target is determined, the focusing target can be used for focusing. The operation of this focusing is recorded as the initial focusing. The initial focusing can be distinguished from at least one real-time focusing at the current temperature.

[0056] After the initial focusing, the focused target can be clearly imaged on the image sensor.

[0057] For the S200, the terminal device may also include a temperature sensor for collecting temperature information. The location of the temperature sensor is not limited and can be determined according to usage requirements. The temperature can be the temperature of the terminal device or the temperature of the lens assembly.

[0058] If the temperature sensor is located in the terminal device away from the lens assembly, it can collect the temperature of the terminal device. If the temperature sensor is located on the lens assembly, or on the outer surface of the lens assembly, it collects the temperature of the lens assembly.

[0059] For example, the number of temperature sensors is not limited; the more sensors there are, the more accurate the current temperature will be as a result of the data collected from multiple temperature sensors.

[0060] The temperature collected by the temperature sensor during the initial focusing will be recorded as the initial temperature.

[0061] The initial image distance can include the distance between the lens assembly and the image sensor after initial focusing at the initial temperature. This initial image distance can also be represented by the position of the lens assembly, such as the relative distance of the lens assembly to the image sensor after initial focusing, denoted as the initial position. The method for determining the initial image distance is not limited; any method that can determine the initial image distance is acceptable, as is any method that can determine the relative distance of the lens assembly to the image sensor.

[0062] For example, the image acquisition module may further include a driving component, which is a driving component for the lens assembly, used to drive the movement of the lens assembly and adjust the relative position of the lens assembly within the image acquisition module. The connection method between the driving component and the lens assembly is not limited; any connection method that enables the driving component to drive the movement of the lens assembly is acceptable. This connection can be direct or indirect, and may include a transmission component, etc.

[0063] For example, the drive component may include a drive motor.

[0064] For example, when determining the current position, the current position can be determined based on the driving information of the driving component. The driving component can be connected to a controller, which can control the driving component to move the lens component. The driving information of the driving component can represent the movement information of the lens component, and the current position of the lens component can be determined based on this driving information. The control signal output by the controller is used to control the driving component to generate a driving signal to drive the lens component. As the control center for controlling the movement of the lens component, the controller can determine the position of the lens component's movement, so the current position, the amount of movement, and the position after movement of the lens component can also be determined based on the control signal.

[0065] The position of the lens assembly relative to the photosensitive assembly can represent the relative distance between the two, and thus the initial image distance can also be determined.

[0066] Exemplarily, the terminal device can further include a position detection assembly for detecting the position of the lens assembly in the image acquisition module. The position detection assembly can include a distance sensor located on the side where the photosensitive assembly is located, which can detect the relative distance of the lens assembly relative to the photosensitive assembly, and thus determine the initial image distance according to the relative distance. The relative distance can be taken as the initial image distance.

[0067] For S300, for the lens assembly, the lens assembly has temperature drift, and is greatly affected by temperature. After the temperature changes, the optical parameters of the lens assembly will change, causing the focus point to change. This will cause the image collected to change, such as out of focus, blur, or change in focus point, when determining the focus point for image acquisition. This can cause the image to be blurred, thereby reducing the image quality, and the image obtained is no longer the image the user needs.

[0068] Reference Figure 2 is an image schematic diagram, Figure 3 is another image schematic diagram.

[0069] Figure 2 The image shown is an image after the focus target is determined, Figure 3 The image shown is an image obtained after the temperature changes, due to the influence of temperature on the lens assembly, causing the parameters in the lens image to change and resulting in out of focus. Figure 3 The image shown is relative to Figure 2 The image shown is more blurred, and the image quality is reduced.

[0070] The temperature sensor can also obtain the current temperature, which can be different from the initial temperature. The current temperature can be higher than the initial temperature, or it can be lower than the initial temperature.

[0071] Exemplarily, the temperature sensor has a sampling period, and can collect the temperature according to the sampling period to obtain the current temperature.

[0072] Exemplarily, the current temperature is the temperature after the initial temperature.

[0073] After the initial focusing, the current temperature and the initial temperature can be different. According to the current temperature, the initial image distance and the initial temperature, at least one real-time focusing is performed on the focus target, so that the focus target can always be clearly imaged in the photosensitive assembly.

[0074] For example, the at least one real-time focusing can include at least one adjustment of the position of the lens assembly. For example, it can be determined whether the current temperature is the same as the initial temperature, and in the case where the two temperatures are different, the position of the lens assembly is adjusted so that the image distance is changed, the focusing target can be clearly imaged on the photosensitive element, and the at least one real-time focusing of the focusing target is completed.

[0075] For example, the at least one real-time focusing can include at least one adjustment of the initial image distance, i.e., adjusting the initial position of the lens assembly after the initial focusing.

[0076] For example, according to the current temperature, the initial image distance, and the initial temperature, the at least one real-time focusing of the focusing target can be performed according to the preset frequency, the current temperature, the initial image distance, and the initial temperature. The real-time focusing here is better the smaller the interval between adjacent two focusings is, and the size of the interval between adjacent two focusings is negatively related to the imaging effect after the real-time focusing. That is, the smaller the interval between adjacent two focusings is, the better the imaging effect after the real-time focusing is, and the larger the interval between adjacent two focusings is, the worse the imaging effect after the real-time focusing is.

[0077] For example, the real-time focusing here can include real-time focusing according to the change between the current temperature and the initial temperature, i.e., the current temperature changes relative to the initial temperature, and at least one real-time focusing is performed according to the temperature change.

[0078] In this way, the focusing target can be refocused according to the change in temperature, the focusing condition can be temperature-compensated, and the defocusing caused by the change in the optical parameters of the lens assembly due to the change in temperature can be reduced. The multiple real-time focusings are more conducive to temperature compensation of the focusing condition, reduce the situation that the focusing target always changes, and maintain the imaging of the focusing target on the photosensitive assembly in a clear state.

[0079] For S400, the imaging can be performed after the at least one real-time focusing is completed, and the image of the focusing target obtained on the photosensitive assembly is clearer.

[0080] Through the above method of the embodiment, after the initial focusing, the initial image distance and the initial temperature corresponding to the initial focusing are taken as references, and according to the current temperature, the initial temperature and the initial image distance, the focusing of the focusing target can be re-performed when the temperature changes. In this way, the focusing effect of the focusing target can be adjusted according to the change of the temperature, so that the focusing of the same focusing target can be realized when the temperature changes, so that the focusing target does not change with the change of the temperature, and the temperature compensation of the focusing condition is realized. The embodiment improves the focusing effect of focusing the focusing target, reduces the situation that the optical reference of the lens assembly changes due to the change of the temperature, so that the focusing target changes, that is, reduces the problem of defocus caused by the change of the temperature, facilitates to improve the imaging quality of the image, and further improves the use experience of the user.

[0081] In one embodiment, referring to Figure 4 , a method for performing at least one real-time focusing, the method comprises:

[0082] S301, determining the temperature change amount of the current temperature and the initial temperature.

[0083] S302, determining the image distance difference corresponding to the temperature change amount.

[0084] S303, determining the target image distance of the current temperature according to the image distance difference and the initial image distance.

[0085] S304, performing at least one real-time focusing on the focusing target according to the target image distance.

[0086] After the current temperature, the initial temperature and the initial image distance are determined, the temperature change amount of the current temperature and the initial temperature, that is, the temperature difference, can be determined. If they are different, there is a temperature change amount, and if they are the same, there is no temperature change amount.

[0087] When the initial temperature and the current temperature are different, the image distance difference corresponding to the temperature change amount of the two can be determined. Each temperature can correspond to a respective image distance, so that the image distance difference corresponding to the image distance of different temperatures can be determined.

[0088] Exemplarily, the image distance change amount corresponding to the temperature change amount, that is, the image distance difference, can be determined according to the preset information. The preset information includes the mapping relationship between the temperature change amount and the image distance difference. Of course, it can also be other ways and is not limited.

[0089] Since the initial image distance has been obtained, the target image distance can be determined according to the initial image distance and the image distance difference, so that at least one real-time focusing is performed on the focusing target according to the target image distance, so that the target object is always in a clear state in the imaging of the light sensing component, thereby reducing the situation that the lens assembly is affected by the change of the temperature.

[0090] Exemplarily, the determining the target image distance can further include:

[0091] The target image distance is determined according to the preset information and the current temperature, the preset information including a mapping relationship between the current temperature and the target image distance, and the corresponding target image distance being found according to the current temperature.

[0092] In an embodiment, when the image distance is represented by a relative position between the lens assembly and the photosensitive assembly, the method can further include:

[0093] The target position of the lens assembly at the initial temperature is determined, i.e., the target position is determined according to the current temperature. The target position can represent the position of the lens assembly relative to the photosensitive assembly. At the current temperature, the lens assembly is located at the target position to focus on the focusing target, so that the focusing target can be clearly displayed.

[0094] Different current temperatures correspond to respective target positions, and the lens assembly is located at different target positions to focus on the same focusing target. After the current temperature changes, the target position corresponding to the changed current temperature can be determined according to the changed current temperature.

[0095] The manner of determining the target position is not limited, and the target position can be determined after the temperature is determined according to the relationship between the temperature and the target position.

[0096] The relationship between the temperature and the target position can be preset, and can be obtained according to experimental test data.

[0097] Exemplarily, the current position and the target position can be determined after the current temperature is obtained, or the current position and the target position can be determined at the same time when the current temperature is obtained. For example, the current position of the lens assembly at the current temperature is obtained, and the target position of the lens assembly at the same temperature is obtained.

[0098] Exemplarily, the imaging mode includes at least any one of the following modes:

[0099] A time-lapse photography mode;

[0100] A movie mode;

[0101] A slow-motion mode;

[0102] A live mode;

[0103] A video mode;

[0104] A photo mode;

[0105] A portrait mode;

[0106] A preview mode.

[0107] In any image acquisition or video acquisition mode, as the device usage time increases, the device may be heated, causing the temperature of the lens assembly to rise, affecting the performance parameters of the lens image. Therefore, in any of the above modes, the methods in the above embodiments and / or the methods in the subsequent embodiments can be performed. That is, the methods of the above embodiments can be applied to different image acquisition modes, and can also be applied to different video acquisition modes, and can also be applied to various preview modes.

[0108] Some modes are exemplified in this embodiment, and other modes for image acquisition or video recording can also be used, which are not limited here. Different terminal devices and / or image acquisition applications may have different modes.

[0109] In one embodiment, S300, the target position of the lens assembly at the current temperature is determined, including:

[0110] According to the preset mapping relationship and the initial temperature, the initial image distance is determined.

[0111] The preset mapping relationship is the relationship between the position of the lens assembly relative to the photosensitive assembly and the temperature of the lens assembly when the focusing target is unchanged.

[0112] In this preset mapping relationship, the image distance (including the distance of the lens assembly relative to the photosensitive assembly) at any temperature can achieve focusing on the focusing target.

[0113] Exemplarily, it can also include:

[0114] The reference mapping relationship between the distance of the lens assembly to the image sensor (including the position of the lens assembly relative to the photosensitive assembly) and the temperature of the lens assembly is determined according to a plurality of different focusing targets, and then the preset mapping relationship is determined according to a plurality of reference mapping relationships.

[0115] For any focusing target, the relationship between the distance of the lens assembly relative to the photosensitive assembly and the temperature when the focusing target is unchanged can be determined.

[0116] For another focusing target, the relationship between the position of the lens assembly relative to the photosensitive assembly and the temperature of the lens assembly can also be determined.

[0117] In this way, the reference mapping relationship corresponding to a plurality of focusing targets can be determined, and the reference mapping relationships are fitted to obtain the preset mapping relationship.

[0118] Exemplarily, the preset mapping relationship includes:

[0119] Linear relationship;

[0120] Non-linear relationship.

[0121] For example, the preset mapping relationship can be obtained by fitting experimental data.

[0122] refer to Figure 5 This is a schematic diagram of a preset mapping relationship, where the horizontal axis represents temperature and the vertical axis represents the position of the lens assembly. Figure 4 The two functions shown are functions with a pre-defined mapping relationship.

[0123] For example, Figure 5 The vertical axis represents the distance between the lens assembly and the focused object, i.e., the object distance, which is negatively correlated with the distance between the lens assembly and the photosensitive assembly. After determining the preset mapping relationship, a first change in the object distance can be obtained based on this preset mapping relationship. Then, a second change in the distance between the lens assembly and the photosensitive assembly can be determined based on the first change in the object distance. The first and second changes are the same but opposite in direction; that is, when the first change decreases, the second change increases, and vice versa.

[0124] According to the imaging formula of a camera, when the focal length remains constant, the object distance and the image distance are negatively correlated.

[0125] Figure 5 Points of different color depths represent different initial temperatures. Figure 4 The preset mapping relationship shown indicates that as the temperature increases, the object distance first increases and then gradually decreases.

[0126] In one embodiment, reference Figure 6 This is a schematic diagram for determining the image distance difference. S302, Determining the image distance difference includes:

[0127] S3021, determine the first reference image distance and the second reference image distance according to the preset mapping relationship. The first reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the initial temperature; the second reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the current temperature.

[0128] S3022, determine the image distance difference based on the first reference image distance and the second reference image distance.

[0129] The preset mapping relationship can be determined based on a focus target other than the currently determined focus target. Therefore, the initial image distance obtained based on this preset mapping relationship is not necessarily the initial image distance of the currently determined focus target. After a temperature change, the change in image distance determined by this preset mapping relationship remains unchanged and can be used as the change in image distance when focusing on the currently determined focus target, i.e., the image distance difference.

[0130] The first reference image distance and the second reference image distance can be determined according to a preset mapping relationship. The reference target can be a focusing object different from the focusing target, for example, a focusing target other than the currently determined focusing target. The object distance corresponding to the reference target is different from the object distance corresponding to the focusing target. For example, the reference target is a focusing object in a long shot, and the focusing target is a focusing object in a close shot. Alternatively, the reference target is a focusing object in a close shot, and the focusing target is a focusing object in a long shot.

[0131] The first reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the initial temperature, and the second reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the current temperature.

[0132] Since the distance between the reference target and the currently determined focusing target and the image acquisition module can be different, they can be located in different distance ranges, and the object distance is negatively correlated with the image distance, directly using the initial image distance obtained according to the preset mapping relationship as the initial image distance when focusing on the focusing target can have deviations, resulting in inaccurate initial image distance.

[0133] According to the first reference image distance and the second reference image distance, the image distance difference between the first reference image distance and the second reference image distance is determined, and the image distance difference is used to determine the target image distance. The target image distance obtained in this way is more accurate, thereby improving the accuracy of focusing, reducing the deviation of focusing on the focusing target, and thereby reducing the out-of-focus situation after temperature change.

[0134] In one embodiment, the image processing method further comprises:

[0135] In the preview state, the focusing target is determined and the focusing target is focused. The initial temperature is the temperature when the initial focusing is performed in the preview state, and the current temperature is the temperature when at least one real-time focusing is performed after switching from the preview state to the video recording state (for example, the video recording state in the time-lapse shooting mode).

[0136] This embodiment is an example of determining a target position in a preview state.

[0137] In the preview state, image acquisition or video recording has not yet started, and focusing can be performed first to determine the focusing target. After the focusing target is determined, the focusing target no longer changes after image acquisition or video recording starts, and the focusing target is always used as the focusing object.

[0138] For example, after the focusing target is determined, the distance between the focusing target and the lens assembly, i.e., the object distance, can be determined.

[0139] After the focusing target is determined, the position of the lens assembly is the current position of the lens assembly, and the temperature of the lens assembly at the time of focusing is the current temperature.

[0140] In this way, the current position and the current temperature at the time of determining the focusing target in the preview process are determined, so as to facilitate determining the target position at the current temperature according to the preset mapping relationship, and then adjusting the position of the lens assembly.

[0141] In an embodiment, the target position includes first position information, and the terminal device needs to perform the following conversion between the first position information and second position information when determining the target position:

[0142] convert the first position information into the second position information;

[0143] convert the second position information into the first position information;

[0144] In the preset position range, the first position information and the second position information have a one-to-one first mapping relationship. Outside the preset position range, the first position information and the second position information have a many-to-one second mapping relationship.

[0145] The first position information and the second position information are information in two different formats, for example, the first position information is analog information, and the second position information is digital information, or the second position information is analog information, and the first position information is digital information. For another example, the first position information is low-layer position information, and the second position information is application-layer position information.

[0146] Because inaccuracy may occur in the conversion process, if there is first position information that exceeds the preset position range, different first position information that exceeds the preset position range may be converted into the same second position information. In this way, when the second position information is converted into the first position information, the correct first position information cannot be obtained, and thus the accurate target position cannot be obtained.

[0147] For example, when the focusing target is a far scene, a target position that exceeds the hyperfocal distance may occur. In this case, no matter where the focusing is, the obtained second position information is the same. When the second position information is converted into the first position information, the first position information corresponding to the second position information cannot be determined, and thus the target position cannot be obtained.

[0148] In this case, determining the target position of the lens assembly at the current temperature includes:

[0149] determining first position information of the lens assembly corresponding to each frame of image in the mth frame to the nth frame of image;

[0150] If the first position information exceeds the preset position range, the target position of the lens assembly when focusing on the focusing target and collecting the i-th image is determined according to the first position information of the lens assembly corresponding to each frame of image collected from the m-th frame to the n-th frame at the current temperature. i is between m and n.

[0151] For example, the m-th frame can be the first frame of image at the beginning of preview image, and the n-th frame can be the end frame of image at the end of image collection, that is, the last frame of image in the image collection process.

[0152] For example, n is greater than m, and the difference between n and m is less than a preset value.

[0153] By the scheme of the embodiment, the abnormal situation in the position information conversion process can be reduced, the defocusing problem caused by temperature change when focusing on the telefocus focusing target can be reduced, and the accuracy of determining the target position is improved.

[0154] In one embodiment, referring to Figure 7 Fig. 1 is a schematic diagram of an image processing device, which comprises:

[0155] A first focusing module 1 is configured to focus on a focusing target for the first time according to the determined focusing target.

[0156] An acquisition module 2 is configured to acquire an initial image distance and an initial temperature. The initial image distance is the distance from the lens assembly to the image sensor after the first focusing, and the initial temperature is the temperature when the first focusing is performed.

[0157] A second focusing module 3 is configured to focus on the focusing target at least once in real time according to the current temperature, the initial image distance and the initial temperature.

[0158] An imaging module 4 is configured to perform imaging after the at least once real-time focusing.

[0159] In one embodiment, the second focusing module 3 comprises:

[0160] A first determining unit is configured to determine the temperature variation amount of the current temperature and the initial temperature.

[0161] A second determining unit is configured to determine the image distance difference corresponding to the temperature variation amount.

[0162] A third determining unit is configured to determine the target image distance of the current temperature according to the image distance difference and the initial image distance.

[0163] A focusing unit is configured to focus on the focusing target at least once in real time according to the target image distance.

[0164] In one embodiment, the acquisition module 2 is configured to:

[0165] determine the initial image distance according to the preset mapping relationship and the initial temperature, wherein the preset mapping relationship is a relationship between a distance from the lens assembly to the image sensor and a temperature when the focusing target is unchanged.

[0166] In one embodiment, the second determination unit includes:

[0167] The reference image distance determination subunit is configured to determine a first reference image distance and a second reference image distance according to the preset mapping relationship, wherein the first reference image distance is a distance from the lens assembly to the image sensor after focusing on a reference target at the initial temperature, and the second reference image distance is a distance from the lens assembly to the image sensor after focusing on the reference target at the current temperature.

[0168] The image distance difference determination subunit is configured to determine the image distance difference according to the first reference image distance and the second reference image distance.

[0169] In one embodiment, the preset mapping relationship includes:

[0170] a linear relationship;

[0171] a nonlinear relationship.

[0172] In one embodiment, the target image distance includes first image distance information.

[0173] The third determination unit includes:

[0174] The first conversion subunit is configured to convert the first image distance information into the second image distance information.

[0175] The second conversion subunit is configured to convert the second image distance information into the first image distance information, wherein a one-to-one first mapping relationship exists between the first image distance information and the second image distance information within a preset image distance range, and a many-to-one second mapping relationship exists between the first image distance information and the second image distance information outside the preset image distance range.

[0176] The first determination subunit is configured to determine the first image distance information corresponding to each frame of image in the mth frame to the nth frame of image.

[0177] The second determination subunit is configured to determine the target image distance when the i th frame of image is collected at the current temperature according to the first image distance information if third image distance information exceeding the preset image distance range exists in the first image distance information.

[0178] In one embodiment, the state of the initial focusing according to the focusing target is a preview state.

[0179] In one embodiment, the apparatus further comprises a focusing target determination module configured to:

[0180] detect a focusing operation, and determine the focusing target according to the focusing operation;

[0181] or,

[0182] determine the focusing target according to a preset focusing parameter.

[0183] In one embodiment, the imaging mode comprises at least one of the following:

[0184] a time-lapse mode;

[0185] a movie mode;

[0186] a slow-motion mode;

[0187] a live mode;

[0188] a video mode;

[0189] a photo mode;

[0190] a portrait mode;

[0191] a preview mode.

[0192] In one embodiment, an example of an application scenario is provided.

[0193] With the development of economy and the progress of technology, mobile phones are used as photographic tools more and more frequently, and the functions of mobile phone cameras are also becoming more and more powerful, and the time-lapse photography mode is one of them. Time-lapse photography is a technique that involves taking video at a low frame rate over a long period of time and then playing it back at a much higher frame rate, creating a time-lapse effect. Through this technique, slow processes that are usually not observable, such as the opening of flowers, the construction of buildings, or the movement of the stars, can be shown. As an important part of mobile phone camera imaging, the quality of the autofocus directly affects the recording effect of time-lapse photography. If the phenomenon of defocusing occurs during recording, the entire video will not be able to be completed.

[0194] The time-lapse photography mode has the following characteristics compared with the normal use mode of the camera:

[0195] (1) The time-lapse photography mode requires device and picture stability;

[0196] (2) The time-lapse photography mode records video for a long time.

[0197] To provide a stable picture, the usual practice is to perform an autofocus once before recording the video (preview stage) and lock the autofocus at the beginning of recording, without triggering the focus again, no matter what changes occur in the picture. The advantage of this is that it can avoid the impact on the stability of the picture due to the frequent triggering of autofocus.

[0198] However, for the image acquisition module of a mobile phone camera, especially for the image acquisition module with high optical magnification (such as a periscope), the temperature drift of the lens will be large. After recording the video for a long time, the temperature of the mobile phone will increase significantly (the temperature may increase by 40+ degrees Celsius). The temperature increase will change the optical parameters of the lens, causing the focal point to change. Therefore, after the autofocus is locked, it will cause the phenomenon of out-of-focus, as shown in Figure 3 .

[0199] This embodiment adjusts the position of the lens assembly based on a temperature compensation algorithm. In the autofocus algorithm, the brief test process and algorithm principle of temperature compensation are as follows:

[0200] a. Place the mobile phone in a thermostat for 5 sets of experiments, and set the temperature of the thermostat to -10°C, 0°C, 10°C, 20°C, and 30°C for each set of experiments. After placing the mobile phone in the thermostat for half an hour, autofocus to the distant view.

[0201] b. Keep the shooting scene unchanged, and count the temperature of the camera through the temperature sensor of the camera. When the temperature increases by one degree, record the temperature data and the lens position data at this time (after the temperature increases, the optical parameters of the lens will change, causing the previously aligned focus lens position to no longer be in focus. The autofocus algorithm will trigger the focus again to get a new focus position).

[0202] c. Count the lens position data corresponding to each temperature data, and the results are as follows Figure 5 . Figure 5 is the measured results of temperature compensation of a certain project. The horizontal coordinate in the figure is the temperature, and the vertical coordinate is the position of the lens after alignment. Through the measured results, the following rules are found:

[0203] ①: Different initial temperatures correspond to different lens positions;

[0204] ②: In the range of the initial temperature increase, the position of the lens does not change;

[0205] ③: When the temperature increases beyond a certain value, the lens position decreases linearly with the increase of the temperature.

[0206] d. According to the three rules in "c", we can fit the test data into two straight lines, i.e. Figure 5The dashed line in the figure. When the camera is just turned on, we count the temperature x at this time, and put it into the linear equation y=k1*x+b1 to get the corresponding focusing position y; In this way, when the camera is used continuously, we can get the corresponding lens position at each temperature.

[0207] e. Select any point on the straight line y=k1*x+b1 as the reference point, such as the blue point (20, 725). Subtract the lens position obtained in "d" from the lens position of the reference point (725). Then we get the lens position (lenspos) value that needs to be compensated for temperature compensation relative to the reference point. The initial image distance and the target image distance can include the lenspos value.

[0208] The above is the basic measurement process and algorithm principle of the temperature compensation algorithm. Based on this, the focusing strategy of the time-lapse photography mode is improved in this scheme, and reference Figure 8 is an example of another image processing method, which includes:

[0209] A. When previewing in time-lapse photography mode (before clicking the video recording button), first perform an automatic focusing, and perform temperature compensation on the focusing result to obtain the focusing lenspos result at the reference temperature.

[0210] B. Use the imaging formula to convert the lenspos result with temperature compensation into the distance result, and report it to the upper APP end.

[0211] C. When starting video recording, the APP end changes the focus mode to manual focusing mode. And the received distance is issued to the bottom layer as the target of manual focusing.

[0212] D. The bottom layer again uses the imaging formula to convert the distance issued by the APP into lenspos, and adds temperature compensation data to get the lenspos result at the current temperature, and completes the focusing. Since it is a manual focusing mode, the automatic focusing algorithm will not trigger the focusing automatically, which avoids the problem of unstable picture caused by frequent triggering of focusing. At the same time, due to the existence of temperature compensation, the problem of defocusing caused by long-time recording temperature is also solved.

[0213] In one embodiment, it also includes:

[0214] E. Due to inaccurate DAC value burning on some devices, when previewing autofocus on a distant scene, a lenspos result exceeding the hyperfocal distance may be obtained. In this case, regardless of the focusing position (e.g., as long as it exceeds the hyperfocal distance lenspos), after mapping by the imaging formula, a uniform distance will be obtained (e.g., the imaging formula is equivalent to a linear mapping between two points, and the hyperfocal distance is equivalent to an endpoint of the linear mapping; therefore, when it exceeds this endpoint, it cannot be mapped to the correct result and can only be mapped to the endpoint). When the recording app sends this distance to the underlying layer and performs mapping (or inverse mapping) again, it can only obtain the lenspos at the hyperfocal distance position. This lenspos is inconsistent with the lenspos obtained from preview autofocus, thus resulting in out-of-focus.

[0215] To solve this problem, refer to Figure 8 The following solution is adopted:

[0216] Add temperature compensation to the lenpos and distance of the most recent 5 frames to obtain the result at the reference temperature, and record it.

[0217] When the distance sent by the app is a hyperfocal distance, and the distance in the 5 recorded frames also contains a hyperfocal distance, the lenspos is not obtained by mapping using the imaging formula; instead, the recorded lenspos is used directly. Finally, temperature compensation is added to this lenspos result to obtain the lenspos result at the current temperature, thus completing the focusing process. This solves the problem without affecting the manual focus mode.

[0218] This solution can resolve the defocusing issue caused by heat generation during long-term video recording without affecting the stability of the time-lapse footage, and can also cover machines with inaccurate recording.

[0219] It should be noted that the terms "first" and "second" in the embodiments of this disclosure are for ease of description and distinction only, and have no other specific meaning.

[0220] Figure 9 This is a block diagram illustrating a terminal device according to an exemplary embodiment. For example, the terminal device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0221] Reference Figure 9The terminal device can include one or more of the following components: a processing component 902, a memory 904, a power component 906, a multimedia component 908, an audio component 908, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0222] The processing component 902 usually controls overall operations of the terminal device, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 902 can include one or more processors 920 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 902 can include one or more modules to facilitate interaction between the processing component 902 and other components. For example, the processing component 902 can include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0223] The memory 904 is configured to store various types of data to support operations of the terminal device. Examples of these data include instructions for any application or method operating on the terminal device, contact data, phonebook data, messages, pictures, videos, and the like. The memory 904 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0224] The power component 906 provides power to various components of the terminal device. The power component 906 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal device.

[0225] The multimedia component 908 includes a screen providing an output interface between the terminal device and the user. In some embodiments, the screen can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide and gestures on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the terminal device is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have a focal length and optical zoom capability.

[0226] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC) that is configured to receive external audio signals when the terminal device is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0227] The I / O interface 912 provides an interface between the processing component 902 and peripheral interface modules, which can be a keyboard, a click wheel, a button, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0228] The sensor component 914 includes one or more sensors to provide various aspects of status assessment for the terminal device. For example, the sensor component 914 can detect an open / closed status of the terminal device, relative positioning of components, such as a display and keypad of the terminal device, a change in position of the terminal device or a component of the terminal device, presence or absence of user contact with the terminal device, terminal device orientation or acceleration / deceleration, and temperature changes of the terminal device. The sensor component 914 can include a proximity sensor configured to detect presence of a nearby object without any physical contact. The sensor component 914 can also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor component 914 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0229] The communication component 916 is configured to facilitate wired or wireless communication between the terminal device and another device. The terminal device can access a wireless network based on a communication standard, such as Wi-Fi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast managing system via the broadcast channel. In an example embodiment, the communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0230] In exemplary embodiments, the terminal device can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for executing the above-described methods.

[0231] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions, such as a memory 904 including executable instructions or a computer program, is also provided, which can be executed by a processor 920 of an apparatus 900 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0232] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform any of the image processing methods according to the embodiments of the present disclosure.

[0233] The embodiments of the present disclosure provide a computer program product, which includes a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium, and the processor executes the computer program or executable instructions, so that the computer device performs any of the image processing methods according to the embodiments of the present disclosure.

[0234] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following, in general, the principles of the present disclosure and including such

[0235] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Perform initial focusing based on the identified focus target; Obtain the initial image distance and initial temperature; wherein, the initial image distance is the distance from the lens assembly to the image sensor after the initial focusing is completed at the initial temperature, and the initial temperature is the temperature at the time of the initial focusing; Based on the current temperature, the initial image distance, and the initial temperature, perform at least one real-time focusing on the target; Imaging is performed after at least one real-time focusing.

2. The method according to claim 1, characterized in that, The step of performing at least one real-time focusing on the target based on the current temperature, the initial image distance, and the initial temperature includes: Determine the temperature change between the current temperature and the initial temperature; Determine the image distance difference corresponding to the temperature change; The target image distance for the current temperature is determined based on the image distance difference and the initial image distance. The at least one real-time focusing is performed on the target object based on the target image distance.

3. The method according to claim 2, characterized in that, The process of obtaining the initial image distance includes: The initial image distance is determined based on a preset mapping relationship and the initial temperature; wherein the preset mapping relationship is the relationship between the distance from the lens assembly to the image sensor and the temperature, assuming the focus target remains unchanged.

4. The method according to claim 3, characterized in that, Determining the image distance difference corresponding to the temperature change includes: Based on the preset mapping relationship, a first reference image distance and a second reference image distance are determined; wherein, the first reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the initial temperature; and the second reference image distance is the distance from the lens assembly to the image sensor after focusing on the reference target at the current temperature. The image distance difference is determined based on the first reference image distance and the second reference image distance.

5. The method according to claim 3 or 4, characterized in that, The preset mapping relationship includes: Linear relationship; Non-linear relationship.

6. The method according to claim 2, characterized in that, The target image distance includes first image distance information; Determining the target image distance at the current temperature includes: Convert the first image distance information into the second image distance information; The second image distance information is converted into the first image distance information; wherein, within a preset image distance range, there is a one-to-one first mapping relationship between the first image distance information and the second image distance information, and outside the preset image distance range, there is a many-to-one second mapping relationship between the first image distance information and the second image distance information. Determine the first image distance information corresponding to each frame in the m-th to n-th frames when acquiring the images; If the first image distance information contains a third image distance information that exceeds the preset image distance range, the target image distance when acquiring the i-th frame image at the current temperature is determined based on the first image distance information.

7. The method according to claim 1, characterized in that, The initial focusing state based on the target is the preview state.

8. The method according to claim 1, characterized in that, The method further includes: Detect the focusing operation, and determine the focusing target based on the focusing operation; or, The focus target is determined based on preset focus parameters.

9. The method according to claim 1, characterized in that, The imaging mode includes at least one of the following: Time-lapse shooting mode; Movie mode; Slow motion mode; Live mode; Video mode; Photo mode; Portrait mode; Preview mode.

10. An image processing apparatus, characterized in that, include: The first focusing module is used to perform initial focusing based on the determined focusing target; An acquisition module is used to acquire the initial image distance and the initial temperature; wherein, the initial image distance is the distance from the lens assembly to the image sensor after the initial focusing is completed, and the initial temperature is the temperature at the time of the initial focusing. The second focusing module is used to perform at least one real-time focusing on the focusing target based on the current temperature, the initial image distance, and the initial temperature. An imaging module for imaging after the at least one real-time focusing.

11. A terminal device, characterized in that, include: A processor and a memory for storing computer programs or executable instructions capable of running on the processor, wherein: When the processor is used to run the computer program or the executable instructions, the executable instructions perform the method described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program or computer-executable instructions, which, when executed by a processor, implement the method described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program or executable instructions, characterized in that, When the computer program or executable instructions are executed by a processor, they implement the method of any one of claims 1 to 9.

Images