Image processing method and device, terminal, storage medium and computer program product

Abstract

The invention relates to an image processing method and device, terminal equipment, a storage medium and a computer program product. The image processing method comprises the steps that the total frame number of a first image to be collected is determined; wherein the total frame number is at least related to the memory of the terminal device and / or the exposure information of the first image; collecting the first image according to the total frame number; and generating a second image according to the collected first image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, terminal device, storage medium, and computer program product. Background Art

[0002] With the development of imaging technologies, more and more electronic devices have imaging functions. Electronic devices with imaging functions usually have an image acquisition module, through which image acquisition can be achieved, and video recording, etc. can also be realized. Summary of the Invention

[0003] The present disclosure provides an image processing method, apparatus, terminal device, and storage medium.

[0004] In a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including: determining the total number of frames of a first image to be acquired; where the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image; acquiring the first image according to the total number of frames; and generating a second image based on the acquired first image.

[0005] In one embodiment, the determining the total number of frames of the first image to be acquired includes: detecting an image acquisition operation; acquiring a first frame of the first image according to the image acquisition operation; and determining the total number of frames according to the exposure information of the first frame of the first image.

[0006] In one embodiment, the exposure information includes an exposure time; and the determining the total number of frames according to the exposure information of the first frame of the first image includes: determining the total number of frames according to the exposure time.

[0007] In one embodiment, the determining the total number of frames according to the exposure information includes: if the exposure time is less than a first reference exposure time, determining the first reference exposure time as the exposure time; if the exposure time is greater than a second reference exposure time, determining the second reference exposure time as the exposure time; where the second reference exposure time is greater than the first reference exposure time; determining the total number of frames according to a third reference exposure time and the exposure time; where the third reference exposure time is greater than the exposure time; and the total number of frames is greater than or equal to 2.

[0008] In one embodiment, the acquiring the first image according to the total number of frames includes: acquiring a second frame of the first image to the last frame of the first image according to the exposure time; where the sum of the number of frames from the first frame of the first image to the last frame of the first image is equal to the total number of frames.

[0009] In one embodiment, determining the total number of frames of the first image to be captured includes: determining the current remaining memory of the terminal device; determining the total number of frames according to the current remaining memory; wherein, the size of the current remaining memory is positively correlated with the total number of frames.

[0010] In one embodiment, determining the total number of frames of the first image to be captured includes: determining a first reference number of frames according to the memory; determining a second reference number of frames according to the exposure information; and taking the minimum value of the first reference number of frames and the second reference number of frames as the total number of frames.

[0011] In one embodiment, before determining the total number of frames of the first image to be captured, the method further includes: determining that the image capture mode is a preset mode; wherein, in the preset mode, the magnification used for capturing the first image is greater than a preset magnification.

[0012] In a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including: a determining module, configured to determine the total number of frames of the first image to be captured; wherein, the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image; a capturing module, configured to capture the first image according to the total number of frames; and a generating module, configured to generate a second image according to the captured first image.

[0013] In a third aspect of the embodiments of the present disclosure, there is provided a terminal device, including:

[0014] a processor and a memory for storing executable instructions that can run on the processor, wherein: when the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0015] In a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0016] In a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program or executable instructions, and when the computer program or executable instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0017] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0018] The solution of the embodiment of the present disclosure determines the total number of frames of the first image to be captured according to the memory of the terminal device and / or the exposure information of the first image, etc., so that the total number of frames can be dynamically determined according to the memory and / or the exposure information of the first image. For example, the total number of frames determined when the remaining memory is less is less than the total number of frames determined when the remaining memory is more, and the total number of frames determined when the exposure time is longer is less than the total number of frames determined when the exposure time is shorter.

[0019] When the total number of frames is small, the memory occupancy can be reduced, thereby reducing the memory overhead. In addition, after the total number of frames is reduced, in the process of generating the second image from these first images with the total number of frames, the processing process of the first image is also reduced, the data processing amount of the first image is reduced, the processing time is reduced, the speed of generating the second image is increased, thereby improving the efficiency of generating the second image, and making the user experience better.

[0020] The second image can also be generated from the first image, thereby improving the image quality of the second image, or when the total number of frames is large, the second image is generated from more frames of the first image, thereby also improving the image quality of the second image.

[0021] The solution of this embodiment takes into account the image quality of the second image, the efficiency of generating the second image, and the performance of the device, and comprehensively improves the user experience.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0024] Figure 1 is a schematic diagram of an image processing method shown according to an exemplary embodiment;

[0025] Figure 2 is a schematic diagram of determining the total number of frames shown according to an exemplary embodiment;

[0026] Figure 3 is a schematic diagram of an image acquisition interface shown according to an exemplary embodiment;

[0027] Figure 4 is a schematic diagram of determining the total number of frames shown according to an exemplary embodiment;

[0028] Figure 5 is a schematic diagram of determining the total number of frames shown according to an exemplary embodiment;

[0029] Figure 6 It is a schematic diagram showing the determination of the total number of frames according to an exemplary embodiment;

[0030] Figure 7 It is a schematic diagram of an image processing apparatus according to an exemplary embodiment;

[0031] Figure 8 It is a schematic diagram of another image processing method according to an exemplary embodiment;

[0032] Figure 9 It is a block diagram of a terminal device according to an exemplary embodiment. Detailed implementation manners

[0033] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the present disclosure. On the contrary, they are merely examples of apparatuses consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] Reference Figure 1 , is a schematic diagram of an image processing method, and the image processing method includes:

[0035] S100: Determine the total number of frames of the first image to be collected; wherein, the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image.

[0036] S200: Collect the first image according to the total number of frames.

[0037] S300: Generate a second image according to the collected first image.

[0038] This method can be at least executed in a terminal device having an image collection and preview function. The terminal device may include a mobile terminal device and a fixed terminal device, etc., that is, the execution subject of this method may at least include a mobile terminal device and a fixed terminal device. The mobile terminal device may include a mobile phone, a tablet computer, an in-vehicle central control device, a wearable device, a smart device, and an aircraft, etc., and the smart device may further include a smart office device, a smart home device, and a robot, etc.

[0039] The terminal device may include an image acquisition module, which may include one or more image acquisition units. The image acquisition units may be cameras, and the models and parameters of different image acquisition units may be different. For example, the image acquisition unit may include at least one of the following: a wide-angle camera, a telephoto camera, an ultra-wide-angle camera, a black-and-white camera, a macro camera, a depth camera (Time-of-Flight, ToF), or other cameras with different parameters.

[0040] The terminal device has a touch display screen, and various operations acting on the touch display screen can be detected through the touch display screen. For example, an image acquisition operation acting on an image acquisition control in the image acquisition interface.

[0041] The terminal device is installed with a camera application. Through the camera application, preview and image acquisition can be performed. The camera application has a preview interface, and the preview interface can be displayed after the camera application is started. Through the preview interface, a preview image of the image to be acquired can be displayed, and the image displayed in the preview interface is the preview image.

[0042] When acquiring an image, the image can be acquired according to the image acquisition operation. For example, after entering the camera application, there is an image acquisition control on the interface displayed by the camera application. When performing image acquisition, the image can be acquired through this image acquisition control.

[0043] The first image is the image to be acquired during the image acquisition process. This first image is used to generate the image stored in the album, and the image stored in the album is the final image to be generated in this solution, that is, the second image. By acquiring the first image according to the total number of frames, and then generating the second image based on the acquired first images of the total number of frames, the image saved to the album after image acquisition through the camera application can be obtained in this way.

[0044] Exemplarily, the first image may be acquired according to the image sampling frame rate of the image acquisition module, including the image acquired according to the sampling frame rate of the image sensor.

[0045] Exemplarily, the format of the first image may be the format of the raw image data. The image information in this format includes the original record of the level of the light signal converted into an electrical signal by the image acquisition sensor, the image data without any processing, and the digital processing of the electrical signal obtained by the photosensitive element, such as the RAW format. The RAW format image records the original information acquired by the image acquisition sensor and also records some metadata generated when the image acquisition sensor acquires the first image. These metadata may include information such as the sensitivity, shutter speed (which can be recorded as the exposure time), aperture value, and white balance.

[0046] During the process of image acquisition, generating a second image by collecting multiple frames of the first image can improve the image quality of the second image. For example, when performing image acquisition on the object to be collected at a magnification exceeding the preset magnification, the object to be collected will be magnified, and the image quality will decrease. By synthesizing multiple frames of the first image into the second image, the image quality of the second image can be improved.

[0047] There can be various ways to determine the total number of frames of the first image to be collected. This total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image. Any method of determining the total number of frames of the first image according to the memory of the terminal device and / or the exposure information of the first image is within the protection scope of this embodiment.

[0048] For example, the total number of frames can be determined according to the current usage status of the memory of the device. The total number of frames determined when the current remaining memory is the first remaining memory is greater than the total number of frames determined when the current remaining memory is the second remaining memory, and the first remaining memory is greater than the second remaining memory.

[0049] During the image acquisition process, the collected first images will be stored in the memory, which is convenient for generating the second image based on the first images subsequently and improving the efficiency of generating the second image. When the number of frames of the first image is relatively large, more memory resources will be occupied. The more frames the first image has, the more memory it occupies, and the fewer frames the first image has, the less memory it occupies.

[0050] Since other applications also need to run during the use of the terminal device, and the operation of the applications also requires memory, and the data required for running the applications is stored in the memory, which is convenient for the operation of the applications. In this case, since the memory is limited, the total number of frames of the first image is dynamically adjusted according to the usage situation of the memory, so as to balance the number of frames of the first image collected and the memory requirements of other applications. When there is more remaining memory, the total number of frames of the first image is increased, so that more frames of the first image can be collected with less impact on the memory used by other applications, thus facilitating the generation of a second image with higher image quality. When there is less remaining memory, the total number of frames of the first image is decreased, which can reduce the memory occupied by the first image, leave more memory for other applications, facilitate the operation of other applications, and reduce the impact on the memory used by other applications due to too many frames of the first image.

[0051] The total number of frames of the first image can also be determined according to the exposure information of the first image. The exposure information can affect the resolution of the first image, thereby affecting the clarity of the first image. Exemplarily, the total number of frames of the first image determined according to the first exposure information is greater than the total number of frames of the first image determined according to the second exposure information, and the clarity of the first image collected according to the second exposure information is higher than that of the first image collected according to the second exposure information.

[0052] The exposure information can include various types, such as the exposure time, and can also include the exposure amount. The exposure amount is related to information such as the exposure time, aperture size, and sensitivity that are related to the exposure information. The method for determining the total number of frames of the first image according to the exposure information can refer to the subsequent embodiments. The exposure time can indicate the opening time of the shutter.

[0053] Exemplarily, the total number of frames of the first image is greater than or equal to 2.

[0054] After determining the total number of frames of the first image, the first image can be collected according to this total number of frames. After collecting the first image with the total number of frames, a second image is generated according to the first image with the collected total number of frames. For example, if the total number of frames is 5, then 5 frames of the first image are collected, and a second image is generated according to these 5 frames of the first image collected.

[0055] The total number of frames of the first image determined by different exposure information and / or different usage states of the memory may be different. In this way, it is possible to dynamically adjust the total number of frames of the first image according to the exposure information and / or different usage states of the memory each time a photo is taken, and then generate the final second image according to the first image with the number of frames.

[0056] In this embodiment, by determining the total number of frames of the first image to be collected according to the memory of the terminal device and / or the exposure information of the first image, etc., the total number of frames can be dynamically determined according to the memory and / or the exposure information of the first image. For example, the total number of frames determined when the remaining memory is less is less than the total number of frames determined when the remaining memory is more, and the total number of frames determined when the exposure time is longer is less than the total number of frames determined when the exposure time is shorter.

[0057] When the total number of frames is small, the occupation of memory can be reduced, thereby reducing the memory overhead. In addition, after the total number of frames is reduced, in the process of generating the second image according to the first image with these total number of frames, the processing process of the first image is also reduced, the data processing amount of the first image is reduced, the processing time is reduced, the speed of generating the second image is increased, thereby improving the efficiency of generating the second image, improving the smoothness of the camera from collecting the first image to generating the second image, and making the user experience better.

[0058] The second image can also be generated from the first image to improve the image quality of the second image. Or when the total number of frames is large, the second image can be generated from more frames of the first image, thereby also improving the image quality of the second image.

[0059] The solution of this embodiment takes into account the image quality of the second image, the efficiency of generating the second image, and the performance of the device, comprehensively improving the user experience.

[0060] In one embodiment, refer to Figure 2 , which is a schematic diagram for determining the total number of frames. S100, determining the total number of frames of the first image to be collected, includes:

[0061] S101, detecting an image acquisition operation.

[0062] S102, acquiring the first frame of the first image according to the image acquisition operation.

[0063] S103, determining the total number of frames according to the exposure information of the first frame of the first image.

[0064] This embodiment is an example of determining the total number of frames of the first image according to the exposure information of the first image.

[0065] An image acquisition operation can be detected. When an image acquisition operation is detected, it indicates that image acquisition is required, and the first image is acquired according to the detected image acquisition operation. The first image is acquired frame by frame. After the first frame of the image is acquired, the exposure information of the first frame of the first image can be determined. When acquiring an image, each frame of the image has exposure information, and the device terminal can obtain the exposure information of each frame of the image.

[0066] Exemplarily, the image acquisition operation can be a user input operation on an image acquisition control. The image acquisition control can be an image acquisition control on the interface in a camera application, or a preset quick button for image acquisition on the terminal device. The image acquisition operation can also be a voice input operation detected by the terminal device for controlling the terminal device to perform image acquisition; it can also be other preset operations for image acquisition, such as air gestures and click or tap operations on the terminal device. The air gesture can be a preset gesture operation acquired by the image acquisition module.

[0067] Refer to Figure 3 , which is a schematic diagram of an image acquisition interface. This image acquisition interface is the image acquisition interface in a camera application, and this interface can be displayed after entering the camera application. Figure 3 The white circular part in the middle and bottom can be the image acquisition control.

[0068] Exemplarily, the exposure information of the first image of the first frame can be automatically adjusted by the terminal device or determined according to the user's exposure information adjustment operation.

[0069] After obtaining the exposure information of the first image of the first frame, determine the total number of frames of the first image according to the exposure information. The exposure information may include the exposure amount, and the exposure amount is related to information such as exposure time, sensitivity, and aperture related to the exposure information. These information will affect the clarity, color, brightness and darkness of the first image, thereby affecting the image quality of the second image. Since the exposure information of the first image of the first frame has been determined, under this exposure information, the clarity of the first image of the first frame can be determined, and the total number of frames of the first image can be determined according to the clarity.

[0070] Exemplarily, there is a first preset association relationship between the exposure information and the clarity. The first preset association relationship can be obtained. According to the exposure information of the first image of the first frame and the first preset association relationship, the clarity of the first image can be determined. There can also be a second preset association relationship between the clarity and the total number of frames. Through the second preset association relationship, the total number of frames of the first image can be determined.

[0071] Exemplarily, the clarity is negatively correlated with the total number of frames of the first image.

[0072] When collecting the first image of the first frame according to different image capture operations, the exposure information of the first image of the first frame may be different. Therefore, the total number of frames of the first image can be adaptively determined according to the exposure information of the first image of the first frame.

[0073] In one embodiment, the exposure information includes the exposure amount.

[0074] S103. Determining the total number of frames according to the exposure information of the first image of the first frame includes:

[0075] Determine the total number of frames according to the exposure amount of the first image of the first frame. The exposure amount is negatively correlated with the total number of frames. The higher the exposure amount, the more light enters the photosensor device, and the easier it is to collect more information. When the amount of information contained in the first image is larger, the second image can be generated according to fewer first images, thereby reducing the number of frames of the first image collected, reducing the total time required to collect the first image, and also reducing the time and computing resources required to generate the second image from the first image, and also reducing the occupied memory, thereby improving the user experience.

[0076] The lower the exposure amount, the less light enters the photosensor device, and the less information may be collected in each frame of the first image. After the amount of information contained in the first image decreases, more first images are required to generate the second image to ensure the image quality of the second image. In this case, more frames of the first image need to be determined to improve the image quality of the second image, thereby enhancing the user experience.

[0077] Through this embodiment, the effect of dynamically determining the total number of frames of the first image adaptively according to the exposure amount of the first frame of the first image is achieved.

[0078] In one embodiment, referring to Figure 4 , which is a schematic diagram for determining the total number of frames. Determining the total number of frames according to the exposure amount includes:

[0079] S10. When the exposure amount is less than the first reference exposure amount, the first reference exposure amount is determined as the exposure amount.

[0080] S20. When the exposure amount is greater than the second reference exposure amount, the second reference exposure amount is determined as the exposure amount; wherein, the second reference exposure amount is greater than the first reference exposure amount.

[0081] S30. Determine the total number of frames according to the third reference exposure amount and the exposure amount; wherein, the third reference exposure amount is greater than the exposure amount; the total number of frames is greater than or equal to 2.

[0082] There is no sequence relationship between S10 and S20. It is possible to determine whether to execute S10 or S20 according to the magnitude relationship between the exposure amount and the first reference exposure amount and the magnitude relationship between the exposure amount and the second reference exposure amount. S10 and S20 will not be executed simultaneously. In this solution, S10 and S30 can be executed, or S20 and S30 can be executed.

[0083] The first reference exposure amount and the second reference exposure amount are reference information for determining the total number of frames of the first image. The first reference exposure amount and the second reference exposure amount are preset and can be adjusted according to actual usage requirements.

[0084] Exemplarily, the first reference exposure amount is the minimum exposure amount, and the second reference exposure amount is the maximum exposure amount.

[0085] When the exposure amount of the first frame of the first image is less than the first reference exposure amount, the first reference exposure amount is determined as the exposure amount. When the exposure amount of the first frame of the first image is greater than the second reference exposure amount, the second reference exposure amount is determined as the exposure amount. This can reduce the influence of insufficient or overexposed exposure amount of the first frame of the first image on determining the total number of frames, thereby improving the accuracy of determining the total number of frames.

[0086] When the exposure amount of the first image in the first frame is greater than the first reference exposure amount and less than the second reference exposure amount, the exposure amount of the first image in the first frame is not adjusted.

[0087] The third reference exposure amount is used to determine the total number of frames of the first image together with the exposure amount of the first image in the first frame. The third reference exposure amount can also be determined according to actual usage requirements. The method of determining the total number of frames of the first image based on the third reference exposure amount and the exposure amount of the first image in the first frame is acceptable without limitation.

[0088] For example, the total number of frames of the first image can be determined according to the difference between the third reference exposure amount and the exposure amount of the first image in the first frame; the total number of frames of the first image can also be determined according to the ratio of the third reference exposure amount to the exposure amount of the first image in the first frame; the total number of frames of the first image can also be determined according to the sum of the third reference exposure amount and the exposure amount of the first image in the first frame; the total number of frames of the first image can also be determined according to the product of the third reference exposure amount and the exposure amount of the first image in the first frame.

[0089] The total number of frames of the first image determined according to the third reference exposure amount and the exposure amount of the first image in the first frame is at least 2 frames, which is convenient for synthesizing the second image and thus improves the image quality of the second image.

[0090] In one embodiment, S200, acquiring the first image according to the total number of frames includes:

[0091] Acquiring the second frame of the first image to the last frame of the first image according to the exposure amount; wherein, the sum of the number of frames from the first frame of the first image to the last frame of the first image is equal to the total number of frames.

[0092] After determining the exposure amount of the first image in the first frame and the total number of frames of the first image, the remaining frames of the first image can be acquired according to the exposure amount of the first image in the first frame. For example, if the total number of frames is 5 frames, the remaining 4 frames of the first image are acquired according to the exposure amount of the first image in the first frame, and the exposure amount of each frame of the first image is the same. This can reduce the color difference between each frame of the first image caused by the difference in exposure amount between each frame of the first image, which is more conducive to synthesizing the second image and thus improves the image quality of the second image.

[0093] In one embodiment, referring to Figure 5 , which is a schematic diagram for determining the total number of frames. Determining the total number of frames according to the exposure information includes:

[0094] S40, if the exposure time is less than the first reference exposure time, determine the first reference exposure time as the exposure time.

[0095] S50. If the exposure time is greater than the second reference exposure time, determine the second reference exposure time as the exposure time, where the second reference exposure time is greater than the first reference exposure time.

[0096] S60. Determine the total number of frames according to the third reference exposure time and the exposure time. Where the third reference exposure time is greater than the exposure time; the total number of frames is greater than or equal to 2.

[0097] This embodiment is an example of determining the total number of frames according to the exposure time.

[0098] There is no sequential relationship between S40 and S50. It is possible to determine whether to execute S40 or S50 according to the magnitude relationship between the exposure amount and the first reference exposure amount and the magnitude relationship between the exposure amount and the second reference exposure amount. S40 and S50 will not be executed simultaneously. In this solution, it is possible to execute S40 and S60, or execute S50 and S60.

[0099] The first reference exposure time and the second reference exposure time are reference information for determining the total number of frames of the first image. The first reference exposure time and the second reference exposure time are preset and can be adjusted according to actual usage requirements.

[0100] Exemplarily, the first reference exposure time is the minimum exposure time, and the second reference exposure time is the maximum exposure time.

[0101] The exposure amount is related to the exposure time and the aperture size. When the aperture size remains unchanged, the exposure time plays a more important role in the exposure information.

[0102] When the exposure time of the first frame of the first image is less than the first reference exposure time, determine the first reference exposure time as the exposure time. When the exposure time of the first frame of the first image is greater than the second reference exposure time, determine the second reference exposure time as the exposure time. This can reduce the impact of insufficient exposure time or overexposure of the first frame of the first image on determining the total number of frames, thereby improving the accuracy of determining the total number of frames.

[0103] When the exposure time of the first frame of the first image is greater than the first reference exposure time and less than the second reference exposure time, do not adjust the exposure time of the first frame of the first image.

[0104] The third reference exposure time is used to determine the total number of frames of the first image together with the exposure time of the first frame of the first image. The third reference exposure time can also be determined according to actual usage requirements. The method of determining the total number of frames of the first image according to the third reference exposure time and the exposure time of the first frame of the first image is fine and is not limited.

[0105] For example, the total number of frames of the first image can be determined based on the difference between the third reference exposure time and the exposure time of the first frame of the first image; the total number of frames of the first image can also be determined based on the ratio of the third reference exposure time to the exposure time of the first frame of the first image; the total number of frames of the first image can also be determined based on the sum of the third reference exposure time and the exposure time of the first frame of the first image; the total number of frames of the first image can also be determined based on the product of the third reference exposure time and the exposure time of the first frame of the first image.

[0106] The total number of frames of the first image determined according to the third reference exposure time and the exposure time of the first frame of the first image is at least 2 frames, which can facilitate the synthesis of the second image and thus improve the image quality of the second image.

[0107] For example, the first reference exposure time is 33,000,000 microseconds, the second reference exposure time is 70,000,000 microseconds, and the third reference exposure time is 240,000,000 microseconds. If the exposure time of the first frame of the first image is 30,000,000 microseconds, the exposure time of the first frame of the first image is adjusted to 33,000,000 microseconds. If the exposure time of the first frame of the first image is 130,000,000 microseconds, the exposure time of the first frame of the first image is adjusted to 70,000,000 microseconds. If the exposure time of the first frame of the first image is 50,000,000 microseconds, the exposure time of the first frame of the first image is not adjusted.

[0108] The total number of frames can be determined according to the following formula:

[0109] Sum=240000000 / CurrentExposureTime-1

[0110] Where Sum represents the total number of frames, and CurrentExposureTime represents the exposure time of the first image of the first frame.

[0111] When determining the total number of frames according to the above formula, Sum is an integer. For example, if Sum is greater than or equal to 8, Sum is determined to be 8; if Sum is equal to or greater than 5, Sum is determined to be 5; and if Sum is greater than or equal to 3, Sum is determined to be 3.

[0112] Exemplarily, the exposure amount of the first frame of the first image is at least related to the exposure time and the aperture size. When the aperture size remains unchanged, S40 to S60 are performed.

[0113] In one embodiment, S200, capturing a first image according to a total number of frames includes:

[0114] The second frame first image to the last frame first image are collected according to the exposure time; wherein the sum of the number of frames from the first frame first image to the last frame first image is equal to the total number of frames.

[0115] After determining the exposure time of the first image in the first frame and the total number of frames of the first image, the first images of the remaining frames can be acquired according to the exposure time of the first image in the first frame. For example, if the total number of frames is 5, then the remaining 4 first images are acquired according to the exposure time of the first image in the first frame, and the exposure time of each first image is the same. This can reduce the color difference between the first images of each frame caused by the difference in exposure time between the first images of each frame, which is more conducive to synthesizing the second image, thereby improving the image quality of the second image.

[0116] In one embodiment, referring to Figure 6 , which is a schematic diagram for determining the total number of frames, S100, determining the total number of frames of the first image to be acquired, includes:

[0117] S104, determining the current remaining memory of the terminal device.

[0118] S105, determining the total number of frames according to the current remaining memory; wherein, the size of the current remaining memory is positively correlated with the total number of frames.

[0119] During the image acquisition process, the acquired first images are stored in the memory, which is convenient for generating the second image according to the first images subsequently and improves the efficiency of generating the second image. When the number of frames of the first image is relatively large, it will occupy more memory resources. The more frames of the first image, the more memory it occupies, and the fewer frames of the first image, the less memory it occupies. Since other applications also need to run during the use of the terminal device, the operation of the applications also requires memory, and the data required for running the applications is stored in the memory, which is convenient for the operation of the applications.

[0120] In this case, since the memory is limited, the total number of frames of the first image is dynamically adjusted by determining the current remaining memory, which can balance the number of frames of the first image acquired and the memory requirements of other applications. The more the current remaining memory, the larger the total number of frames of the first image. In this way, more frames of the first image can be acquired while reducing the impact on the memory used by other applications, thus facilitating the generation of a second image with higher image quality. The less the current remaining memory, the fewer the total number of frames of the first image. This can reduce the memory occupied by the first image, leave more memory for other applications, facilitate the operation of other applications, and reduce the impact on the memory used by other applications due to too many frames of the first image.

[0121] The total number of frames determined in this case is greater than or equal to 2.

[0122] In one embodiment, the total number of frames can also be less than 8.

[0123] In one embodiment, S100, determining the total number of frames of the first image to be captured, includes:

[0124] Determining a first reference number of frames according to the memory, determining a second reference number of frames according to the exposure information, and taking the minimum value of the first reference number of frames and the second reference number of frames as the total number of frames.

[0125] In this embodiment, an example of determining the total number of frames by combining the memory and the exposure information of the first image can determine the first reference number of frames according to the memory. For example, determine the first reference number of frames according to the current remaining memory. Determine the second reference number of frames according to the exposure information, and determine the smallest of the two numbers of frames as the total number of frames.

[0126] For the method of determining the first reference number of frames according to the current remaining memory, reference can be made to the content in the above embodiment of determining the total number of frames according to the memory. For the method of determining the second reference number of frames according to the exposure information, reference can be made to the content in the above embodiment of determining the total number of frames according to the exposure information.

[0127] This can reduce the number of frames of the first image, thereby reducing the occupied memory, leaving more content for other applications, and facilitating the use of other applications. It also reduces the amount of data processing when generating the second image from the first image, improves the efficiency of generating the second image, reduces the time for collecting the first image, makes the speed of obtaining the second image faster, and improves the user experience.

[0128] In one embodiment, before determining the total number of frames of the first image to be captured, the method further includes:

[0129] Determining that the image capture mode is a preset mode; wherein, in the preset mode, the magnification used for capturing the first image is greater than a preset magnification.

[0130] The preset mode can be a Super-Resolution (SR) mode. The enabling conditions of this mode may be different in different terminal devices, and it can enter the preset mode after the magnification reaches the preset magnification. For example, the preset magnification is 1.4X. Refer to Figure 3 , for a schematic diagram of previewing in the preset mode, Figure 3 As shown, the magnification is 2X, which is greater than the preset magnification of 1.4X.

[0131] In the preset mode, due to the relatively large magnification, when capturing an image of the target object, there may be a situation of blurred image or poor image quality. Through S100 to S300 and the corresponding further limited solutions, multiple frames of the first image are captured to generate the second image, improving the image quality of the second image, and thus improving the image quality of the finally captured image of the target object.

[0132] In one embodiment, refer toFigure 7 , is a schematic diagram of an image processing device, the device comprising:

[0133] Determination module 1, used to determine the total number of frames of the first image to be collected; wherein the total number of frames is at least related to the memory of the terminal device and / or the exposure time of the first image;

[0134] Acquisition module 2, used for acquiring a first image according to the total number of frames;

[0135] The generating module 3 is configured to generate a second image based on the collected first image.

[0136] In one embodiment, the determination module 1 includes:

[0137] A detection unit, configured to detect an image acquisition operation;

[0138] An acquisition unit, configured to acquire a first frame of a first image according to an image acquisition operation;

[0139] The first determining unit is configured to determine the total number of frames according to the exposure information of the first frame of the first image.

[0140] In one embodiment, the first determining unit includes:

[0141] A first determining subunit is configured to determine the first reference exposure time as the exposure time if the exposure time is less than the first reference exposure time when the aperture size remains unchanged;

[0142] a second determining subunit, configured to determine the second reference exposure time as the exposure time if the exposure time is greater than the second reference exposure time; wherein the second reference exposure time is greater than the first reference exposure time;

[0143] The third determining subunit is configured to determine the total number of frames according to a third reference exposure time and the exposure time; wherein the third reference exposure time is greater than the exposure time; and the total number of frames is greater than or equal to 2.

[0144] In one embodiment, the acquisition module 2 is used to acquire the second frame first image to the last frame first image according to the exposure time; wherein the sum of the frame numbers of the first frame first image to the last frame first image is equal to the total frame number.

[0145] In one embodiment, the determination module 1 includes:

[0146] A second determining unit, configured to determine the current remaining memory of the terminal device;

[0147] The third determining unit is configured to determine the total number of frames according to the current remaining memory; wherein the size of the current remaining memory is positively correlated with the total number of frames.

[0148] In one embodiment, the determination module 1 includes:

[0149] a fourth determining unit, configured to determine a first reference frame number according to the memory;

[0150] a fifth determining unit, configured to determine a second reference frame number according to the exposure information;

[0151] The sixth determining unit is configured to use the minimum value of the first reference frame number and the second reference frame number as the total frame number.

[0152] In one embodiment, the apparatus further comprises:

[0153] The preset mode determination module is used to determine that the image acquisition mode is a preset mode before determining the total number of frames of the first image to be acquired; wherein, in the preset mode, the magnification used to acquire the first image is greater than the preset magnification.

[0154] In one embodiment, reference Figure 8 , a schematic diagram of another image processing method.

[0155] Iterative updates to mobile phone hardware have resulted in qualitative improvements in more accessories. Both the increase in aperture and the improvement in telephoto lenses have led to rapid developments in imaging quality. Consequently, requirements for image framing and clarity under telephoto lenses are also increasing. The addition of new resolution algorithms and optimization of the current SR (super resolution) algorithm pathways are all aimed at continuously reducing performance to ensure that mobile imaging systems better meet user expectations.

[0156] To effectively improve the performance of super-resolution (SR) photography (for example, scenes with a telephoto lens at 1.4x zoom or above) and reduce memory consumption, the current SR photography frame settings are dynamically adjusted based on the exposure time. The number of frames captured is controlled in real time according to the exposure time and exposure amount. This variable number of frames reduces the frame capture time and the additional time and performance consumption caused by a large number of frames in the processing flow. Reducing the number of frames effectively reduces memory consumption, thus achieving dual optimization of performance and memory. This allows for more efficient SR photography without compromising image quality, enhancing the user's photo-taking experience.

[0157] Currently, most mobile camera photography, especially telephoto lenses, uses super-resolution (SR) algorithms to process captured images frame by frame. The number of input frames required by the algorithm is typically determined by coordination between the phone manufacturer and the algorithm manufacturer. This process takes into account multiple factors, such as hardware performance and the image quality requirements of the algorithm processing, and is a variable condition.

[0158] As the algorithm library is iteratively updated, the parameter information required inside the image increases, which also reduces the processing pressure of the algorithm relatively, and the places where dynamic improvement can be made gradually increase. Therefore, considering memory reduction and performance improvement, on the premise of ensuring the smoothness and clarity of SR photography, this solution dynamically improves the performance index based on the exposure time, which will be able to better improve the SR photography performance.

[0159] The method includes the following steps:

[0160] 1. Open the camera to trigger SR mode photography, and the camera app sends the current corresponding scene tag: SR. Generate an image acquisition request according to the image acquisition operation, including the photography request sent by the APP.

[0161] 2. Configure a new logical dynamic frame acquisition scheme before sending the meta related to the tag Tag to the Hal layer.

[0162] Add an exposure time dynamic frame selection algorithm logic to the original getSrCheckerSum function.

[0163] First, detect the current exposure time CurrentExposureTime (by default, all frames in SR mode use the exposure information of the first frame). If it is less than 33000000, set it to 33000000. If it is greater than 70000000, still set it to 70000000, that is, determine the range interval of the exposure time to ensure the normal operation of the algorithm.

[0164] Divide 240000000 by the processed CurrentExposureTime and subtract 1 from the obtained number.

[0165] Sum = 240000000 / CurrentExposureTime - 1.

[0166] Judge Sum, and select the actual number of frames according to its value. If Sum is greater than or equal to 8, determine the total number of frames as 8 frames. If Sum is greater than or equal to 5, determine the total number of frames as 5 frames. If Sum is greater than or equal to 3, determine the total number of frames as 3 frames. The establishment conditions of the three are mutually exclusive, and there are no interference options.

[0167] Pass the finally calculated Sum value into the newly added getMultiFrameNums function. Its function will be called during the process of sending to the farmework, and the corresponding Sum value will be transmitted to the data sent downward.

[0168] 3. The HAL layer creates a functional process in the SR mode based on the parameters carrying Sum data sent by the camera app, and notifies the sensor to complete the image output request of the Request according to this standard, and outputs the corresponding Sum-frame RAW data.

[0169] 4. Perform frame selection operations on this RAW format data. After completing the conversion from Bayer format to YUV format, input the Sum frame to the SR algorithm library for synthesis processing.

[0170] 5. The algorithm library in the SR mode transmits the synthesized frame of YUV data to the VT Camera for format conversion from YUV to JPEG, and generates the image seen by the user and saves it to the album.

[0171] In the key step 2 of this embodiment, by adding a dynamic frame selection algorithm logic, according to the current scene exposure time, it is determined how many frames of data need to be selected for algorithm processing in the current SR mode. In this way, it can take into account both the current lighting conditions and the image quality requirements, and can also better optimize performance and memory, making the photo-taking effect and speed better, and enabling users to better experience the technology brought by the imaging system.

[0172] The solution of this embodiment sets the number of frames through the dynamic frame selection algorithm, effectively utilizes the advantages of software and hardware to save memory and improve processing speed, and greatly improves the photo-taking performance in the SR mode, enabling it to complete multiple SR photo-takings in a shorter time.

[0173] Without affecting clarity and resolution, it gives users a better photo-taking experience, reduces the problem of longer photo-taking time caused by more photo-taking frames in the SR mode, and makes the performance better.

[0174] The solution of this embodiment mainly dynamically sets the number of SR photo-taking frames based on the ExposureTime parameter. By different numbers of frames taken under different exposure times, it reduces the redundant memory overhead, and by reducing the number of frames taken, it reduces the frame selection time and the per-frame processing time of the frames, effectively improving the performance parameters and meeting the high-performance requirements. On the premise of ensuring the current SR photo-taking resolution and clarity, it efficiently and quickly completes the photo-taking request, generates an imaging effect that is exactly the same as when the default total number of frames is 8, and stores it in the album for users to use.

[0175] It should be noted that the "first" and "second" in the embodiments of the present disclosure are only for convenience of expression and distinction, and have no other specific meanings.

[0176] Figure 9 It is a block diagram of a terminal device shown according to an exemplary embodiment. For example, the terminal device can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0177] Referring to Figure 9 , the terminal device may include one or more of the following components: a processing component 902, a memory 904, a power component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0178] The processing component 902 generally controls the overall operation of the terminal device, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0179] The memory 904 is configured to store various types of data to support the operation of the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0180] The power component 906 provides power to various components of the terminal device. The power component 906 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the terminal device.

[0181] The multimedia component 908 includes a screen that provides an output interface between the terminal device and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0182] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a Microphone (MIC) that is configured to receive external audio signals when the terminal device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 further includes a speaker for outputting audio signals.

[0183] The I / O interface 912 provides an interface between the processing component 902 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0184] The sensor assembly 914 includes one or more sensors for providing status assessments of various aspects for the terminal device. For example, the sensor assembly 914 can detect the on / off state of the terminal device, the relative positioning of components, such as the display and keypad of the terminal device, the sensor assembly 914 can also detect changes in the position of the terminal device or a component of the terminal device, the presence or absence of user contact with the terminal device, the orientation or acceleration / deceleration of the terminal device, and temperature changes of the terminal device. The sensor assembly 914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0185] The communication component 916 is configured to facilitate communication between the terminal device and other devices in a wired or wireless manner. The terminal device can access a wireless network based on communication standards, such as Wi-Fi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0186] In an exemplary embodiment, the terminal device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above method.

[0187] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided. For example, a memory 904 including executable instructions or a computer program, and the above instructions or computer program can be executed by a processor 920 of the device 900 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0188] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute any one of the above image processing methods in the embodiments of the present disclosure. For example, the method includes:

[0189] Determine the total number of frames of the first image to be captured; wherein the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image; capture the first image according to the total number of frames; generate a second image according to the captured first image.

[0190] The embodiments of the present disclosure provide a computer program product, which includes: a computer program or executable instructions, and the computer program or executable instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium, and the processor executes the computer program or executable instructions, so that the computer device executes any one of the above image processing methods in the embodiments of the present disclosure.

[0191] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0192] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, Including: Determine the total number of frames of the first image to be captured; wherein, the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image; Capture the first image according to the total number of frames; Generate a second image based on the captured first image.

2. The method according to claim 1, wherein The determining the total number of frames of the first image to be captured includes: Detect an image capture operation; Capture the first frame of the first image according to the image capture operation; Determine the total number of frames according to the exposure information of the first frame of the first image.

3. The method according to claim 2, wherein The exposure information includes the exposure time; The determining the total number of frames according to the exposure information of the first frame of the first image includes: Determine the total number of frames according to the exposure time.

4. The method according to claim 3, wherein The determining the total number of frames according to the exposure time includes: If the exposure time is less than the first reference exposure time, determine the first reference exposure time as the exposure time; If the exposure time is greater than the second reference exposure time, determine the second reference exposure time as the exposure time; wherein, the second reference exposure time is greater than the first reference exposure time; Determine the total number of frames according to the third reference exposure time and the exposure time; wherein, the third reference exposure time is greater than the exposure time; the total number of frames is greater than or equal to 2.

5. The method according to claim 3, characterized in that The capturing the first image according to the total number of frames includes: Capture the second frame to the last frame of the first image according to the exposure time; wherein, the sum of the number of frames from the first frame to the last frame of the first image is equal to the total number of frames.

6. The method according to claim 1, wherein The determining the total number of frames of the first image to be captured includes: Determine the current remaining memory of the terminal device; Determine the total number of frames according to the current remaining memory; wherein, the size of the current remaining memory is positively correlated with the total number of frames.

7. The method according to claim 1, characterized in that The determining the total number of frames of the first image to be captured includes: Determine a first reference number of frames according to the memory; Determine a second reference number of frames according to the exposure information; Take the minimum value of the first reference number of frames and the second reference number of frames as the total number of frames.

8. The method according to claim 1, characterized in that, Before the determining the total number of frames of the first image to be captured, the method further includes: Determine that the image capture mode is a preset mode; wherein, in the preset mode, the magnification used for capturing the first image is greater than a preset magnification.

9. An image processing apparatus, characterized in that, Including: A determining module, configured to determine the total number of frames of the first image to be captured; wherein, the total number of frames is at least related to the memory of the terminal device and / or the exposure information of the first image; A capturing module, configured to capture the first image according to the total number of frames; A generating module, configured to generate a second image based on the captured first image.

10. A terminal device, characterized in that, Including: A processor and a memory for storing a computer program or executable instructions that can run on the processor, wherein: When the processor is used to run the computer program or the executable instructions, the executable instructions execute the method according to any one of claims 1 to 8 above.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program or computer-executable instructions, and when the computer program or the computer-executable instructions are executed by a processor, the method described in any one of claims 1 to 8 above is implemented.

12. A computer program product, comprising a computer program or executable instructions, characterized in that, When the computer program or the executable instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images