Image processing method, image processing device and electronic equipment

Abstract

The invention discloses an image processing method, an image processing device and electronic equipment, and belongs to the technical field of data processing. The image processing method comprises: controlling a first image signal processor to acquire a first data stream through a first path, the first data stream comprising a normally exposed first image frame, and controlling the first image signal processor to acquire a second data stream through a second path, the second data stream comprising an abnormally exposed first image frame; controlling the preview interface to display a normally-exposed first image frame, wherein the normally-exposed first image frame comprises a non-overexposed image frame and a non-underexposed image frame; and determining a first image based on the normally exposed first image frame and the abnormally exposed first image frame, the first image being an image of high dynamic range imaging, the abnormally exposed first image frame comprising an overexposed image frame and / or an underexposed image frame.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to an image processing method, an image processing device and an electronic device. Background Art

[0002] When shooting in backlit scenes or in relatively dimly lit scenes, a High Dynamic Range Imaging (HDR) algorithm will be superimposed to ensure high dynamic range imaging.

[0003] Specifically, image frames with different exposures are captured, and a high dynamic range imaging algorithm for multi-frame synthesis is used to synthesize a film with a high dynamic range imaging effect.

[0004] The preview image displayed on the preview interface is an image frame with normal exposure. Therefore, after capturing image frames with different exposures, the image frames with abnormal exposure need to be eliminated. After eliminating the image frames with abnormal exposure, the preview image displayed on the preview interface will be stuck, affecting the shooting experience. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, an image processing device and an electronic device, which can solve the problem that the preview image displayed on the preview interface is stuck, affecting the shooting experience.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, wherein a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, and the image processing method includes: controlling the first image signal processor to obtain a first data stream through the first path, the first data stream including a first image frame with normal exposure, and controlling the first image signal processor to obtain a second data stream through the second path, the second data stream including a first image frame with abnormal exposure; controlling a preview interface to display a first image frame with normal exposure, the first image frame with normal exposure including an image frame that is neither overexposed nor underexposed; determining a first image based on the first image frame with normal exposure and the first image frame with abnormal exposure, the first image being an image with high dynamic range imaging, the first image frame with abnormal exposure including an overexposed image frame and / or an underexposed image frame.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, wherein a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, and the image processing device includes: a control module, used to control the first image signal processor to obtain a first data stream through the first path, the first data stream including a normally exposed first image frame, and to control the first image signal processor to obtain a second data stream through the second path, the second data stream including an abnormally exposed first image frame; a display module, used to control a preview interface to display a normally exposed first image frame, the normally exposed first image frame including an image frame that is neither overexposed nor underexposed; a determination module, used to determine a first image based on the normally exposed first image frame and the abnormally exposed first image frame, the first image being an image of high dynamic range imaging, the abnormally exposed first image frame including an overexposed image frame and / or an underexposed image frame.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method of the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0012] In an embodiment of the present application, an image processing method is proposed, wherein a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, so that the first image signal processor can obtain a first data stream and a second data stream from the camera sensor through the first path and the second path, respectively, wherein the first data stream includes a first image frame with normal exposure, and the second data stream includes a first image frame with abnormal exposure. After obtaining the first data stream and the second data stream, a preview interface is controlled to display the first image frame with normal exposure contained in the first data stream, and a first image is determined based on the first image frame with normal exposure and the first image frame with abnormal exposure to obtain an image with high dynamic range imaging. In this process, while obtaining an image with high dynamic range imaging, the jamming of the preview image displayed on the preview interface can be improved, thereby improving the shooting experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram showing a flow chart of an image processing method in an embodiment of the present application is shown;

[0014] Figure 2 One of the schematic diagrams showing the image processing process in the embodiment of the present application;

[0015] Figure 3 The second schematic diagram shows the image processing process in the embodiment of the present application;

[0016] Figure 4 A schematic block diagram of an image processing device in an embodiment of the present application is shown;

[0017] Figure 5 A schematic block diagram of an electronic device in an embodiment of the present application is shown;

[0018] Figure 6 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application is shown.

[0019] in, Figure 2 and Figure 3 The corresponding relationship between the reference numerals and the component names is as follows:

[0020] 202 is a camera sensor, 204 is a first image signal processor, and 206 is a second image signal processor. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the embodiments in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] The image processing method, image processing device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0024] In one embodiment, the electronic device includes a camera sensor and a first image signal processor, and the camera sensor is connected to the first image signal processor through a first path and a second path respectively. The camera sensor can be understood as a photosensitive element in the electronic device for sensing ambient light information and obtaining an ambient image, and the first path and the second path are data transmission channels, and the first image signal processor uses different frame capture methods to capture image frames from the camera sensor through the first path and the second path respectively.

[0025] like Figure 1 As shown, an image processing method is proposed, comprising:

[0026] Step 102, controlling the first image signal processor to obtain a first data stream through a first path, the first data stream including a first image frame with normal exposure, and controlling the first image signal processor to obtain a second data stream through a second path, the second data stream including a first image frame with abnormal exposure.

[0027] In the above embodiment, the first data stream is a data stream composed of first image frames with normal exposure. Similarly, the second data stream is a data stream composed of first image frames with abnormal exposure.

[0028] Exemplarily, the normally exposed first image frame includes a normally exposed first sub-image frame, a normally exposed second sub-image frame, a normally exposed third sub-image frame, a normally exposed fourth sub-image frame, ..., and a normally exposed Nth sub-image frame, where N is a positive integer greater than or equal to 2. The preceding and following data formed by the normally exposed first sub-image frame, the normally exposed second sub-image frame, the normally exposed third sub-image frame, the normally exposed fourth sub-image frame, ..., and the normally exposed Nth sub-image frame are sequentially captured as the first data stream.

[0029] Similarly, the abnormally exposed first image frame includes the abnormally exposed first sub-image frame, the abnormally exposed second sub-image frame, the abnormally exposed third sub-image frame, the abnormally exposed fourth sub-image frame, ..., the abnormally exposed Nth sub-image frame, where N is a positive integer greater than or equal to 2. The preceding and following data formed by the abnormally exposed first sub-image frame, the abnormally exposed second sub-image frame, the abnormally exposed third sub-image frame, the abnormally exposed fourth sub-image frame, ..., the abnormally exposed Nth sub-image frame are captured in sequence as the second data stream.

[0030] Step 104 , controlling the preview interface to display a first image frame with normal exposure, where the first image frame with normal exposure includes an image frame that is neither overexposed nor underexposed.

[0031] In this embodiment, the preview interface can be understood as an interface for previewing and displaying the first image frame of normal exposure, which is used to display the picture presented by the camera sensor.

[0032] In the above embodiment, the normally exposed first image frame may be understood as an image frame that is neither overexposed nor underexposed.

[0033] Among them, overexposure refers to the excessive brightness and whiteness of the picture caused by reasons such as too large aperture and too slow shutter speed, while underexposure is the opposite of overexposure, which will not be elaborated here.

[0034] Step 106 , determining a first image based on the normally exposed first image frame and the abnormally exposed first image frame, wherein the first image is a high dynamic range imaging image, and the abnormally exposed first image frame includes an overexposed image frame and / or an underexposed image frame.

[0035] Among them, High Dynamic Range Imaging (HDR) refers to high dynamic lighting rendering. High dynamic range imaging is a processing technology that improves image brightness and contrast. Compared with ordinary images, high dynamic range imaging can provide more dynamic range and image details. The final HDR image is synthesized by using low dynamic range (LDR) images with the best details corresponding to each exposure time, which can better reflect the visual effects in the real environment.

[0036] In an embodiment of the present application, an image processing method is proposed, wherein a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, so that the first image signal processor can obtain a first data stream and a second data stream from the camera sensor through the first path and the second path, respectively, wherein the first data stream includes a first image frame with normal exposure, and the second data stream includes a first image frame with abnormal exposure. After obtaining the first data stream and the second data stream, a preview interface is controlled to display the first image frame with normal exposure contained in the first data stream, and a first image is determined based on the first image frame with normal exposure and the first image frame with abnormal exposure to obtain an image with high dynamic range imaging. In this process, while obtaining an image with high dynamic range imaging, the jamming of the preview image displayed on the preview interface can be improved, thereby improving the shooting experience.

[0037] In some embodiments of the present application, a first image is determined based on a normally exposed first image frame and an abnormally exposed first image frame, specifically including: using an image-sensing domain algorithm to process the normally exposed first image frame and the abnormally exposed first image frame, and using a true color domain algorithm to process a processing result obtained by the image-sensing domain algorithm to obtain the first image.

[0038] The format of the first image may be a Joint Photographic Experts Group (JPEG) format, wherein JPEG is an image compression technology standard developed by an expert group of the same name.

[0039] In some embodiments of the present application, the first image is stored in an album to facilitate user search.

[0040] In some embodiments of the present application, controlling the preview interface to display the normally exposed first image frame specifically includes: performing preview processing on the normally exposed first image frame to obtain a preview image; and displaying the preview image on the preview interface.

[0041] The preview processing may be understood as one or more image processing including noise reduction, blurring, and adding filter effects.

[0042] In this embodiment, the preview processing is performed on the normally exposed first image frame so that the preview image obtained by the processing can meet the display requirements of the preview interface, and finally the display of the preview image is realized.

[0043] In the above embodiment, the preview processing can be understood as using an algorithm corresponding to the preview processing to process the normally exposed first image frame to obtain a preview image.

[0044] Specifically, the algorithm corresponding to the preview processing may be selected according to the preview effect selected by the user.

[0045] In this embodiment, corresponding preview processing can be performed on the normally exposed first image frame according to user needs, so as to present a preview effect that meets the user needs.

[0046] In some embodiments of the present application, the preview effect may be one or more of beautification, blurring, and preview 3D rendering.

[0047] In some embodiments of the present application, controlling the preview interface to display the normally exposed first image frame specifically includes: performing anti-shake processing on the normally exposed first image frame to obtain a processing result; performing preview processing on the processing result to obtain a preview image; and displaying the preview image on the preview interface.

[0048] In this embodiment, by performing anti-shake processing on the normally exposed first image frame, the picture displayed on the preview interface is kept smooth during the video shooting process, thereby ensuring the shooting experience of video shooting.

[0049] In the above embodiments, the anti-shake processing may be electronic anti-shake, optical anti-shake, or electronic anti-shake and optical anti-shake are used in sequence.

[0050] In the above embodiment, after the first image frame of normal exposure is subjected to anti-shake processing, the processing result is previewed so that the preview image obtained by processing can meet the display requirements of the preview interface, and finally the display of the preview image is realized.

[0051] In some embodiments of the present application, after the processing results are previewed and a preview image is obtained, the image processing method further includes: sending the preview image to a second image signal processor; controlling the second image signal processor to output the preview image so that the preview interface displays the preview image, and controlling the second image signal processor to generate a video file based on the preview image.

[0052] In this embodiment, based on the first image signal processor, a second image signal processor is set again, so that the preview image can be processed by the second image signal processor, so that the preview image can be divided into two output paths after processing, wherein the first output path is used to output the preview image to the preview interface so as to display the preview image using the preview interface, and the second output path is used to generate a video file.

[0053] In this process, it can ensure that the preview interface can display the preview image smoothly while also directly generating a smooth video file, thereby improving shooting efficiency.

[0054] In some embodiments of the present application, the video file is aligned using the timestamp of the preview image to ensure the smoothness of the video file.

[0055] In some embodiments of the present application, the first image signal processor and the second image signal processor may be two independent image signal processors, or may be an integrated processor, or may be integrated in the same processing chip. The specific situation may be selected according to actual usage needs and will not be elaborated here.

[0056] In some embodiments of the present application, the image processing method also includes: obtaining a first delay time, the first delay time is the time taken for a target image frame to be output from a camera sensor and displayed on a preview interface, and the target image frame is an image frame historically displayed on the preview interface; obtaining the first time, obtaining a second delay time, and the second delay time is the sum of the first delay time and the first time; when the third delay time is less than the second delay time, after a delay of a fourth time, outputting a preview image for display on the preview interface, and the fourth time is the difference between the second delay time and the third delay time; when the third delay time is greater than or equal to the second delay time, outputting a preview image for display on the preview interface; wherein the third delay time is the predicted processing time of the first image frame of normal exposure, and the processing time is the time taken for the first image frame of normal exposure to be output from the camera sensor and displayed on the preview interface.

[0057] In this embodiment, the time required for different image frames to be sent to the preview interface is different, which will also cause the preview images displayed on the preview interface to freeze, affecting the shooting experience.

[0058] Based on this, obtain a first delay duration to obtain the delay duration corresponding to the image frames previously displayed on the preview interface. After predicting the third delay duration, determine the relationship among the first duration, the first delay duration, and the third delay duration, so as to determine the timing of sending the first image frame with normal exposure, and ensure the smoothness of the preview interface display by adjusting the timing of sending the first image frame with normal exposure.

[0059] Specifically, obtain a second delay duration, where the second delay duration is the sum of the first delay duration and the first duration. If the third delay duration is less than the second delay duration, it is considered that the delay duration corresponding to the first image frame with normal exposure is relatively small, and the preview image needs to wait for a fourth duration before being sent to the preview interface, so as to ensure the smoothness of the preview interface display. On the contrary, if the third delay duration is greater than or equal to the second delay duration, it is considered that the delay duration corresponding to the first image frame with normal exposure is relatively large, and the preview image needs to be immediately sent to the preview interface for display on the preview interface, improving the display freeze to ensure the display effect.

[0060] Exemplarily, if the first delay duration is b milliseconds, the third delay duration is a milliseconds, and the first duration is c milliseconds, then if a < b + c, the preview image needs to wait for b + c - a milliseconds before being sent for preview display to ensure the overall delay is stable; if a ≥ b + c, it is directly sent for preview display.

[0061] Among them, the first duration can be set according to actual usage needs and will not be elaborated here.

[0062] In one of the embodiments, the second delay duration is less than or equal to the maximum delay duration from when the camera sensor outputs a frame to when it is displayed on the preview interface.

[0063] In some embodiments of the present application, the first data stream further includes a second image frame with abnormal exposure. Controlling the preview interface to display the first image frame with normal exposure specifically includes: obtaining a third image frame and a fourth image frame in the first data stream, where the third image frame is the image frame before the second image frame, the fourth image frame is the image frame after the second image frame, and the third image frame and the fourth image frame are image frames with normal exposure; performing interpolation processing on the third image frame and the fourth image frame to obtain a fifth image frame with normal exposure; replacing the second image frame with abnormal exposure in the first data stream with the fifth image frame with normal exposure; and controlling the preview interface to display the first image frame with normal exposure and the fifth image frame with normal exposure.

[0064] In this embodiment, when the first data stream includes an abnormally exposed second image frame, an interpolation method can be used to generate a fifth image frame based on normally exposed third and fourth image frames, and the abnormally exposed second image frame can be replaced with the normally exposed fifth image frame, and the normally exposed first and fifth image frames can be displayed using a preview interface.

[0065] In this process, the first data stream can be displayed in the preview interface without frame loss, thereby ensuring smoothness of the preview interface display.

[0066] Exemplarily, when 5 underexposed image frames appear in the first data stream, in order to ensure the preview effect, the first data stream is interpolated to replace the 5 underexposed image frames to ensure the smoothness of the display of the first data stream.

[0067] In some embodiments of the present application, determining the first image based on the normally exposed first image frame and the abnormally exposed first image frame specifically includes: determining the first image based on the normally exposed first image frame, the abnormally exposed first image frame, and the abnormally exposed second image frame.

[0068] In this embodiment, an image-sensing domain algorithm is used to process the normally exposed first image frame, the abnormally exposed first image frame, and the abnormally exposed second image frame, and a true color domain algorithm is used to process the processing result obtained by the image-sensing domain algorithm to obtain a first image.

[0069] In this process, in the process of obtaining the first image, image frames with abnormal exposure are captured from the first path and the second path at the same time, thereby ensuring the accuracy of the first image.

[0070] In one embodiment, if Figure 2 As shown, specifically, the camera sensor 202 outputs two-way exposure data, one for normal / abnormal preview exposure and one for underexposed / overexposed frames. The normally exposed frames are processed by the first image signal processor 204 and then sent to the preview channel for processing and finally displayed on the screen. At the same time, the normally exposed frames, underexposed frames and overexposed frames are selected according to the requirements of the HDR algorithm, and an image with a high dynamic range imaging is obtained based on the HDR algorithm.

[0071] Among them, according to the needs of the camera sensor 202 frame output path, if all channels output normally exposed image frames, no additional frame interpolation processing is required. If the need for HDR frame capture requires that both channels output abnormally exposed image frames at the same time, in order to ensure that the preview display effect is not affected while ensuring smoothness, it is necessary to dynamically interpolate the preview data stream and send the interpolation results to the preview display, so as to use the interpolation algorithm to ensure the overall smoothness of the preview interface.

[0072] Among them, Yuv is a color encoding method, Y represents brightness, u and v represent chrominance. The raw domain, also known as RAWDomain, is the original data without any processing.

[0073] For example, the normal / abnormal exposure outputs 5 frames of underexposure, while the underexposure / overexposure outputs 4 frames of normal exposure and 1 frame of overexposure. Obviously, for the 5th frame, both exposures are abnormal. In this case, interpolation processing is required.

[0074] The preview data stream is processed by the preview algorithm to present different preview effects. At the same time, the image frame is processed by the image domain algorithm (HDR) and the true color domain algorithm, and finally packaged into a JPEG format image for display in the album.

[0075] Among them, in the frame interval filtering process, the preview display terminal is used to calculate the delay data, and the delay data is smoothed and filtered. Specifically, through the differential autoregressive integrated moving average model (ARIMA), also called the summed autoregressive moving average model, the delay data of the previous frames is used to predict the delay data of the next frame to obtain smooth preview delay data.

[0076] In this embodiment, based on the two-way exposure data output by the camera sensor, one way is used for preview data. The preview uses normal exposure data and interpolated frame data to ensure that there is no frame loss in the preview. At the same time, the filtering algorithm of the preview path is used to ensure smooth preview display. At the same time, image frames with abnormal preview exposure and normal exposure are captured, and processed using the photo HDR algorithm to obtain the final film, thereby ensuring the overall smoothness of the preview without the user feeling any freeze, and satisfying the user's demand for HDR effect of the film.

[0077] In one embodiment, if Figure 3 As shown, specifically, the camera sensor 202 outputs two channels of exposure data, one channel for normal / abnormal preview exposure, and one channel for under-exposure / over-exposure frames. The normally exposed frames are processed by the first image signal processor 204 and then sent to the preview channel for processing and finally displayed on the screen. According to the requirements of HDR dynamic range, there may also be a situation where the two channels of data simultaneously capture abnormally exposed image frames, and the abnormally exposed image frames will obtain high dynamic range imaging images based on the HDR algorithm.

[0078] Among them, according to the needs of the camera sensor 202 for taking pictures and outputting frames, if all channels output normally exposed image frames, no additional frame interpolation processing is required. If two channels need to output abnormally exposed image frames at the same time due to the need for HDR frame capture, in order to ensure smoothness while not affecting the preview display effect, it is necessary to dynamically interpolate the preview data stream and send the interpolation results to the preview interface for display, so as to use the interpolation algorithm to ensure the overall smoothness of the preview.

[0079] The preview data stream is processed by the preview algorithm to present different preview effects. At the same time, the image will be processed by the image domain algorithm (HDR) and the true color domain algorithm, and finally packaged into a JPEG format image for display in the album.

[0080] After the preview data stream is processed by the preview algorithm, it needs to pass through the second image signal processor 206 to divide the preview stream into two paths, one for preview display and the other for video encoding and saving video files. The video encoding data stream is generally aligned using the timestamp of the frame output by the camera sensor 202 to ensure the smoothness of the video file. For the preview data stream, after going through the jitter algorithm and the preview algorithm, it will go through a preview filtering algorithm to achieve frame alignment to ensure the overall smoothness of the preview.

[0081] Among them, the preview filtering algorithm is also the processing flow of the frame interval filtering process mentioned above, which will not be repeated here.

[0082] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0083] In some embodiments of the present application, Figure 4 As shown, an image processing device 400 is provided, wherein a camera sensor is connected to a first image signal processor through a first path and a second path respectively, and the image processing device 400 comprises: a control module 402, configured to control the first image signal processor to obtain a first data stream through the first path, wherein the first data stream comprises a first image frame with normal exposure, and to control the first image signal processor to obtain a second data stream through the second path, wherein the second data stream comprises a first image frame with abnormal exposure; a display module 404, configured to control a preview interface to display a first image frame with normal exposure, wherein the first image frame with normal exposure comprises an image frame that is neither overexposed nor underexposed; and a determination module 406, configured to determine a first image based on the first image frame with normal exposure and the first image frame with abnormal exposure, wherein the first image is an image with high dynamic range imaging, and the first image frame with abnormal exposure comprises an overexposed image frame and / or an underexposed image frame.

[0084] In an embodiment of the present application, a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, so that the first image signal processor can obtain a first data stream and a second data stream from the camera sensor through the first path and the second path, respectively, wherein the first data stream includes a first image frame with normal exposure, and the second data stream includes a first image frame with abnormal exposure. After obtaining the first data stream and the second data stream, the preview interface is controlled to display the first image frame with normal exposure contained in the first data stream, and a first image is determined based on the first image frame with normal exposure and the first image frame with abnormal exposure to obtain an image with high dynamic range imaging. In this process, while obtaining an image with high dynamic range imaging, the freeze of the preview image displayed on the preview interface can be improved, thereby improving the shooting experience.

[0085] In some embodiments of the present application, the display module 404 is specifically used to: perform preview processing on the normally exposed first image frame to obtain a preview image; and display the preview image on the preview interface.

[0086] In some embodiments of the present application, the display module 404 is specifically used to: perform anti-shake processing on the normally exposed first image frame to obtain a processing result; perform preview processing on the processing result to obtain a preview image; and display the preview image on the preview interface.

[0087] In some embodiments of the present application, the control module 402 is also used to: send the preview image to the second image signal processor; control the second image signal processor to output the preview image so that the preview interface displays the preview image, and control the second image signal processor to generate a video file based on the preview image.

[0088] In some embodiments of the present application, the control module 402 is also used to: obtain a first delay duration, the first delay duration is the duration taken for a target image frame to be output from a camera sensor and displayed on a preview interface, and the target image frame is an image frame historically displayed on the preview interface; obtain the first duration; obtain a second delay duration, the second delay duration is the sum of the first delay duration and the first duration; when the third delay duration is less than the second delay duration, after a delay of a fourth duration, output a preview image for display on the preview interface, and the fourth duration is the difference between the second delay duration and the third delay duration; when the third delay duration is greater than or equal to the second delay duration, output a preview image for display on the preview interface; wherein the third delay duration is the predicted processing duration of the first image frame of normal exposure, and the processing duration is the duration taken for the first image frame of normal exposure to be output from the camera sensor and displayed on the preview interface.

[0089] In some embodiments of the present application, the first data stream also includes an abnormally exposed second image frame, and the display module 404 is specifically used to: obtain a third image frame and a fourth image frame in the first data stream, the third image frame is an image frame located before the second image frame, the fourth image frame is an image frame located after the second image frame, and the third image frame and the fourth image frame are normally exposed image frames; perform interpolation processing according to the third image frame and the fourth image frame to obtain a normally exposed fifth image frame; use the normally exposed fifth image frame to replace the abnormally exposed second image frame in the first data stream; control the preview interface to display the normally exposed first image frame and the normally exposed fifth image frame.

[0090] In some embodiments of the present application, the determination module 406 is specifically configured to determine the first image based on the normally exposed first image frame, the abnormally exposed first image frame, and the abnormally exposed second image frame.

[0091] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0092] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0093] The image processing device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the image processing method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0094] like Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 502 and a memory 504, wherein the memory 504 stores programs or instructions that can be executed on the processor 502, and when the program or instructions are executed by the processor 502, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0095] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0096] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0097] like Figure 6 As shown, the electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610 and other components.

[0098] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0099] In some embodiments of the present application, the camera sensor is connected to the first image signal processor through a first path and a second path respectively, and the processor 610 is used to control the first image signal processor to obtain a first data stream through the first path, the first data stream including a normally exposed first image frame, and control the first image signal processor to obtain a second data stream through the second path, the second data stream including an abnormally exposed first image frame; control the preview interface to display the normally exposed first image frame, the normally exposed first image frame including an image frame that is not overexposed and not underexposed; determine the first image based on the normally exposed first image frame and the abnormally exposed first image frame, the first image is a high dynamic range imaging image, and the abnormally exposed first image frame includes an overexposed image frame and / or an underexposed image frame.

[0100] In some embodiments of the present application, the processor 610 is specifically used to perform preview processing on the normally exposed first image frame to obtain a preview image; and display the preview image on the preview interface.

[0101] In some embodiments of the present application, the processor 610 is specifically used to perform anti-shake processing on the normally exposed first image frame to obtain a processing result; preview the processing result to obtain a preview image; and display the preview image on the preview interface.

[0102] In some embodiments of the present application, the processor 610 is also used to send the preview image to the second image signal processor; control the second image signal processor to output the preview image so that the preview interface displays the preview image, and control the second image signal processor to generate a video file based on the preview image.

[0103] In some embodiments of the present application, the processor 610 is also used to obtain a first delay duration, the first delay duration is the duration taken for a target image frame to be output from a camera sensor and displayed on a preview interface, and the target image frame is an image frame historically displayed on the preview interface; obtain the first duration; obtain the second delay duration, the second delay duration is the sum of the first delay duration and the first duration; when the third delay duration is less than the second delay duration, after a delay of a fourth duration, a preview image for display on the preview interface is output, and the fourth duration is the difference between the second delay duration and the third delay duration; when the third delay duration is greater than or equal to the second delay duration, a preview image for display on the preview interface is output; wherein the third delay duration is the predicted processing duration of the first image frame of normal exposure, and the processing duration is the duration taken for the first image frame of normal exposure to be output from the camera sensor and displayed on the preview interface.

[0104] In some embodiments of the present application, the first data stream also includes an abnormally exposed second image frame, and the processor 610 is specifically used to: obtain a third image frame and a fourth image frame in the first data stream, the third image frame is an image frame located before the second image frame, the fourth image frame is an image frame located after the second image frame, and the third image frame and the fourth image frame are normally exposed image frames; perform interpolation processing according to the third image frame and the fourth image frame to obtain a normally exposed fifth image frame; replace the abnormally exposed second image frame in the first data stream with the normally exposed fifth image frame; and control the preview interface to display the normally exposed first image frame and the normally exposed fifth image frame.

[0105] In some embodiments of the present application, the processor 610 is specifically configured to determine the first image based on the normally exposed first image frame, the abnormally exposed first image frame, and the abnormally exposed second image frame.

[0106] In an embodiment of the present application, a camera sensor is connected to a first image signal processor through a first path and a second path, respectively, so that the first image signal processor can obtain a first data stream and a second data stream from the camera sensor through the first path and the second path, respectively, wherein the first data stream includes a first image frame with normal exposure, and the second data stream includes a first image frame with abnormal exposure. After obtaining the first data stream and the second data stream, the preview interface is controlled to display the first image frame with normal exposure contained in the first data stream, and a first image is determined based on the first image frame with normal exposure and the first image frame with abnormal exposure to obtain an image with high dynamic range imaging. In this process, while obtaining an image with high dynamic range imaging, the freeze of the preview image displayed on the preview interface can be improved, thereby improving the shooting experience.

[0107] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0108] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0109] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0110] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0111] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0112] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0113] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0114] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0115] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "including one" do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the embodiments of the present application are essentially or partly contributed to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0117] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The camera sensor is connected to the first image signal processor through a first path and a second path respectively, and the image processing method includes: Controlling the first image signal processor to acquire a first data stream through a first path, wherein the first data stream includes a first image frame with normal exposure, and controlling the first image signal processor to acquire a second data stream through a second path, wherein the second data stream includes a first image frame with abnormal exposure; Controlling the preview interface to display the normally exposed first image frame, wherein the normally exposed first image frame includes an image frame that is neither overexposed nor underexposed; A first image is determined based on the normally exposed first image frame and the abnormally exposed first image frame, wherein the first image is a high dynamic range imaging image, and the abnormally exposed first image frame includes an overexposed image frame and / or an underexposed image frame.

2. The image processing method according to claim 1, characterized in that: The controlling the preview interface to display the first image frame of normal exposure specifically includes: Performing preview processing on the normally exposed first image frame to obtain a preview image; The preview image is displayed on the preview interface.

3. The image processing method according to claim 1, characterized in that: The controlling the preview interface to display the first image frame of normal exposure specifically includes: Performing anti-shake processing on the normally exposed first image frame to obtain a processing result; Performing preview processing on the processing result to obtain a preview image; The preview image is displayed on the preview interface.

4. The image processing method according to claim 3, characterized in that: After performing preview processing on the processing result to obtain a preview image, the image processing method further includes: sending the preview image to a second image signal processor; The second image signal processor is controlled to output the preview image so that the preview interface displays the preview image, and the second image signal processor is controlled to generate a video file based on the preview image.

5. The image processing method according to claim 2 or 3, characterized in that: The image processing method further comprises: Acquire a first delay duration, where the first delay duration is the duration taken for a target image frame to be output from the camera sensor and displayed on the preview interface, and the target image frame is an image frame historically displayed on the preview interface; Get the first duration; Obtain a second delay duration, where the second delay duration is the sum of the first delay duration and the first duration; When the third delay time is shorter than the second delay time, the preview image for display on the preview interface is output after a fourth delay time, where the fourth delay time is the difference between the second delay time and the third delay time; When the third delay time is greater than or equal to the second delay time, outputting the preview image for display on the preview interface; Among them, the third delay duration is the predicted processing duration of the normally exposed first image frame, and the processing duration is the duration taken for the normally exposed first image frame to be output from the camera sensor to be displayed on the preview interface.

6. The image processing method according to any one of claims 1 to 4, characterized in that: The first data stream further includes a second image frame with abnormal exposure, and the controlling preview interface to display the first image frame with normal exposure specifically includes: Acquire a third image frame and a fourth image frame in the first data stream, wherein the third image frame is an image frame located before the second image frame, the fourth image frame is an image frame located after the second image frame, and the third image frame and the fourth image frame are image frames that are normally exposed; Performing interpolation processing according to the third image frame and the fourth image frame to obtain a fifth image frame with normal exposure; Using the normally exposed fifth image frame to replace the abnormally exposed second image frame in the first data stream; The preview interface is controlled to display the normally exposed first image frame and the normally exposed fifth image frame.

7. The image processing method according to claim 6, characterized in that: The determining the first image based on the normally exposed first image frame and the abnormally exposed first image frame specifically includes: A first image is determined based on the normally exposed first image frame, the abnormally exposed first image frame, and the abnormally exposed second image frame.

8. An image processing device, characterized in that: The camera sensor is connected to the first image signal processor through a first path and a second path respectively, and the image processing device includes: a control module, configured to control the first image signal processor to acquire a first data stream through a first path, wherein the first data stream includes a first image frame with normal exposure, and to control the first image signal processor to acquire a second data stream through a second path, wherein the second data stream includes the first image frame with abnormal exposure; A display module, used for controlling the preview interface to display the normally exposed first image frame, wherein the normally exposed first image frame includes an image frame that is neither overexposed nor underexposed; A determination module is used to determine a first image based on the normally exposed first image frame and the abnormally exposed first image frame, wherein the first image is a high dynamic range imaging image, and the abnormally exposed first image frame includes an overexposed image frame and / or an underexposed image frame.

9. The image processing device according to claim 8, characterized in that: The display module is specifically used for: Performing preview processing on the normally exposed first image frame to obtain a preview image; The preview image is displayed on the preview interface.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images