Image processing method, electronic device, storage medium, chip system and computer program product

Abstract

The embodiment of the invention provides an image processing method, electronic equipment, a storage medium, a chip system and a computer program product, and relates to the technical field of terminals, the method comprises the following steps: obtaining a first image in a video recording process; performing first processing on the first image to obtain a second image; the first processing is used for processing the single-frame image by using intra-frame information of the single-frame image; performing second processing on the second image according to the first information to obtain and display a third image, and storing the second information and the second image; wherein the first information is obtained based on association information between the first image and a frames of images before the first image, the second information is obtained based on association information between the first image and b frames of images before the first image, and a is smaller than b. Therefore, in the recording process, the image of the preview stream is processed in real time to ensure the chirality of the preview image; and meanwhile, the image of the video stream is stored and subjected to delayed processing, so that the memory pressure in the video recording process is reduced.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, electronic device, storage medium, chip system and computer program product. Background Art

[0002] An electronic device may support a camera function, and a user may use the electronic device to take photos and / or record videos. Taking video recording as an example, after launching a camera application, in response to a trigger operation on a recording control, the electronic device may display a preview image on the display screen and save the preview image to obtain a recorded video.

[0003] Possible implementation: During the video recording process, the smoothness of the electronic device display image is poor, affecting the user experience. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, an electronic device, a storage medium, a chip system, and a computer program product, which are applied to the field of terminal technology. In a video recording scenario, the image processing process is segmented; the image of the preview stream is processed in real time to improve the hand tracking of the preview image; and the image of the video stream is processed in a delayed manner to reduce the memory pressure during the recording process.

[0005] In a first aspect, an embodiment of the present application proposes an image processing method. The method includes: obtaining a first image during a video recording process; performing a first processing on the first image to obtain a second image; the first processing is used to process the single-frame image using the intra-frame information of the single-frame image; performing a second processing on the second image based on the first information to obtain and display a third image, and saving the second information and the second image; wherein the first information is obtained based on the association information between the first image and the a-frame image before the first image, the second information is obtained based on the association information between the first image and the b-frame image before the first image, a and b are both positive integers, and a is less than b, the third image is used for preview display during the video recording process, and the second information and the second image are used to obtain the second processing when the recorded video is recorded.

[0006] The first image may be, for example, Figure 7 or Figure 11 The first processing can be, for example, Figure 7 The processing performed by the image front-end processing module, the multi-camera spatial alignment module, the motion estimation module, the image back-end processing module, the video function feature module and the format conversion module; or, Figure 11 The second image may be, for example, image 2 or image 5. The third image may be, for example, image 2 or image 5. Figure 7or Figure 11 The first information may be information of the preview stream image, such as preview inter-frame anti-shake information or preview noise reduction information. The second information may be information of the video stream image, such as video inter-frame anti-shake information or video noise reduction information. Figure 7 1; b in the embodiment shown can be, for example Figure 7 N in the embodiment shown (N is for example 30); a can be for example Figure 11 a; b in the embodiment shown can be, for example Figure 11 b in the embodiment shown. It can be seen that the value of a is smaller than the value of b.

[0007] It should be noted that the embodiments of this application Figure 7 The second processing is taken as an example to illustrate the solution, Figure 11 The second processing is multi-frame noise reduction as an example to illustrate the solution. In actual scenarios, electronic devices can also postpone the second processing of multiple video stream image processing processes involving inter-frame processing. This is only for the convenience of explaining the segmented processing architecture. Figure 11 The image stabilization processing module is defined as the first processing in the image stabilization processing module. The image stabilization module can also refer to Figure 7 The relevant descriptions in the illustrated embodiments are also deferred. Similarly, the second processing may also be HDR effect processing, video enhancement algorithm, super-resolution reconstruction algorithm and / or blurring algorithm processing involved in the video function feature module, etc. This embodiment of the present application will not be further described in detail.

[0008] In addition, the first processing also includes some multi-frame processing that is not used to distinguish between the preview stream and the recording stream. For example, in some multi-frame processing, the number of frame images cached for processing the preview stream is roughly the same as the number of frame images cached for processing the recording stream. At this time, the first processing will not divert the image (diversion can be understood as dividing the image into a preview stream and a recording stream).

[0009] In this way, during the recording process, the electronic device processes the image of the preview stream in real time to ensure the tracking of the preview image; at the same time, the image of the video stream is saved first and processed later, reducing the memory pressure during the recording process, thereby reducing the problem of preview screen freezes and high device temperature.

[0010] Saving the second information and the second image includes: encoding the second information and the second image to obtain and save the first video (for example, recorded video 1 or recorded video 3); wherein the first video includes a first media track (for example, a video track) and a second media track (for example, a video inter-frame stabilization information track or a video noise reduction information track); during the recording process, the second image is stored in the first media track, and the second information is stored in the second media track.

[0011] In this way, the electronic device can encode the second information and the second image into one video, thereby improving the synchronization between the second image and the second information, simplifying the management and transmission of media resources, and improving transmission efficiency.

[0012] Optionally, after obtaining the first video, the method further includes: responding to an operation for playing the first video (for example, Figure 2 The trigger operation for the thumbnail control 101 in the interface shown in a, or Figure 2 (a trigger operation for video 201 in the interface shown in c in the figure) performs a second processing (for example, inter-frame stabilization or multi-frame noise reduction) on the second image according to the second information (for example, inter-frame stabilization information or video noise reduction information) to obtain and display a fourth image (for example, image 3 or image 6); after traversing and processing the images in the first video, the first video is replaced with a second video (for example, recorded video 2), the second video includes the fourth image, and the second video does not include the second image and the second information (it can be understood that after the replacement, the electronic device no longer saves the recorded video 1 to reduce memory usage).

[0013] In this way, when a subsequent user uses the first video, the electronic device can perform multi-frame processing on the images in the first video, such as inter-frame anti-shake or multi-frame noise reduction, thereby reducing memory pressure during the recording process. At the same time, when subsequent users view the recorded video, they can watch the recorded video with better image quality; when the user does not use the first video, the images in the first video will not be processed by multiple frames, further reducing the image calculation pressure.

[0014] Optional, see Figure 7 The processing flow shown in the figure, the second processing includes a first electronic anti-shake EIS processing (for example, inter-frame anti-shake processing); the first EIS processing is used to perform anti-shake processing on the first image based on the association information between consecutive frame images; the first information is a first matrix (for example, preview inter-frame anti-shake information, matrix A) obtained based on the meta of the first image and the meta data of the a frame image before the first image; the electronic device includes a first module, which performs a second processing on the second image according to the first information to obtain a third image, including: the first module performs a first EIS processing on the second image based on a preset matrix of the second image (for example, a unit matrix, matrix C) and the first matrix to obtain the third image; the first module supports coordinate transformation of the second image, and the first module includes a GPU or an IPE (for example Figure 7 wherein the electronic device may set the first module to be a GPU or an IPE according to power consumption or running memory occupancy.

[0015] In this way, during the video recording process, a segmented processing architecture is adopted, and the first module can perform anti-shake processing on the image of the preview stream through the anti-shake information between preview frames to ensure the hand tracking of the preview image.

[0016] Optionally, the second information is a second matrix (for example, video inter-frame anti-shake information, matrix B) obtained based on the meta data of the first image and the meta data of the b-frame image before the first image; the electronic device also includes an encoder and a decoder, and the first video is encoded by the encoder; the second image is subjected to a second processing according to the second information to obtain a fourth image, including: the decoder decodes the first video to obtain a second image and a second matrix; wherein the second image is decoded from the first media track, the second matrix is ​​decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second matrix; the GPU performs a first EIS processing on the second image based on a preset matrix of the second image (for example, a unit matrix, matrix C) and the second matrix to obtain a fourth image; it can be understood that the processing process during the recording process may involve IPE; during the video playback process, the power consumption used by calling IPE is relatively large, that is, Figure 7 The image calculation module in the preview stream is not suitable for processing the video stream image during playback. Here, the GPU can be used instead of IPE.

[0017] In this way, when the recorded video is subsequently used, a segmented processing architecture is adopted, and the electronic device can perform anti-shake processing on the recorded video through the anti-shake information between video frames to improve the image quality of the recorded video.

[0018] Optionally, the meta data of the first image includes one or more of the following: sensor data, timestamp and shooting parameters when the first image is acquired; the meta data of the i-th frame image in the b-frame image before the first image includes one or more of the following: sensor data, timestamp and shooting parameters when the i-th frame image is acquired, where i is less than b; the second matrix is ​​obtained in the following manner: based on the sensor data of the i-th frame image and the sensor data of the first image (for example, the sensor data of image 1 in step S703; it is understandable that the first image can be the original image, and the sensor data of the image in the processing process is the same as the sensor data of the original image of the same frame), a rotation matrix, a scaling matrix and a translation matrix are obtained respectively; a transformation matrix is ​​obtained based on the rotation matrix, the scaling matrix and the translation matrix; after traversing the b-frame image before the first image, multiple transformation matrices are processed into a second matrix (see step S703).

[0019] In this way, during the recording process, the electronic device can calculate and save the anti-shake information used to describe the recorded video (such as the anti-shake information between video frames); so that the anti-shake information can be used in subsequent playback scenarios to perform anti-shake processing on the recorded video and improve the image quality of the recorded video.

[0020] Optionally, the electronic device further includes a second module (eg Figure 7The processing module located before the image stabilization module in the process shown), the third module (for example, the module 2 and module 3 for calculating the inter-frame stabilization data in the image stabilization module) and the fourth module (for example Figure 7 a processing module located after the image stabilization module in the shown process); performing a first processing on the first image to obtain a second image, including: after obtaining the first image, the second module performs pre-processing in the first processing on the first image to obtain a fifth image (for example, image 1); the pre-processing includes one or more of the following: IFE processing (corresponding to the image front-end processing module), SAT processing (corresponding to the multi-camera spatial alignment module), GME processing (corresponding to the motion estimation module) and the second EIS processing (for example, module 1 in the image stabilization module for implementing intra-frame correction); the second EIS processing is used to perform stabilization processing on the first image using the intra-frame information of the first image; the third module calculates a first matrix (calculated by module 2) and a second matrix (calculated by module 3); the fourth module performs post-processing in the first processing on the fifth image to obtain a second image (for example, image 2); the post-processing includes one or more of the following: IPE processing (corresponding to the image back-end processing module), skin beautification processing (corresponding to the video function characteristic module), blur processing (corresponding to the video function characteristic module) and image format conversion processing (corresponding to the format conversion module).

[0021] In this way, the electronic device can adopt a segmented processing architecture. When performing the second processing on the image, it first calculates the data required for the second processing of the preview stream and the video stream, and does not perform the second processing on the image, thereby avoiding diversion due to the different second processing of the preview stream and the video stream; subsequently, after completing the first processing, the second information is used to divert the image to the preview stream and the video stream; reducing the processing modules involved after the diversion, thereby reducing the power consumption of image processing.

[0022] Optional, see Figure 11 In the processing flow shown, the second processing also includes noise reduction processing; the first information is a first noise reduction parameter (such as preview noise reduction information) obtained based on the meta data of the first image and the meta data of the a-frame image before the first image; the electronic device includes a fifth module, which performs a second processing on the second image (such as image 5) according to the first information to obtain a third image (such as a preview image), including: the fifth module performs noise reduction processing on the second image based on the first noise reduction parameter to obtain the third image; the fifth module includes: GPU or IPE.

[0023] In this way, during the video recording process, a segmented processing architecture is adopted, and the fifth module can perform noise reduction processing on the image of the preview stream by using the preview noise reduction information to ensure the chirality of the preview image.

[0024] Optionally, the second information is a second noise reduction parameter (such as video noise reduction information) obtained based on the meta data of the first image and the meta data of the b-frame image before the first image; the electronic device also includes an encoder and a decoder, and the first video is encoded by the encoder; the second image is subjected to a second processing according to the second information to obtain a fourth image, including: the decoder decodes the first video to obtain a second image and a second noise reduction parameter; wherein the second image is decoded from the first media track, the second noise reduction parameter is decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second noise reduction parameter; the sixth module processes the second image based on the second noise reduction parameter to obtain a fourth image, and the sixth module includes a GPU or a decoder. Here, a filter can be set in the GPU or the codec, and the GPU or the decoder can be used to perform multi-frame noise reduction processing during the playback of the first video. Optionally, the electronic device can also use the encoder to perform multi-frame noise reduction processing on the recorded video during the recording process, and the embodiment of the present application does not limit this.

[0025] In this way, when the recorded video is subsequently used, a segmented processing architecture is adopted, and the electronic device can perform multi-frame noise reduction processing on the recorded video through the video noise reduction information to improve the image quality of the recorded video.

[0026] Optionally, the electronic device further includes a seventh module (eg Figure 11 The processing module located before the multi-frame noise reduction module in the process shown), the eighth module (eg, the multi-frame noise reduction module) and the ninth module (eg, Figure 11 a processing module located after the multi-frame noise reduction module in the shown process); performing a first processing on the first image to obtain a second image, including: after obtaining the first image, the seventh module performs pre-processing in the first processing on the first image to obtain a sixth image (for example, image 4); the pre-processing includes one or more of the following: IFE processing (corresponding to the image front-end processing module), SAT processing (corresponding to the multi-camera spatial alignment module), GME processing (corresponding to the motion estimation module) and EIS processing (corresponding to the image stabilization module); the eighth module obtains the first noise reduction parameter and the second noise reduction parameter; the ninth module performs post-processing in the first processing on the sixth image to obtain a second image (for example, image 5); the post-processing includes one or more of the following: IPE processing (corresponding to the image back-end processing module), skin beautification processing (corresponding to the video function characteristic module), blur processing (corresponding to the video function characteristic module) and image format conversion processing (corresponding to the format conversion module).

[0027] In this way, the electronic device can adopt a segmented processing architecture. When performing the second processing on the image, it first calculates the data required for the second processing of the preview stream and the video stream, and does not perform the second processing on the image, thereby avoiding diversion due to the different second processing of the preview stream and the video stream; subsequently, after completing the first processing, the second information is used to divert the image to the preview stream and the video stream; reducing the processing modules involved after the diversion, thereby reducing the power consumption of image processing.

[0028] In a second aspect, an embodiment of the present application provides an electronic device, which may also be referred to as a terminal device, terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0029] The electronic device comprises: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method of the first aspect.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, it enables the computer to execute the method of the first aspect.

[0032] In the fifth aspect, an embodiment of the present application provides a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through lines, and the at least one processor is used to run computer programs or instructions to execute the method of the first aspect.

[0033] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the interface for the video recording process provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of an interface for playing recorded videos provided in an embodiment of the present application;

[0036] Figure 3 A flowchart of an image stabilization process in a possible implementation is shown;

[0037] Figure 4 1 is a flow chart of another image stabilization process in a possible implementation;

[0038] Figure 5 A schematic structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of the software structure of the electronic device 100 provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of a process for image stabilization provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of the structure of the video 1 provided in an embodiment of the present application;

[0042] Figure 9 A schematic diagram of inter-frame anti-shake and intra-frame correction provided in an embodiment of the present application;

[0043] Figure 10 A schematic diagram of a possible implementation of an image noise reduction process;

[0044] Figure 11 A schematic diagram of a process for image noise reduction provided in an embodiment of the present application;

[0045] Figure 12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0047] 1. Electronic devices

[0048] The electronic devices of the embodiments of the present application may include handheld devices, vehicle-mounted devices, etc. with image processing functions. For example, some electronic devices include: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices (such as car computers), wearable devices, electronic devices in 5G or future evolution of public land mobile communications (PLMCs), and the like. The electronic devices in the network (PLMN) are not limited to this in the embodiments of the present application.

[0049] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as hearing aids, glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0050] In addition, in the embodiment of the present application, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects through communication technology, thereby realizing the intelligent interconnection of man and machine and the interconnection of things.

[0051] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0052] 2. Other terms

[0053] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0055] It should be noted that the "at..." in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after the situation occurs, and the embodiments of the present application do not specifically limit this. In addition, the display interface provided in the embodiments of the present application is only an example, and the display interface can also include more or less content.

[0056] The following combination Figure 1 Describe the recording process.

[0057] The electronic device can take photos and / or record videos based on the camera application. In response to the trigger operation for starting the camera application, the electronic device can enter the initial photo taking interface, such as Figure 1 The interface shown in a. Figure 1 The interface shown in a in the figure can display a top menu area, a viewing area, and a bottom menu area. The top menu area can display one or more function controls, such as flash controls, AI photography controls, high dynamic range (HDR) controls, filter controls, and setting controls. The viewing area can display a preview image of the object being photographed. The bottom menu area can display function controls, shooting mode controls, thumbnail controls 101, shutter controls 102, and lens conversion controls 103, etc.; among them, function controls such as magnification controls; shooting mode controls such as aperture controls, night scene controls, portrait controls, photo controls, video controls, movie controls, and professional controls. The thumbnail control 101 can be used to quickly browse and manage previews of taken photos or videos. The shutter control 102 can be used to take pictures or start recording videos; the lens conversion control 103 can be used to switch between the front camera and the rear camera. Among them, the shooting mode corresponding to the initial photo interface can be a photo mode.

[0058] In response to a click operation on the video control in the shooting mode, the electronic device can switch the shooting mode to the video mode and display the following information: Figure 1 The interface shown in b. Figure 1 In the interface shown in b, the video mode control can be displayed in the center to indicate that the video mode is currently in progress. When a trigger operation is received for the shutter control 102, the electronic device can start recording and display the following information: Figure 1 The interface shown in c.

[0059] exist Figure 1 In the interface shown in c, the electronic device can display the end recording control 104, the pause recording control 105, the recording time and the preview image of the object being photographed. When receiving a click operation on the end recording control 104, the electronic device can end the recording, save the video, and display the following information: Figure 1 The interface shown in d. Figure 1 In the interface shown in d, the thumbnail control 101 displays a thumbnail of the video.

[0060] At this point, the electronic device can record videos based on the recording function of the camera application.

[0061] The following combination Figure 2 Describes the process of viewing recorded videos.

[0062] In a possible implementation, the electronic device can view the recorded video based on the thumbnail control 101 in the camera application. For example, after the recording is finished, the electronic device can display the following Figure 2 The interface shown in a in FIG. includes a thumbnail control 101. When a trigger operation for the thumbnail control 101 is received, the electronic device may display the following Figure 2 The interface shown in b. Figure 2 The interface shown in b can play the recorded video in full screen.

[0063] In another possible implementation, the electronic device can view recorded videos based on the gallery application. For example, in response to the operation for starting the gallery application, the electronic device can display the gallery application interface, such as Figure 2 The interface shown in c. Figure 2 The interface shown in c includes one or more pictures or videos, for example, video 201. When a trigger operation for video 201 is received, the electronic device can play video 201, such as Figure 2 The interface shown in b.

[0064] In a possible implementation, the electronic device may refer to Figure 1 However, during the recording process, the electronic device may experience poor image smoothness and high temperature.

[0065] The following are combined Figure 3 and Figure 4 Describe the cause of this problem scenario.

[0066] During recording, the electronic device executes two processes: a preview stream and a recording stream. The preview stream provides immediate visual feedback, allowing the user to check what is being recorded. The recording stream is responsible for storing the captured video data and creating a recorded video file.

[0067] In one possible implementation, the electronic device may be configured as follows: Figure 3 The video recording process shown obtains preview stream and video stream respectively.

[0068] The recording process may involve: camera, image front-end processing module (also called image front end, IFE), multi-camera spatial alignment module (also called spatial align transaction, SAT), motion estimation module (also called global motion estimation, GME), image stabilization module (also called electronic image stabilization, EIS), image back-end processing module (also called image processing engine, IPE), video function feature module and format conversion module, etc.

[0069] The camera can be used to capture original images, such as raw images. The camera can include an image sensor. The image sensor can be used to convert captured light into electrical signals, thereby capturing an image of the object being photographed.

[0070] The image front-end processing module can be used to receive the original image from the image sensor and perform preliminary processing on the original image. The preliminary processing can include color correction, downsampling, demosaicing, and statistical 3A data.

[0071] The multi-camera spatial alignment module can be used to spatially align images captured by different cameras, thus achieving multi-camera image fusion.

[0072] The motion estimation module can be used to estimate the overall motion between two or more image frames.

[0073] The image stabilization module can be used to compensate for slight camera movements and reduce shaking and blur when shooting.

[0074] The image backend processing module can be an image processing engine responsible for performing more complex image processing tasks, such as hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement, etc.

[0075] The video function feature module can be used to process the functions in the video recording mode, which may include skin beautification and blurring. Figure 1 The interface shown in b also includes a blur control 106 and a skin beautification control 107. When the blur control 106 is turned on, the electronic device can blur the image based on the blur algorithm to highlight the subject; when the skin beautification control 107 is turned on, the electronic device can beautify the image based on the skin beautification algorithm to beautify the person.

[0076] The format conversion module can be used to convert the image format. The image format can be processed based on IPE, such as converting an RGB image to a YUV image, or converting a YUV image to JPEG or PNG format, etc., to facilitate image display and storage.

[0077] See also Figure 3After starting the camera application, the electronic device collects multiple original images based on the image sensors of one or more cameras. The image sensor transmits the original image to the image front-end processing module, which can perform color correction, downsampling, demosaicing, statistical 3A data and other algorithm processing on the original image. The image front-end processing module transmits the processed image to the multi-camera control alignment module, which fuses the processed images from multiple cameras. The multi-camera control alignment module transmits the fused image to the motion estimation module, which analyzes and estimates the motion vector between the images and uses the estimated vector to compensate the image. The motion estimation module transmits the image to the image stabilization module, which can obtain the measurement data reported by the gyroscope sensor (such as three-axis angular velocity data). The image stabilization module can use the measurement data to process the image to reduce the jitter between images.

[0078] The image stabilization module passes the processed image to the image backend processing module, which performs hardware noise reduction, image cropping, noise reduction, color processing, and detail enhancement. The image backend processing module then passes the processed image to the video function feature module, which performs image processing such as skin beautification and blurring. The video function feature module then passes the image to the format conversion module, which converts the image to a preset format.

[0079] Afterwards, the electronic device can display the image and sequentially display multiple preview images on the display screen, such as Figure 1 In the a interface to Figure 1 As shown in the d interface.

[0080] The above is the processing process of the preview stream. When the electronic device displays the preview image in real time, it can start recording the video in response to the operation for recording.

[0081] For example, at time 1, the electronic device is Figure 1 In the interface shown in b, a click operation on the shutter control 102 is received, and the electronic device can save the images at time 1 and after. Figure 3 After the format conversion module converts the image format, the electronic device can save the image at time 1 and after. Figure 1 In the interface shown in c, a trigger operation for the end recording control 104 is received; in response to the trigger operation, the electronic device can stop saving the image and obtain a video composed of frame images obtained between time 1 and time 2.

[0082] from Figure 3As can be seen, the images in the preview stream and the video stream are processed sequentially by the camera, image front-end processing module, multi-camera spatial alignment module, motion estimation module, image stabilization module, image back-end processing module, video feature module, and format conversion module. Within the same frame, the preview stream and the video stream images are identical. However, in practical applications, the preview stream images must maintain chirality during recording. Therefore, electronic devices must quickly process and display the preview stream images in real time to maintain smooth preview images. As stored images, the video stream images require higher image quality. Therefore, to ensure high image quality for the video stream, the image processing process is complex and takes a long time, which cannot meet the chirality requirements for the preview stream images. To ensure chirality for the preview stream images, the image processing process needs to be simplified and reduced, but this will result in a decrease in image quality, affecting the quality of the video stream.

[0083] On this basis, another possible implementation is provided, which is to split the processing of preview stream and video stream, reduce the processing time of preview stream and improve the hand tracking of preview image without affecting the quality of video. Taking the image stabilization module as an example, the electronic device can adopt Figure 4 The video recording process shown obtains preview stream and video stream respectively.

[0084] The recording process may involve: camera, image front-end processing module, multi-camera spatial alignment module, motion estimation module, image stabilization module (preview image stabilization module and video image stabilization module), image back-end processing module, video function feature module and format conversion module, etc.

[0085] Among them, the preview image stabilization module can be used to perform stabilization processing on the image of the preview stream to reduce the shaking and blurring of the preview image. The video image stabilization module can be used to perform stabilization processing on the image of the recording stream to reduce the shaking and blurring of the video image. When performing stabilization processing on the image, the image stabilization module can cache multiple frames of images before the frame image, and use the multiple frames of images to process the frame image. In order to improve the tracking of the preview image during the recording process, the preview image stabilization module can be set to cache fewer frame images (for example, 1 frame) to reduce the time of running the electronic stabilization algorithm. In order to improve the clarity of the video image in the recorded video, the video image stabilization module can be set to cache more frame images (for example, 30 frames).

[0086] After launching the camera app, the electronic device captures multiple raw images using the image sensors of one or more cameras. These images are then passed to the image front-end processing module, which performs algorithms such as color correction, downsampling, de-mosaicing, and 3A data statistics. The front-end processing module then passes the processed images to the multi-camera alignment module, which fuses the processed images from multiple cameras.

[0087] After the images are spatially aligned, the electronic device can split the processed images into two streams: one for preview and the other for video recording.

[0088] For the preview stream, the multi-camera alignment module passes the image to the preview image stabilization module. The preview image stabilization module obtains sensor data and uses it to process the image to reduce jitter between images. For example, the preview image stabilization module obtains and caches the previous frame and uses the previous frame and sensor data to perform stabilization on the next frame. The preview image stabilization module passes the processed image to the image backend processing module, which performs hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement, and other processing on the image. The image backend processing module passes the processed image to the video function feature module, which performs skin beautification and blurring on the image. The video function feature module passes the image to the format conversion module, which converts the image to a preset format.

[0089] Afterwards, the electronic device may display the image and sequentially display multiple preview images on the display screen. This process may be, for example, Figure 1 In the a interface to Figure 1 The preview screen is shown in the d interface.

[0090] For video streams, the multi-camera alignment module passes the fused image to the motion estimation module, which analyzes and estimates the motion vectors between multiple frames and uses these vectors to compensate for the image. The motion estimation module then passes the image to the video stabilization module, which receives sensor data and processes it to reduce jitter between frames.

[0091] For example, at time 3, the electronic device is Figure 1 In the interface shown in b, a trigger operation for the shutter control 102 is received, and the electronic device can process the image at time 3. The video image stabilization module obtains and caches the 30 frames of images before time 3, and uses the first 30 frames of images and the measurement data to perform stabilization processing on the image at time 3.

[0092] The video image anti-shake module delivers the processed image to the image back-end processing module, and the image back-end processing module performs hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement, and the like on the image. The image back-end processing module delivers the processed image to the video function characteristic module, and the video function characteristic module performs skin beautification, blurring, and the like on the image. The video function characteristic module delivers the image to the format conversion module, and the format conversion module converts the image into a preset format. At time 4, a triggering operation on the end recording control 104 is received in the interface shown by c in FIG. 4; in response to the triggering operation, the electronic device can stop saving the image, and obtain a video composed of frame images acquired between time 3 and time 4. Figure 1

[0093] It can be seen that the method can reduce the processing time of the preview stream and improve the hand following property of the preview image by reducing the buffered frame images. However, because the anti-shake processing is located relatively early in the processing flow, a large amount of additional processing flow is introduced, which increases the memory amount and power consumption. For example, the image processed by the preview image anti-shake module and the image processed by the video image anti-shake module are different, which requires that the two images are processed by the image back-end processing module, the video function characteristic module, and the format conversion module, respectively.

[0094] During the recording process, the large memory increment and power consumption can affect the smoothness of the preview image, and at the same time, cause the electronic device to overheat, affecting the user's thermal experience.

[0095] Based on this, the embodiment of the present application provides an image processing method. Because of the hand following property of the preview image and the high definition property of the video image, and the video image has a lower requirement for the hand following property, during the video recording process, the electronic device can process and display the preview image in real time on the display screen. During the video recording process, the electronic device does not require real-time processing of the video image, and therefore part of the processing flow of the video image, such as anti-shake processing, multi-frame noise reduction processing, HDR effect processing, blurring processing, video enhancement processing, and super-resolution processing, can not be performed. Subsequently, when the video is played, the electronic device can complete the part of the processing flow of the video image to play high-quality video images. In this way, during the recording process, the processor can process the flow related to the hand following property, and during the playing, the processor can process the definition of the video image. On the basis of not affecting the high definition of the video image, the memory pressure during the recording process is reduced, and the problem scenario of the preview image being stuck and the device temperature being high is reduced.

[0096] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiments of the present application is introduced as follows:

[0097] Figure 5 ​1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identification module (SIM) card interface 195, and an embedded secure element (eSE) 196. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0098] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0099] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0100] The electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. In the embodiment of the present application, the electronic device 100 can reuse the image processing capabilities of the GPU to implement delayed processing of the video stream image.

[0101] The display screen 194 is used to display images, videos, etc. In the embodiment of the present application, the display screen 194 can be used to display Figure 1 The preview image shown can also be used to display Figure 2 In the embodiment of the present application, the display screen 194 can be used to display the corresponding interfaces of the camera application and the gallery application.

[0102] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0103] The ISP is used to process data fed back by the camera 193. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be set in the camera 193.

[0104] The camera 193 is used to capture still images or videos.

[0105] Video codecs are used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc. In an embodiment of the present application, the electronic device 100 can use the encoding capability of the encoder to encode image processing data and unprocessed video stream images; for example, the video inter-frame anti-shake information and the video stream images that have not undergone inter-frame anti-shake processing are encoded into a video. The electronic device 100 can also use the decoding capability of the decoder to decode the video into image processing data and unprocessed video stream images, so that the image processing data can be used to perform related processing on the unprocessed video stream images.

[0106] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0107] In the embodiment of the present application, after recording is completed, the electronic device 100 can store the recorded video including image processing data (e.g., inter-frame stabilization information) and unprocessed (e.g., not stabilized) recorded stream images based on the internal memory 121. During playback, the electronic device 100 can perform relevant image processing (e.g., inter-frame stabilization) on the recorded video based on the internal memory 121 storage and update the recorded video after stabilization.

[0108] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the gyro sensor 180B can be used to determine the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes). In the embodiment of the present application, the gyro sensor 180B can be used for anti-shake shooting.

[0109] The acceleration sensor 180E can detect the magnitude of acceleration in various directions (generally three axes) of the electronic device 100. In the embodiment of the present application, the acceleration sensor 180E can be used for anti-shake photography.

[0110] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture, etc. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software structure of the electronic device 100 .

[0111] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0112] A layered architecture divides software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system can include: an application layer (applications), an application framework layer, a hardware abstraction layer (HAL), and a kernel layer, where the kernel layer can become a driver layer.

[0113] The application layer can include a series of application packages.

[0114] As shown in Figure 6 , the application package can include camera, gallery and other applications. Among them, in the embodiment of the application, the camera application can support video recording, and during the recording process, the camera application can smoothly display the preview image, and perform deferred processing on part of the processing flow of the video stream image, for example, interframe anti-shake process, multi-frame noise reduction process, etc.

[0115] The gallery application can be used to store recorded videos, wherein the first recorded video stored in the gallery application after recording can include unprocessed frame images and related processing data; taking anti-shake as an example, the recorded video includes frame images without anti-shake processing and video interframe anti-shake information. In the case of first playing or first sharing the video by the gallery application, the electronic device can use the video interframe anti-shake information to perform interframe anti-shake processing on the frame images, and the gallery application plays and updates the recorded video after anti-shake processing.

[0116] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The application framework layer includes some pre-defined functions.

[0117] As shown in Figure 6 , the application framework layer can include a window manager, a content provider, a resource manager, a view system, a notification manager, a camera access interface, and a rendering module, etc.

[0118] The camera access interface enables the application to manage the camera and access the camera device. For example, managing the camera to take pictures, etc.

[0119] The rendering module is responsible for processing graphical data for output and display, including 3D model rendering, image processing, and visual effects generation. The rendering module includes the Open Graphics Library (OpenGL), the Open Source Computer Vision Library (OpenCV), and the Open Computing Language Library (OpenCL).

[0120] In the embodiment of the present application, the rendering module may be OpenGL. During playback of recorded video, the GPU may invoke the rendering capability of OpenGL and perform matrix transformation on the frame image based on the inter-frame anti-shake information of the video, thereby achieving the effect of video anti-shake.

[0121] Hardware abstraction layer, the Android system can load the corresponding library modules for the device hardware, thereby enabling the application framework layer to access the device hardware. The hardware abstraction layer can contain multiple algorithm library modules. For example, image front-end processing module, multi-camera spatial alignment module, motion estimation module, image anti-shake module, image back-end processing module, video function feature module, format conversion module, multi-frame noise reduction module and image calculation module. Among them, the image front-end processing module to the format conversion module can be seen in Figure 3 The relevant description in will not be repeated here.

[0122] The multi-frame noise reduction module can be used to perform noise reduction processing on the preview stream image and the video stream image, and the image calculation module can be used to perform anti-shake processing on the preview stream image according to the anti-shake information between preview frames to obtain an anti-shake preview image.

[0123] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware, making it work. The kernel layer can include display drivers, camera drivers, and sensor drivers, etc., which are not limited in this embodiment of the application.

[0124] The hardware layer can include ISP, GPU, codec, camera, and sensors, etc.

[0125] The following combination Figure 7-11 The image processing method in the embodiment of the present application is described.

[0126] Taking the anti-shake algorithm as an example, Figure 7 FIG. 1 shows a flow chart of an image processing method provided in an embodiment of the present application; FIG. Figure 7 As shown:

[0127] S701 . In response to a video recording operation, obtain image 1 .

[0128] The video recording operation can correspond to Figure 1 In the interface shown in b, a click operation on the shutter control 102 is received. The recording operation can also be a specific gesture, voice command, shortcut operation, etc. for triggering recording, which is not limited in this embodiment of the present application.

[0129] Image 1 may be an image in the preview stream and / or video stream that has been processed by a preset process 1; the preset process 1 is in the processing of the preview stream and / or video stream and is before the anti-shake processing.

[0130] by Figure 7 For example, the modules involved in preview process 1 include but are not limited to: a camera, an image front-end processing module, a multi-camera spatial alignment module, and a motion estimation module; Image 1 may be an image processed by the motion estimation module. The acquisition process of Image 1 may be, for example: the camera captures the original image and passes the original image to the image front-end processing module; the image front-end processing module performs IFE processing on the original image and passes the processed image to the multi-camera spatial alignment module; the multi-camera spatial alignment module spatially aligns images from different cameras at the same time and passes the processed image to the motion estimation module; the motion estimation module compensates the image based on the motion vector to obtain Image 1.

[0131] Optionally, in actual scenarios, the preview stream or the video stream may include more or fewer image processing modules before the anti-shake process, and the present embodiment does not limit this. Image 1 may be the image before it flows into the image anti-shake module.

[0132] S702 : The image stabilization module obtains preview inter-frame stabilization information of the image 1 according to the metadata (meta) of the image 1 and the meta data of the previous frame image.

[0133] Meta data may include sensor data and timestamps. Sensor data includes, for example, gyroscope data, accelerometer data, and visual sensor data; wherein, gyroscope data can be used to record the rotational angular velocity of each frame of image; accelerometer data can be used to record the linear velocity of each frame of image; and visual sensor data is used to estimate the scale of the subject. Timestamps can be used to record the capture time of each frame of image. The preview inter-frame anti-shake information can be data used to perform inter-frame anti-shake on images in the preview stream; in an embodiment of the present application, the preview inter-frame anti-shake information can be in the form of a matrix. The preview inter-frame anti-shake information can be, for example, matrix A.

[0134] It is understandable that due to the chirality requirements of the images in the preview stream, the electronic device needs to process the preview stream images in real time and quickly. Therefore, in step S702, the metadata of the previous frame image can be used to perform inter-frame stabilization on Image 1, resulting in fewer cached frames.

[0135] Specifically, obtaining preview inter-frame anti-shake information of image 1 based on the meta data of image 1 and the meta data of the previous frame image may include the following steps: caching the meta data of image 1 and the meta data of the previous frame image; preprocessing the meta data, and the preprocessing may include filtering, aligning sensor data with image frame timestamps, etc.; obtaining a rotation matrix between the previous frame image and image 1 based on the gyroscope data of image 1 and the gyroscope data of the previous frame image; obtaining a translation matrix between the previous frame image and image 1 based on the accelerometer data of image 1 and the accelerometer data of the previous frame image; obtaining a scaling matrix between the previous frame image and image 1 based on the visual sensor data of image 1 and the visual sensor data of the previous frame image; and calculating a transformation matrix (e.g., matrix A) between the previous frame image and image 1 based on the rotation matrix, the translation matrix, and the scaling matrix.

[0136] Among them, the matrix A can satisfy the following format:

[0137]

[0138] After obtaining the preview inter-frame anti-shake information, the electronic device caches the preview inter-frame anti-shake information of the image 1 and currently does not use the preview inter-frame anti-shake information to process the image 1.

[0139] S703 : The image stabilization module obtains inter-frame video stabilization information of image 1 according to the meta data of image 1 and the meta data of the previous N (N is a positive integer greater than 1) frames of image.

[0140] The video inter-frame anti-shake information may be data used to perform inter-frame anti-shake on images in a video stream; in an embodiment of the present application, the video inter-frame anti-shake information may be in the form of a matrix. The video inter-frame anti-shake information may be, for example, matrix B.

[0141] It is understandable that since the images in the video stream do not need to be displayed in real time but require higher accuracy and stability, in step S703, the meta of the previous N frames can be used to perform inter-frame anti-shake on image 1, where N can be 30, for example.

[0142] Specifically, obtaining inter-frame video stabilization information of image 1 based on the meta data of image 1 and the meta data of the previous N frames of images may include the following steps: caching the meta data of image 1 and the meta data of the previous N frames of images; preprocessing the meta data, which may include filtering, aligning sensor data with image frame timestamps, etc.; for any frame of the previous N frames of images, obtaining a rotation matrix based on the gyroscope data of the frame image and the gyroscope data of image 1; obtaining a translation matrix based on the accelerometer data of the frame image and the accelerometer data of image 1; obtaining a scaling matrix based on the visual sensor data of the frame image and the visual sensor data of image 1; calculating the transformation matrix between the two frames of images based on the rotation matrix, the translation matrix, and the scaling matrix; after traversing the N frames of images, smoothing the N-1 transformation matrices to obtain a processed transformation matrix (for example, matrix B).

[0143] In one possible implementation, smoothing the N-1 transformation matrices can be performed by taking a weighted average of the N-1 transformation matrices. In some embodiments, the weights of the N-1 transformation matrices can be set based on the temporal distance between the corresponding frame image and Image 1, with the weight being negatively correlated with the temporal distance. For example, the greater the difference between the timestamp of the frame image corresponding to the transformation matrix and the timestamp of Image 1, the smaller the weight; conversely, the greater the difference, the greater the weight.

[0144] In another possible implementation, a Kalman filter may be used to smooth the N-1 transformation matrices to reduce noise and uncertainty in the transformation matrices. This embodiment of the present application does not impose any limitation on this.

[0145] Among them, the matrix B can satisfy the following format:

[0146]

[0147] After obtaining the inter-frame anti-shake information, the electronic device caches the inter-frame anti-shake information of image 1 and does not currently use the inter-frame anti-shake information to process image 1. Subsequently, the image anti-shake module transmits image 1 to the image back-end processing module.

[0148] S704: Process image 1 to obtain image 2.

[0149] Image 1 may be an image from the preview stream and / or video stream that has been processed by preset process 2; preset process 2 is performed in the preview stream and / or video stream processing, after the stabilization process. For example, the processing of preset process 2 involves an image backend processing module, a video function feature module, and a format conversion module; the processing may correspond to image backend processing, function feature processing, and format conversion processing. Examples of image backend processing include noise reduction, color processing, and detail enhancement; examples of function feature processing include skin beautification and blur processing; and examples of format conversion processing include converting an RGB format image to a YUV format image.

[0150] It is understandable that after the image stabilization module calculates the preview frame inter-frame stabilization information and the video frame inter-frame stabilization information, the image stabilization module does not perform stabilization processing on Image 1, but passes Image 1 to the subsequent image processing module.

[0151] This is because if the image stabilization module performs stabilization on Image 1, two different images will be obtained: one for the preview stream and one for the video stream. Subsequently, the electronic device needs to use the image backend processing module to the format conversion module to post-process the two images separately. However, in the embodiment of the present application, the image stabilization module does not process Image 1. After passing through the image stabilization module, the image in the preview stream and the video stream remains Image 1. Therefore, the image backend processing module to the format conversion module only needs to process Image 1 once.

[0152] and Figure 4 Comparing the processing flows shown, it can be seen that in the embodiment of this application, the image processing of the preview stream and the image processing of the video stream have more overlap. For example, in the overall image processing flow, the processing steps after the image stabilization process can overlap. This reduces the processing steps of the electronic device, improves image processing efficiency, and reduces computing pressure.

[0153] It should be noted that in the embodiment of the present application, the electronic device may, based on the hand-following characteristics of the preview stream image, preferentially calculate the preview inter-frame anti-shake information, and then calculate the video inter-frame anti-shake information. After the calculation is completed, the image 1 is passed to the image back-end processing module, for example, steps S702, S703 and S704 are executed in sequence. In other scenarios, since the inter-frame anti-shake information does not need to participate in the calculation of step S704, the image anti-shake module may also pass the image 1 to the image back-end processing module after obtaining the meta information, without waiting until the calculation is completed; for example, the image anti-shake module and the image back-end processing module run on different processors respectively, and the image anti-shake module calculates the inter-frame anti-shake information based on the meta data. At the same time, the image back-end processing module processes the image 1; that is, steps S702 and S703 can be executed simultaneously with S704. The embodiment of the present application does not limit the execution order of the relevant steps.

[0154] It should be noted that in step S705, the image anti-shake module can directly pass the image 1 to the subsequent processing flow after calculating the preview inter-frame anti-shake data and the video inter-frame anti-shake data. Alternatively, in step S705, the image anti-shake module can also perform intra-frame correction on the image 1; the intra-frame correction is an internal processing on a single frame image, and the processed preview stream and the video stream still share the intra-frame corrected image 1; the effect achieved is consistent with the above description, and the subsequent embodiments of the present application will give a detailed description of the intra-frame correction and the inter-frame anti-shake (see Figure 9 ).

[0155] S705, the image calculation module performs anti-shake processing on the image 2 using the preview inter-frame anti-shake information to obtain and send a preview image.

[0156] The image calculation module can be an IPE. It should be noted that the image calculation module and the image backend processing module both rely on the functions of the IPE, but they are not the same. Among them, the image calculation module can apply the matrix transformation capability of the IPE; and the image backend processing module can apply the noise reduction, color processing, and detail enhancement capabilities of the IPE.

[0157] The image calculation module can obtain the image 2 from the format conversion module, and obtain the preview inter-frame anti-shake information from the image anti-shake module. The image calculation module performs anti-shake processing on the image 2 using the preview inter-frame anti-shake information. Among them, the process of anti-shake processing can be, for example: the image calculation module obtains the image 2 and the matrix (for example, matrix A) of the image 2; for any pixel region in the image 2, the coordinate matrix of the pixel region is processed using the matrix A and a preset matrix to obtain the anti-shake coordinate matrix 1 of the pixel region, wherein the pixel region includes one or more pixel points; and the image is reconstructed according to the anti-shake coordinate matrix 1 of any pixel region to obtain a preview image.

[0158] After that, the image calculation module can pass the preview image to the display end, which can include a media player, and the media player displays the preview image. At this point, during the video recording process, the electronic device completes the processing flow of the preview image, and displays the preview image on the display screen.

[0159] S706, the encoder encodes the image 2 and the video inter-frame anti-shake information of the image 2 to obtain a video recording video 1, and stores the video recording video 1.

[0160] The video recording 1 includes multiple frames of images, and video interframe stabilization information corresponding to any frame of image. The multiple frames of images can be, for example, image 2. The processing of other frames of images in the video recording 1 can refer to the image 2. The embodiments of the present application do not repeat the description of this. The electronic device can include a media framework (MediaCodec), which is a module provided by Android for encoding and decoding audio and video. It includes encoders and decoders, etc. During video recording, the MediaCodec can create an encoder to encode the image.

[0161] During video recording, the electronic device can perform video encoding and packaging on the video recording 1; during the video encoding and packaging, the video file usually contains multiple types of data, which are set in a specific format to facilitate storage and playback.

[0162] For example, the video recording 1 can include a file type (ftyp), media data (mdat), and a movie media type. The format of the video recording 1 can be as shown in Figure 8 .

[0163] Among them, the file type is used to identify the type of file, help the decoder or player to identify the file format, and determine whether the file content can be correctly parsed and played. The media data can be used to store the encoded file data, and the media data can be, for example, audio, video, and subtitles, etc. The media type can include header information (movie header) and multiple media tracks; the header information can be used to record the global information of the video recording 1, for example, including the time scale, duration, etc. of the file; the media track is used to represent different media streams or data streams.

[0164] In the embodiments of the present application, the video recording 1 can include a video composed of frames of images and video interframe stabilization information of the frames of images. During the encoding of the video, the encoder can also encode the video interframe stabilization information into the video in the form of a media track.

[0165] During the encoding, the encoder can create multiple media tracks and write media data in the multiple media tracks, one media track corresponding to one type of media data. The multiple media tracks can include a video track, an audio track, a subtitle track, and a video interframe stabilization information track, etc.

[0166] Referring to Figure 8: In the recorded video 1, media track 1 can be a video track, and the media data written on the video track can be multiple frames of images (such as image 2). Media track 2 can be an audio track, and the media data written on the audio track can be audio data recorded during the recording process. Optionally, media track 3 can be a subtitle track, and the media data written on the subtitle track can be subtitles; for example, the electronic device supports the AI ​​subtitle function, and the electronic device can convert the audio data into text data in real time during the recording process. Media track 4 can be a video inter-frame stabilization information track, and the media data written on the video inter-frame stabilization information track can be matrix data. Among them, for the video inter-frame stabilization information track, the encoder can write the video inter-frame stabilization information into the video inter-frame stabilization information track based on the timestamp corresponding to the video inter-frame stabilization information.

[0167] Optionally, an embodiment of the present application may include one or more video inter-frame anti-shake information tracks, and / or one or more video tracks. It is understandable that the camera application of the electronic device may support multi-lens recording (front camera and rear camera recording at the same time), picture-in-picture recording, etc., which can achieve multiple results in one recording. After the recording is completed, the electronic device can obtain one or more videos, that is, the recorded video 1 may include one or more videos, and accordingly, the recorded video 1 corresponds to one or more groups of video inter-frame anti-shake information. During encoding, the encoder can create multiple video tracks and video inter-frame anti-shake information tracks. The embodiment of the present application does not limit this.

[0168] After encoding, the electronic device can save the recorded video 1 to a gallery application.

[0169] In an embodiment of the present application, the encoder may encode frame by frame, and after obtaining a frame image and the video inter-frame anti-shake information corresponding to the frame image, encode the frame image and the video inter-frame anti-shake information into corresponding media tracks respectively.

[0170] S707 : In response to the operation for ending the recording, save the recorded video 1 to the gallery application.

[0171] The operation for ending the recording can be, for example, Figure 1 In the interface shown in c, the trigger operation of the end recording control 104 is performed. After the recording is completed, the electronic device can save the recorded video 1 to the gallery application (such as Figure 2 ), and displaying recorded videos 1 as thumbnails in the camera app (as shown in c in Figure 2 (The interface shown in a).

[0172] At this point, the electronic device completes the video preview and recording process. At this point, the frame images in recorded video 1 have not yet been stabilized. Later, if the user needs to view or use recorded video 1, the electronic device may perform stabilization on recorded video 1 to improve the quality of the frame images in recorded video 1. If the user does not view or use recorded video 1, the electronic device may not perform stabilization on recorded video 1 until the electronic device uses recorded video 1 or deletes it.

[0173] Exemplarily, the process of using the recorded video 1 is described below in conjunction with steps S708-S710.

[0174] S708 : In response to the operation for playing the recorded video 1 , the decoder decodes the recorded video 1 .

[0175] The operation for playing the recorded video 1 can be, for example: Figure 2 In the interface shown in c, click on video 1; or, if video 1 is the latest shot and recorded image, Figure 2 In the interface shown in a, a click operation is performed on the thumbnail control 101.

[0176] During video playback, MediaCodec can create a decoder to decode the encoded images. The decoder can decode recorded video 1 into multiple frames and corresponding inter-frame stabilization information. The decoder can then pass the multiple frames and the corresponding inter-frame stabilization information to the GPU.

[0177] S709 : The GPU performs anti-shake processing on the recorded video 1 using the inter-frame anti-shake information, and plays the recorded video 2.

[0178] The GPU is used to process image data. During video playback, the GPU performs image rendering. Electronic devices can reuse the GPU's capabilities to perform stabilization on multiple frames. For example, the GPU can call upon a rendering module to perform texture transformation on the frame image. During this texture transformation, the frame image's preset matrix is ​​processed using inter-frame stabilization information to achieve stabilization.

[0179] The preset matrix (also called texture transformation matrix) can be used to describe the rotation, translation, and scaling of the frame image. In the embodiment of the present application, since the electronic device does not perform editing operations such as rotation, translation, and scaling on the recorded video 1 after obtaining the recorded video 1, the preset matrix of the recorded video 1 can be a unit matrix. The preset matrix is, for example, matrix C, which can meet the following format:

[0180]

[0181] Taking the rendering module as OpenGL and the frame image as image 2 as an example, the anti-shake processing process may include the following steps: obtaining the inter-video frame anti-shake information of image 2 (for example, matrix B) and the preset matrix of image 2 (for example, matrix C); obtaining the target matrix of image 2 based on the inter-video frame anti-shake information and the preset matrix, the target matrix can be used to represent the rotation, translation and scaling of the image after anti-shake, and the target matrix can be the product of matrix B and matrix C; processing image 2 based on the target matrix to obtain image 3.

[0182] The preset matrix can be the mSTMatrix obtained by OpenGL through the SurfaceTexture.getTransformMatrix() function. In one possible implementation, image 2 is processed based on the target matrix to obtain image 3. For example, for any pixel area in image 2, the coordinate matrix of the pixel area is processed using matrix B to obtain the stabilized coordinate matrix 2 of the pixel area, where the pixel area includes one or more pixels; and the image is reconstructed according to the stabilized coordinate matrix 2 of any pixel area to obtain image 3.

[0183] Perform the above operation on any frame image in the video 1 to obtain the video 2. It can be understood that the video 2 can be the video after the video 1 is anti-shake processed, and the video 2 no longer includes the video frame anti-shake information, that is, the format of the video 2 no longer includes the video frame anti-shake information. Figure 8 The video inter-frame stabilization information track is shown.

[0184] S710: Save recorded video 2 in the gallery application and delete recorded video 1.

[0185] After initially playing recorded video 1, the electronic device can obtain recorded video 2 with an anti-shake effect. To reduce processing power consumption and save storage space during subsequent playback of recorded video 1, the electronic device can update recorded video 1 to recorded video 2. Subsequently, when the electronic device receives a trigger operation to play recorded video 2, it can use the normal video playback process to play recorded video 2.

[0186] In this way, part of the image processing process during recording is transferred to the video playback process, reducing the memory resources and computing resources occupied by the recording process, improving the smoothness of the preview stream image, and avoiding the impact of high power consumption on the user's thermal experience.

[0187] Optionally, in step S708, the image stabilization process is described by taking the playback of recorded video 1 as an example. In some scenarios, the user may share recorded video 1 to other devices without viewing recorded video 1. Other devices may not have the ability to convert recorded video 1 into recorded video 2, which may cause other devices to be unable to watch recorded video 1. Therefore, when receiving an operation for sharing recorded video 1, the electronic device may also call the GPU to perform stabilization processing on recorded video 1 to obtain recorded video 2; after saving recorded video 2 to the gallery application, share recorded video 2 to other devices. This process is similar to the process of converting recorded video 1 to recorded video 2 during playback, and the embodiment of the present application will not describe this process in detail.

[0188] Optionally, since the GPU also has the ability to perform matrix transformation on images, the execution subject of step S705 in the embodiment of the present application can also be the GPU. Figure 7 As shown:

[0189] After steps S701-S704, the method further includes:

[0190] S711. The GPU performs anti-shake processing on the image 2 using the preview inter-frame anti-shake information, and obtains and displays a preview image.

[0191] After step S711, the process further includes steps S706-S710.

[0192] This process can be seen in the description of step S705. Step S705 can be understood as performing stabilization on Image 2 during the image processing process, while step S711 can be understood as performing stabilization on Image 2 during the process of rendering and displaying the preview image. Both processes are essentially similar, performing stabilization on Image 2 to produce a preview image.

[0193] In an embodiment of the present application, the electronic device may set a module for anti-shake processing on the preview image according to the respective computing power consumption of the GPU and the image computing module during the video recording process. For example, if the power consumption of using the GPU to process image 2 is less than the power consumption of using the image computing module to process image 2, then after step S704, the electronic device may execute S711; otherwise, after step S704, the electronic device may execute S705. Alternatively, the image computing module may run on the CPU, and the electronic device may set a module for anti-shake processing on the preview image according to the occupancy of the running memory of the CPU and GPU. For another example, if the current running memory occupancy of the CPU is high, then after step S704, the electronic device may execute S711; otherwise, after step S704, the electronic device may execute S705. The embodiment of the present application may also select the corresponding module for processing image 2 according to other strategies, which will not be described in detail in the embodiment of the present application.

[0194] The following combination Figure 9 The anti-shake process in the embodiment of the present application is described. Figure 9 As shown:

[0195] exist Figure 9 In the figure, the dotted rectangle in the upper left corner can be considered the subject being recorded. During the recording process, the electronic device can obtain raw images from frames 1 to 4. After processing the raw images using the image front-end processing module, the multi-camera spatial alignment module, and the motion estimation module, the electronic device obtains Image 1. The image stabilization module can also obtain Image 1.

[0196] During EIS (Electronic Image Stabilization) processing, the stabilization process includes two processes: intra-frame correction and inter-frame stabilization. Intra-frame correction can be used to compensate for jitter within a single frame; inter-frame stabilization can be used to compensate for jitter between two or more frames. The image stabilization module can include three submodules: module 1, module 2, and module 3. Module 1 can be used to process intra-frame correction of images; module 2 can be used to calculate inter-frame stabilization of preview stream images; and module 3 can be used to calculate inter-frame stabilization of recorded stream images.

[0197] The anti-shake processing process is described below with reference to steps S901 to S904.

[0198] Exemplarily, at S901, after obtaining image 1, module 1 may perform intra-frame correction on any frame of image 1 to obtain an intra-frame corrected image 1. Module 1 may identify local displacements in image 1 by analyzing feature points and / or textures of pixel regions in image 1, and correct these displacements.

[0199] For example, Figure 9 In any of the images 1 from the first to the fourth frame, it can be seen that the pixel area where the subject (the dot-filled rectangle) is located in image 1 is deformed due to optical distortion, sensor defects, or other factors during the recording process. For example, radial or tangential distortion causes image distortion. After performing intra-frame correction on image 1, the degree of deformation of the subject in image 1 is reduced, and image quality is improved.

[0200] S902 , module 2 obtains the matrix A of the current frame (preview inter-frame anti-shake information) based on the meta data of the current frame and the meta data of the previous frame.

[0201] S903 , module 3 obtains the matrix B (video inter-frame anti-shake information) of the current frame based on the meta data of the current frame and the meta data of the previous 30 frames.

[0202] S904 : The electronic device processes the image 1 after inter-frame correction to obtain image 2 .

[0203] Steps S902-S904 may refer to the relevant descriptions in steps S702-S704 and will not be repeated here.

[0204] S905 . During the video recording process, use matrix A to perform anti-shake processing on image 2 to obtain a preview image.

[0205] S906 : During playback, use matrix B to perform anti-shake processing on image 2 to obtain image 3.

[0206] Steps S905-S906 may refer to the relevant descriptions in steps S705-S710 and will not be repeated here.

[0207] It is understandable that during the video recording process, the shaking and / or movement of the user's handheld electronic device may affect the relative position of the subject in the image and the shooting angle. The shooting angles of the subjects in each frame image are not the same. For example, after intra-frame correction, the subject in the first frame image 1 is rectangular, the subject in the second frame image 1 is trapezoidal, the subject in the third frame image 1 is trapezoidal, and the subject in the fourth frame image 1 is trapezoidal, etc. In addition, the position of the subject in each frame image is not the same. For example, after intra-frame correction, the subject in the first frame image 1 is biased to the right, the subject in the second frame image 1 is biased to the left, the subject in the third frame image 1 is biased to the lower right, and the subject in the fourth frame image 1 is biased to the upper right, etc.

[0208] Inter-frame anti-shake can obtain inter-frame anti-shake data through the meta data of two or more adjacent frames of images; the inter-frame anti-shake data can be used to transform the coordinates of the subject in the image through operations such as rotation, scaling, and translation, thereby reducing the difference in perspective and position of the subject between adjacent images. For example, Figure 9 As shown, during playback, after inter-frame stabilization, the subject in the first to fourth frames of the image 3 is roughly centered and has roughly the same viewing angle, thereby achieving the effect of reducing shaking and improving image stability.

[0209] It should be noted that the above embodiments use the anti-shake algorithm as an example to illustrate the image processing method provided by the embodiments of the present application. The image processing method shown in the embodiments of the present application can also be applied to the processing of other multi-frame image algorithms, such as: multi-frame noise reduction algorithm, HDR effect processing, blur algorithm, video enhancement algorithm and super-resolution reconstruction algorithm.

[0210] The following uses a multi-frame noise reduction algorithm as an example to illustrate possible implementations and the image processing method provided in the embodiments of the present application.

[0211] Figure 10 A multi-frame noise reduction process in a possible implementation is shown.

[0212] It is understandable that in Figure 4 Based on the illustrated embodiment, the electronic device may further include a multi-frame noise reduction module, which may include a preview multi-frame noise reduction module and a video multi-frame noise reduction module. The preview multi-frame noise reduction module may be used to perform noise reduction on preview stream images, while the video multi-frame noise reduction module may be used to perform noise reduction on recorded stream images.

[0213] Similar to the anti-shake process, during the noise reduction process, due to the chirality requirements of the preview stream image and the high quality requirements of the video stream image, the preview multi-frame noise reduction module can cache a small number of frame images to perform multi-frame noise reduction on the preview stream image, for example, 1-5 frames; the video multi-frame noise reduction module can cache a larger number of frame images to perform multi-frame noise reduction on the video stream image, for example, 5-10 frames.

[0214] After the camera collects the raw data, the electronic device can process the raw image through the image front-end processing module and the multi-camera spatial alignment module to obtain the processed image (see Figure 4 The process shown, Figure 10 (not shown). At this point, since multi-frame noise reduction involves processing both the preview stream and the recorded video stream, the subsequent processing process is divided into two paths. One path is processed by the preview image stabilization module, the preview multi-frame noise reduction module, the image back-end processing module, the video function and characteristics module, and the format conversion mode to obtain a preview image; the other path is processed by the motion estimation module, the video image stabilization module, the video multi-frame noise reduction module, the image back-end processing module, the video function and characteristics module, and the format conversion mode to obtain a recorded video.

[0215] It can be seen that in the possible image processing process, the multi-frame noise reduction processing position is relatively early, and the preview stream image and the video stream image are separated too early. As a result, multiple modules need to process the preview stream image and the video stream image separately. There are many processing flows, which increases the memory amount and power consumption.

[0216] Figure 11 FIG. 4 shows an image processing process of multi-frame noise reduction provided by an embodiment of the present application, such as Figure 11 As shown:

[0217] S1101. In response to a video recording operation, obtain image 4.

[0218] This step may refer to the relevant description in step S701 , wherein the image 4 may be understood as the image processed by the preset process 1 and the image stabilization module.

[0219] Optionally, the image stabilization process and the multi-frame noise reduction process can also be associated. For example, after processing by the preset process 1 (for example, the camera, the image front-end processing module, the multi-camera spatial alignment module, and the motion estimation module), image 4 is obtained. The image stabilization module uses the steps shown in steps S702 and S703 to extract the inter-frame stabilization information, and does not use the inter-frame stabilization information to perform stabilization processing on image 4. Afterwards, the image stabilization module passes image 4 to the multi-frame noise reduction module; in addition, the image stabilization module can also pass the preview inter-frame stabilization information to the image calculation module, and pass the video inter-frame stabilization information to the encoder.

[0220] S1102 : The multi-frame noise reduction module obtains preview noise reduction information of image 4 based on the meta data of image 4 and the meta data of the previous a (a is a positive integer) frames of image.

[0221] Metadata can include sensor data, timestamps, and shooting parameters. Shooting parameters can include exposure time, ISO value, aperture size, and other information. Preview noise reduction information can be used to perform multi-frame noise reduction on images in the preview stream. This information can include noise model parameters. Noise model parameters can be used to describe the noise characteristics in the image, such as the mean and variance of the noise. These parameters can be used to determine the strength and effectiveness of the multi-frame noise reduction algorithm.

[0222] The noise model parameters may be obtained according to shooting parameters and / or sensor characteristics, wherein the sensor characteristics include, for example, the sensor type, sensor size, pixel size, etc.

[0223] After obtaining the preview noise reduction information, the electronic device may cache the preview noise reduction information.

[0224] S1103 , the multi-frame noise reduction module obtains video noise reduction information of image 4 according to the meta data of image 4 and the meta data of the previous b (b is a positive integer greater than a) frames of image.

[0225] The video noise reduction information may be data used to perform multi-frame noise reduction on images in a video stream.

[0226] It is understood that this process is similar to step S1102, but because the images in the video stream do not need to be displayed in real time but require higher accuracy and stability, in step S1103, the meta of the previous b frames can be used to perform multi-frame fusion on image 4, and the value of b is greater than the value of a.

[0227] After obtaining the above-mentioned video noise reduction information, the electronic device may cache the video noise reduction information and transmit the image 4 to the image back-end processing module.

[0228] S1104 , process image 4 to obtain image 5.

[0229] Image 5 may be an image from the preview stream and / or video stream that has been processed by preset process 2. Preset process 2 is performed on the preview stream and / or video stream, after multi-frame noise reduction. For example, the processing of preset process 2 involves an image backend processing module, a video function feature module, and a format conversion module. The processing may correspond to image backend processing, function feature processing, and format conversion processing. Examples of image backend processing include noise reduction, color processing, and detail enhancement; examples of function feature processing include skin beautification and blurring; and examples of format conversion include converting an RGB image to a YUV image.

[0230] It is understandable that after the multi-frame noise reduction module calculates the preview noise reduction information and the video noise reduction information, the multi-frame noise reduction module does not perform noise reduction processing on the image 4, but passes the image 4 to the subsequent image processing module.

[0231] This is because: if the multi-frame noise reduction module performs multi-frame noise reduction on image 4, two different images will be obtained: one for the preview stream and one for the video stream. Subsequently, the electronic device needs to use the image back-end processing module to the format conversion module to post-process the two images separately. However, in the embodiment of the present application, the multi-frame noise reduction module does not perform noise reduction on image 1. After the multi-frame noise reduction module, the image in the preview stream and the video stream remains image 4. Therefore, the image back-end processing module to the format conversion module only needs to process image 4 once.

[0232] and Figure 10 Comparing the processing flows shown, it can be seen that in the embodiment of this application, the image processing of the preview stream and the image processing of the recorded stream have more overlap. For example, in the overall image processing flow, the processing steps after the multi-frame noise reduction process can overlap. This reduces the processing steps of the electronic device, improves image processing efficiency, and reduces computing pressure.

[0233] S1105 : The image calculation module or GPU performs multi-frame noise reduction processing on the image 4 using the preview noise reduction information to obtain and display a preview image.

[0234] The image computing module may be an IPE. For example, the IPE may have the capabilities of matrix transformation and image denoising.

[0235] After obtaining the preview noise reduction information and image 5, the image calculation module or GPU can perform noise reduction processing on image 5 based on the noise model parameters in the preview noise reduction information to obtain a noise-reduced preview image. The electronic device then displays the preview image.

[0236] S1106 : The encoder encodes the image 5 and the video noise reduction information of the image 5 to obtain the recorded video 3 , and stores the recorded video 3 .

[0237] Video 3 includes multiple frames of images and video noise reduction information corresponding to any frame of image. Multiple frames of image can be, for example, image 5. The processing process of other frames of image in video 3 can refer to image 5. This embodiment of the application will not be repeated here.

[0238] During the video recording process, the electronic device may encode and package the video 3. During the video encoding and packaging process, the video file usually contains multiple types of data, which are set into a specific format for easy storage and playback. For example, the recorded video 3 may include: file type, media data, and media type. This process can be referred to Figure 8 The description of the embodiment shown and step S706 will not be repeated here.

[0239] It should be noted that, unlike the image stabilization process, in this embodiment of the present application, the recorded video 3 may include multiple media tracks, such as a video track, an audio track, a subtitle track, and a video noise reduction information track. The media data written to the video noise reduction information track may be noise model parameters. Specifically, for the video noise reduction information track, the encoder may write the video noise reduction information to the video noise reduction information track based on the timestamp corresponding to the video noise reduction information.

[0240] Optionally, as described above, the image stabilization process and the multi-frame noise reduction process can be associated. In step S1106, the media track of the recorded video 3 may also include a video inter-frame stabilization information track, so that the GPU can perform inter-frame stabilization and multi-frame noise reduction on image 5 during subsequent playback.

[0241] S1107 : In response to the operation for ending the recording, save the recorded video 3 to the gallery application.

[0242] This process can refer to the relevant description in step S707 and will not be repeated here.

[0243] Exemplarily, the process of using the recorded video 1 is described below in conjunction with steps S1108-S1110.

[0244] S1108 : In response to the operation for playing the recorded video 3 , the decoder decodes the recorded video 3 .

[0245] After decoding the recorded video 3, multiple frames of images and the video noise reduction information corresponding to any frame of image can be obtained. This process can be referred to the relevant description in step S708 and will not be repeated here.

[0246] S1109 : The GPU performs noise reduction processing on the recorded video 3 using the video noise reduction information, and plays the recorded video 4.

[0247] The GPU can be used to process image data. For example, the GPU can perform noise reduction on images during video playback. Electronic devices can reuse the GPU's capabilities to perform noise reduction on multiple frames. For example, the GPU can call a noise reduction algorithm API to perform noise reduction on a frame. After the GPU performs noise reduction on image 5, image 6 is obtained. The electronic device then obtains recorded video 4, which includes image 6. The electronic device can then use the GPU and a player to play recorded video 4.

[0248] Optionally, the codec may also include a filter for image noise reduction. In step S1109, during the playback of the recorded video, the decoder may use the video noise reduction information to perform noise reduction processing on the recorded video 3 to obtain the recorded video 4.

[0249] S1110 . Save recorded video 4 in the gallery application and delete recorded video 3.

[0250] The process can be referred to the relevant description in step S710 and will not be repeated here.

[0251] In this way, part of the image processing process during recording is transferred to the video playback process, reducing the memory resources and computing resources occupied by the recording process, improving the smoothness of the preview stream image, and avoiding the impact of high power consumption on the user's thermal experience.

[0252] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0253] The image processing method according to the embodiment of the present application has been described above. The following describes the apparatus for performing the above-described image processing method provided in the embodiment of the present application. Those skilled in the art will appreciate that the method and apparatus may be combined and referenced with each other, and the relevant apparatus provided in the embodiment of the present application may perform the steps in the above-described image processing method.

[0254] like Figure 12 As shown, the image processing apparatus 1200 can be used in a communication device, circuit, hardware component, or chip, and includes a display unit 1201 and a processing unit 1202. The display unit 1201 is used to support the display step performed by the image processing apparatus 1200; the processing unit 1202 is used to support the information processing step performed by the image processing apparatus 1200.

[0255] In a possible implementation, the image processing apparatus 1200 may also include a communication unit 1203. Specifically, the communication unit is used to support the image processing apparatus 1200 in executing the steps of sending and receiving data. The communication unit 1203 may be an input or output interface, pin, or circuit.

[0256] In one possible embodiment, the image processing apparatus may further include a storage unit 1204. The processing unit 1202 and the storage unit 1204 are connected via a circuit. The storage unit 1204 may include one or more memories, which may be devices in one or more devices or circuits for storing programs or data. The storage unit 1204 may exist independently and be connected to the processing unit 1202 of the image processing apparatus via a communication circuit. The storage unit 1204 may also be integrated with the processing unit 1202.

[0257] The storage unit 1204 can store computer-executable instructions for the method in the terminal device, so that the processing unit 1202 executes the method in the above embodiment. The storage unit 1204 can be a register, a cache, or a RAM, etc. The storage unit 1204 can be integrated with the processing unit 1202. The storage unit 1204 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions. The storage unit 1204 can be independent of the processing unit 1202.

[0258] The image processing method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include electronic devices. The specific device form of the electronic devices can refer to the above related descriptions and will not be repeated here.

[0259] An embodiment of the present application provides an electronic device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the above method.

[0260] The present embodiment provides a chip system comprising at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is configured to execute a computer program or instruction to perform the above-described method. The implementation principles and technical effects thereof are similar to those of the above-described related embodiments and are not further described here.

[0261] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0262] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium intended to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and optical disc as used herein include optical disc, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0263] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0264] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Used in electronic equipment, including: During the video recording process, a first image is obtained; Performing a first processing on the first image to obtain a second image; the first processing is used to process the single-frame image using intra-frame information of the single-frame image; The second image is subjected to a second processing according to the first information to obtain and display a third image, and the second information and the second image are saved; wherein the first information is obtained based on the association information between the first image and the a-frame image before the first image, and the second information is obtained based on the association information between the first image and the b-frame image before the first image, a and b are both positive integers, and a is less than b, the third image is used for preview display during recording, and the second information and the second image are used for performing the second processing when obtaining the recorded video.

2. The method according to claim 1, characterized in that The saving of the second information and the second image includes: The second information and the second image are encoded to obtain and save a first video; wherein the first video includes a first media track and a second media track; during the recording process, the second image is stored in the first media track, and the second information is stored in the second media track.

3. The method according to claim 2, characterized in that After obtaining the first video, the method further includes: In response to an operation for playing the first video, performing the second processing on the second image according to the second information to obtain and display a fourth image; The first video is replaced with a second video, the second video includes the fourth image, and the second video does not include the second image and the second information.

4. The method according to any one of claims 1 to 3, characterized in that The second processing includes a first electronic image stabilization (EIS) processing; the first EIS processing is used to perform an anti-shake processing on the first image based on association information between consecutive frame images; the first information is a first matrix obtained based on metadata (meta) of the first image and metadata of a frame image preceding the first image; The electronic device includes a first module, and performing a second process on the second image according to the first information to obtain a third image includes: The first module performs a first EIS processing on the second image based on a preset matrix of the second image and the first matrix to obtain the third image; The first module supports coordinate transformation of the second image, and the first module includes a graphics processor GPU or an image processing engine IPE.

5. The method according to claim 4, characterized in that The second information is a second matrix obtained based on the meta data of the first image and the meta data of a b-frame image preceding the first image; the electronic device further includes an encoder and a decoder, and the first video is encoded by the encoder; The performing the second processing on the second image according to the second information to obtain a fourth image includes: The decoder decodes the first video to obtain the second image and a second matrix; wherein the second image is decoded from the first media track, the second matrix is ​​decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second matrix; The GPU performs a first EIS process on the second image based on a preset matrix of the second image and the second matrix to obtain the fourth image.

6. The method according to claim 4 or 5, characterized in that The meta data of the first image includes one or more of the following: sensor data and a timestamp when the first image is acquired; the meta data of the i-th frame of the b-frame image preceding the first image includes one or more of the following: sensor data, a timestamp, and shooting parameters when the i-th frame of the image is acquired, where i is less than b; The second matrix is ​​obtained by: obtaining a rotation matrix, a scaling matrix, and a translation matrix based on the sensor data of the i-th frame image and the sensor data of the first image; Obtaining a transformation matrix based on the rotation matrix, the scaling matrix, and the translation matrix; After traversing the b-frame images preceding the first image, multiple transformation matrices are processed into the second matrix.

7. The method according to any one of claims 4 to 6, characterized in that The electronic device further includes a second module, a third module, and a fourth module; and performing a first process on the first image to obtain a second image includes: After obtaining the first image, the second module performs pre-processing in the first processing on the first image to obtain a fifth image; the pre-processing includes one or more of the following: image front-end IFE processing, spatial alignment SAT processing, motion estimation GME processing, and second EIS processing; the second EIS processing is used to perform anti-shake processing on the first image using intra-frame information of the first image; The third module calculates and obtains the first matrix and the second matrix; The fourth module performs post-processing in the first processing on the fifth image to obtain the second image; the post-processing includes one or more of the following: IPE processing, skin beautification processing, blurring processing and image format conversion processing.

8. The method according to any one of claims 1 to 3, characterized in that The second processing includes noise reduction processing; the first information is a first noise reduction parameter obtained based on the meta data of the first image and the meta data of the a-frame image before the first image; The electronic device includes a fifth module, and performing a second process on the second image according to the first information to obtain a third image includes: The fifth module performs noise reduction processing on the second image based on the first noise reduction parameter to obtain the third image; the fifth module includes: a GPU or an IPE.

9. The method according to claim 8, characterized in that The second information is a second noise reduction parameter obtained based on the meta data of the first image and the meta data of a b-frame image preceding the first image; the electronic device further includes an encoder and a decoder, and the first video is encoded by the encoder; The performing the second processing on the second image according to the second information to obtain a fourth image includes: The decoder decodes the first video to obtain the second image and the second noise reduction parameter; wherein the second image is obtained by decoding the first media track, the second noise reduction parameter is obtained by decoding the second media track, and the timestamp of the second image corresponds to the timestamp of the second noise reduction parameter; The sixth module processes the second image based on the second noise reduction parameter to obtain the fourth image, and the sixth module includes a GPU or a decoder.

10. The method according to claim 8 or 9, characterized in that The electronic device further includes a seventh module, an eighth module, and a ninth module; and performing a first process on the first image to obtain a second image includes: After obtaining the first image, the seventh module performs pre-processing in the first processing on the first image to obtain a sixth image; the pre-processing includes one or more of the following: IFE processing, SAT processing, GME processing, and EIS processing; The eighth module obtains the first noise reduction parameter and the second noise reduction parameter; The ninth module performs post-processing in the first processing on the sixth image to obtain the second image; the post-processing includes one or more of the following: IPE processing, skin beautification processing, blurring processing and image format conversion processing.

11. An electronic device, characterized in that: include: processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A chip system, characterized in that: The system comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is configured to run a computer program or instruction to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 10.

Images