Image processing methods, electronic devices, storage media, chip systems, and computer program products

Abstract

This application provides an image processing method, electronic device, storage medium, chip system, and computer program product, relating to the field of terminal technology. The method includes: obtaining a first image during recording; performing a first processing on the first image to obtain a second image; the first processing being used to process the single-frame image using intra-frame information; performing a second processing on the second image based on the first information to obtain and display a third image; and saving the second information and the second image; wherein the first information is obtained based on the correlation information between the first image and a preceding frame, and the second information is obtained based on the correlation information between the first image and b preceding frames, where a is less than b. In this way, during recording, the preview stream image is processed in real-time to ensure the responsiveness of the preview image; simultaneously, the recorded stream image is saved and processed later to reduce memory pressure during recording.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, storage medium, chip system, and computer program product. Background Technology

[0002] Electronic devices can support camera functions, allowing users to take photos and / or record videos. Taking video recording as an example, after launching the camera application, in response to a trigger operation on the recording control, the electronic device can display a preview on the screen and save the preview, thus obtaining the recorded video.

[0003] One possible consequence is that during video recording, the electronic device's display may experience poor smoothness, negatively impacting the user experience. Summary of the Invention

[0004] This application provides an image processing method, electronic device, storage medium, chip system, and computer program product, applied in the field of terminal technology. In a video recording scenario, the image processing process is segmented; real-time image processing of the preview stream improves the responsiveness of the preview image; and delayed image processing of the video stream reduces memory pressure during recording.

[0005] In a first aspect, embodiments of this application propose an image processing method. The method includes: obtaining a first image during video recording; performing a first processing on the first image to obtain a second image; the first processing being used to process the single-frame image using intra-frame information; performing a second processing on the second image based on the first information to obtain and display a third image; and saving the second information and the second image; wherein the first information is obtained based on the association information between the first image and a preceding frame image, the second information is obtained based on the association information between the first image and b preceding frame images, a and b are both positive integers, and a is less than b; the third image is used for preview display during video recording; and the second information and the second image are used to perform the second processing when obtaining the recorded video.

[0006] The first image may be, for example... Figure 7 or Figure 11 The raw image captured by the camera. The first processing step may be, for example, Figure 7 The processing performed by the image front-end processing module, multi-camera spatial alignment module, motion estimation module, image back-end processing module, video feature module, and format conversion module; or, Figure 11 The processing performed by the image front-end processing module, multi-camera spatial alignment module, motion estimation module, image stabilization module, image back-end processing module, video function feature module, and format conversion module. The second image can be, for example, image 2 or image 5. The third image can be, for example... Figure 7or Figure 11 The preview image in the video stream. The first information can be information about the preview stream image, such as preview inter-frame stabilization information or preview noise reduction information. The second information can be information about the recorded video stream image, such as video inter-frame stabilization information or video noise reduction information. 'a' can be, for example... Figure 7 In the illustrated embodiment, 1;b can be, for example... Figure 7 In the illustrated embodiment, N (N is, for example, 30); a can be, for example... Figure 11 In the illustrated embodiment, a and b can be, for example... Figure 11 In the illustrated embodiment, b. It can be seen that the value of a is less than the value of b.

[0007] It should be noted that the embodiments of this application Figure 7 The second processing method, inter-frame stabilization, is used as an example to illustrate the solution. Figure 11 The solution will be illustrated using multi-frame noise reduction as an example of the second processing step. In real-world scenarios, electronic devices may also postpone the second processing step, which involves inter-frame processing, in multiple video stream image processing steps. This is only used here for the sake of illustrating the segmented processing architecture. Figure 11 The image stabilization module is defined as the first processing step, and the image stabilization module can also be referred to... Figure 7 The relevant descriptions in the illustrated embodiments are also processed later. Similarly, the second processing could also be HDR effect processing, video enhancement algorithms, super-resolution reconstruction algorithms, and / or bokeh algorithm processing involved in video functional feature modules, etc. This application's embodiments will not elaborate on these aspects further.

[0008] In addition, the first processing also includes some multi-frame processing that is not used to distinguish between the preview stream and the recording stream. For example, in some multi-frame processing, the number of frame images cached for processing the preview stream is roughly the same as the number of frame images cached for processing the recording stream. In this case, the first processing will also not split the images (splitting can be understood as dividing the images into preview stream and recording stream).

[0009] In this way, during the recording process, the electronic device processes the preview stream images in real time to ensure the smoothness of the preview images; at the same time, the recorded stream images are saved first and processed later to reduce memory pressure during recording, thereby reducing the occurrence of problems such as stuttering in the preview and high device temperature.

[0010] Saving the second information and the second image includes: encoding the second information and the second image to obtain and save a first video (e.g., video recording 1 or video recording 3); wherein the first video includes a first media track (e.g., a video track) and a second media track (e.g., a video frame stabilization information track or a video noise reduction information track); during the recording process, the second image is stored in the first media track, and the second information is stored in the second media track.

[0011] In this way, electronic devices can encode the second information and the second image into a single video, improving the synchronization between the second image and the second information, simplifying the management and transmission of media resources, and improving transmission efficiency.

[0012] Optionally, after obtaining the first video, the method further includes: responding to an operation for playing the first video (e.g., Figure 2 The triggering operation for thumbnail control 101 in the interface shown in Figure a, or Figure 2 In the interface shown in c), the trigger operation for video 201 is performed. Based on the second information (such as video frame stabilization information or video noise reduction information), the second image is processed in the second way (such as frame stabilization or multi-frame noise reduction) to obtain and display the fourth image (such as image 3 or image 6). After traversing and processing the images in the first video, the first video is replaced with the second video (such as video recording 2). The second video includes the fourth image, but does not include the second image and the second information (it is understood that after the replacement, the electronic device no longer saves video recording 1 to reduce memory usage).

[0013] In this way, when a user uses the first video, the electronic device can perform multi-frame processing on the images in the first video, such as inter-frame stabilization or multi-frame noise reduction, thereby reducing memory pressure during recording. At the same time, when the user views the recorded video later, they can watch a video with better image quality. When the user does not use the first video, the electronic device does not perform multi-frame processing on the images in the first video, further reducing image computing pressure.

[0014] Optional, see Figure 7 The processing flow shown includes a second processing step comprising a first electronic image stabilization (EIS) process (e.g., inter-frame stabilization); the first EIS process is used to perform image stabilization on a first image based on the correlation information between consecutive frame images; the first information is a first matrix (e.g., preview inter-frame stabilization information, matrix A) obtained based on the meta data of the first image and the meta data of the a-frame images preceding the first image; the electronic device includes a first module that performs a second processing on a second image based on the first information to obtain a third image, including: the first module performs the first EIS processing on the second image based on a preset matrix (e.g., identity matrix, matrix C) and the first matrix to obtain the third image; the first module supports coordinate transformation of the second image, and the first module includes a GPU or IPE (e.g., Figure 7 The first module is configured as either a GPU or an IPE, depending on the power consumption or memory usage of the electronic device.

[0015] In this way, a segmented processing architecture is adopted during the recording process. The first module can perform image stabilization processing on the preview stream image through the inter-frame stabilization information to ensure the smoothness of the preview image.

[0016] Optionally, the second information is a second matrix (e.g., video frame stabilization information, matrix B) obtained based on the meta data of the first image and the meta data of the b-frame images preceding the first image; the electronic device also includes an encoder and a decoder, and the first video is encoded by the encoder; the second image is processed according to the second information to obtain a fourth image, including: the decoder decodes the first video to obtain the second image and the second matrix; wherein, the second image is decoded from the first media track, the second matrix is ​​decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second matrix; the GPU performs a first EIS processing on the second image based on a preset matrix (e.g., identity matrix, matrix C) and the second matrix to obtain the fourth image; it is understood that the processing during recording may involve IPE; during video playback, the power consumption of calling IPE is relatively large, i.e. Figure 7 The image processing module in the preview stream is not suitable for processing the recorded stream images during playback. Here, the GPU can be used instead of IPE.

[0017] In this way, when using the recorded video later, the segmented processing architecture allows the electronic device to perform image stabilization on the recorded video through inter-frame image stabilization information, thereby improving the image quality of the recorded video.

[0018] Optionally, the meta data of the first image includes one or more of the following: sensor data, timestamp, and shooting parameters when the first image was acquired; the meta data of the i-th frame image in the b-frame images preceding the first image includes one or more of the following: sensor data, timestamp, and shooting parameters when the i-th frame image was acquired, where i is less than b; the second matrix is ​​obtained in the following manner: based on the sensor data of the i-th frame image and the sensor data of the first image (e.g., the sensor data of image 1 in step S703; it is understood that the first image can be the original image, and the sensor data of the image during processing is the same as the sensor data of the original image of the same frame), the rotation matrix, scaling matrix, and translation matrix are obtained respectively; the transformation matrix is ​​obtained based on the rotation matrix, scaling matrix, and translation matrix; after traversing the b-frame images preceding the first image, the multiple transformation matrices are processed into the second matrix (see step S703).

[0019] In this way, during the recording process, the electronic device can calculate and save the image stabilization information (such as inter-frame image stabilization information) used to describe the recorded video; so that the image stabilization information can be used to perform image stabilization processing on the recorded video in subsequent playback scenarios, thereby improving the image quality of the recorded video.

[0020] Optionally, the electronic device may also include a second module (e.g. Figure 7The flowchart shows the processing modules preceding the image stabilization module, the third module (e.g., modules 2 and 3 in the image stabilization module for calculating inter-frame stabilization data), and the fourth module (e.g., ... Figure 7 The process shown is a processing module located after the image stabilization module; the first image is processed to obtain a second image, including: after obtaining the first image, the second module performs preprocessing on the first image to obtain a fifth image (e.g., image 1); the preprocessing includes one or more of the following: IFE processing (corresponding to the image front-end processing module), SAT processing (corresponding to the multi-camera spatial alignment module), GME processing (corresponding to the motion estimation module) and second EIS processing (e.g., module 1 in the image stabilization module used to implement intra-frame correction); the second EIS processing is used to perform image stabilization processing on the first image using the intra-frame information of the first image; the third module calculates a first matrix (calculated by module 2) and a second matrix (calculated by module 3); the fourth module performs postprocessing on the fifth image to obtain a second image (e.g., image 2); the postprocessing includes one or more of the following: IPE processing (corresponding to the image back-end processing module), skin smoothing processing (corresponding to the video function feature module), bokeh processing (corresponding to the video function feature module) and image format conversion processing (corresponding to the format conversion module).

[0021] In this way, the electronic device can adopt a segmented processing architecture. When performing the second processing on the image, the data required for the second processing of the preview stream and the video stream are calculated first, and the image is not processed in the second way. This avoids the splitting of the preview stream and the video stream due to the different second processing of the preview stream and the video stream. After completing the first processing, the second information is used to split the image into the preview stream and the video stream. This reduces the number of processing modules involved after splitting, thereby reducing the power consumption of image processing.

[0022] Optional, see Figure 11 The processing flow shown includes a second processing step that further includes noise reduction processing; the first information is a first noise reduction parameter (e.g., preview noise reduction information) obtained based on the meta data of the first image and the meta data of the a-frame image preceding the first image; the electronic device includes a fifth module that performs a second processing on the second image (e.g., image 5) according to the first information to obtain a third image (e.g., preview image), including: the fifth module performs noise reduction processing on the second image based on the first noise reduction parameter to obtain the third image; the fifth module includes: a GPU or an IPE.

[0023] In this way, a segmented processing architecture is adopted during the recording process. The fifth module can perform noise reduction processing on the preview stream image through the preview noise reduction information to ensure the smoothness of the preview image.

[0024] Optionally, the second information is a second noise reduction parameter (e.g., video noise reduction information) obtained based on the meta data of the first image and the meta data of the b-frame images preceding the first image; the electronic device also includes an encoder and a decoder, the first video being encoded by the encoder; the second image is processed according to the second information to obtain a fourth image, including: the decoder decoding the first video to obtain the second image and the second noise reduction parameter; wherein, the second image is decoded from the first media track, the second noise reduction parameter is decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second noise reduction parameter; a sixth module processes the second image based on the second noise reduction parameter to obtain the fourth image, the sixth module including a GPU or a decoder. Here, a filter can be set in the GPU or encoder / decoder, and multi-frame noise reduction processing can be performed using the GPU or decoder during the playback of the first video. Optionally, the electronic device can also use an encoder to perform multi-frame noise reduction processing on the recorded video during the recording process, and this embodiment of the application does not limit this.

[0025] In this way, when using the recorded video later, the segmented processing architecture allows electronic devices to perform multi-frame noise reduction processing on the recorded video using video noise reduction information, thereby improving the image quality of the recorded video.

[0026] Optionally, the electronic device may also include a seventh module (e.g. Figure 11 The flowchart shows the processing modules located before the multi-frame noise reduction module, the eighth module (e.g., the multi-frame noise reduction module), and the ninth module (e.g., the processing module before the multi-frame noise reduction module). Figure 11 The process shown is as follows: a seventh module performs a first processing step on the first image to obtain a second image, including: after obtaining the first image, the seventh module performs preprocessing on the first image to obtain a sixth image (e.g., image 4); the preprocessing includes one or more of the following: IFE processing (corresponding to the image front-end processing module), SAT processing (corresponding to the multi-camera spatial alignment module), GME processing (corresponding to the motion estimation module), and EIS processing (corresponding to the image stabilization module); the eighth module obtains the first noise reduction parameter and the second noise reduction parameter; the ninth module performs postprocessing on the sixth image to obtain the second image (e.g., image 5); the postprocessing includes one or more of the following: IPE processing (corresponding to the image back-end processing module), skin smoothing processing (corresponding to the video function feature module), bokeh processing (corresponding to the video function feature module), and image format conversion processing (corresponding to the format conversion module).

[0027] In this way, the electronic device can adopt a segmented processing architecture. When performing the second processing on the image, the data required for the second processing of the preview stream and the video stream are calculated first, and the image is not processed in the second way. This avoids the splitting of the preview stream and the video stream due to the different second processing of the preview stream and the video stream. After completing the first processing, the second information is used to split the image into the preview stream and the video stream. This reduces the number of processing modules involved after splitting, thereby reducing the power consumption of image processing.

[0028] Secondly, embodiments of this application provide an electronic device, which may also be referred to as a terminal device, terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on.

[0029] The electronic device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the method as described in the first aspect.

[0030] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method as described in the first aspect.

[0031] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when run, causes a computer to perform the method as described in the first aspect.

[0032] Fifthly, embodiments of this application provide a chip system including at least one processor and a communication interface, the communication interface and at least one processor being interconnected via a line, the at least one processor being used to run computer programs or instructions to perform the method as described in the first aspect.

[0033] It should be understood that the second to fifth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0034] Figure 1 A schematic diagram of the recording process provided in an embodiment of this application;

[0035] Figure 2 A schematic diagram of the interface for playing recorded videos provided in an embodiment of this application;

[0036] Figure 3 This is a schematic diagram of one possible image stabilization process.

[0037] Figure 4 This is a schematic diagram of another possible image stabilization process in a possible implementation;

[0038] Figure 5 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application;

[0039] Figure 6 A schematic diagram of the software structure of the electronic device 100 provided in the embodiments of this application;

[0040] Figure 7 This is a schematic diagram of an image stabilization process provided in an embodiment of this application;

[0041] Figure 8 This is a schematic diagram of the structure of the video recording 1 provided in an embodiment of this application;

[0042] Figure 9 A schematic diagram illustrating inter-frame stabilization and intra-frame correction provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of one possible image noise reduction process.

[0044] Figure 11 This is a schematic diagram of an image noise reduction process provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0046] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0047] 1. Electronic equipment

[0048] The electronic devices in this application embodiment may include handheld devices with image processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices (such as vehicle infotainment systems), wearable devices, electronic devices in 5G, or future evolutions of public land mobile communications. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0049] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as hearing aids, glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0050] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects through communication technology, thereby realizing the intelligent interconnection of humans and machines and the interconnection of things.

[0051] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0052] 2. Other terms

[0053] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0055] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the display interface provided in the embodiments of this application is merely an example, and the display interface may include more or less content.

[0056] The following is combined with Figure 1 Explain the video recording process.

[0057] The electronic device can take photos and / or record videos based on a camera application. In response to a trigger operation to launch the camera application, the electronic device can enter the initial photo-taking interface, such as... Figure 1 The interface shown in Figure 'a'. Figure 1 The interface shown in Figure 'a' displays a top menu area, a framing area, and a bottom menu area. The top menu area can display one or more function controls, such as flash controls, AI photography controls, high dynamic range (HDR) controls, filter controls, and settings controls. The framing area displays a preview image of the object being photographed. The bottom menu area displays function controls, shooting mode controls, thumbnail controls 101, shutter controls 102, and lens switching controls 103. Function controls include, for example, magnification controls; shooting mode controls include, for example, aperture controls, night scene controls, portrait controls, photo controls, video controls, movie controls, and professional controls. Thumbnail controls 101 can be used to quickly browse and manage previews of taken photos or videos. Shutter controls 102 can be used to take pictures or start recording videos; lens switching controls 103 can be used to switch between the front and rear cameras. The initial shooting interface can be set to photo mode.

[0058] In response to a click on the recording controls in shooting mode, the electronic device can switch the shooting mode to video recording mode and display as shown below. Figure 1 The interface shown in b is shown in the image. Figure 1 In the interface shown in b, the recording mode control can be displayed in the center to indicate that the device is currently in recording mode. When a trigger operation is received for the shutter control 102, the electronic device can start recording and display as shown in the image. Figure 1 The interface shown in c is shown in the image.

[0059] exist Figure 1 In the interface shown in Figure c, the electronic device can display an end-recording control 104, a pause-recording control 105, the recording time, and a preview image of the object being filmed. When a click operation is received on the end-recording control 104, the electronic device can stop recording, save the video, and display the following: Figure 1 The interface shown in d is shown in the image. Figure 1 In the interface shown by d, the thumbnail control 101 displays a thumbnail of the video.

[0060] Now, electronic devices can record video using the recording function of camera applications.

[0061] The following is combined with Figure 2 The process of viewing the recorded video is explained.

[0062] In one possible implementation, the electronic device can view the recorded video based on a thumbnail control 101 in the camera application. For example, after recording ends, the electronic device can display something like... Figure 2 The interface shown in Figure a includes a thumbnail control 101. When a trigger operation is received on the thumbnail control 101, the electronic device can display as shown in Figure a. Figure 2 The interface shown in b is shown in the image. Figure 2 The interface shown in b can play recorded videos in full screen.

[0063] In another possible implementation, the electronic device can view recorded videos based on a gallery application. For example, in response to an operation to launch the gallery application, the electronic device can display the gallery application interface, such as... Figure 2 The interface shown in c is shown in the image. Figure 2 The interface shown as 'c' includes one or more images or videos, such as video 201. When a trigger operation is received for video 201, the electronic device can play video 201, such as... Figure 2 The interface shown in b is shown in the image.

[0064] In a possible implementation, the electronic device may refer to Figure 1 Record the scene shown. However, during the recording process, the electronic device may experience poor image smoothness and high temperature.

[0065] The following are combined with Figure 3 and Figure 4 Explain the reasons for the occurrence of this problem.

[0066] During recording, the electronic device can execute two streams: a preview stream and a recording stream. The preview stream provides real-time visual feedback, allowing the user to view the footage being recorded. The recording stream stores the captured video data, creating a recorded video file.

[0067] In one possible implementation, the electronic device may employ, for example... Figure 3 The recording process shown yields a preview stream and a recording stream, respectively.

[0068] The recording process may involve: camera, image front end (IFE), multi-camera spatial alignment transaction (SAT), motion estimation module (GME), image stabilization module (EIS), image processing engine (IPE), video feature module, and format conversion module, etc.

[0069] The camera can be used to capture raw images, such as RAW images. The camera may include an image sensor. The image sensor can be used to convert captured light into electrical signals, thereby capturing an image of the object being photographed.

[0070] The image front-end processing module can be used to receive raw images from the image sensor and perform preliminary processing on the raw images, such as color correction, downsampling, depigmentation, and statistical analysis of 3A data.

[0071] The multi-camera spatial alignment module can be used to spatially align images captured by different cameras, enabling multi-camera image fusion.

[0072] The motion estimation module can be used to estimate the overall motion between two or more image frames.

[0073] Image stabilization modules can be used to compensate for minor camera movements and reduce blurring during shooting.

[0074] The image backend processing module can serve as the image processing engine, responsible for performing more complex image processing tasks, such as hardware noise reduction, image cropping, noise reduction, color processing, and detail enhancement.

[0075] The video function feature module can be used to process functions in recording mode, which may include skin smoothing and blurring functions. For example... Figure 1 The interface shown in b also includes a blur control 106 and a beautification control 107. When the blur control 106 is turned on, the electronic device can blur the image based on the blur algorithm to highlight the subject. When the beautification control 107 is turned on, the electronic device can beautify the image based on the beautification algorithm to enhance the appearance of the person.

[0076] The format conversion module can be used to convert the format of images. The image format can be processed based on IPE, such as converting RGB images to YUV images, and YUV images to JPEG or PNG formats, to facilitate image display and saving.

[0077] See Figure 3After launching the camera application, the electronic device acquires multiple raw images using image sensors from one or more cameras. The image sensors then transmit these raw images to an image front-end processing module, which performs algorithms such as color correction, downsampling, de-mosaicing, and statistical 3A data processing on the raw images. The image front-end processing module then transmits the processed images to a multi-camera control alignment module, which fuses the processed images from multiple cameras. The multi-camera control alignment module then transmits the fused image to a motion estimation module, which analyzes and estimates the motion vectors between the images and uses the estimated vectors to compensate for image jitter. Finally, the motion estimation module transmits the images to an image stabilization module, which acquires measurement data (such as three-axis angular velocity data) reported by the gyroscope sensor and uses this measurement data to process the images to reduce jitter between them.

[0078] The image stabilization module transmits the processed image to the image back-end processing module, which performs hardware noise reduction, image cropping, color processing, and detail enhancement. The image back-end processing module then transmits the processed image to the video features module, which performs skin smoothing and blurring effects. Finally, the video features module transmits the image to the format conversion module, which converts the image to a preset format.

[0079] The electronic device can then send the images to the display screen, showing multiple preview images sequentially, such as... Figure 1 The a interface in the middle Figure 1 As shown in the d interface.

[0080] The above describes the processing procedure for the preview stream. When the preview image is displayed in real-time on the electronic device, video recording can begin in response to an operation used for recording.

[0081] For example, at time 1, the electronic device is Figure 1 When a click operation is received on the shutter control 102 in the interface shown in b, the electronic device can save the image at time 1 and later. See also Figure 3 After the format conversion module converts the image format, the electronic device can save images from time 1 onwards. At time 2, Figure 1 The interface shown in c receives a trigger operation for the end recording control 104; in response to the trigger operation, the electronic device can stop saving images and obtain a video composed of frame images acquired between time 1 and time 2.

[0082] from Figure 3As can be seen, both the preview stream and the recorded stream images undergo sequential processing by modules including the camera, image front-end processing module, multi-camera spatial alignment module, motion estimation module, image stabilization module, image back-end processing module, video function feature module, and format conversion module. Within the same frame, the preview stream and recorded stream images are identical. However, in practical applications, the preview stream image needs to maintain smoothness during recording. Therefore, the electronic device must quickly process and display the preview stream image in real time to ensure the smoothness of the preview. The recorded stream image, as the stored image, requires higher image quality. Therefore, ensuring high-quality images in the recorded stream requires a complex image processing process and a long processing time, which cannot meet the smoothness requirements of the preview stream image. Conversely, ensuring smoothness in the preview stream requires simplifying the image processing process and reducing processing time, but this would correspondingly decrease image quality, affecting the image quality of the recorded stream.

[0083] Building upon this, another possible implementation is provided: by separating the processing of the preview stream and the recording stream, the processing time of the preview stream is reduced, improving the responsiveness of the preview image without affecting the quality of the recorded video. Taking image stabilization modules as an example, electronic devices can employ methods such as... Figure 4 The recording process shown yields a preview stream and a recording stream, respectively.

[0084] The recording process may involve: camera, image front-end processing module, multi-camera spatial alignment module, motion estimation module, image stabilization module (preview image stabilization module and video image stabilization module), image back-end processing module, video function feature module and format conversion module, etc.

[0085] The preview image stabilization module is used to stabilize images in the preview stream, reducing blurriness and jitter. The video image stabilization module is used to stabilize images in the recorded stream, reducing blurriness and jitter. When stabilizing images, the image stabilization module can cache multiple frames preceding the current frame and use these multiple frames to process the current frame. To improve the responsiveness of the preview image during recording, the preview image stabilization module can be configured to cache fewer frames (e.g., 1 frame) to reduce the time spent running the electronic image stabilization algorithm. Conversely, to improve the clarity of video images in the recorded video, the video image stabilization module can be configured to cache more frames (e.g., 30 frames).

[0086] After launching the camera application, the electronic device captures multiple raw images using image sensors from one or more cameras. The image sensors then transmit these raw images to an image front-end processing module, which performs algorithms such as color correction, downsampling, de-mosaicing, and statistical 3A data processing on the raw images. The image front-end processing module then transmits the processed images to a multi-camera control alignment module, which fuses the processed images from the multiple cameras.

[0087] After spatial alignment of the images, the electronic device can split the processed images into two streams: one for preview and one for recording.

[0088] For the preview stream, the multi-camera control alignment module passes the image to the preview image stabilization module. This module acquires sensor data and uses it to process the image, reducing image jitter. For example, it obtains and caches the previous frame and uses it along with sensor data to stabilize the next frame. The stabilization module then passes the processed image to the image backend processing module, which performs hardware noise reduction, image cropping, color processing, and detail enhancement. Finally, the image backend processing module passes the processed image to the video feature module, which performs skin smoothing and blurring effects. The video feature module then passes the image to the format conversion module, which converts the image to a preset format.

[0089] The electronic device can then send the images to the display screen, sequentially showing multiple preview images. This process can be, for example... Figure 1 The a interface in the middle Figure 1 The preview screen shown in the d interface.

[0090] For the video stream, the multi-camera control alignment module passes the fused image to the motion estimation module. The motion estimation module analyzes and estimates the motion vectors between multiple frames and uses the estimated vectors to compensate for image jitter. The motion estimation module then passes the image to the video image stabilization module. The video image stabilization module acquires sensor data and uses this data to process the image to reduce jitter between frames.

[0091] For example, at time 3, the electronic device is Figure 1 The interface shown in b receives a trigger operation for the shutter control 102, allowing the electronic device to process the image at time 3. The video image stabilization module obtains and buffers 30 frames of images prior to time 3, and uses the previous 30 frames and measurement data to perform image stabilization processing on the image at time 3.

[0092] The video image stabilization module transmits the processed image to the image back-end processing module, which performs hardware noise reduction, image cropping, color processing, and detail enhancement. The image back-end processing module then transmits the processed image to the video feature module, which performs skin smoothing and blurring effects. Finally, the video feature module transmits the image to the format conversion module, which converts the image to a preset format. At time 4, in Figure 1 The interface shown in c receives a trigger operation for the end recording control 104; in response to the trigger operation, the electronic device can stop saving images and obtain a video composed of frame images acquired between time 3 and time 4.

[0093] As can be seen, this method can reduce the processing time of the preview stream and improve the responsiveness of the preview image by reducing the number of cached frame images. However, since the image stabilization process is located relatively early in the processing flow, it introduces more additional processing steps, increasing memory usage and power consumption. For example, the image processed by the preview image stabilization module is different from the image processed by the video image stabilization module, which means that the two images need to be processed separately by the image back-end processing module, the video feature module, and the format conversion module.

[0094] During recording, significant memory increases and power consumption may affect the smoothness of the preview footage and cause the electronic device to overheat, impacting the user's thermal experience.

[0095] Based on this, this application provides an image processing method. Due to the responsiveness of preview images and the high-definition nature of video images, and the relatively low requirement for responsiveness in video images, the electronic device can process and display the preview image in real time during recording. However, during recording, real-time processing of the video image is not required; therefore, certain video image processing steps, such as image stabilization, multi-frame noise reduction, HDR effect processing, bokeh effect, video enhancement, and super-resolution processing, can be omitted. Subsequently, during video playback, the electronic device can complete these processing steps to play high-quality video images. Thus, during recording, the processor can handle processes related to responsiveness; during playback, the processor can handle the clarity of the video image. This reduces memory pressure during recording without affecting the high definition of the video image, thereby reducing issues such as stuttering in the preview and high device temperature.

[0096] To better understand the embodiments of this application, the structure of the electronic device of this application is described below:

[0097] Figure 5A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an embedded secure element (eSE) chip 196, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0098] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0099] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0100] The electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. In this embodiment, the electronic device 100 can reuse the image processing capabilities of the GPU to perform delayed processing on the video stream images.

[0101] The display screen 194 is used to display images, videos, etc. In this embodiment, the display screen 194 can be used to display, for example... Figure 1 The preview image shown can also be used to display images such as... Figure 2 The video image shown. In this embodiment, the display screen 194 can be used to display the corresponding interfaces of the camera application and the gallery application.

[0102] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0103] The ISP is used to process data fed back by the camera 193. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0104] Camera 193 is used to capture still images or videos.

[0105] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc. In this embodiment, electronic device 100 may use the encoding capability of an encoder to encode image processing data and unprocessed video stream images; for example, encoding inter-frame stabilization information and unprocessed video stream images into a recorded video. Electronic device 100 may also use the decoding capability of a decoder to decode the recorded video into image processing data and unprocessed video stream images, so that the image processing data can be used to perform relevant processing on the unprocessed video stream images.

[0106] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0107] In this embodiment of the application, after recording is completed, the electronic device 100 can store a recorded video, including image processing data (e.g., inter-frame stabilization information) and unprocessed (e.g., unstabilized) video stream images, in its internal memory 121. During playback, the electronic device 100 can perform relevant image processing (e.g., inter-frame stabilization) on the recorded video stored in its internal memory 121 and update the stabilized recorded video.

[0108] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). In this embodiment, the gyroscope sensor 180B can be used for image stabilization.

[0109] The accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). In this embodiment, the accelerometer 180E can be used for image stabilization.

[0110] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0111] Figure 6 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0112] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system may include: an application layer, an application framework layer, a hardware abstraction layer (HAL), and a kernel layer, where the kernel layer may also be called the driver layer.

[0113] The application layer can include a series of application packages.

[0114] like Figure 6 As shown, the application package may include applications such as a camera and a gallery. In this embodiment, the camera application can support video recording, and during recording, the camera application can smoothly display preview images and perform delayed processing on some aspects of the video stream image processing, such as inter-frame stabilization and multi-frame noise reduction.

[0115] The gallery app can be used to store recorded videos. Upon completion of recording, the first video stored in the gallery app may include unprocessed frame images and related processing data. For example, with image stabilization, the recorded video includes unstabilized frame images as well as inter-frame stabilization information. When the gallery app plays or shares the recorded video for the first time, the electronic device can use the inter-frame stabilization information to perform inter-frame stabilization on the frame images, and the gallery app will then play and update the stabilized video.

[0116] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0117] like Figure 6 As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, camera access interface, and rendering module, etc.

[0118] The camera access interface allows applications to manage the camera and access the camera device, such as managing the camera to capture images.

[0119] The rendering module is responsible for processing the output and display of graphical data, including 3D model rendering, image processing, and visual effects generation. The rendering module includes open graphics libraries (OpenGL), open source computer vision libraries (OpenCV), and open computing language libraries (OpenCL).

[0120] In this embodiment, the rendering module can be OpenGL. During the playback of the recorded video, the GPU can utilize OpenGL's rendering capabilities to perform matrix transformations on the frame images based on the inter-frame stabilization information, thereby achieving video stabilization.

[0121] The Hardware Abstraction Layer (HAL) allows the Android system to load corresponding library modules for device hardware, enabling the application framework layer to access the device hardware. The HAL can contain multiple algorithm library modules, such as image front-end processing modules, multi-camera spatial alignment modules, motion estimation modules, image stabilization modules, image back-end processing modules, video feature modules, format conversion modules, multi-frame noise reduction modules, and image calculation modules. For details on the image front-end processing module and the format conversion module, please refer to [link to relevant documentation]. Figure 3 The relevant descriptions in the document will not be repeated here.

[0122] The multi-frame noise reduction module can be used to reduce noise in the preview stream image and the video stream image, and the image calculation module can be used to stabilize the preview stream image based on the inter-frame stabilization information to obtain the stabilized preview image.

[0123] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to function. The kernel layer may include display drivers, camera drivers, and sensor drivers, etc., but this application does not limit this.

[0124] The hardware layer can include ISP, GPU, codec, camera, and sensors, etc.

[0125] The following is combined with Figures 7-11 The image processing methods in the embodiments of this application will be described.

[0126] Taking image stabilization algorithms as an example, Figure 7 This illustration shows a flowchart of an image processing method provided in an embodiment of this application; as shown Figure 7 As shown:

[0127] S701, In response to the recording operation, acquire image 1.

[0128] The recording operation can correspond to in Figure 1 The interface shown in b receives a click operation on the shutter control 102. The recording operation can also be a specific gesture, voice command, shortcut operation, etc. to trigger recording. This application embodiment does not limit this.

[0129] Image 1 can be an image processed by preset process 1 in the preview stream and / or video stream; preset process 1 is located before the image stabilization process in the preview stream and / or video stream processing.

[0130] by Figure 7 For example, the modules involved in preview process 1 include, but are not limited to: camera, image front-end processing module, multi-camera spatial alignment module, and motion estimation module; Image 1 can be the image processed by the motion estimation module. The acquisition process of Image 1 can be, for example: the camera captures the original image and transmits the original image to the image front-end processing module; the image front-end processing module performs IFE processing on the original image and transmits the processed image to the multi-camera spatial alignment module; the multi-camera spatial alignment module spatially aligns the images from different cameras at the same time and transmits the processed image to the motion estimation module; the motion estimation module compensates for the image based on the motion vector to obtain Image 1.

[0131] Optionally, in real-world scenarios, the preview stream or video stream may include more or fewer image processing modules before the image stabilization process; this embodiment of the application does not impose any limitations on this. Image 1 can be an image before it flows into the image stabilization module.

[0132] S702, The image stabilization module obtains the preview inter-frame stabilization information of image 1 based on the metadata of image 1 and the metadata of the previous frame image.

[0133] The meta data may include sensor data and timestamps. Sensor data includes, for example, gyroscope data, accelerometer data, and visual sensor data; gyroscope data can be used to record the rotational angular velocity of each frame; accelerometer data can be used to record the linear velocity of each frame; and visual sensor data is used to estimate the scaling of the subject. Timestamps can be used to record the capture time of each frame. Preview inter-frame stabilization information can be data used for inter-frame stabilization of images in the preview stream; in this embodiment, the preview inter-frame stabilization information can be in the form of a matrix. For example, preview inter-frame stabilization information can be matrix A.

[0134] Understandably, due to the requirement for responsiveness in the images of the preview stream, electronic devices need to process the images in the preview stream quickly in real time. Therefore, in step S702, the meta of the previous frame image can be used to perform inter-frame stabilization on image 1, resulting in fewer buffered frame images.

[0135] Specifically, obtaining the inter-frame stabilization information for image 1 based on the meta data of image 1 and the meta data of the previous frame image can include the following steps: caching the meta data of image 1 and the meta data of the previous frame image; preprocessing the meta data, which may include filtering, aligning sensor data with image frame timestamps, etc.; obtaining the rotation matrix between the previous frame image and image 1 based on the gyroscope data of image 1 and the gyroscope data of the previous frame image; obtaining the translation matrix between the previous frame image and image 1 based on the accelerometer data of image 1 and the accelerometer data of the previous frame image; obtaining the scaling matrix between the previous frame image and image 1 based on the visual sensor data of image 1 and the visual sensor data of the previous frame image; and calculating the transformation matrix (e.g., matrix A) between the previous frame image and image 1 based on the rotation matrix, translation matrix, and scaling matrix.

[0136] Matrix A can satisfy the following format:

[0137]

[0138] After obtaining the preview inter-frame stabilization information, the electronic device caches the preview inter-frame stabilization information of image 1 and does not currently use the preview inter-frame stabilization information to process image 1.

[0139] S703 The image stabilization module obtains the inter-frame stabilization information of image 1 based on the meta data of image 1 and the meta data of the previous N (N is a positive integer greater than 1) frames.

[0140] Inter-frame stabilization information can be data used to perform inter-frame stabilization on images in a recorded stream; in this embodiment, the inter-frame stabilization information can be in the form of a matrix. For example, the inter-frame stabilization information can be matrix B.

[0141] Understandably, since the images in the video stream do not need to be displayed in real time, but require higher precision and stability, step S703 can use the meta of the first N frames to perform inter-frame stabilization on image 1, where N can be, for example, 30.

[0142] Specifically, based on the meta data of image 1 and the meta data of the previous N frames, the inter-frame stabilization information of image 1 is obtained, which may include the following steps: caching the meta data of image 1 and the meta data of the previous N frames; preprocessing the meta data, which may include filtering, aligning sensor data with image frame timestamps, etc.; for any frame of the previous N frames, obtaining a rotation matrix based on the gyroscope data of that frame and the gyroscope data of image 1; obtaining a translation matrix based on the accelerometer data of that frame and the accelerometer data of image 1; obtaining a scaling matrix based on the visual sensor data of that frame and the visual sensor data of image 1; calculating the transformation matrix between the two frames based on the rotation matrix, translation matrix, and scaling matrix; after traversing the N frames, smoothing the N-1 transformation matrices to obtain the processed transformation matrix (e.g., matrix B).

[0143] One possible implementation involves smoothing the N-1 transformation matrices, which can be achieved by taking a weighted average of the N-1 transformation matrices. In some embodiments, the weights of the N-1 transformation matrices can be set based on the temporal distance between the corresponding frame image and image 1, with the weights being negatively correlated with the temporal distance. For example, the larger the difference between the timestamp of the frame image corresponding to the transformation matrix and the timestamp of image 1, the smaller the weight; conversely, the larger the weight.

[0144] In another possible implementation, Kalman filtering can be used to smooth the N-1 transformation matrices to reduce noise and uncertainty in the transformation matrices. This application does not limit this approach.

[0145] Matrix B can satisfy the following format:

[0146]

[0147] After obtaining the inter-frame stabilization information, the electronic device buffers the inter-frame stabilization information for image 1, and does not currently use the inter-frame stabilization information to process image 1. Then, the image stabilization module transmits image 1 to the image back-end processing module.

[0148] S704. Process image 1 to obtain image 2.

[0149] Image 1 can be an image processed by preset process 2 in the preview stream and / or video stream; preset process 2 is located after image stabilization processing in the preview stream and / or video stream. For example, the processing of preset process 2 involves an image back-end processing module, a video feature module, and a format conversion module, etc.; the processing can correspond to image back-end processing, feature processing, and format conversion processing, etc. Image back-end processing includes, for example, noise reduction, color processing, and detail enhancement; feature processing includes, for example, skin smoothing and blurring; format conversion processing includes, for example, converting an RGB format image to a YUV format image.

[0150] It is understandable that after the image stabilization module calculates the inter-frame stabilization information of the preview frame and the inter-frame stabilization information of the video frame, the image stabilization module will not perform stabilization processing on image 1, but will instead pass image 1 to the subsequent image processing module.

[0151] This is because if the image stabilization module performs stabilization on image 1, it will produce two different images: one for the preview stream and one for the recording stream. Subsequently, the electronic device needs to use the image back-end processing module to the format conversion module to perform post-processing on the two images respectively. However, in this embodiment, the image stabilization module does not process image 1. After passing through the image stabilization module, the image in the preview stream and the recording stream is still image 1. This means that the image back-end processing module to the format conversion module only needs to process image 1 once.

[0152] and Figure 4 A comparison of the processing flows shown reveals that in the embodiments of this application, there is more overlap between the image processing of the preview stream and the image processing of the video stream. For example, in the overall image processing flow, some processing steps after image stabilization may overlap. This reduces the processing steps of the electronic device, improves image processing efficiency, and reduces computational pressure.

[0153] It should be noted that, in this embodiment, the electronic device can prioritize calculating the inter-frame stabilization information of the preview frame based on the tracking characteristics of the preview stream image, and then calculate the inter-frame stabilization information of the video frame. After the calculation is completed, image 1 is passed to the image back-end processing module, for example, by executing steps S702, S703, and S704 sequentially. In other scenarios, since the inter-frame stabilization information does not need to participate in the calculation of step S704, the image stabilization module can also pass image 1 to the image back-end processing module after obtaining the meta information, without waiting for the calculation to be completed; for example, the image stabilization module and the image back-end processing module run on different processors. The image stabilization module calculates the inter-frame stabilization information based on the meta data, while the image back-end processing module processes image 1; that is, steps S702 and S703 can be executed simultaneously with S704. This embodiment does not limit the execution order of the relevant steps.

[0154] It should be noted that in step S705, the image stabilization module can directly pass image 1 to the subsequent processing flow after calculating the preview inter-frame stabilization data and the video inter-frame stabilization data. Alternatively, in step S705, the image stabilization module can also perform intra-frame correction on image 1; intra-frame correction is an internal processing of a single frame image, and after processing, the preview stream and the recording stream still share the intra-frame corrected image 1; the achieved effect is consistent with the above description. The embodiments of this application will provide a detailed description of intra-frame correction and inter-frame stabilization later (see...). Figure 9 (This will not be elaborated upon here.)

[0155] S705 The image calculation module uses the preview inter-frame stabilization information to perform stabilization processing on image 2, and then sends the preview image to be displayed.

[0156] The image computation module can be an IPE (Integrated Image Processing) module. It should be noted that while both the image computation module and the image back-end processing module rely on the functionality of the IPE, they are not entirely the same. The image computation module can utilize the matrix transformation capabilities of the IPE, while the image back-end processing module can utilize the noise reduction, color processing, and detail enhancement capabilities of the IPE.

[0157] The image calculation module obtains Image 2 from the format conversion module and preview inter-frame stabilization information from the image stabilization module. The image calculation module uses the preview inter-frame stabilization information to perform stabilization processing on Image 2. The stabilization process can be as follows: the image calculation module obtains Image 2 and its matrix (e.g., matrix A); for any pixel region in Image 2, the coordinate matrix of the pixel region is processed using matrix A and a preset matrix to obtain the stabilized coordinate matrix 1 for that pixel region, where each pixel region includes one or more pixels; the image is reconstructed based on the stabilized coordinate matrix 1 of any pixel region to obtain the preview image.

[0158] Next, the image processing module can transmit the preview image to the display device, which may include a media player to display the preview image. Thus, during recording, the electronic device completes the processing of the preview image and displays it on the screen.

[0159] S706, The encoder encodes the inter-frame stabilization information of image 2 and image 2 to obtain video recording 1, and stores video recording 1.

[0160] Video recording 1 includes multiple frames and inter-frame stabilization information corresponding to any given frame. For example, image 2 can be one of the multiple frames. The processing of other frames in video recording 1 can be referred to image 2. This will not be repeated in the embodiments of this application. The electronic device may include a MediaCodec, a module provided by Android for encoding and decoding audio and video. It includes encoders and decoders, etc. During recording, MediaCodec can create encoders to encode images.

[0161] During the recording process, the electronic device can encode and encapsulate the recorded video 1. During the video encoding and encapsulation process, the video file usually contains multiple types of data, which are set into a specific format for easy storage and playback.

[0162] For example, recorded video 1 may include: file type (ftyp), media data (mdat), and movie media type. The format of recorded video 1 may be as follows: Figure 8 As shown.

[0163] The file type identifies the type of file, helping decoders or players recognize the file format and determine whether the file content can be correctly parsed and played. Media data stores encoded file data, such as audio, video, and subtitles. The media type may include a movie header and multiple media tracks; the header information records global information about the recorded video, such as the file's timescale and duration; media tracks represent different media streams or data streams.

[0164] In this embodiment, the recorded video 1 may include a video composed of frame images and inter-frame stabilization information of the frame images. During the video encoding process, the encoder may also encode the inter-frame stabilization information into the video in the form of media tracks.

[0165] During the encoding process, the encoder can create multiple media tracks and write media data into these tracks, with each media track corresponding to a type of media data. These media tracks can include video tracks, audio tracks, subtitle tracks, and video frame stabilization information tracks, among others.

[0166] See Figure 8In the recorded video 1, media track 1 can be a video track, and the media data written on the video track can be multiple frames of images (e.g., image 2). Media track 2 can be an audio track, and the media data written on the audio track can be audio data recorded during the recording process. Optionally, media track 3 can be a subtitle track, and the media data written on the subtitle track can be subtitles; for example, if the electronic device supports AI subtitle functionality, the electronic device can convert audio data into text data in real time during the recording process. Media track 4 can be a video frame stabilization information track, and the media data written on the video frame stabilization information track can be matrix data. Specifically, for the video frame stabilization information track, the encoder can write the video frame stabilization information into the video frame stabilization information track based on the timestamp corresponding to the video frame stabilization information.

[0167] Optionally, embodiments of this application may include one or more video frame stabilization information tracks, and / or one or more video tracks. It is understood that the camera application of the electronic device can support multi-camera recording (simultaneous recording by the front and rear cameras), picture-in-picture recording, etc., enabling multiple recordings at once. After recording ends, the electronic device can obtain one or more videos; that is, recorded video 1 may include one or more videos, and correspondingly, recorded video 1 contains one or more sets of video frame stabilization information. During encoding, the encoder can create multiple video tracks and video frame stabilization information tracks. Embodiments of this application do not impose limitations on this.

[0168] After encoding, the electronic device can save the recorded video 1 to the gallery application.

[0169] In this embodiment of the application, the encoder can encode frame by frame. After obtaining a frame image and the corresponding video frame stabilization information, the frame image and the video frame stabilization information are encoded into the corresponding media tracks respectively.

[0170] S707, In response to the operation to end recording, save the recorded video 1 to the Gallery application.

[0171] The operation to end recording can be, for example, in Figure 1 In the interface shown in Figure c, the operation for triggering the end-recording control 104 is described. After recording ends, the electronic device can save the recorded video 1 to the gallery application (such as...). Figure 2 The interface shown in c), and the recorded video 1 displayed as a thumbnail in the camera application (as shown in c). Figure 2 (The interface shown in 'a').

[0172] At this point, the electronic device has completed the video preview and recording process. At this stage, the frames in recorded video 1 have not yet undergone image stabilization. Subsequently, the electronic device can apply image stabilization to recorded video 1 when the user needs to view or use it, thereby improving the quality of the frames in recorded video 1. If the user does not view or use recorded video 1, the electronic device may not apply image stabilization until the electronic device uses or deletes recorded video 1.

[0173] For example, the usage process of the recorded video 1 will be described below with reference to steps S708-S710.

[0174] S708, In response to the operation for playing the recorded video 1, the decoder decodes the recorded video 1.

[0175] The operation for playing video recording 1 can be, for example: in Figure 2 In the interface shown in c, the click operation is performed on video recording 1; or, if video recording 1 is the most recently captured and recorded image, in... Figure 2 In the interface shown in Figure a, the click operation is performed on the thumbnail control 101.

[0176] During video playback, MediaCodec can create a decoder to decode the encoded images. The decoder can decode the recorded video 1 into multiple frames and the corresponding inter-frame stabilization information. The decoder can then pass the multiple frames and the corresponding inter-frame stabilization information to the GPU.

[0177] The S709 and GPU use inter-frame stabilization information to stabilize the recorded video 1 and then play the recorded video 2.

[0178] GPUs can be used to process image data. During video playback, GPUs can perform image rendering; electronic devices can reuse the capabilities of GPUs to perform image stabilization on multiple frames. For example, the GPU can call the rendering module to perform texture transformation on frame images; during texture transformation, the preset matrix of the frame images is processed using inter-frame stabilization information to achieve the stabilization effect.

[0179] The preset matrix (also known as the texture transformation matrix) can be used to describe the rotation, translation, and scaling of the frame image. In this embodiment, since the electronic device does not perform editing operations such as rotation, translation, and scaling on the recorded video 1 after obtaining it, the preset matrix of the recorded video 1 can be an identity matrix. The preset matrix is, for example, matrix C, which can satisfy the following format:

[0180]

[0181] Taking OpenGL as the rendering module and image 2 as the frame image as an example, the image stabilization process may include the following steps: obtaining the inter-frame stabilization information of image 2 (e.g., matrix B) and the preset matrix of image 2 (e.g., matrix C); obtaining the target matrix of image 2 based on the inter-frame stabilization information and the preset matrix. The target matrix can be used to characterize the rotation, translation and scaling of the image after image stabilization. The target matrix can be the product of matrix B and matrix C; processing image 2 based on the target matrix to obtain image 3.

[0182] The preset matrix can be mSTMatrix obtained by OpenGL through the SurfaceTexture.getTransformMatrix() function. In one possible implementation, image 2 is processed based on the target matrix to obtain image 3. For example, for any pixel region in image 2, matrix B is used to process the coordinate matrix of the pixel region to obtain the stabilized coordinate matrix 2 of that pixel region, where the pixel region includes one or more pixels; the image is then reconstructed based on the stabilized coordinate matrix 2 of any pixel region to obtain image 3.

[0183] Performing the above operation on any frame of recorded video 1 yields recorded video 2. It is understood that recorded video 2 can be a video after image stabilization processing of recorded video 1; recorded video 2 no longer includes inter-frame image stabilization information, meaning that the format of recorded video 2 no longer includes... Figure 8 The image shows the inter-frame stabilization information track.

[0184] S710. Save video recording 2 in the Gallery application and delete video recording 1.

[0185] After the initial playback of video recording 1, the electronic device can obtain video recording 2 with image stabilization. To reduce the processing power consumption of subsequent playbacks of video recording 1 and save storage space, the electronic device can update video recording 1 to video recording 2. Subsequently, when the electronic device receives a trigger operation to play video recording 2, it can play video recording 2 using the regular video playback process.

[0186] In this way, some of the image processing steps during recording are moved to the video playback process, reducing the memory and computing resources used in the recording process, improving the smoothness of the preview stream, and avoiding the impact of high power consumption on the user's thermal experience.

[0187] Optionally, step S708 uses playing video recording 1 as an example to illustrate the image stabilization process. In some scenarios, a user may share video recording 1 to other devices without having viewed it before. Other devices may not have the ability to convert video recording 1 into video recording 2, potentially preventing them from viewing video recording 1. Therefore, upon receiving an operation to share video recording 1, the electronic device can also utilize the GPU to perform image stabilization on video recording 1, resulting in video recording 2; after saving video recording 2 to the gallery application, it can then share it to other devices. This process is similar to the process of converting video recording 1 into video recording 2 during playback, and this embodiment will not describe this process in detail.

[0188] Optionally, since GPUs also have the ability to perform matrix transformations on images, the execution entity of step S705 in this embodiment can also be a GPU. For example... Figure 7 As shown:

[0189] Following steps S701-S704, the method further includes:

[0190] The S711 and GPU use the preview inter-frame stabilization information to perform stabilization processing on image 2, and then send the preview image to be displayed.

[0191] After step S711, the method further includes steps S706-S710.

[0192] This process can be referred to in the relevant description of step S705. Step S705 can be understood as performing image stabilization on image 2 during the image processing flow; step S711 can be understood as performing image stabilization on image 2 during the process of rendering and displaying the preview image. They are essentially similar, both involving image stabilization to obtain the preview image.

[0193] In this embodiment, the electronic device can set the module for image stabilization processing of the preview image based on the respective computing power consumption of the GPU and the image computing module during recording. For example, if the power consumption of processing image 2 using the GPU is less than the power consumption of processing image 2 using the image computing module, the electronic device can execute S711 after step S704; otherwise, the electronic device can execute S705 after step S704. Alternatively, the image computing module can run on the CPU, and the electronic device can set the module for image stabilization processing of the preview image based on the memory usage of the CPU and GPU. For example, if the current CPU memory usage is high, the electronic device can execute S711 after step S704; otherwise, the electronic device can execute S705 after step S704. This embodiment can also select the corresponding module for processing image 2 according to other strategies, which will not be elaborated further in this embodiment.

[0194] The following is combined with Figure 9 The anti-shake process in the embodiments of this application will be described. For example... Figure 9 As shown:

[0195] exist Figure 9 In the image, the rectangle filled with dots in the upper left corner can be considered the subject being filmed during the recording process. During the recording of this subject, the electronic device obtains the raw images from frame 1 to frame 4. After processing the raw images using the image front-end processing module, the multi-camera spatial alignment module, and the motion estimation module, the electronic device obtains image 1. The image stabilization module then obtains image 1.

[0196] In the EIS image stabilization process, stabilization can include two processes: intra-frame correction and inter-frame stabilization. Intra-frame correction can compensate for jitter within a single frame of an image; inter-frame stabilization can compensate for jitter between two or more frames of images. The image stabilization module can include three sub-modules, for example, module 1, module 2, and module 3; module 1 can be used to process intra-frame correction of images; module 2 can be used to calculate inter-frame stabilization of preview stream images; and module 3 can be used to calculate inter-frame stabilization of recorded stream images.

[0197] The image stabilization process is explained below with reference to steps S901-S904.

[0198] For example, in S901, after obtaining image 1, module 1 can perform intra-frame correction on any frame of image 1 to obtain intra-frame corrected image 1. Specifically, module 1 can identify local displacements in image 1 by analyzing feature points and / or textures in pixel regions of image 1, and correct these displacements.

[0199] For example, Figure 9 In any frame of image 1 from the first to the fourth frame, it can be seen that due to optical distortion, sensor defects, or other factors during the recording process, the pixel area containing the main subject (the rectangle filled with dots) in image 1 is deformed, for example, by radial or tangential distortion causing image distortion. After intra-frame correction of image 1, the degree of deformation of the main subject in the intra-frame corrected image 1 is reduced, and the image quality is improved.

[0200] S902 and Module 2 obtain the matrix A (preview of inter-frame stabilization information) of the current frame based on the meta data of the current frame and the meta data of the previous frame.

[0201] S903 and Module 3 obtain the matrix B (inter-frame stabilization information) of the current frame based on the meta data of the current frame and the meta data of the previous 30 frames.

[0202] S904. The electronic device processes the inter-frame corrected image 1 to obtain image 2.

[0203] Steps S902-S904 can be referred to the relevant descriptions in steps S702-S704, and will not be repeated here.

[0204] S905. During the recording process, matrix A is used to perform image stabilization on image 2 to obtain a preview image.

[0205] S906. During playback, matrix B is used to perform image stabilization on image 2 to obtain image 3.

[0206] Steps S905-S906 can be referred to the relevant descriptions in steps S705-S710, and will not be repeated here.

[0207] It is understandable. During video recording, the shaking and / or movement of the user's handheld electronic device may affect the relative position of the subject in the image and the shooting angle. The shooting angle of the subject in each frame is not exactly the same. For example, after intra-frame correction, the subject in the first frame (Image 1) is rectangular, the subject in the second frame (Image 1) is trapezoidal, the subject in the third frame (Image 1) is trapezoidal, and the subject in the fourth frame (Image 1) is trapezoidal, etc. In addition, the position of the subject in each frame is not exactly the same. For example, after intra-frame correction, the subject in the first frame (Image 1) is biased to the right, the subject in the second frame (Image 1) is biased to the left, the subject in the third frame (Image 1) is biased to the lower right, and the subject in the fourth frame (Image 1) is biased to the upper right, etc.

[0208] Inter-frame image stabilization can be achieved by extracting meta-data from two or more adjacent frames. This stabilization data can then be used to perform coordinate transformations on the subject within the image through operations such as rotation, scaling, and translation, reducing differences in the subject's viewpoint and position between adjacent images. For example, ... Figure 9 As shown, during playback, after inter-frame stabilization, the main subject in images 3 (first to fourth frames) is roughly centered, and the viewing angle is roughly the same. This achieves the effect of reducing jitter and improving image stability.

[0209] It should be noted that the above embodiments use an image stabilization algorithm as an example to illustrate the image processing method provided in this application. The image processing method shown in this application can also be applied to the processing of other multi-frame image algorithms, such as multi-frame noise reduction algorithms, HDR effect processing, bokeh algorithms, video enhancement algorithms, and super-resolution reconstruction algorithms.

[0210] The following uses a multi-frame noise reduction algorithm as an example to illustrate possible implementations and the image processing methods provided in the embodiments of this application.

[0211] Figure 10 This illustrates a possible multi-frame noise reduction process in an implementation.

[0212] Understandably, in Figure 4 Based on the illustrated embodiment, the electronic device may further include a multi-frame noise reduction module, which may include a preview multi-frame noise reduction module and a video multi-frame noise reduction module. The preview multi-frame noise reduction module can be used to perform noise reduction processing on the preview stream image, and the video multi-frame noise reduction module can be used to perform noise reduction processing on the recorded stream image.

[0213] Similar to the image stabilization process, during the noise reduction process, due to the requirements for the responsiveness of the preview stream image and the high quality requirements of the video stream image, the preview multi-frame noise reduction module can cache a small number of frames to perform multi-frame noise reduction processing on the preview stream image, for example, 1-5 frames; the video multi-frame noise reduction module can cache a larger number of frames to perform multi-frame noise reduction processing on the video stream image, for example, 5-10 frames.

[0214] After the camera captures the raw data, the electronic device can process the raw image through the image front-end processing module and the multi-camera spatial alignment module to obtain the processed image (see reference). Figure 4 The process shown is as follows: Figure 10 (Not shown in the image). At this point, since multi-frame noise reduction involves processing the preview stream and the recording stream, the subsequent processing is divided into two paths. One path processes the preview image through the preview image stabilization module, the preview multi-frame noise reduction module, the image back-end processing module, the video function feature module, and the format conversion mode to obtain the preview image. The other path processes the motion estimation module, the video image stabilization module, the video multi-frame noise reduction module, the image back-end processing module, the video function feature module, and the format conversion mode to obtain the recorded video.

[0215] It can be seen that in the possible image processing, multi-frame noise reduction is processed relatively early. If the preview stream image and the video stream image are separated too early, multiple subsequent modules will need to process the preview stream image and the video stream image separately, resulting in a large number of processing steps and increased memory usage and power consumption.

[0216] Figure 11 This application illustrates a multi-frame noise reduction image processing procedure provided by an embodiment of the present application, such as... Figure 11 As shown:

[0217] S1101, In response to the recording operation, acquire image 4.

[0218] This step can be referred to in the relevant description in step S701, where image 4 can be understood as the image after being processed by the preset process 1 and the image stabilization module.

[0219] Optionally, the image stabilization process and the multi-frame noise reduction process can also be linked. For example, after processing by a preset process 1 (e.g., camera, image front-end processing module, multi-camera spatial alignment module, and motion estimation module), image 4 is obtained. The image stabilization module extracts inter-frame stabilization information using steps such as S702 and S703, but does not use the inter-frame stabilization information to perform stabilization processing on image 4. Then, the image stabilization module passes image 4 to the multi-frame noise reduction module; furthermore, the image stabilization module can also pass the preview inter-frame stabilization information to the image calculation module, and the video inter-frame stabilization information to the encoder.

[0220] S1102, the multi-frame noise reduction module obtains the preview noise reduction information of image 4 based on the meta data of image 4 and the meta data of the previous a (a is a positive integer) frames.

[0221] Meta data can include sensor data, timestamps, and shooting parameters. Shooting parameters can include exposure time, ISO value, aperture size, and other information. Preview noise reduction information can be used to perform multi-frame noise reduction on images in the preview stream. Preview noise reduction information can include noise model parameters. Noise model parameters can be used to describe the noise characteristics in the image, such as the mean and variance of the noise, and these parameters can be used to determine the strength and effectiveness of the multi-frame noise reduction algorithm.

[0222] Noise model parameters can be obtained based on shooting parameters and / or sensor characteristics, such as sensor type, sensor size, pixel size, etc.

[0223] After obtaining the above preview noise reduction information, the electronic device can cache the preview noise reduction information.

[0224] S1103 The multi-frame noise reduction module obtains the video noise reduction information of image 4 based on the meta data of image 4 and the meta data of the previous b (b is a positive integer greater than a) frames.

[0225] Video noise reduction information can be data used to perform multi-frame noise reduction on images in a video stream.

[0226] Understandably, this process is similar to step S1102, but since the images in the video stream do not need to be displayed in real time, but require higher precision and stability, step S1103 can use the meta of the first b frames to perform multi-frame fusion on image 4, and the value of b is greater than the value of a.

[0227] After obtaining the above video noise reduction information, the electronic device can cache the video noise reduction information and transmit image 4 to the image back-end processing module.

[0228] S1104. Process image 4 to obtain image 5.

[0229] Image 5 can be an image processed by preset process 2 in the preview stream and / or video stream; preset process 2 is located after multi-frame noise reduction processing in the preview stream and / or video stream processing. For example, the processing of preset process 2 involves an image back-end processing module, a video feature module, and a format conversion module, etc.; the processing can correspond to image back-end processing, feature processing, and format conversion processing, etc. Image back-end processing includes, for example, noise reduction, color processing, and detail enhancement; feature processing includes, for example, skin smoothing and blurring; format conversion processing includes, for example, converting an RGB format image to a YUV format image.

[0230] Understandably, after the multi-frame noise reduction module calculates the preview noise reduction information and the video noise reduction information, the multi-frame noise reduction module will not perform noise reduction processing on image 4, but will instead pass image 4 to the subsequent image processing module.

[0231] This is because if the multi-frame noise reduction module performs multi-frame noise reduction on image 4, it will obtain two different images: one for the preview stream and one for the recording stream. Subsequently, the electronic device needs to use the image back-end processing module to the format conversion module to perform post-processing on the two images respectively. However, in this embodiment, the multi-frame noise reduction module does not perform noise reduction on image 1. After the multi-frame noise reduction module, the image in the preview stream and the recording stream is still image 4. This allows the image back-end processing module to process image 4 only once.

[0232] and Figure 10 A comparison of the processing flows shown reveals that the image processing procedures for the preview stream and the video stream overlap significantly in the embodiments of this application. For example, some processing steps following multi-frame noise reduction may overlap in the overall image processing flow. This reduces the processing steps of the electronic device, improves image processing efficiency, and reduces computational pressure.

[0233] S1105, the image computing module or GPU uses the preview noise reduction information to perform multi-frame noise reduction processing on image 4, and then sends the preview image to be displayed.

[0234] The image computation module can be an IPE (Integrated Image Processor). For example, an IPE can have matrix transformation and image denoising capabilities.

[0235] After the image processing module or GPU obtains the preview denoising information and image 5, it can perform denoising processing on image 5 based on the noise model parameters in the preview denoising information to obtain the denoised preview image. Then, the electronic device displays the preview image.

[0236] S1106. The encoder encodes the video noise reduction information of image 5 to obtain video recording 3, and stores video recording 3.

[0237] The recorded video 3 includes multiple frames of images, as well as video noise reduction information corresponding to any one of the frames. For example, the multiple frames can be represented by image 5. The processing procedure for the other frames in the recorded video 3 can be referred to image 5. This embodiment will not repeat the description here.

[0238] During recording, the electronic device can encode and encapsulate the recorded video (3). During encoding and encapsulation, the video file typically contains various types of data, which are formatted in a specific way for storage and playback. For example, recorded video (3) may include: file type, media data, and media type. This process can be referenced... Figure 8 The descriptions of the embodiments and steps S706 shown are not repeated here.

[0239] It should be noted that, unlike the image stabilization process, in this embodiment, the recorded video 3 may include multiple media tracks, such as a video track, an audio track, a subtitle track, and a video noise reduction information track. The media data written on the video noise reduction information track can be noise model parameters. Specifically, for the video noise reduction information track, the encoder can write the video noise reduction information into the video noise reduction information track based on the timestamp corresponding to the video noise reduction information.

[0240] Optionally, as described above, the image stabilization process and the multi-frame noise reduction process can be associated. In step S1106, the media track of the recorded video 3 may also include an inter-frame stabilization information track, so that the GPU can perform inter-frame stabilization processing and multi-frame noise reduction processing on the image 5 respectively during subsequent playback.

[0241] S1107. In response to the operation to end recording, save the recorded video 3 to the Gallery application.

[0242] The process can be referred to in the relevant description in step S707, and will not be repeated here.

[0243] For example, the usage process of the recorded video 1 will be described below with reference to steps S1108-S1110.

[0244] S1108, In response to the operation for playing the recorded video 3, the decoder decodes the recorded video 3.

[0245] After decoding the recorded video 3, multiple frames of images and the corresponding video noise reduction information for any one frame can be obtained. This process can be referred to the relevant description in step S708, and will not be repeated here.

[0246] S1109, the GPU uses video noise reduction information to perform noise reduction processing on the recorded video 3, and then plays the recorded video 4.

[0247] GPUs can be used to process image data. For example, a GPU can perform noise reduction on images during video playback; electronic devices can reuse the GPU's capabilities to perform noise reduction on multiple frames of images. For instance, the GPU can call the API of a noise reduction algorithm to perform noise reduction on frame images. After the GPU performs noise reduction on image 5, image 6 is obtained. The electronic device can then obtain recorded video 4, which includes image 6. Afterward, the electronic device can use the GPU and a player to play recorded video 4.

[0248] Optionally, the codec may also include a filter for image noise reduction. In step S1109, during the playback of the recorded video, the decoder may also use video noise reduction information to perform noise reduction processing on the recorded video 3 to obtain the recorded video 4.

[0249] S1110. Save video recording 4 in the Gallery application and delete video recording 3.

[0250] The process can be referred to in the relevant description in step S710, and will not be repeated here.

[0251] In this way, some of the image processing steps during recording are moved to the video playback process, reducing the memory and computing resources used in the recording process, improving the smoothness of the preview stream, and avoiding the impact of high power consumption on the user's thermal experience.

[0252] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0253] The image processing method of the present application embodiments has been described above. The apparatus for performing the above image processing method provided in the present application embodiments will now be described. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced in each other, and the related apparatus provided in the present application embodiments can perform the steps in the above image processing method.

[0254] like Figure 12 As shown, the image processing apparatus 1200 can be used in communication equipment, circuits, hardware components, or chips. The image processing apparatus includes a display unit 1201 and a processing unit 1202. The display unit 1201 supports the display steps performed by the image processing apparatus 1200; the processing unit 1202 supports the information processing steps performed by the image processing apparatus 1200.

[0255] In a possible implementation, the image processing device 1200 may also include a communication unit 1203. Specifically, the communication unit supports the image processing device 1200 in performing data transmission and data reception steps. The communication unit 1203 may be an input or output interface, pins, or circuits, etc.

[0256] In one possible embodiment, the image processing apparatus may further include a storage unit 1204. The processing unit 1202 and the storage unit 1204 are connected via a line. The storage unit 1204 may include one or more memories, which may be devices in one or more devices or circuits used for storing programs or data. The storage unit 1204 may exist independently and be connected to the processing unit 1202 of the image processing apparatus via a communication line. Alternatively, the storage unit 1204 may be integrated with the processing unit 1202.

[0257] Storage unit 1204 may store computer-executable instructions for the methods in the terminal device, so that processing unit 1202 executes the methods in the above embodiments. Storage unit 1204 may be a register, cache, or RAM, etc., and storage unit 1204 may be integrated with processing unit 1202. Storage unit 1204 may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, and storage unit 1204 may be independent of processing unit 1202.

[0258] The image processing method provided in this application can be applied to electronic devices with communication functions. Electronic devices include electronic devices in general; the specific device form of an electronic device can be referred to the above-described related features, and will not be repeated here.

[0259] This application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the above-described method.

[0260] This application provides a chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to execute the above-described method. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0261] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0262] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0263] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0264] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0265] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, including: The first image is obtained during the recording process; The first image is processed to obtain a second image; the second image is a shared image in the preview stream and the recording stream; the first processing is used to process the single-frame image using intra-frame information of the single-frame image; First information and second information are acquired. After acquiring the first information and second information, and during the process of performing a first processing on the first image to obtain the second image, the first information and second information are not used to process the second image. The first information is obtained based on the association information between the first image and a preceding frame (a), and the second information is obtained based on the association information between the first image and b preceding frame (b). Both a and b are positive integers, and a is less than b. In the preview stream, the second image is processed according to the first information to obtain and display a third image. The third image is used for preview display during recording. In the video stream, the second information and the second image are encoded to obtain and save a first video; wherein the first video includes a first media track and a second media track; during the recording process, the second image is stored in the first media track, and the second information is stored in the second media track; wherein the second information and the second image are used to perform the second processing when the recorded video is obtained after the recording process ends; After obtaining the first video, the following is also included: In response to the operation for playing the first video, the second image is processed according to the second information to obtain and display the fourth image; The first video is replaced with a second video, the second video including the fourth image, and the second video not including the second image and the second information.

2. The method according to claim 1, characterized in that, The second processing includes a first electronic image stabilization (EIS) process; the first EIS process is used to perform image stabilization on the first image based on the correlation information between consecutive frame images; the first information is a first matrix obtained based on the metadata (meta) of the first image and the metadata of the a-frame images preceding the first image; The electronic device includes a first module, wherein the step of performing a second processing on the second image based on the first information to obtain a third image includes: The first module performs a first EIS process on the second image based on a preset matrix of the second image and the first matrix to obtain the third image; The first module supports coordinate transformation of the second image, and the first module includes a graphics processing unit (GPU) or an image processing engine (IPE).

3. The method according to claim 2, characterized in that, The second information is a second matrix obtained based on the meta data of the first image and the meta data of the b-frame images preceding the first image; the electronic device also includes an encoder and a decoder, and the first video is encoded by the encoder; The step of performing the second processing on the second image based on the second information to obtain the fourth image includes: The decoder decodes the first video to obtain the second image and the second matrix; wherein the second image is decoded from the first media track, the second matrix is ​​decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second matrix; The GPU performs a first EIS process on the second image based on a preset matrix and the second matrix to obtain the fourth image.

4. The method according to claim 2 or 3, characterized in that, The meta data of the first image includes one or more of the following: sensor data and timestamp when the first image was acquired; the meta data of the i-th frame in the b-frame images preceding the first image includes one or more of the following: sensor data, timestamp and shooting parameters when the i-th frame image was acquired, where i is less than b; The second matrix is ​​obtained in the following manner: based on the sensor data of the i-th frame image and the sensor data of the first image, the rotation matrix, scaling matrix and translation matrix are obtained respectively; The transformation matrix is ​​obtained based on the rotation matrix, the scaling matrix, and the translation matrix; After traversing the b frames preceding the first image, the multiple transformation matrices are processed into the second matrix.

5. The method according to any one of claims 2-4, characterized in that, The electronic device further includes a second module, a third module, and a fourth module; the first processing of the first image to obtain the second image includes: After obtaining the first image, the second module performs the preprocessing in the first processing on the first image to obtain the fifth image; the preprocessing includes one or more of the following: image front-end IFE processing, spatial alignment SAT processing, motion estimation GME processing, and second EIS processing; the second EIS processing is used to perform image stabilization processing on the first image using the intra-frame information of the first image. The third module calculates the first matrix and the second matrix; The fourth module performs the post-processing of the first processing on the fifth image to obtain the second image; the post-processing includes one or more of the following: IPE processing, skin smoothing processing, blurring processing and image format conversion processing.

6. The method according to claim 1, characterized in that, The second processing includes noise reduction processing; the first information is a first noise reduction parameter obtained based on the meta data of the first image and the meta data of the a-frame images preceding the first image; The electronic device includes a fifth module, wherein the step of performing a second process on the second image based on the first information to obtain a third image includes: The fifth module performs noise reduction processing on the second image based on the first noise reduction parameters to obtain the third image; the fifth module includes: GPU or IPE.

7. The method according to claim 6, characterized in that, The electronic device further includes a sixth module, wherein the second information is a second noise reduction parameter obtained based on the meta data of the first image and the meta data of the b-frame images preceding the first image; the electronic device further includes an encoder and a decoder, wherein the first video is encoded by the encoder; The step of performing the second processing on the second image based on the second information to obtain the fourth image includes: The decoder decodes the first video to obtain the second image and the second noise reduction parameter; wherein the second image is decoded from the first media track, the second noise reduction parameter is decoded from the second media track, and the timestamp of the second image corresponds to the timestamp of the second noise reduction parameter; The sixth module processes the second image based on the second noise reduction parameters to obtain the fourth image. The sixth module includes a GPU or a decoder.

8. The method according to claim 6 or 7, characterized in that, The electronic device further includes a seventh module, an eighth module, and a ninth module; the first processing of the first image to obtain a second image includes: After obtaining the first image, the seventh module performs the preprocessing in the first processing on the first image to obtain the sixth image; the preprocessing includes one or more of the following: IFE processing, SAT processing, GME processing and EIS processing; The eighth module obtains the first noise reduction parameter and the second noise reduction parameter; The ninth module performs the post-processing of the first processing on the sixth image to obtain the second image; the post-processing includes one or more of the following: IPE processing, skin smoothing processing, blurring processing and image format conversion processing.

9. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

11. A chip system, characterized in that, It includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected via a line, the at least one processor being configured to run a computer program or instructions to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1-8.

Images