Image frame processing method and related equipment

Abstract

The embodiment of the invention provides an image frame processing method and related equipment. The method comprises the following steps: acquiring a current image frame of a to-be-returned frame and to-be-returned frame time of the current image frame; judging whether frame loss exists between the current image frame and an adjacent previous image frame or not; if frame loss does not exist between the current image frame and the adjacent previous image frame, judging whether the time difference between the to-be-returned frame time of the current image frame and the frame returning time of the adjacent previous image frame is smaller than the period of collecting the image frames by the image sensor; and if the time difference between the to-be-returned frame time of the current image frame and the frame returning time of the adjacent previous image frame is smaller than the period of collecting the image frame by the image sensor, delaying a first preset compensation time and then returning the frame of the current image frame. According to the embodiment of the invention, the frame returning time of the image frame is optimized, so that the frame returning time is uniform, the fluency of a picture displayed on a preview interface or a recording interface of a camera is ensured, and the user experience is effectively optimized.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image frame processing method and related equipment. Background Art

[0002] With the development of terminal technology, smart terminal devices such as smartphones and tablets all have shooting functions, allowing users to record various scenes by taking photos or videos, and users have higher and higher requirements for shooting experience, so it is necessary to process the captured image data in real time, such as beauty, noise reduction, etc. However, it takes a certain amount of time to process the image data during the shooting process, which may cause a delay in the display of the image data, and then cause the preview interface or the recording interface to freeze, thereby affecting the user experience. Summary of the invention

[0003] In view of the above, it is necessary to provide an image frame processing method and related equipment to solve the problem that it takes a certain amount of time to process the image data during the shooting process, which causes a delay in the display of the image data, and then causes the preview interface or the recording interface to become stuck.

[0004] In a first aspect, an embodiment of the present application provides an image frame processing method, which is applied to an electronic device, and the method includes: obtaining a current image frame of a to-be-returned frame and a to-be-returned frame time of the current image frame; determining whether there is frame loss between the current image frame and an adjacent previous image frame; if there is no frame loss between the current image frame and an adjacent previous image frame, determining whether the time difference between the to-be-returned frame time of the current image frame and the return frame time of the adjacent previous image frame is less than a period of image sensor acquisition of image frames; if the time difference between the to-be-returned frame time of the current image frame and the return frame time of the adjacent previous image frame is less than a period of image sensor acquisition of image frames, returning the current image frame after a delay of a first preset compensation time.

[0005] Through the above technical solution, the frame return time of the image frame is optimized to make the frame return time uniform, thereby ensuring the smoothness of the display of the camera preview interface or recording interface, and effectively optimizing the user experience.

[0006] In a possible implementation, the method further includes: if the time difference between the waiting frame return time of the current image frame and the frame return time of the adjacent previous image frame is greater than or equal to the period of the image sensor capturing image frames, returning the current image frame and updating the frame return time.

[0007] Through the above technical solution, when the image frame return time is late, the image frame is directly returned, so that the image frame is drawn, rendered and displayed in time to ensure the smoothness of the picture.

[0008] In a possible implementation, the obtaining of the current image frame of the frame to be returned and the return time of the frame to be returned of the current image frame includes: the camera provider uses the image frame processed by the image processing node as the current image frame of the frame to be returned, and uses the time when it receives the image frame processed by the image processing node as the return time of the frame to be returned of the current image frame.

[0009] Through the above technical solution, the return time of the frame to be returned of the current image frame can be accurately determined.

[0010] In a possible implementation, the determining whether there is a lost frame between the current image frame and the adjacent previous image frame includes: determining whether the time interval between the current image frame and the adjacent previous image frame is greater than the period of the image sensor for collecting image frames; if the time interval between the current image frame and the adjacent previous image frame is greater than the period of the image sensor for collecting image frames, it is determined that there is a lost frame in the image sensor between the current image frame and the adjacent previous image frame; or if the time interval between the current image frame and the adjacent previous image frame is less than or equal to the period of the image sensor for collecting image frames, it is determined that there is no lost frame in the image sensor between the current image frame and the adjacent previous image frame.

[0011] Through the above technical solution, it can be accurately determined whether there is a lost frame in the image sensor when collecting image frames according to the frame output time interval between adjacent image frames.

[0012] In a possible implementation, the first preset compensation time is the difference between the period of the image sensor for collecting image frames and the time difference between the return time of the frame to be returned of the current image frame and the return time of the adjacent previous image frame.

[0013] Through the above technical solution, the delayed return time of the current image frame is determined according to the return time of the previous image frame, so that the return times of different image frames are uniform, ensuring the smoothness of the display screen.

[0014] In a possible implementation, the method further includes: if there is a lost frame between the current image frame and the adjacent previous image frame, determining whether the time difference between the return time of the frame to be returned of the current image frame and the return time of the adjacent previous image frame is less than a preset multiple of the period of the image sensor for collecting image frames, where the preset multiple is the number of lost frames between the current image frame and the adjacent previous image frame plus one; if the time difference between the return time of the frame to be returned of the current image frame and the return time of the adjacent previous image frame is less than the preset multiple of the period of the image sensor for collecting image frames, the current image frame is returned after delaying the second preset compensation time, and the return time is updated.

[0015] Through the above technical solution, when there is a lost frame in the image sensor, the return timing of the image frame can be determined according to the number of lost frames, ensuring the smoothness of the display screen.

[0016] In a possible implementation, the second preset compensation time is the difference between a preset multiple of the period for the image sensor to collect image frames and the time difference between the time for the current image frame to be returned and the time for the previous adjacent image frame to be returned.

[0017] Through the above technical solution, when the image sensor drops frames, the delayed return time of the current image frame is determined according to the return time of the previous image frame and the number of dropped frames, so that the return times of different image frames are uniform, ensuring the smoothness of the displayed picture.

[0018] In a possible implementation, the method further includes: if the time difference between the time for the current image frame to be returned and the time for the previous adjacent image frame to be returned is greater than or equal to a preset multiple of the period for the image sensor to collect image frames, return the current image frame and update the return time.

[0019] Through the above technical solution, when the return time of the image frame is relatively late, the image frame is directly returned, so as to perform rendering and sending display on the image frame in time, ensuring the smoothness of the picture.

[0020] In a possible implementation, the method further includes: determining whether the current image frame is the first image frame output by the image sensor of the electronic device; if the current image frame is the first image frame output by the image sensor of the electronic device, returning the current image frame and recording the return time; or if the current image frame is not the first image frame output by the image sensor of the electronic device, determining whether there are dropped frames between the current image frame and the previous adjacent image frame.

[0021] Through the above technical solution, when the current image frame is the first image frame output by the image sensor, the current image frame can be directly returned without determining whether to delay the return of the first image frame, effectively improving the return efficiency.

[0022] In a second aspect, an embodiment of the present application provides an image frame processing method applied to an electronic device. The method includes: the framework layer receives an image frame and records the return timestamp and the output timestamp of each image frame; according to the return timestamp and the output timestamp of each image frame, determining the maximum delay time between the output time and the return time of the image frame within a preset time period; determining a preset delay time according to the maximum delay time and a preset change threshold of the delay time; obtaining the current image frame to be returned and the time for the current image frame to be returned; if the actual delay time between the time for the current image frame to be returned and the output time is less than the preset delay time, returning the current image frame after delaying for a third preset compensation time.

[0023] Through the above technical solution, the delay time between the frame return and frame output of image frames within a certain time period can be sampled, the preset delay time can be determined according to the maximum delay time and the preset delay time change threshold, and the frame return timing of the current image frame can be determined according to the preset delay time, so as to improve the uniformity of the frame return time of the image frames and ensure the smoothness of the display screen.

[0024] In a possible implementation manner, the determining the maximum delay time between the frame output time and the frame return time of the image frames within the preset time period according to the frame return timestamp and the frame output timestamp of each image frame includes: calculating the difference between the frame return timestamp and the frame output timestamp of each image frame to obtain the delay time between the frame output time and the frame return time of each image frame; determining the maximum delay time according to the delay time between the frame output time and the frame return time of all the image frames within the preset time period.

[0025] Through the above technical solution, the maximum delay time between the frame return and the frame output of all the image frames within the preset time period can be accurately determined.

[0026] In a possible implementation manner, the determining the preset delay time according to the maximum delay time and the preset change threshold of the delay time includes: calculating the difference between the maximum delay time and the preset change threshold of the delay time to obtain the preset delay time.

[0027] Through the above technical solution, a suitable preset delay time can be determined according to the maximum delay time between the frame return and the frame output of all the image frames within the preset time period and the preset change threshold of the delay time, so as to ensure the uniformity of the frame return time of the image frames.

[0028] In a possible implementation manner, the method further includes: judging whether the difference between the maximum delay time and the preset change threshold of the delay time is greater than the delay time threshold; if the difference between the maximum delay time and the preset change threshold of the delay time is greater than the delay time threshold, determining the delay time threshold as the preset delay time; or if the difference between the maximum delay time and the preset change threshold of the delay time is less than or equal to the delay time threshold, determining the difference between the maximum delay time and the preset change threshold of the delay time as the preset delay time.

[0029] Through the above technical solution, it can be ensured that the preset delay time does not exceed the delay time threshold and the preset delay time is prevented from being too large.

[0030] In a possible implementation manner, the method further includes: determining the average delay time between the frame output time and the frame return time of the image frames within the preset time period according to the frame return timestamp and the frame output timestamp of each image frame.

[0031] Through the above technical solution, the average delay time between the frame output time and the frame return time of the image frames within a preset time period can be accurately determined.

[0032] In a possible implementation, the method further includes: calculating the difference between the preset delay time and the average delay time, and determining whether the absolute value of the difference between the preset delay time and the average delay time is less than or equal to a preset value; if the absolute value of the difference between the preset delay time and the average delay time is less than or equal to the preset value, determining that the preset delay time is valid.

[0033] Through the above technical solution, when the average delay time is within an acceptable range, the difference between the maximum delay time and the preset change threshold of the delay time can be determined as the preset delay time, ensuring the uniformity of the frame return time of the image frames.

[0034] In a possible implementation, the third preset compensation time is the absolute value of the difference between the preset delay time and the actual delay time, and the actual delay time is the difference between the frame return time to be awaited and the frame output time of the current image frame.

[0035] Through the above technical solution, the delayed frame return time of the current image frame can be accurately determined according to the preset delay time and the actual delay time, ensuring the uniformity of the frame return time of the image frames.

[0036] In a possible implementation, the method further includes: if the actual delay time between the frame output time and the frame return time of the current image frame is greater than or equal to the preset delay time, performing frame return on the current image frame.

[0037] Through the above technical solution, when the frame return time of the current image frame is relatively late, the frame return can be directly performed on the current image frame, ensuring the smoothness of the display screen.

[0038] In a third aspect, the present application provides an electronic device, which includes a memory and a processor: wherein, the memory is used for storing program instructions; the processor is used for reading and executing the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device is enabled to execute the above-mentioned image frame processing method.

[0039] In a fourth aspect, the present application provides a chip, which is coupled to the memory in the electronic device, and the chip is used for controlling the processor of the electronic device to execute the above-mentioned image frame processing method.

[0040] In a fifth aspect, the present application provides a computer storage medium, which stores program instructions, and when the program instructions are run on an electronic device, the processor of the electronic device is enabled to execute the above-mentioned image frame processing method.

[0041] In addition, for the technical effects brought by the third to fifth aspects, reference may be made to the descriptions related to the methods of each design in the above method section, which will not be elaborated here. Description of the Drawings

[0042] Figure 1 It is a schematic diagram of an image frame processing flow in the related art.

[0043] Figure 2 It is another schematic diagram of an image frame processing flow in the related art.

[0044] Figure 3 It is a schematic diagram of a preview interface in a multi-mirror mode provided by an embodiment of the present application.

[0045] Figure 4 It is a software architecture diagram of an electronic device provided by an embodiment of the present application.

[0046] Figure 5 It is a software architecture diagram of the camera service of an electronic device provided by an embodiment of the present application.

[0047] Figure 6 It is a schematic diagram of a frame buffer during the drawing and rendering process provided by an embodiment of the present application.

[0048] Figure 7 It is a flowchart of an image frame processing method provided by an embodiment of the present application.

[0049] Figure 8 It is a schematic diagram of the timing of frame output and display of an image frame before frame stabilization provided by an embodiment of the present application.

[0050] Figure 9 It is a schematic diagram of the timing of frame output and display of an image frame after frame stabilization provided by an embodiment of the present application.

[0051] Figure 10 It is a flowchart of an image frame processing method provided by another embodiment of the present application.

[0052] Figure 11 It is a partial processing timing schematic diagram of an image frame processing method provided by an embodiment of the present application.

[0053] Figure 12 It is a partial processing timing schematic diagram of an image frame processing method provided by another embodiment of the present application.

[0054] Figure 13 It is a hardware architecture diagram of an electronic device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0055] In the embodiments of the present application, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to mean as an example, illustration, or explanation. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that unless otherwise stated in this application, " / " means "or". For example, A / B may mean A or B. The "and / or" in this application is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, these three situations. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b, or c may mean: a, b, c, a and b, a and c, b and c, a, b, and c, these seven situations. Without conflict, the following embodiments and the features in the embodiments may be combined with each other.

[0057] For ease of understanding, some explanations of concepts related to the embodiments of the present application are given exemplarily for reference:

[0058] Frame: A single image or picture that is the smallest unit in interface display. One frame can be an image, and quickly and continuously displaying multiple consecutive frames can form a video. Processes such as drawing, rendering, and composition are usually required before the interface display frame.

[0059] Frame rate: The number of frames for refreshing images within 1 second, which can also be understood as the number of times the display screen of an electronic device refreshes the screen per second. A high frame rate can obtain smoother and more realistic animations. The more frames per second, the smoother the displayed picture.

[0060] Frame drawing: The image drawing of the display interface. The display interface can be composed of one or more views. Each view can be drawn by visual controls of the view system. Each view is composed of sub-views, and one sub-view corresponds to a widget in the view. For example, one sub-view corresponds to a symbol in the display interface.

[0061] Frame rendering: Perform coloring operations on the drawn view or add 3D effects, etc. For example, the 3D effects can be lighting effects, shadow effects, texture effects, etc.

[0062] Frame composition: The process of combining one or more rendered views into a display interface.

[0063] Vertical synchronization (VSync) signal: A signal used to control the start of processes such as frame drawing, rendering, composition, and display. To ensure smooth display and avoid phenomena such as display stuttering, electronic devices usually perform display based on the VSync signal, thereby synchronizing processes such as image drawing, rendering, composition, and display screen refreshing.

[0064] The VSync signal is a periodic signal, and the period of the VSync signal can be set according to the display screen refresh rate. For example, when the display screen refresh rate is 60 Hz, the VSync signal period can be 16.6 ms, that is, the electronic device generates a control signal every 16.6 ms to trigger the VSync signal periodically. Another example is that when the display screen refresh rate is 90 Hz, the VSync signal period can be 11.1 ms, that is, the electronic device generates a control signal every 11.1 ms to trigger the VSync signal periodically.

[0065] The VSync signal includes software VSync signals, such as VSync-APP signal and VSync-SF signal, and the VSync signal also includes hardware VSync signals, such as VSync-HW signal. Among them, the VSync-APP signal is used to trigger the drawing and rendering process; the VSync-SF signal is used to trigger the composition process. The VSync-HW signal is used to trigger the display screen refreshing process. The software VSync signal and the hardware VSync signal maintain cycle synchronization. Taking the change of 60 Hz and 120 Hz as an example, if the period of the VSync-HW signal switches from 60 Hz to 120 Hz, the periods of the VSync-APP signal and the VSync-SF signal change synchronously, from 60 Hz to 120 Hz.

[0066] With the development of terminal technology, intelligent terminal devices such as smartphones and tablets all have a shooting function, enabling users to record various scenes by taking photos or shooting videos, and users' requirements for shooting experience are also getting higher and higher. Therefore, it is necessary to perform real-time processing on the captured image data, such as beauty and noise reduction. However, processing the image data during shooting takes a certain amount of time, which may cause a delay in the display of the image data, and then cause stuttering in the preview interface or recording interface, thus affecting the user experience.

[0067] Refer to Figure 1 As shown, it is a schematic diagram of the image frame processing flow in the related art. When the camera application of the electronic device receives a shooting request, the camera service allocates the processing of the image frame, generates a camera pipeline (camera pipeline) in response to the shooting request, including the image sensor acquiring the image frame, transmitting the image frame to nodes node1 to nodeX for processing, and transmitting the processed image frame to the framework layer for drawing and rendering. The drawn and rendered image frame is transmitted to the composition service (SurfaceFlinger), and the composition service composes the image frame onto the display screen, and the display screen displays the image frame. The drawing and rendering service draws and renders the display interface according to the display screen refresh rate and the VSync signal. When receiving the VSync signal, it determines whether the layer queue (BufferQueue) includes a frame buffer (cached image frame). If the layer queue includes a frame buffer, it draws and renders and displays the image frame at the front of the layer queue. If the layer queue does not include a frame buffer, the process of drawing and rendering and displaying is not executed. The image sensor usually outputs frames according to a pre-configured frequency, such as 30Hz, 60Hz, 120Hz, 240Hz, etc. Outputting frames includes acquiring an image frame and sending the acquired image frame to the next node (such as an image processing node) for processing. However, the processing of the image frame by the node may cause a delay when the image frame is transmitted to the drawing and rendering service of the framework layer, that is, the frame return time is delayed, resulting in the drawing and rendering service being unable to draw and display the image frame in time. When the drawing and rendering cycle of the drawing and rendering service for the image frame is greater than the frame output cycle of the image sensor, the situation of dropped display frames will occur.

[0068] Refer to Figure 2 As shown, it is another schematic diagram of the image frame processing flow in the related art. When the camera application of the electronic device receives a shooting request in the multi-camera mode, the camera service allocates the processing of the image frames acquired by multiple lenses, generates a camera pipeline in response to the shooting request in the multi-camera mode, including: image sensor 1 acquires an image frame, transmits the image frame to nodes a1 to ax for processing, and transmits the processed image frame to the framework layer; image sensor 2 acquires an image frame, transmits the image frame to nodes b1 to bx for processing, and transmits the processed image frame to the framework layer. The drawing and rendering service of the framework layer draws and renders the image frames of image sensors 1 and 2, transmits the drawn and rendered image frames to the composition service, and the composition service composes the image frames onto the display screen, and the display screen displays the image frames acquired by different image sensors in a split-screen or picture-in-picture manner. Since the processing of the image frames by nodes a1 to ax and nodes b1 to bx takes a certain amount of time, and the time taken may be different, the transmission to the framework layer is out of sync, and as a result, the split-screen or picture-in-picture display is out of sync.

[0069] For example, referring to Figure 3 As shown, split screen 1 and split screen 2 are respectively the images captured by two lenses of the electronic device. If the frame return times of the image sensors of the two lenses are out of sync after the frames are output, the images in split screen 1 and split screen 2 will be out of sync, thus affecting the user's visual experience. It can be understood that the multi-lens mode in the embodiments of the present application is only used for illustrative purposes, and in fact, it is not limited to the multi-lens mode or the main character mode, and can also be applied to other shooting modes.

[0070] In order to avoid the camera preview interface or the recording interface from being stuck due to dropped frames in display, or to avoid the images captured by different lenses in the multi-lens mode from being out of sync during recording, the embodiments of the present application provide an image frame processing method to optimize the frame return time of the image frame (for example, the time when the image frame is transmitted to the framework layer), so that the frame return time of the image frame is uniform, thereby ensuring the smoothness of the images displayed on the camera preview interface or the recording interface, and effectively optimizing the user experience.

[0071] Referring to Figure 4 As shown, it is the software architecture diagram of the electronic device provided by the embodiments of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system is divided into four layers, from top to bottom, which are the application layer 101, the framework layer 102, the Android runtime and the system library 103, the hardware abstraction layer 104, the kernel layer 105, and the hardware layer 106.

[0072] The application layer 101 may include a series of application packages. For example, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and device control service.

[0073] The framework layer 102 provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0074] Among them, the window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make this data accessible to application programs. The said data may include videos, images, audios, dialed and received calls, browsing histories and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide the communication functions of the electronic device. For example, the management of call states (including connection, disconnection, etc.). The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables application programs to display notification information in the status bar, can be used to convey notification-type messages, can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the completion of a download, message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt tone, the electronic device vibrates, the indicator light flashes, etc.

[0075] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system. The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android.

[0076] The application layer 101 and the framework layer 102 run in the virtual machine. The virtual machine executes the Java files of the application layer and the framework layer as binary files. The virtual machine is used to execute functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0077] The system library 103 can include multiple functional modules. For example, a surface manager, Media Libraries, a 3D graphics processing library (such as: OpenGL ES), a 2D graphics engine (such as SGL), etc.

[0078] Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is the drawing engine for 2D drawing.

[0079] The hardware abstraction layer 104 runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer.

[0080] The kernel layer 105 is the layer between the hardware and the software. The kernel layer 105 includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

[0081] The kernel layer 105 is the core of the operating system of the electronic device, is the first layer of software extension based on the hardware, provides the most basic functions of the operating system, is the basis for the operating system to work, is responsible for managing the system's processes, memory, device drivers, files, and network systems, and determines the performance and stability of the system. For example, the kernel can determine the operation time of an application program for a certain part of the hardware.

[0082] The kernel layer 105 includes programs closely related to the hardware, such as interrupt handlers, device drivers, etc., and also includes basic, common, and frequently running modules, such as a clock management module, a process scheduling module, etc., and also includes key data structures. The kernel layer can be set in the processor or solidified in the internal memory.

[0083] The hardware layer 106 includes the hardware of the electronic device, such as a display screen, buttons, a camera, etc.

[0084] Refer to Figure 5 As shown, it is the software architecture diagram of the camera service of the electronic device provided by the embodiment of the present application. In response to the user's operation of opening the camera application, the camera application creates a camera process, instantiates the camera service (CameraService), the cache queue corresponding to the camera application (CameraBufferQuene), the cache reading service (BlastBufferQueue, BBQ), the synthesis service (SurfaceFlinger), the camera provider (CameraProvider), the hardware composer (Hardware Composer, HWC), the signal center (DispSync), etc., and creates a layer of the control to be displayed in the camera process, such as Figure 3Layers of control widgets 1, 2, 3, 4, etc. Among them, control widgets 1, 2, 3, 4 can be control widgets for selecting shooting modes, shooting controls, controls for setting shooting parameters, controls for opening the gallery application, and lens switching controls.

[0085] In an embodiment of the present application, HWC periodically generates a VSync-HW signal according to the refresh frequency of the display screen. DispSync assigns a corresponding VSync-APP signal to the user interface (UI) thread corresponding to the camera application according to the periodically generated VSync-HW signal, and periodically sends the generated VSync-APP signal to the UI thread of the camera application, so that after receiving the VSync-APP signal, the UI thread triggers the processing of drawing and rendering, such as the drawing and rendering of control widgets 1, 2, 3, 4, etc.

[0086] In an embodiment of the present application, DispSync also assigns a corresponding VSync-SF signal to SurfaceFlinger according to the periodically generated VSync-HW signal, and periodically sends the generated VSync-SF signal to SurfaceFlinger, so that after receiving the VSync-SF signal, SurfaceFlinger obtains an image frame from the CameraBufferQuene through BBQ, and then performs a synthesis process on the obtained image frame. Among them, the image frames cached in the CameraBufferQuene can be image frames obtained after the CameraProvider processes the image frames in the original image stream collected by the camera. The collection of the original image stream can be continuously collected by the camera according to the image preview request generated by the camera application. Among them, the generation of the image preview request can be generated by the camera application according to the current shooting mode after the camera application is started.

[0087] In an embodiment of the present application, the detection algorithms and image processing logics corresponding to different shooting modes may be different. Therefore, when generating a corresponding image preview request according to the shooting mode and transmitting the image preview request to the CameraProvider through the CameraService, the CameraProvider can perform corresponding detection and processing on the image frames in the original image stream collected by the camera according to the image preview request.

[0088] In an embodiment of the present application, after receiving an image preview request transmitted by the CameraService, the CameraProvider transmits the image preview request to the camera driver. The camera driver then sends the image preview request to the camera and drives the camera to collect image frames according to the continuously transmitted image preview requests.

[0089] In an embodiment of the present application, after the camera collects image frames according to the continuously transmitted image preview requests, it continuously reports the image frames in the raw image stream to the camera driver. Then, the camera driver transmits the image frames to the CameraProvider at a preset frame rate. The CameraProvider performs corresponding detection and processing on the image frames. The CameraProvider transmits the processed image frames to the CameraService, and the CameraService adds the image frames to the CameraBufferQuene, that is Figure 5 the queue operation in

[0090] In an embodiment of the present application, the SurfaceFlinger performs composite processing on the image frames cached in the CameraBufferQuene, which is triggered by the VSync-SF signal periodically generated by DispSync. After receiving the VSync-SF signal, the SurfaceFlinger obtains the cached image frames from the CameraBufferQuene through the BBQ, for example Figure 5 the acquire operation in Figure 5 the Apply / setTransactionState operation in

[0091] If the SurfaceFlinger obtains the cached image frames from the CameraBufferQuene through the BBQ, it performs composite processing on the image frames to obtain the composite image corresponding to the image frames, and transmits the obtained composite image to the HWC. At the same time, the SurfaceFlinger sends an instruction to destroy (for example Figure 5 the release operation in

[0092] Among them, the dequeue function is used to remove the first function in the specified queue of each matching element and execute the removed function. For example, it deletes the image frame at the head of the queue. When SurfaceFlinger obtains the cached image frame from the CameraBufferQuene through the BBQ, it can obtain it from the head of the queue. Therefore, after synthesizing the image, by calling the dequeue function in the above manner, the used image frame can be deleted from the CameraBufferQuene.

[0093] The HWC transfers the synthesized image sent by SurfaceFlinger to the display driver, and the display driver drives the display screen to display the picture corresponding to the synthesized image. Taking the control 1 in the camera application as an example, which is a Surface View control for displaying the preview image, the picture obtained through the above processing will finally be displayed in the control 1 on the display screen.

[0094] According to Figure 5 As shown in the preview stream display process, from the acquisition of the image frame to its final display in the preview interface, it mainly involves the following links: the link where the camera transmits the acquired image frame to the CameraProvider through the camera driver (frame output link), the link where the CameraProvider processes the original image frame, and the link where the processed image frame by the CameraProvider is transmitted to the CameraService (frame return link), the link where the CameraService adds the processed image frame to the CameraBufferQuene (caching link), the link where SurFaceFlinger extracts the image frame from the CameraBufferQuene through the BBQ for synthesis (synthesis link), and the link where the hardware synthesizer transmits the picture synthesized by SurFaceFlinger to the display driver, and then drives the display screen to display the synthesized picture (display link).

[0095] Based on the above process, the synthesis operation of the image frame by the SurfaceFlinger thread (SF thread) is triggered based on the VSync - SF signal, and the operation of the display driver finally driving the display screen to display the picture corresponding to the image frame is triggered based on the VSync - HW signal, and the time interval between these two signals is usually fixed, for example, it is one VSync signal period. Refer to Figure 6 As shown, assuming that the time intervals between any two adjacent time points from time point t1 to time point t8 are the same, for example, it is one VSync signal period. If the SF thread receives the VSync - SF signal at each time point from t1 to t8, it will obtain the corresponding image frame through the BBQ for synthesis.

[0096] If no frames are dropped, the composition time of each frame is completed within a fixed time. For example, when it is completed within one VSync signal cycle, for example, within t1 to t2, the composition process of frame 1 is completed; within t2 to t3, the composition process of frame 2 is completed; within t3 to t4, the composition process of frame 3 is completed; within t4 to t5, the composition process of frame 4 is completed; within t5 to t6, the composition process of frame 5 is completed; within t6 to t7, the composition process of frame 6 is completed; within t7 to t8, the composition process of frame 7 is completed, and so on. When the VSync - SF signal lags by one VSync signal cycle, a VSync - HW signal will be received at time point t2. At this time, since the SF thread has completed the composition process of frame 1, the hardware compositor receives the composed frame 1 from the SF thread and then transmits it to the display driver for display, that is, the display driver will drive the display screen to display the picture corresponding to frame 1 within the VSync signal cycle from t2 - t3.

[0097] After receiving the next VSync - HW signal at time point t3, since the SF thread has completed the composition process of frame 2 at this time, the hardware compositor receives the composed frame 2 from the SF thread and then transmits it to the display driver for display, that is, the display driver can drive the display screen to display the picture corresponding to frame 2 within the VSync signal cycle from t3 to t4. Similarly, the display driver can drive the display screen to display the picture corresponding to frame 3 within the VSync signal cycle from t4 to t5, display the picture corresponding to frame 4 within the VSync signal cycle from t5 to t6, display the picture corresponding to frame 5 within the VSync signal cycle from t6 to t7, and display the picture corresponding to frame 6 within the VSync signal cycle from t7 to t8.

[0098] In the case of no frame drops and the composition process not timing out, as the user's shooting requirements continue to increase, the improvement of image pixels, and the increase in the detection and processing links involved in the shooting mode, the duration of the image processing process will also increase, resulting in uneven frame return time or display time, thus causing stuttering in the preview interface or recording interface displayed on the display screen.

[0099] Refer to Figure 7 As shown, it is a flowchart of an image frame processing method provided by an embodiment of the present application. The method is applied to an electronic device, and the image frame processing method includes:

[0100] S101, obtain the current image frame to be returned and the time to return the current image frame.

[0101] In an embodiment of the present application, after the user starts the camera application, the camera application automatically generates a preview shooting request and sends it layer by layer, or in response to a shooting request input by the user, sends the shooting request layer by layer. For example, when the shooting control of the camera application is triggered, it can be determined that the camera application has received the shooting request; or when the shooting shortcut key of the electronic device is triggered, it can be determined that the camera application has received the shooting request. The process of the camera application sending the shooting request layer by layer may include: the camera application in the application layer sends the shooting request to the camera service in the framework layer, the camera service sends the shooting request to the camera provider in the hardware abstraction layer, the camera provider sends the shooting request to the camera driver in the kernel layer, the camera driver sends the shooting request to the camera in the hardware layer, and drives the image sensor of the camera to collect an image frame, such as an image of the scene in front of the shooting camera, and transmits the collected image frame to the camera driver, and the camera driver transmits the image frame to the camera provider.

[0102] Then, the camera provider transmits the image frame collected by the image sensor to multiple image processing nodes in the hardware abstraction layer for processing. After the image processing nodes complete the processing of the image frame, they return the processed image frame to the camera provider. The camera provider uses the image frame processed by the image processing nodes as the current image frame of the frame to be returned, and takes the time when it receives the image frame processed by the image processing nodes as the return time of the current image frame, and records the return time of the current image frame as g_curtime. Here, returning the frame means that the camera provider transmits the processed image frame to the camera service in the framework layer, and the camera service transmits the image frame to the rendering service for rendering and display. Here, the return time of the frame and other times in the embodiments of the present application are all times measured by a timer after the camera application is started.

[0103] S102, determine whether the current image frame is the first image frame output by the image sensor. If the current image frame is the first image frame output by the image sensor, execute S103; if the current image frame is not the first image frame output by the image sensor, execute S104.

[0104] In an embodiment of the present application, the camera provider determines whether the current image frame is the first image frame output by the image sensor in response to the current shooting request. If the current image frame is the first image frame output by the image sensor in response to the current shooting request, it is determined that the current image frame is the first image frame output by the image sensor; if the current image frame is not the first image frame output by the image sensor in response to the current shooting request (for example, the second image frame is output), it is determined that the current image frame is not the first image frame output by the image sensor.

[0105] S103, Return the current image frame and record the return time.

[0106] In an embodiment of the present application, if the current image frame is the first image frame output by the image sensor, directly return the current image frame, record the time to be returned g_curtime as the actual return time g_pretime. At this time, the current image frame is the previous adjacent image frame relative to the next adjacent image frame.

[0107] S104, Determine whether there is a lost frame between the current image frame and the previous adjacent image frame. If there is no lost frame between the current image frame and the previous adjacent image frame, execute S105; if there is a lost frame between the current image frame and the previous adjacent image frame, execute S108.

[0108] In an embodiment of the present application, after receiving a shooting request, the image sensor responds to the shooting request, performs image acquisition according to a preset sampling frequency, and sends the acquired image frame to the camera driver. The preset sampling frequency can be 30Hz, 60Hz, 120Hz or other frequencies. And according to the preset sampling frequency of the image sensor, the period for the image sensor to acquire image frames can be determined. For example, if the preset sampling frequency is 30Hz, the period for the image sensor to acquire image frames is 33.3ms (milliseconds); if the preset sampling frequency is 60Hz, the period for the image sensor to acquire image frames is 16.6ms.

[0109] In an embodiment of the present application, each time the camera provider receives an image frame output (i.e., frame out) by the image sensor, it records the timestamp corresponding to the image frame. According to the timestamp of the current image frame output by the image sensor and the timestamp of the previous adjacent image frame, it determines the time interval between the current image frame and the previous adjacent image frame, and determines whether the time interval between the current image frame and the previous adjacent image frame is greater than the period for the image sensor to acquire image frames. If the time interval between the current image frame and the previous adjacent image frame is greater than the period for the image sensor to acquire image frames, it is determined that there is a lost frame in the image sensor between the current image frame and the previous adjacent image frame; if the time interval between the current image frame and the previous adjacent image frame is less than or equal to the period for the image sensor to acquire image frames, it is determined that there is no lost frame in the image sensor between the current image frame and the previous adjacent image frame.

[0110] S105. Determine whether the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is less than the period of the image sensor for acquiring image frames. If the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is less than the period of the image sensor for acquiring image frames, execute S106; if the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is greater than or equal to the period of the image sensor for acquiring image frames, execute S107.

[0111] In an embodiment of the present application, the return frame time of the current image frame is g_curtime, the return frame time of the adjacent previous image frame is g_pretime, and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is time_interval, and time_interval = g_curtime - g_pretime. For example, if g_curtime is 45 ms and g_pretime is 10 ms, then time_interval is 35 ms.

[0112] S106. After delaying for the first preset compensation time, return the current image frame and update the return frame time.

[0113] In an embodiment of the present application, the first preset compensation time is the difference between the period of the image sensor for acquiring image frames and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame. For example, if the period of the image sensor for acquiring image frames is 33.3 ms and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is 20 ms, then the first preset compensation time is 33.3 ms - 20 ms = 13.3 ms. After returning the frame, update the return frame time, record the return frame time g_curtime as the actual return frame time g_pretime. At this time, the current image frame is the adjacent previous image frame relative to the adjacent next image frame.

[0114] S107. Return the current image frame and update the return frame time.

[0115] In an embodiment of the present application, if the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is greater than or equal to the period of the image sensor for acquiring image frames, directly return the current image frame. After returning the frame, update the return frame time, record the return frame time g_curtime as the actual return frame time g_pretime. At this time, the current image frame is the adjacent previous image frame relative to the adjacent next image frame.

[0116] S108, Determine whether the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is less than a preset multiple of the period of the image sensor for acquiring image frames. If the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is less than the preset multiple of the period of the image sensor for acquiring image frames, execute S109; if the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is greater than or equal to the preset multiple of the period of the image sensor for acquiring image frames, execute S110.

[0117] In an embodiment of the present application, the preset multiple is the number of dropped frames between the current image frame and the adjacent previous image frame plus one. Subtract the output frame timestamp of the adjacent previous image frame from the output frame timestamp of the current image frame to obtain the time difference between the output frame time of the current image frame and the output frame time of the adjacent previous image frame. Subtract one from the integer part of the value obtained by dividing the time difference between the output frame time of the current image frame and the output frame time of the adjacent previous image frame by the period of the image sensor for acquiring image frames to obtain the number of dropped frames between the current image frame and the adjacent previous image frame.

[0118] S109, After delaying the second preset compensation time, return the current image frame and update the return frame time.

[0119] In an embodiment of the present application, the second preset compensation time is the difference between the preset multiple of the period of the image sensor for acquiring image frames and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame. For example, if the period of the image sensor for acquiring image frames is 33.3 ms and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is 60 ms, then the second preset compensation time is 66.6 ms - 60 ms = 6.6 ms. After returning the frame, update the return frame time, record the return frame time g_curtime as the actual return frame time g_pretime. At this time, the current image frame is the adjacent previous image frame relative to the adjacent next image frame.

[0120] S110, Return the current image frame and update the return frame time.

[0121] In an embodiment of the present application, if the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is less than the preset multiple of the period of the image sensor for acquiring image frames, directly return the current image frame. After returning the frame, update the return frame time, record the return frame time g_curtime as the actual return frame time g_pretime. At this time, the current image frame is the adjacent previous image frame relative to the adjacent next image frame.

[0122] Refer to Figure 8As shown in the figure, it is a timing schematic diagram of the output and display of the pre-stabilization image frame provided by the embodiment of the present application. After the image sensor outputs the image frame for the first time, the rendering service of the hardware abstraction layer renders and displays the first image frame after the first image frame is returned. If the return time of the image frame is less than the theoretical preview frame interval period, it means that the return time of the image frame is too early. At this time, if the frame is directly returned, it may overwrite the frame buffer that has not been completely rendered in the cache queue, resulting in interface lag. If the return time of the image frame is greater than the theoretical preview frame interval period, it means that the return time of the image frame is too late. At this time, if the frame is not directly returned, the rendering service cannot render the next interface, resulting in interface lag.

[0123] Refer to Figure 9 As shown in the figure, it is a timing schematic diagram of the output and display of the post-stabilization image frame provided by the embodiment of the present application.

[0124] After the image sensor outputs the image frame for the first time, the rendering service of the hardware abstraction layer renders and displays the first image frame after the first image frame is returned. If the return time of the image frame is less than the theoretical preview frame interval period, it means that the return time of the image frame is too early. At this time, the frame return is delayed, so that after the frame buffer in the cache queue is completely rendered, it is added to the frame buffer waiting for rendering. If the return time of the image frame is greater than the theoretical preview frame interval period, it means that the return time of the image frame is too late. At this time, the frame is directly returned, so that the rendering service can render the next interface in time, avoiding interface lag.

[0125] Refer to Figure 10 As shown in the figure, it is a flowchart of an image frame processing method provided by another embodiment of the present application. The method is applied to an electronic device, and the image frame processing method includes:

[0126] S201, the framework layer receives the image frame and records the return timestamp and output timestamp of each image frame.

[0127] In an embodiment of the present application, after the user activates the camera application, the camera application automatically generates a preview shooting request and sends it layer by layer, or in response to a shooting request input by the user, sends the shooting request layer by layer. For example, when the shooting control of the camera application is triggered, it can be determined that the camera application has received the shooting request; or when the shooting shortcut key of the electronic device is triggered, it can be determined that the camera application has received the shooting request. The process of the camera application sending the shooting request layer by layer may include: the camera application in the application layer sends the shooting request to the camera service in the framework layer, the camera service sends the shooting request to the camera provider in the hardware abstraction layer, the camera provider sends the shooting request to the camera driver in the kernel layer, the camera driver sends the shooting request to the camera in the hardware layer, and drives the image sensor of the camera to collect an image frame, such as an image of the scene in front of the shooting camera, and transmits the collected image frame to the camera driver, and the camera driver transmits the image frame to the camera provider.

[0128] Then, the camera provider transmits the image frame collected by the image sensor to multiple image processing nodes in the hardware abstraction layer for processing. After the image processing nodes complete the processing of the image frame, they return the processed image frame to the camera provider. The camera provider sends the image frame processed by the image processing nodes to the camera service in the framework layer for display. The camera service uses the timestamp of receiving the image frame as the return frame timestamp of the image frame and records it as g_curtime. Here, returning the frame means that the camera provider transmits the processed image frame to the camera service in the framework layer, and the camera service transmits the image frame to the rendering service for rendering and display. Here, the return frame time and other times in the embodiments of the present application are times measured by a timer after the camera application is activated.

[0129] In an embodiment of the present application, the image sensor collects images according to a preset sampling frequency and sends the collected image frames to the camera driver, thereby performing the out-frame of the image frames. Each time the image sensor collects an image frame, it records the out-frame timestamp sensortimestamp of the image frame. For example, the preset sampling frequency is 30Hz, the out-frame timestamp of the first image frame is 0ms, the out-frame timestamp of the second image frame is 33.3ms, the out-frame timestamp of the third image frame is 66.6ms, and so on.

[0130] S202, according to the return frame timestamp and the out-frame timestamp of each image frame, determine the maximum delay time between the out-frame time and the return frame time of the image frames within a preset time period.

[0131] In an embodiment of the present application, by calculating the difference between the return frame timestamp g_curtime and the output frame timestamp sensortimestamp of each image frame, the delay time offset between the output frame time and the return frame time of each image frame is obtained. According to the delay time offset between the output frame time and the return frame time of all image frames within a preset time period, the maximum delay time maxoffset is determined. Wherein, the preset time period can be 5 minutes, 10 minutes, 15 minutes or other time.

[0132] In another embodiment of the present application, the average delay time avarageoffset can also be determined according to the delay time offset between the output frame time and the return frame time of all image frames within a preset time period.

[0133] In another embodiment of the present application, the minimum delay time minoffset can also be determined according to the delay time offset between the output frame time and the return frame time of all image frames within a preset time period.

[0134] S203. Determine the preset delay time according to the maximum delay time and the preset change threshold of the delay time.

[0135] In an embodiment of the present application, the preset delay time is the difference between the maximum delay time and the preset change threshold of the delay time. For example, if the maximum delay time is 115ms and the preset change threshold of the delay time is 15ms, the preset delay time is 100ms.

[0136] In another embodiment of the present application, calculate the difference between the maximum delay time and the preset change threshold of the delay time, and determine whether the difference between the maximum delay time and the preset change threshold of the delay time is greater than the delay time threshold. If the difference between the maximum delay time and the preset change threshold of the delay time is greater than the delay time threshold, determine the delay time threshold as the preset delay time; if the difference between the maximum delay time and the preset change threshold of the delay time is less than or equal to the delay time threshold, determine the difference between the maximum delay time and the preset change threshold of the delay time as the preset delay time. Wherein, the delay time threshold is the maximum delay time that the human eye can feel, such as 110ms, 120ms, 130ms or other time.

[0137] In another embodiment of the present application, S203 can be replaced by: determining the preset delay time according to the maximum delay time, the average delay time and the preset change threshold of the delay time.

[0138] In another embodiment of the present application, after determining the preset delay time, calculate the difference between the preset delay time and the average delay time, and determine whether the absolute value of the difference between the preset delay time and the average delay time is less than or equal to a preset value. If the absolute value of the difference between the preset delay time and the average delay time is less than or equal to the preset value, determine that the preset delay time is valid, and determine this preset delay time as the final preset delay time, that is, determine the difference between the maximum delay time and the preset change threshold of the delay time as the preset delay time; if the absolute value of the difference between the preset delay time and the average delay time is greater than the preset value, abandon the setting of the preset delay time, and directly return the current image frame when receiving the current image frame of the frame to be returned. Among them, the preset value is 15ms, 20ms, 25ms or other values.

[0139] In one embodiment of the present application, the preset change threshold of the delay time is greater than or equal to 0 and less than or equal to the difference between the maximum delay time and the average delay time.

[0140] S204, obtain the current image frame of the frame to be returned and the time of the frame to be returned of the current image frame.

[0141] The specific implementation manner of S204 is the same as that of S101, and will not be elaborated here.

[0142] S205, determine whether the actual delay time between the time of the frame to be returned of the current image frame and the frame output time is less than the preset delay time. If the actual delay time between the time of the frame to be returned of the current image frame and the frame output time is less than the preset delay time, execute S206; if the actual delay time between the time of the frame to be returned of the current image frame and the frame output time is greater than or equal to the preset delay time, execute S207.

[0143] In one embodiment of the present application, by calculating the difference between the time of the frame to be returned of the current image frame and the frame output time, the actual delay time between the time of the frame to be returned of the current image frame and the frame output time is obtained.

[0144] S206, return the current image frame after delaying the third preset compensation time.

[0145] In one embodiment of the present application, the third preset compensation time is the absolute value of the difference between the preset delay time and the actual delay time. For example, if the actual delay time is 120ms and the preset delay time is 100ms, then the third preset compensation time is 20ms.

[0146] It can be understood that if the actual delay time between the frame return time and the frame output time of the current image frame is less than the preset delay time, in order to prevent the current image frame from being cached in the cache queue in advance and overwriting the previous image frame that has not completed rendering and display, the frame return of the current image frame is performed after delaying the third preset compensation time.

[0147] S207, perform frame return on the current image frame.

[0148] It can be understood that if the actual delay time between the frame output time and the frame return time of the current image frame is greater than or equal to the preset delay time, in order to prevent the rendering service from being unable to perform rendering and display for a long time, the frame return of the current image frame is directly performed.

[0149] Refer to Figure 11 As shown, it is a partial processing timing diagram of the image frame processing method provided by an embodiment of the present application.

[0150] S301, the electronic device receives the operation of the user to open the camera application and select the shooting mode.

[0151] S302, open the camera application and set the shooting mode.

[0152] S303, the camera application responds to the shooting request and allocates the camera pipeline. Among them, the shooting request can be a preview shooting request automatically generated after the camera application is opened, or a shooting request input by the user triggering the shooting control.

[0153] S304, the camera application sends the shooting request to the camera module. Among them, the camera module is a camera including an image sensor.

[0154] S305, the camera module responds to the shooting request and performs a shooting operation to collect an image frame. Among them, the camera module collects image frames based on a preset sampling frequency.

[0155] S306, record the frame output timestamp of each image frame during the shooting process.

[0156] S307, record the frame return timestamp of each image frame. Among them, the frame return timestamp is obtained by recording the timestamp of the processed image frame received by the recording framework layer from the hardware abstraction layer.

[0157] S308, according to the frame output timestamps and frame return timestamps of multiple image frames, calculate the minimum offset time, maximum offset time, and average offset time between the frame output time and the frame return time of the image frames within the threshold time. Among them, the threshold time is the preset time period in the above embodiment, and the offset time is the delay time in the above embodiment.

[0158] S309, cache the minimum offset time, maximum offset time, average offset time, and the frame rate of the image frames acquired by the image sensor.

[0159] Through the above embodiments of the present application, sample the frame output time and frame return time of the image frames within the threshold time (i.e., the preset time period), and determine the minimum offset time, maximum offset time, and average offset time between the frame output and frame return of the image frames within the threshold time, which serve as the basis for calculating the preset offset time in the subsequent process.

[0160] Refer to Figure 12 As shown, it is another schematic diagram of the processing timing sequence of the image frame processing method provided by an embodiment of the present application.

[0161] S401, the electronic device receives the operation of the user to open the camera application and select the shooting mode.

[0162] S402, open the camera application and set the shooting mode.

[0163] S403, the camera application responds to the shooting request and allocates the camera pipeline. Among them, the shooting request can be a preview shooting request automatically generated after the camera application is opened, or a shooting request input by the user triggering the shooting control.

[0164] S404, the camera application sends the shooting request to the camera module.

[0165] S405, the camera module responds to the shooting request to perform the shooting operation and acquire the image frames.

[0166] S406, set the calibration offset time according to the cached minimum offset time, maximum offset time, and average offset time. Among them, the calibration offset time is the preset delay time in the above embodiments, and the determination method of the calibration offset time is the same as that of the preset delay time, which will not be elaborated here.

[0167] S407, record the frame output timestamp of each image frame during the shooting process.

[0168] S408, record the frame return timestamp to be awaited for each image frame.

[0169] S409. Determine the frame return delay time of the current image frame according to the calibration offset time and the actual offset time. Specifically, calculate the difference between the timestamp of the frame to be returned and the timestamp of the frame output for the current image frame to obtain the actual offset time between the frame return time and the frame output time of the current image frame, and determine whether the actual offset time between the frame return time and the frame output time of the current image frame is less than the calibration offset time. If the actual offset time between the frame return time and the frame output time of the current image frame is less than the calibration offset time, delay the third preset compensation time to return the current image frame. The third preset compensation time is the absolute value of the difference between the calibration offset time and the actual offset time.

[0170] S411. If the current offset time between the frame output time and the frame return time of the current image frame is greater than or equal to the calibration offset time, directly return the current image frame.

[0171] Through the above embodiments of the present application, calculate the preset delay time according to the minimum delay time, the maximum delay time, and the average delay time between the frame output and the frame return within the threshold time, and determine the frame return timing of the current image frame according to the actual frame return time and the preset delay time of the current image frame, so as to ensure the uniformity of the frame return time of the image frame and improve the smoothness of the picture.

[0172] The embodiments of the present application also provide an electronic device 100. Refer to Figure 13 As shown, the electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a netbook, a cellular phone, a Personal Digital Assistant (PDA), an Augmented Reality (AR) device, a Virtual Reality (VR) device, an Artificial Intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The embodiments of the present application do not impose special restrictions on the specific type of the electronic device 100.

[0173] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0174] It can be understood that the structure schematically shown in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0175] The processor 110 may include one or more processing units. For example, the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and / or a Neural-network Processing Unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0176] The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0177] A memory can also be provided in the processor 110 for storing instructions and data. In an embodiment of the present application, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can directly call them from the memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0178] In an embodiment of the present application, the processor 110 may include one or more interfaces. The interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0179] The I2C interface is a bidirectional synchronous serial bus, including a Serial Data Line (SDA) and a Serial Clock Line (SCL). In an embodiment of the present application, the processor 110 may include multiple groups of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: The processor 110 can be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 to communicate with the touch sensor 180K through the I2C bus interface to implement the touch function of the electronic device 100.

[0180] The I2S interface can be used for audio communication. In an embodiment of the present application, the processor 110 may include multiple groups of I2S buses. The processor 110 can be coupled to the audio module 170 through the I2S bus to achieve communication between the processor 110 and the audio module 170. In an embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0181] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In an embodiment of the present application, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In an embodiment of the present application, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to achieve the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0182] The UART interface is a general-purpose serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In an embodiment of the present application, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example: The processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to achieve the Bluetooth function. In an embodiment of the present application, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to achieve the function of playing music through a Bluetooth headset.

[0183] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes the Camera Serial Interface (CSI), the Display Serial Interface (DSI), etc. In an embodiment of the present application, the processor 110 and the camera 193 communicate through the CSI interface to achieve the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to achieve the display function of the electronic device 100.

[0184] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In an embodiment of the present application, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0185] The USB interface 130 is an interface compliant with the USB standard specification, which can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. The interface can also be used to connect other electronic devices 100, such as AR devices, etc.

[0186] It can be understood that the interface connection relationships between the modules schematically shown in the embodiments of the present invention are only illustrative descriptions and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0187] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 through the power management module 141.

[0188] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0189] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0190] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0191] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through Antenna 1, and perform processing such as filtering and amplifying on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate it out. In an embodiment of the present application, at least some functional modules of the mobile communication module 150 can be disposed in the processor 110. In an embodiment of the present application, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0192] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In an embodiment of the present application, the modulation and demodulation processor can be an independent device. In some other embodiments, the modulation and demodulation processor can be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0193] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0194] In an embodiment of the present application, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0195] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image frame processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0196] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a Microled, a Micro-OLED, a quantum dot light-emitting diode (Quantum Dot Light Emitting Diodes, QLED), etc. In an embodiment of the present application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0197] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0198] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In an embodiment of the present application, the ISP can be set in the camera 193.

[0199] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, etc. format. In an embodiment of the present application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0200] The digital signal processor is used to process digital signals. In addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0201] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0202] The NPU is a Neural-Network (NN) computing processor. By learning from the biological neural network structure, such as learning from the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0203] The internal memory 121 may include one or more Random Access Memories (RAM) and one or more Non-Volatile Memories (NVM).

[0204] The random access memory may include Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM, for example, the fifth-generation DDR SDRAM is generally called DDR5 SDRAM), etc.;

[0205] The non-volatile memory may include disk storage devices, flash memory.

[0206] Flash memory can be classified according to its operating principle into NOR Flash, NAND Flash, 3D NAND Flash, etc. According to the number of potential levels of storage cells, it can include Single-Level Cell (SLC), Multi-Level Cell (MLC), Triple-Level Cell (TLC), Quad-Level Cell (QLC), etc. According to storage specifications, it can include Universal Flash Storage (UFS), embedded Multi Media Card (eMMC), etc.

[0207] The random access memory can be directly read and written by the processor 110. It can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store data of users and application programs, etc.

[0208] The non-volatile memory can also store executable programs and data of users and application programs, etc. It can be pre-loaded into the random access memory for the processor 110 to directly read and write.

[0209] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0210] The internal memory 121 or the external memory interface 120 is used to store one or more computer programs. One or more computer programs are configured to be executed by the processor 110. One or more computer programs include a plurality of instructions. When the plurality of instructions are executed by the processor 110, the screen display detection method executed on the electronic device 100 in the above embodiments can be implemented to realize the screen display detection function of the electronic device 100.

[0211] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor, etc. Such as music playback, recording, etc.

[0212] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In an embodiment of the present application, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0213] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or a hands-free call through the speaker 170A.

[0214] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the user can listen to the voice by bringing the receiver 170B close to the ear.

[0215] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by bringing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to implement functions such as collecting sound signals, noise reduction, identifying the sound source, and implementing a directional recording function.

[0216] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0217] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive key inputs to generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0218] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playing, etc.) can correspond to different vibration feedback effects. For touch operations on different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0219] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0220] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to achieve functions such as calls and data communication. In an embodiment of the present application, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. The embodiments of the present application also provide a computer storage medium, in which computer instructions are stored. When the computer instructions run on the electronic device 100, the electronic device 100 is caused to execute the above-related method steps to implement the image frame processing method in the above embodiment.

[0221] The embodiments of the present application also provide a computer program product. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the image frame processing method in the above embodiment.

[0222] In addition, the embodiments of the present application also provide a device, which can specifically be a chip, component or module. The device can include a processor and a memory connected to each other; wherein, the memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the image frame processing method in each of the above method embodiments.

[0223] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0224] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0225] In several embodiments provided in this application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0226] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0228] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An image frame processing method, applied to an electronic device, characterized in that, The method includes: Obtaining a current image frame of a frame to be returned and the time of the frame to be returned of the current image frame; Determining whether there is a lost frame between the current image frame and the adjacent previous image frame; If there is no lost frame between the current image frame and the adjacent previous image frame, determining whether the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame is less than the period of the image sensor of the electronic device for collecting image frames; If the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame is less than the period of the image sensor for collecting image frames, returning the current image frame after delaying a first preset compensation time.

2. The image frame processing method according to claim 1, wherein The method further includes: If the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame is greater than or equal to the period of the image sensor for collecting image frames, returning the current image frame and updating the time of the frame returned.

3. The image frame processing method according to claim 1, characterized in that, The obtaining of the current image frame of the frame to be returned and the time of the frame to be returned of the current image frame includes: The hardware abstraction layer of the electronic device uses the image frame processed by the image processing node as the current image frame of the frame to be returned, and uses the time when the image frame processed by the image processing node is received as the time of the frame to be returned of the current image frame.

4. The image frame processing method according to claim 1, characterized in that, The determining whether there is a lost frame between the current image frame and the adjacent previous image frame includes: Determining whether the time interval between the current image frame and the adjacent previous image frame is greater than the period of the image sensor for collecting image frames; If the time interval between the current image frame and the adjacent previous image frame is greater than the period of the image sensor for collecting image frames, determining that there is a lost frame of the image sensor between the current image frame and the adjacent previous image frame; or If the time interval between the current image frame and the adjacent previous image frame is less than or equal to the period of the image sensor for collecting image frames, determining that there is no lost frame of the image sensor between the current image frame and the adjacent previous image frame.

5. The image frame processing method according to claim 1, characterized in that The first preset compensation time is the difference between the period of the image sensor for collecting image frames and the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame.

6. The image frame processing method according to claim 1, wherein The method further includes: If there is a lost frame between the current image frame and the adjacent previous image frame, determining whether the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame is less than a preset multiple of the period of the image sensor for collecting image frames, where the preset multiple is the number of lost frames between the current image frame and the adjacent previous image frame plus one; If the time difference between the time of the frame to be returned of the current image frame and the time of the frame returned of the adjacent previous image frame is less than the preset multiple of the period of the image sensor for collecting image frames, returning the current image frame after delaying a second preset compensation time and updating the time of the frame returned.

7. The image frame processing method according to claim 6, wherein The second preset compensation time is the difference between a preset multiple of the period of the image sensor for collecting image frames and the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame.

8. The image frame processing method according to claim 1, characterized in that, The method further includes: If the time difference between the return frame time of the current image frame and the return frame time of the adjacent previous image frame is greater than or equal to a preset multiple of the period of the image sensor for collecting image frames, return the current image frame and update the return frame time.

9. The image frame processing method according to claim 1, wherein The method further includes: Determine whether the current image frame is the first image frame output by the image sensor; If the current image frame is the first image frame output by the image sensor, return the current image frame and record the return frame time; or If the current image frame is not the first image frame output by the image sensor, determine whether there is a lost frame between the current image frame and the adjacent previous image frame.

10. An image frame processing method, applied to an electronic device, characterized in that, The method includes: The framework layer of the electronic device receives the image frames and records the return frame timestamp and the output frame timestamp of each image frame; According to the return frame timestamp and the output frame timestamp of each image frame, determine the maximum delay time between the output frame time and the return frame time of the image frames within a preset time period; Determine a preset delay time according to the maximum delay time and a preset change threshold of the delay time; Obtain the current image frame to be returned and the return frame time of the current image frame; If the actual delay time between the return frame time and the output frame time of the current image frame is less than the preset delay time, return the current image frame after delaying for a third preset compensation time.

11. The image frame processing method according to claim 10, characterized in that, The step of determining the maximum delay time between the output frame time and the return frame time of the image frames within a preset time period according to the return frame timestamp and the output frame timestamp of each image frame includes: By calculating the difference between the return frame timestamp and the output frame timestamp of each image frame, obtain the delay time between the output frame time and the return frame time of each image frame; According to the delay time between the output frame time and the return frame time of all the image frames within the preset time period, determine the maximum delay time.

12. The image frame processing method according to claim 10, wherein Determining the preset delay time according to the maximum delay time and the preset change threshold of the delay time includes: By calculating the difference between the maximum delay time and the preset change threshold of the delay time, obtain the preset delay time.

13. The image frame processing method according to claim 12, wherein The method further includes: Determine whether the difference between the maximum delay time and the preset change threshold of the delay time is greater than a delay time threshold; If the difference between the maximum delay time and the preset change threshold of the delay time is greater than the delay time threshold, determine the delay time threshold as the preset delay time; or If the difference between the maximum delay time and the preset change threshold of the delay time is less than or equal to the delay time threshold, determine the difference between the maximum delay time and the preset change threshold of the delay time as the preset delay time.

14. The image frame processing method according to claim 13, wherein The method further includes: Determine the average delay time between the frame output time and the frame return time of the image frames within the preset time period according to the frame return timestamp and the frame output timestamp of each image frame.

15. The image frame processing method according to claim 14, characterized in that, The method further includes: Calculate the difference between the preset delay time and the average delay time, and determine whether the absolute value of the difference between the preset delay time and the average delay time is less than or equal to a preset value; If the absolute value of the difference between the preset delay time and the average delay time is less than or equal to the preset value, determine that the preset delay time is valid.

16. The image frame processing method according to claim 10, characterized in that, The third preset compensation time is the absolute value of the difference between the preset delay time and the actual delay time, and the actual delay time is the difference between the frame return time to be awaited and the frame output time of the current image frame.

17. The image frame processing method according to claim 10, wherein The method further includes: If the actual delay time between the frame output time and the frame return time of the current image frame is greater than or equal to the preset delay time, perform frame return on the current image frame.

18. An electronic device, characterized in that, The electronic device includes a memory and a processor: Wherein, the memory is used for storing program instructions; The processor is used for reading and executing the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device executes the image frame processing method according to any one of claims 1 to 17.

19. A chip, coupled to a memory in an electronic device, characterized in that, The chip is used for controlling the electronic device to execute the image frame processing method according to any one of claims 1 to 17.

20. A computer storage medium, characterized in that, The computer storage medium stores program instructions. When the program instructions run on an electronic device, the processor of the electronic device executes the image frame processing method according to any one of claims 1 to 17.

Images