Data processing method and electronic device

Abstract

The present application relates to the technical field of electronic devices. Provided in the embodiments of the present application are a data processing method and an electronic device. The method can effectively reduce display lag caused by the switching of image output modes. The method may comprise: at a first moment, starting a camera application, and an image sensor outputting an image on the basis of a first image output mode and a first exposure duration; at a second moment, acquiring first control information, the first control information comprising first image output parameters, the first image output parameters comprising a second exposure duration and a second image output mode, the second exposure duration being shorter than the first exposure duration, the second image output mode being different from the first image output mode, and the second moment being later than the first moment; at a third moment, the image sensor performing exposure on the basis of the second exposure duration; and at a fourth moment, the image sensor outputting an image in the second image output mode.

Description

Data processing method and electronic device Technical Field

[0001] The embodiments of the present application relate to the technical field of electronic devices, and in particular, to a data processing method and electronic device. Background Art

[0002] An electronic device can control a camera to provide a shooting function through a camera application. The camera can be provided with an image sensor for converting optical signals into electrical signals. The image sensor can operate according to a configured image output mode. Different image output modes can correspond to different shooting scenarios. Therefore, in different scenarios, the image sensor can capture corresponding images according to the corresponding image output mode.

[0003] When the shooting scene changes, the electronic device can control the image sensor to switch to the image output mode. In the existing solution implementation, after the electronic device generates an instruction to switch to the image output mode, the display will freeze.

[0004] Summary of the Invention

[0005] The present application provides a data processing method and electronic device, which can effectively reduce display freezes caused by switching between image output modes.

[0006] To achieve the above technical objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a data processing method is provided, which is applied to an electronic device, the electronic device including a camera application and an image sensor, the method comprising: at a first moment, starting the camera application. The image sensor outputs an image based on a first image output mode and a first exposure time. At a second moment, first control information is obtained, the first control information including a first image output parameter, the first image output parameter including a second exposure time and a second image output mode, wherein the second exposure time is less than the first exposure time, the second image output mode is different from the first image output mode, and the second moment is later than the first moment. At a third moment, the image sensor performs exposure based on the second exposure time, wherein the third moment is later than the second moment. At a fourth moment, the image sensor outputs an image based on the second image output mode, wherein the fourth moment is later than the third moment.

[0008] Based on this solution, the electronic device can use a shorter first exposure time to output images in the second output mode. This shortens the display time of the last frame of the image output in the first output mode on the display screen, thereby optimizing the lag problem that occurs when switching output modes.

[0009] Optionally, the image sensor outputs an image based on the second image output mode, including: the image sensor outputs the Nth frame image based on the second image output mode, wherein the first exposure time is the same as the exposure time of the N-1th frame, and the Nth frame image is the first frame image output by the image sensor after switching to the second image output mode.

[0010] Optionally, the second moment is earlier than the moment corresponding to the SOF (Start of Frame Delimiter) of the N-1th frame image.

[0011] Optionally, the third moment is between the moment corresponding to the EOF (end of frame delimiter) of the N-1th frame image and the moment corresponding to the SOF of the Nth frame image.

[0012] Optionally, the fourth moment is the moment corresponding to the SOF of the Nth frame image.

[0013] Optionally, the first control information corresponds to information of RequestN+1.

[0014] Through the above optional implementation, when the solution provided in this example takes effect, the exposure time of the Nth frame image is shortened, thereby reducing the display time of the N-1th frame image on the display screen. This can optimize the lag caused by the prolonged display of the N-1th frame image on the display screen. In addition, through the above possible designs, the timing of each operation processing in this example is also specifically limited.

[0015] Optionally, the method further includes: at a fifth moment, writing the first control information into a first register, the first register being a register of the image sensor, wherein the fifth moment is later than the second moment and earlier than the third moment.

[0016] Optionally, the first control information includes a first address and a first value, wherein the first address is an address at which the image sensor stores the exposure duration, and the first value corresponds to the second exposure duration. Writing the first control information into the first register includes: writing the first value to the first address of the first register.

[0017] Optionally, the fifth moment is between a moment corresponding to the SOF and a moment corresponding to the EOF of the N-1th frame image.

[0018] Optionally, the electronic device is configured with a camera request management module (CRM). The obtaining of the first control information includes: the CRM obtaining the first control information at the second moment. The writing of the first control information into the first register includes: the CRM writing the first control information into the first register at the fifth moment.

[0019] Thus, the electronic device can write the shortened exposure time into the register via the CRM according to the effective sequence. The image output parameter of the switching frame that triggers seamless switching is effective for 1 frame. Thus, through the above solution, when the N-1th frame image is output, the shortened exposure time is written to the register, allowing the sensor to use the shortened exposure time when exposing the Nth frame image.

[0020] Optionally, before generating the first control information, the method further includes: determining the second exposure duration.

[0021] Optionally, before determining the second exposure duration, the method further includes: obtaining the first exposure duration. Determining the second exposure duration includes: determining the second exposure duration based on the first exposure duration, such that the second exposure duration is less than the first exposure duration.

[0022] Optionally, the method further includes: obtaining a first exposure gain, wherein the first exposure gain is an exposure gain used by the image sensor when outputting an image based on the first output mode. Determining the second exposure duration based on the first exposure duration includes: determining the second exposure duration based on the first exposure duration and the first exposure gain.

[0023] Optionally, the first control information further includes a second exposure gain, wherein a product of the second exposure gain and the second exposure duration is equal to a product of the first exposure gain and the first exposure duration. Exposing, by the image sensor, based on the second exposure duration includes: exposing, by the image sensor, based on the second exposure duration and the second exposure gain.

[0024] Optionally, the second exposure gain is the maximum exposure gain of the image sensor.

[0025] Thus, several different implementations for determining the shortened exposure duration are provided. It should be noted that in the above possible implementations, the second exposure duration is not infinitely shortened. Through the above implementations, it can be seen that the total exposure (i.e., exposure duration multiplied by exposure gain) remains unchanged before and after the exposure duration is shortened. This allows for shortening the exposure duration while avoiding any undesirable effects on image exposure.

[0026] According to a second aspect, a data processing method is provided, which is applied to an electronic device, the electronic device including a camera application and an image sensor, the method comprising: at a first moment, starting the camera application. The image sensor outputs an image based on a first output mode. At a second moment, first control information is obtained, the first control information including a first output parameter, the first output parameter including a first frame length and a second output mode, wherein the first frame length is less than the second frame length, the second frame length is the frame length configured for the image sensor when outputting an image based on the second output mode, the second output mode is different from the first output mode, and the second moment is later than the first moment. At a fourth moment, the image sensor outputs an image based on the second output mode and the first frame length, wherein the fourth moment is later than the third moment.

[0027] Based on this solution, a solution is provided for shortening the display duration of the first frame of image displayed on the display screen after switching to the output image. This solution can shorten the frame length of the first frame of image output according to the second output image mode. It can be understood that the first frame of image is not sent for display. After switching the output image mode, the image actually displayed on the display screen is the second frame of image output after switching the output image mode. Therefore, shortening the frame length of the first frame of image can effectively control the display duration of the first frame of image displayed on the display screen without affecting the image quality.

[0028] Optionally, the fourth moment is a moment corresponding to the SOF of the Nth frame image. The second frame length is a frame length configured for the image sensor when outputting the Nth frame image based on the second image output mode. The image sensor outputs an image based on the second image output mode and the first frame length, comprising: the image sensor outputs the Nth frame image based on the second image output mode and the first frame length, wherein the Nth frame image is the first frame image output by the image sensor after switching to the second image output mode.

[0029] Optionally, the second moment is earlier than the moment corresponding to the SOF of the N-1th frame image.

[0030] Optionally, the first control information corresponds to information of RequestN+1.

[0031] Based on this solution, it is clarified that the first frame after switching the image output mode can be the Nth frame. In this way, the frame length of the Nth frame can be controlled, thereby shortening the image output time of the Nth frame. Correspondingly, since the display time of the N+1th frame image on the display screen corresponds to the image output time of the Nth frame and the image output time of the N+1th frame, the display time of the N+1th frame on the display screen can be effectively shortened. This can further optimize the lag problem after switching the image output mode.

[0032] It should be noted that in this example, the "configured" frame length can be interpreted as the default frame length. For example, the configured frame length for outputting the Nth frame is also the default frame length for outputting the Nth frame.

[0033] Optionally, the method further includes: at a fifth moment, writing the first control information into a first register, the first register being a register of the image sensor, wherein the fifth moment is later than the second moment and earlier than the third moment.

[0034] Optionally, the first control information includes a second address and a second value, wherein the second address is an address at which the image sensor stores a frame length, and the second value corresponds to the first frame length. Writing the first control information into the first register includes: writing the second value to the second address of the first register.

[0035] Optionally, the electronic device is configured with a camera request management module (CRM). The obtaining of the first control information includes: the CRM obtaining the first control information at the second moment. The writing of the first control information into the first register includes: the CRM writing the first control information into the first register at the fifth moment.

[0036] Optionally, the fifth moment is between a moment corresponding to the SOF and a moment corresponding to the EOF of the N-1th frame image.

[0037] In this example, the following solution is also implemented to provide a specific method for determining the shortened first frame length.

[0038] Optionally, the first frame length is smaller than the second frame length, and the first frame length is greater than or equal to half of the second frame length.

[0039] Optionally, before generating the first control information, the method further includes: determining the first frame length.

[0040] Optionally, before determining the first frame length, the method further includes: acquiring the second frame length. The determining the first frame length includes: determining the first frame length based on the second frame length.

[0041] Optionally, obtaining the second frame length includes: reading a parameter configuration file to obtain the second frame length, wherein the parameter configuration file is stored in the electronic device.

[0042] Optionally, determining the first frame length based on the second frame length includes: determining half of the second frame length as the first frame length.

[0043] Through the above solution example, the frame length of the Nth frame image is controlled. In the following implementation, the frame length of the N+1th frame image can also be controlled to further shorten the display time of the N+1th frame.

[0044] Optionally, the method further includes: generating second control information at a sixth moment, the second control information including second image output parameters, the second image output parameters including a third frame length, wherein the third frame length is less than a fourth frame length, the fourth frame length being a frame length configured for the image sensor when outputting the (N+1)th image frame based on the second image output mode, and the sixth moment is later than the second moment. At a seventh moment, the image sensor outputs the (N+1)th image frame based on the second image output mode and the third frame length, and the seventh moment is later than the fourth moment.

[0045] Optionally, the third frame length is greater than or equal to half of the fourth frame length.

[0046] Optionally, the sixth moment is earlier than the moment corresponding to the SOF of the Nth frame image.

[0047] Optionally, the seventh moment is the moment corresponding to the SOF of the N+1th frame image.

[0048] Optionally, the method further includes: writing the third frame length into the register of the image sensor at an eighth moment, wherein the eighth moment is between the moment corresponding to the SOF and the moment corresponding to the EOF of the Nth frame image.

[0049] Optionally, the second control information corresponds to information of RequestN+2.

[0050] It is understood that in other embodiments, instead of controlling the frame length of the Nth frame, only the frame length of the N+1th frame may be controlled. For example, the frame length of the N+1th frame may be shortened. This can also shorten the display time of the N+1th frame on the display screen, thereby reducing lag.

[0051] According to a third aspect, an electronic device is provided, comprising: a memory, one or more processors, and one or more cameras. Each camera is configured with a corresponding image sensor. The memory, the processor, and the image sensor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method provided in the first aspect and any possible design thereof, and / or executes the method provided in the second aspect and any possible design thereof to control the image sensor to output an image. In some implementations, the electronic device is further configured with one or more display screens. In this way, by implementing the various solutions provided in the embodiments of the present application, the jamming of the image stream (such as a preview stream) displayed on the display screen before and after triggering the seamless switching can be effectively controlled.

[0052] In a fourth aspect, the present application further provides a chip system, which is applied to an electronic device; the chip system may include one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line, and the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, where the signal includes a computer instruction stored in the memory. When the processor executes the above-mentioned computer instructions, the electronic device executes the method provided in the first aspect and any possible design thereof, and / or executes the method provided in the second aspect and any possible design thereof, to control the image sensor to output an image.

[0053] In a fifth aspect, the present application also provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method provided in the first aspect and any possible design thereof, and / or executes the method provided in the second aspect and any possible design thereof, to control the image sensor to output an image.

[0054] In a sixth aspect, the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the method provided in the first aspect and any possible design thereof, and / or execute the method provided in the second aspect and any possible design thereof, to control the image sensor to output an image.

[0055] It can be understood that the solutions provided in the third to sixth aspects of the present application can respectively correspond to the first aspect and any possible design thereof, and / or the second aspect and any possible design thereof, so the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a schematic diagram of an interface interaction;

[0057] FIG2 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0058] FIG3 is a schematic diagram of a flow chart of an interaction between modules provided in an embodiment of the present application;

[0059] FIG4 is a timing diagram of an inter-module instruction transmission provided by an embodiment of the present application;

[0060] FIG5 is a schematic diagram of switching an image output mode provided in an embodiment of the present application;

[0061] FIG6 is a schematic diagram of switching an image output mode provided in an embodiment of the present application;

[0062] FIG7 is a timing diagram of an inter-module instruction transmission provided by an embodiment of the present application;

[0063] FIG8 is a comparative diagram of a solution provided in an embodiment of the present application before and after it takes effect;

[0064] FIG9 is a schematic diagram of an inter-module interaction process provided in an embodiment of the present application;

[0065] FIG10 is a comparative diagram of a solution provided in an embodiment of the present application before and after it takes effect;

[0066] FIG11 is a schematic diagram of an inter-module interaction process provided in an embodiment of the present application;

[0067] FIG12 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0068] FIG13 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0069] FIG14 is a schematic diagram showing the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0071] Electronic devices can provide users with a shooting function through software and hardware configured therein.

[0072] For example, the software configured in the electronic device may include a camera application, and the hardware may include one or more cameras.

[0073] Take the mobile phone as an example of an electronic device.

[0074] Referring to Figure 1 , an example of interface interaction is provided. Through the example of interface interaction shown in Figure 1 , the electronic device can provide the user with a shooting function under the user's instruction.

[0075] As shown in Figure 1, an electronic device may display an interface 01. The interface 01 may be the main interface of the electronic device. The interface 01 may include one or more icons of installed applications. For example, the interface 01 may include an icon 101 for a camera application.

[0076] The user may input a click operation to the icon 101 to instruct the electronic device to execute a camera application.

[0077] Correspondingly, the electronic device can run a camera application and switch to display the interface of the camera application.

[0078] In the example shown in Figure 1, the camera application interface may include Interface 02. In Interface 02, the electronic device may display a preview image to the user. Interface 02 may also include one or more function buttons. For example, the focus options bar may display the current focus and selectable focus. Another example is the lower portion of Interface 02, which displays the current shooting mode. This shooting mode may include "Photo" or "Video."

[0079] Before displaying the preview image on interface 02, the electronic device can obtain image data (e.g., a RAW image) through the camera according to instructions issued by the camera application. The electronic device can also process the RAW image to obtain a corresponding image for display. In this way, when the preview image is needed on interface 02, the electronic device's display screen can display it based on the image for display.

[0080] In this application, a camera may include components such as a lens and an image sensor. The lens is used to acquire light signals, and the image sensor is used to convert light signals into electrical signals. The electrical signals may include display information (such as color, grayscale, etc.) indicating each pixel in the RAW image. The generation of electrical signals by the image sensor can also be referred to as image output.

[0081] In this way, the electronic device processes and obtains a corresponding frame of image for display each time the camera's image sensor outputs an image. This cycle repeats, and by controlling the image sensor to output images multiple times, the electronic device can obtain multiple consecutive frames of image for display. Based on this, the electronic device can display a continuous preview image to the user on the display.

[0082] In the following description, the image sensor is referred to as sensor or sensor.

[0083] Referring to Figure 2, which is a schematic diagram of the composition of an electronic device provided by an embodiment of the present application, the electronic device can control the image sensor to continuously output images through software modules such as a camera application.

[0084] As shown in Figure 2, in this example, an electronic device may be configured with an operating system. From a software perspective, the operating system may have a layered architecture. For example, the electronic device may include an application (APP) layer, a framework layer, a hardware abstraction layer (HAL), a kernel layer, and the like. Each of these layers is described below.

[0085] The APP layer is also known as the application layer. The application layer can include a series of application packages. For example, an application package may include a camera application. In other implementations, the APP layer may also include applications such as gallery, calendar, map, WLAN, music, SMS, call, navigation, Bluetooth, and video.

[0086] The Framework layer, also known as the application framework layer, provides an application programming interface (API) and programming framework for applications in the application layer. The Framework layer includes some predefined functions.

[0087] Exemplarily, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

[0088] In the example shown in FIG2 , the Framework layer may include a camera service (CameraService). In some implementations, after the electronic device starts a camera application, the camera service may be started in the Framework layer according to a start instruction sent by the camera application.

[0089] The HAL layer, also known as the hardware abstraction layer, runs in user space, encapsulates kernel drivers, and provides a calling interface to upper layers.

[0090] In this example, the HAL may be configured with a decision (Policy) module and a CAMX framework module.

[0091] In some embodiments, the decision module can be used to decide the drawing mode.

[0092] The output modes may include Binning, HDR, Remosaic, Idcg, etc. In different implementations, the specific implementation of the output mode may be different. The embodiments of the present application do not limit the specific implementation of the output mode.

[0093] For example, in Binning output mode, after the sensor captures the raw pixel array, it adds the induced charges corresponding to adjacent pixels in the raw pixel array together to form a single pixel point. This Binning output mode combines several adjacent pixels and uses them as a single pixel. This allows the raw image data output by the sensor to maintain the same field of view (FOV) while reducing output resolution, increasing the photosensitive area, and improving sensitivity to light in dark areas. Binning output mode is typically the default output mode.

[0094] In Idcg output mode, the sensor's dynamic range can be improved. This dynamic range is the sensor's ability to simultaneously represent highlights and shadows in an image. A larger sensor's dynamic range indicates a greater ability to represent both highlights and shadows. In Idcg output mode, the sensor uses the same exposure time to simultaneously acquire a high-gain (HCG) map and a low-gain (LCG) map corresponding to the same frame of raw image data. The sensor then fuses the high-gain and low-gain maps into a single image, which is then output as the raw image data. Similar to Idcg output mode, in HDR output mode, the sensor can output raw image data with brighter highlights.

[0095] Compared with the Binning output mode, the Idcg output mode has a larger dynamic range. Of course, the corresponding power consumption is also higher, about 1.5 times that of the binning output mode.

[0096] In the Remosaic output mode, the sensor uses the raw pixel array collected by the 4-cell sensor as the output raw image data. That is, in the Remosaic output mode, the raw image data received by the camera driver is not synthesized by binning pixels. That is, during the period when the Remosaic output mode is enabled, the raw image data obtained by the camera driver cannot be directly recognized and processed. The raw image data needs to be converted into a standard Bayer format image. This conversion process is called remosaic. Exemplarily, the raw image data collected by the sensor can be converted into remosaic image data by the (sensor front end, SFE). Compared with the raw image data obtained in the binning output mode, the above-mentioned remosaic image data has more pixels and higher definition, and is more suitable for shooting scenes where the user instructs to enlarge the shooting picture (that is, increase the zoom ratio).

[0097] Different output modes can be suitable for different shooting scenes / shooting modes.

[0098] For example, when the shooting mode is the photo mode, the electronic device can control the sensor to output the image according to Binning, or control the sensor to output the image according to Remosaic, or control the sensor to output the image according to Idcg.

[0099] For another example, when the shooting mode is the video mode, the electronic device can control the sensor to output the image according to Binning, or control the sensor to output the image according to Idcg.

[0100] In different image output modes, the resolution, dynamic range, and / or clarity of the RAW image output by the sensor may vary. Therefore, the electronic device can control the sensor's image output mode to obtain a RAW image that matches the current scene.

[0101] In other embodiments, the decision module may also determine the camera to be used.

[0102] It is understandable that the electronic device may be configured with multiple cameras. Then, the decision module can determine the camera to be used based on the instructions issued by the upper layer (such as the framework layer). For example, the decision module can determine the CameraID of the camera to be used.

[0103] In this example, the CAMX framework module may provide a variety of functional interfaces and / or functional components related to the shooting function.

[0104] In some embodiments, the CAMX framework module may include a sensor node and an image front end (IFE) node.

[0105] The sensor node can generate control instructions for the sensor based on the current output mode to be used. For example, the control instruction for the sensor may include the CameraID, the register address of the corresponding sensor, and the value to be written to the register. The corresponding sensor may be the sensor of the camera to be used (i.e., the camera indicated by the CameraID). By generating and passing the register address and value downward, the control instruction is transmitted to the sensor. This allows the sensor to output images according to the parameters indicated by the control instruction (such as the output mode) after the register takes effect.

[0106] The IFE node generates control instructions for the IFE based on the current output mode. This is similar to the working mechanism of the Sensor node. This allows the IFE to process the RAW image after acquiring it using the processing mechanism corresponding to the current output mode.

[0107] In the example shown in FIG. 2 , the IFE node and the Sensor node may be included in the software layer (SWL) of the CAMX framework module.

[0108] The CAMX framework module may also be configured with an external software interface, for example, in the Camera Service Layer (CSL).

[0109] In this example, a cmxCSL module may be configured in the CSL. The cmxCSL module may correspond to the above software interface and provide downward (eg, toward the kernel layer) communication capabilities.

[0110] In this example, the kernel layer may be a layer configured between hardware and software.

[0111] The kernel layer may include a Linux basic driver. For example, the Linux basic driver may include Video for Linux 2 (V4L2). V4L2 is a kernel driver for video devices in Linux. V4L2 can receive information from upper layers (such as the Framework layer). V4L2 can also transmit the received information to the device driver configured in the kernel layer.

[0112] The kernel layer may be configured with device drivers for various hardware devices. For example, the kernel layer may be configured with sensor drivers corresponding to various cameras. The kernel layer may also be configured with an image front end (IFE) driver. If an electronic device is configured with multiple cameras, the kernel layer may be configured with a corresponding sensor driver and IFE driver for each camera. In other cases, sensor drivers and / or IFE drivers for different cameras may also be shared.

[0113] In some embodiments, the kernel layer may also be configured with a camera request manager (CameraRequestManger, CRM). The CRM can be used to manage the control information of components such as sensors and IFE based on information from the upper layer. For example, CMR can send register write instructions to the sensor driver, IFE driver, etc. of the camera that needs to be used based on the CameraID. This allows the camera that needs to be used to perform operations such as outputting images and IFE image processing based on the effectiveness of the write registers. For another example, for the camera that needs to be used, CMR can transmit corresponding register write instructions to the sensor driver, IFE driver, etc. based on the effectiveness timing of the registers in each component of the camera under different output modes. This allows each component to achieve frame synchronization of outputting images and image processing.

[0114] In the example of FIG. 2 , hardware components (e.g., a hardware layer) in the electronic device are also provided. The hardware layer may include components such as an IFE and a sensor. In the case where the electronic device is configured with multiple cameras, each camera may be configured with a corresponding IFE and sensor. In some implementations, different cameras may also share the same IFE and / or sensor.

[0115] In this way, through the various software layers in the operating system and the cooperation with the hardware layer, the sensor can be controlled to produce images, which are then displayed on the screen accordingly.

[0116] 3 , based on the software composition shown in FIG2 , an example is given of module interaction during the image output process of the electronic device.

[0117] As shown in Figure 3, the camera application of the electronic device can send indication information 31 to the camera service. For example, in conjunction with Figure 1, upon receiving user operation OP1, the electronic device launches the camera application. Thus, the camera application can generate and send indication information 31 upon startup. In some implementations, the camera application can also instruct the default camera's sensor and / or IFE to power on and operate through other control information.

[0118] In some embodiments, the instruction information 31 may include a start instruction, so that the electronic device can start the camera service in the Framework layer.

[0119] In other embodiments, the indication information 31 may include parameter information related to image output, such as the currently used shooting mode, the currently effective focal length parameter, etc.

[0120] Shooting modes may include photo mode, video mode, etc. The currently effective focal length parameter corresponds to the effective focal length displayed in the focal length option bar in interface 02 of Figure 1 . For example, in the example shown in Figure 1 , the currently effective focal length parameter may indicate a focal length of 1x. After the user inputs an operation to adjust the focal length (such as clicking 2x in the focal length option bar), the currently effective focal length parameter changes accordingly (e.g., changes to a focal length of 2x).

[0121] In other embodiments, the indication information 31 may include an identifier of a camera to be used. For example, the indication information 31 may include a CameraID of a camera to be used.

[0122] The camera service may receive the indication information 31, generate and send the indication information 32 to the decision module in the HAL layer.

[0123] For example, the indication information 32 may include parameter information related to image output issued by the camera application. For example, the indication information 32 may include the currently used shooting mode, the currently effective focal length parameter, etc.

[0124] In some embodiments, the indication information 32 may also include the CameraID of the camera to be used.

[0125] In the case where the indication information 31 carries a CameraID, the camera service may carry the same CameraID in the generated indication information 32. In other implementations, the CameraID carried in the indication information 32 may also be determined by the camera service according to a preset policy.

[0126] The decision module determines the image output mode to be used according to the instruction information 32. In combination with the above description, the image output mode may include Binning, HDR, Remosaic, Idcg, etc.

[0127] For example, in some embodiments, the decision module may determine the image output mode to be used based on information such as the currently used shooting mode and focal length parameters.

[0128] In other embodiments, the decision module may determine the image output mode based on the currently used shooting mode, focal length parameters, and ambient brightness. The ambient brightness may be obtained by the decision module based on the RAW image that has already been output, or the ambient brightness may be obtained by the decision module based on other sensors configured in the electronic device (such as a color temperature sensor).

[0129] In this example, the decision module may also obtain the CameraID of a camera to be used (eg, a target camera).

[0130] In some implementations, the decision module may also obtain the CameraID of the camera to be used according to the instruction information 32 .

[0131] For example, the decision module may determine the CameraID of the camera to be used according to the CameraID carried in the indication information 32 .

[0132] For another example, the decision module can determine the CameraID of the camera to be used based on a preset strategy.

[0133] Afterwards, the decision module can transmit the image output mode information and CameraID to the sensor node and IFEnode respectively, so that other components can control their respective hardware.

[0134] For example, the decision module may send image output information 33 to the sensor node and the IFE node respectively. The image output information 33 may include information on the image output mode and the CameraID of the camera to be used.

[0135] In this way, the sensor node can generate instructions according to the image output information 33, and then control the sensor of the camera to be used to output the corresponding image.

[0136] The IFE node can generate instructions according to the image output information 33, and then control the IFE of the camera to be used to process the RAW image.

[0137] Take the relevant processing mechanism of sensor nodes as an example.

[0138] The sensor node may generate control information 34a according to the image output information 33. The control information 34a may be control information of a register indicating the CameraID of the camera to be used.

[0139] Exemplarily, in some embodiments, the control information 34a may include at least one of the following: exposure information, information on the image output frame length, and information on the image output mode.

[0140] In other embodiments, the control information 34a may also only include exposure information.

[0141] Exposure information can include exposure time and gain. Exposure time and gain indicate the exposure level of a frame. A greater exposure level means a longer time to produce the image. Conversely, a smaller exposure level means a shorter time to produce the image.

[0142] In some implementations of the present application, VTS can be used to identify the output frame length. VTS has a linear relationship with the output frame length. The VTS can be used to indicate the number of rows that need to be exposed during the output of a frame of image. In some implementations, the value of VTS can correspond to different output frame lengths. The larger the VTS is, the longer it takes to output the frame image. Conversely, the smaller the VTS is, the shorter it takes to output the frame image. Take the output frame rate of 30fps as an example. The frame length indicated by the default VTS (Default VTS) is approximately 33ms. In some implementations, the VTS can also be replaced by a frame length line (FrameLengthLine).

[0143] The output mode can correspond to any of the output modes in the above examples. In the following examples, the currently used output mode is modeA.

[0144] In a specific implementation, each item of information in the control information 34a can be represented by a register address and a value corresponding to the register address.

[0145] For example, the control information 34a includes the above-mentioned exposure information, information on the image output frame length, and information on the image output mode.

[0146] As a possible implementation, the control information 34a may include an address AD1 and a corresponding value V1; an address AD2 and a corresponding value V2; and an address AD3 and a corresponding value V3.

[0147] The address AD1 is the register address of the target sensor storing the image output mode, the value V1 indicates the image output mode mode A. The target sensor is the sensor of the camera indicated by CameraID.

[0148] Address AD2 is the register address of the target sensor that stores the image output frame length, and value V2 indicates that the VTS of the image output is V2.

[0149] Address AD3 is the register address of the target sensor for storing exposure information, and value V3 indicates that the exposure information of the image is V3.

[0150] In this example, the sensor node may send the generated control information 34a to the camera to request management.

[0151] For example, the sensor node can transmit the control information 34a downward by sending a request message (such as Request). For example, the sensor node can carry the control information 34a in the Request and send it to the CRM.

[0152] 2 , in some implementations, the sensor node may call cmxCSL in the CSL to send a Request message carrying the control information 34 a to the kernel layer.

[0153] V4L2 is configured in the Linux basic driver of the kernel layer, and the Request information including the control information 34a can be transmitted to the CRM.

[0154] The CRM can send the corresponding parameters to the sensor driver of the target sensor according to the effective timing of each parameter carried in the control information 34a. In some implementations, the CRM can transmit some or all parameters in the control information 34a to the sensor driver through a Request message.

[0155] For example, the control information 34a includes information on the image output mode. The information on the image output mode may include an address AD1 and a corresponding value V1. The parameters of the image output mode may be configured to take effect after 2 frames.

[0156] Thus, the CRM can send the output mode parameters for the Nth frame to the sensor driver before the N-1th frame begins output (e.g., during the N-2nd frame output). Upon receiving the output mode parameters, the sensor driver can directly write them into the corresponding register. In other words, the target sensor's address AD1 can be written with the value V1 during the N-2nd frame output. After two frames have passed, output is performed according to mode A corresponding to value V1.

[0157] After the N-2 frame is output, the sensor can proceed to output the N-1 frame. After outputting the N-2 and N-1 frames, when the N frame is output, the value V1 written to address AD1 takes effect. The sensor outputs the image according to mode A corresponding to value V1.

[0158] For example, the control information 34a includes the parameters related to the exposure information. The parameters related to the exposure information may include the address AD3 and the corresponding value V3. The exposure information may be configured to take effect after 2 frames.

[0159] Thus, the CRM can send the exposure information for frame N to the sensor driver before frame N-1 begins output (e.g., during frame N-2). Upon receiving the exposure information, the sensor driver can directly write the parameter into the corresponding register. In other words, the target sensor's address AD3 can be written with the value V3 during frame N-2 output. After two frames have passed, exposure is performed based on the exposure information corresponding to value V3.

[0160] Similar to the previous example, after the N-2 frame is output, the sensor can proceed to the N-1 frame. After two frames have been output, when the N frame is output, the value V3 written to address AD3 takes effect. The sensor performs exposure according to the exposure information corresponding to value V3.

[0161] As an example, referring to FIG4 , a timing comparison diagram of the transmission of Request information in an electronic device is shown when two frames are valid.

[0162] In Figure 4, UMD refers to a software module in the HAL layer. For example, a UMD corresponds to a sensor node. KMD refers to a software module in the kernel layer. For example, a KMD corresponds to a CRM. MIPI corresponds to the timing of sensor image output. dp2 indicates that the parameters in the Request message will take effect two frames later.

[0163] It's understood that during the sensor's image output process, the data for a complete frame of image data can be divided by the start-of-frame (SOF) and end-of-frame (EOF) delimiters. For example, when the sensor outputs a frame of image data, the corresponding data stream may start with the SOF and end with the EOF. The data stream of the RAW image may be included between the SOF and EOF.

[0164] Take the delivery of RequestN-2 in frame N-2 as an example.

[0165] As shown in Figure 4, the KMD can issue RequestN-2 corresponding to frame N-2 during the image output process of frame N-4 (i.e., after the SOF and before the EOF of frame N-4), so that the sensor driver can write the parameters in RequestN-2 to the corresponding register before the image output of frame N-4 is completed. In this way, after the image output of frames N-4 and N-3 is completed, the parameters in RequestN-2 take effect. During the image output process of frame N-2, the image sensor can output the image according to the parameters in RequestN-2.

[0166] It is understood that the KMD may receive RequestN-2 from the UMD before sending RequestN-2. For example, as shown in Figure 4, the KMD may receive RequestN-2 from the UMD during the rendering of frame N-5. In other embodiments, the timing of transmitting RequestN-2 between the UMD and the KMD may be any other time before the rendering of frame N-4 begins. The specific implementation is not limited.

[0167] In the example shown in Figure 4 above, the parameters in the Request message take effect two frames later. In other implementations, the Request message may also include parameters that take effect one frame later. For example, if a parameter that takes effect one frame later is written in frame N, it will take effect when the image is output in frame N+1.

[0168] For example, the control information 34a includes parameters related to the output frame length. The parameters related to the output frame length may include address AD2 and corresponding value V2. The parameters related to the output frame length may be configured to take effect after 1 frame.

[0169] Thus, the CRM can send the relevant parameters for the output frame length of the Nth frame to the sensor driver before the Nth frame begins output (e.g., during the output of the N-1th frame). Upon receiving the relevant parameters, the sensor driver can directly write them into the corresponding register. In other words, the address AD2 of the target sensor can be written with the value V2 during the output of the N-1th frame. After indicating one frame, the image is output according to the frame length corresponding to the value V2.

[0170] Therefore, after the N-1th frame image is output, after 1 frame is output, when the Nth frame image is output, the value V2 written to the address AD2 takes effect. The sensor outputs the image according to the frame length corresponding to the value V2.

[0171] The above example illustrates the signaling transmission process for sensor nodes. Similarly, the IFE node can generate control information 34b corresponding to the IFE based on the image output information 33. The IFE node can send this control information 34b to the CRM. In response, the CRM can write the parameters to the IFE registers based on their corresponding timings, driven by the IFE. This facilitates frame synchronization of the parameters used during IFE processing of the RAW image.

[0172] In this way, as shown in Figures 3 and 4, the electronic device can control the sensor to output images according to the corresponding image output mode, image output frame length, and exposure information through internal software instructions.

[0173] It should be noted that the above description uses the example of each frame's image output mode being carried in the Request for that frame. In other embodiments, if the output mode of the current frame is the same as that of the previous frame, that is, the output mode has not changed, then the Request for that frame may not carry the output mode information. Accordingly, if the KMD and sensor do not receive a new output mode, they can continue to use the existing output mode for output.

[0174] For example, if the decision module determines that the output mode for both frame N-2 and frame N-1 is mode A, then Request N-1 corresponding to frame N-1 can contain only parameters such as exposure information, without the need for mode A. Based on this Request N-1, the sensor can continue to use mode A for outputting frame N-1.

[0175] In some current implementations, electronic devices can also provide a seamless switching solution for image output modes. This allows the electronic device to control the image sensor to switch to an image output mode that matches the new scene when the environmental scene changes, thereby obtaining better image output and display effects. In this application, this seamless switching solution for image output modes can be referred to as a seamless switching solution, or simply seamless switching.

[0176] For example, when changes occur in ambient brightness, dynamic range, effective focal length, etc., the electronic device's decision module can determine which image output mode to switch to. For example, switching mode A to mode B. The decision module can include information about the image output mode corresponding to mode B in the newly issued image output information, allowing sensor nodes and IFE nodes to control hardware to subsequently output images according to mode B.

[0177] Referring to Figure 5, before and after switching between different modes, the time between the two frames of image output may include a reset period. The sensor can adjust parameters when switching from mode A to mode B during the reset period.

[0178] In the current implementation, there will be a long pause in image output when switching from mode A to mode B. This will appear on the display as a pause in the preview stream.

[0179] As shown in Figure 6, actual measurements show that, using a 30fps output frame rate as an example, before seamless switching is triggered, the sensor can output a frame in 33ms using mode A. However, when seamless switching is triggered, two long frames, one approximately 59ms long and the other approximately 66ms long, appear.

[0180] The 59ms long frame 1 corresponds to the first frame after seamless switching is triggered. The 66ms long frame 1 corresponds to the second frame after seamless switching is triggered. Normality can be restored in the third frame. According to mode B, a frame of image is output and displayed in 33ms.

[0181] In this way, after the seamless switching is triggered, there will be a freeze of at least two frames of images on the display.

[0182] In this application, the following describes the mechanism for sending Request information during the image output mode switching process, in conjunction with the timing comparison provided in Figure 7. This can determine the cause of the jamming shown in Figure 6. The existing solution shown in Figure 7 can also be referred to as Solution 1.

[0183] In the example shown in FIG. 7 , it is taken as an example that the decision module determines to switch from mode A to mode B for outputting the Nth frame image.

[0184] As shown in Figure 7, the UMD can send RequestN before the N-2 frame image starts to be output. In this way, RequestN can be written into the sensor register by the KMD during the N-2 frame image output process.

[0185] In combination with the above description, in this example, the RequestN may include relevant parameters valid for two frames, for example, the exposure information Expo1 of the Nth frame image.

[0186] In this way, after completing the output of the N-2 frame image, the sensor can continue to output the N-1 frame image. After the sensor is written to RequestN, the parameters of RequestN will take effect after 2 frames of image output.

[0187] It should be noted that in the seamless switching solution, new configuration parameters can be configured to take effect after 1 frame.

[0188] When the UMD determines that it needs to switch to mode B for image output after the Nth frame, it can send a RequestN+1 to the KMD before the SOF of the N-1th frame arrives (that is, before the N-1th frame starts to be output). The RequestN+1 can carry the parameters required for the Nth frame after switching to mode B.

[0189] For example, the RequestN+1 may include at least one of the following:

[0190] Information indicating that the image is output according to mode B, exposure information of the Nth frame image (such as exposure information Expo2), and information indicating the output frame length of the Nth frame image (such as VTS1).

[0191] In the example shown in FIG. 7 , the UMD may send the RequestN+1 during the process of outputting the N-2 frame image.

[0192] The KMD can write RequestN+1 to the sensor's register during the image output process of the next frame (e.g., frame N-1) after receiving RequestN+1. For example, the value corresponding to modeB can be written to the register address storing the image output mode. This instructs the sensor to switch to modeB for image output in the next frame.

[0193] Therefore, after RequestN+1 is written to the register, after the N-1th frame is output, during the process of outputting the Nth frame, the output mode (such as modeB) indicated by RequestN+1 takes effect. Accordingly, the sensor can output the Nth frame image according to the parameters indicated by RequestN+1.

[0194] That is, when the Nth frame is output, the parameters that actually take effect are those configured in RequestN+1. For example, when the Nth frame is output, the sensor can output the image based on information such as output mode modeB, exposure information Expo2, and output frame length VTS1.

[0195] The Nth frame image may be the first frame image output according to mode B. The parameters in RequestN+1 corresponding to the Nth frame image may be effective for one frame.

[0196] After that, for subsequent frames (e.g., frame N+1), the default delayed frame effect mechanism can be restored. For example, parameters such as the image output mode and exposure information can be delayed for 2 frames. Alternatively, parameters such as the image output frame length (e.g., VTS) can be delayed for 1 frame.

[0197] Take the output frame length included in RequestN+2 as VTS2 as an example.

[0198] When outputting the N+1th frame image, the sensor may use modeB indicated by RequestN+1, exposure information Exop1, and VTS2 indicated by RequestN+2 to output the image.

[0199] Thus, seamless switching can be completed by the N-1th frame and the Nth frame. The N-1th frame can also be called the first frame after the seamless switching is triggered. The frame displayed after the first frame can be the second frame after the seamless switching is triggered.

[0200] The first frame after the seamless switching is triggered may correspond to the long frame 1 shown in FIG. 6 .

[0201] As shown in Figure 5 , after the sensor finishes outputting the N-1th frame, the mode B output mode indicated by RequestN+1 takes effect, and the sensor can immediately begin the mode switching process. For example, after completing the output of the N-1th frame, the sensor can switch to mode B and perform related output operations within the reset duration. Thereafter, the sensor can immediately begin exposing the Nth frame according to the exposure information Expo2 indicated by RequestN+1.

[0202] Thus, before the EOF of the Nth frame arrives, the duration of the first frame after triggering the seamless switch can include: the output duration of the N-1th frame (e.g., 33ms), the reset duration (e.g., 1 to 2ms), and the exposure duration of the Nth frame (e.g., the exposure duration corresponding to the exposure information Expo2). As a result, the display time of this first frame will be significantly longer than the normal 33ms, forming the first long frame (e.g., long frame 1) as shown in Figure 6.

[0203] In the above example, the second frame after the seamless switching is triggered may correspond to the long frame 2 shown in FIG. 6 .

[0204] In current technical implementations, after triggering seamless switching, the Nth frame output by the new mode B may experience image anomalies due to unstable matching between multiple modules, such as the sensor, algorithm, and AE. Therefore, the electronic device (such as its decision module) can mark the Nth frame as not to be displayed. Correspondingly, the RequestN can be a dummy request (DummyRequest), which corresponds to not displaying the Nth frame.

[0205] In this way, after the sensor outputs the Nth frame image according to RequestN+1, the electronic device may not display the Nth frame image.

[0206] Correspondingly, in order to achieve smooth switching of the preview stream without displaying the Nth frame image, the electronic device may adjust the timestamp of the N+1th frame image to the timestamp of the Nth frame image. For example, the electronic device may adjust the timestamp of the SOF of the N+1th frame image to the timestamp of the SOF of the Nth frame image.

[0207] Thus, the electronic device can directly display the (N+1)th frame image when it needs to display the (N)th frame image.

[0208] Thus, the actual duration of the N+1th frame displayed on the display screen corresponds to the duration between the SOF of the Nth frame and the EOF of the N+1th frame. This constitutes Long Frame 2 as shown in Figure 6. As can be seen, the duration of Long Frame 2 is close to the display duration of two consecutive frames.

[0209] The solution of the embodiment of the present application can effectively overcome the above-mentioned problem of display freeze on the display screen after triggering seamless switching.

[0210] For example, in some embodiments of the present application, by adjusting and optimizing the duration information of the Nth frame image, the duration of long frame 1 is shortened. In other embodiments of the present application, by adjusting and optimizing the output frame length of the Nth frame image and / or the N+1th frame image, the duration of long frame 2 is shortened. In other embodiments of the present application, the exposure information and output frame length of the Nth frame image, as well as the output frame length of the N+1th frame image, can also be optimized simultaneously to optimize the display freeze after switching to mode B.

[0211] The following explains them separately.

[0212] In some embodiments of the present application, optimization of the exposure duration of the Nth frame image is taken as an example.

[0213] Refer to Figure 8. In the conventional solution shown in Solution 1, Long Frame 1 corresponds to the SOF of the N-1th frame image to the SOF of the N-th frame image. Specifically, Long Frame 1 includes the output time of the N-1th frame image (i.e., the duration between the SOF of the N-1th frame image and the EOF of the N-1th frame image), the reset duration, and the exposure duration of the Nth frame.

[0214] Through the solution provided in the embodiments of this application, the exposure duration of frame N is optimized. The optimized exposure duration of frame N is shorter than the exposure duration of the original frame N in solution 1. This shortens the time between the SOF of frame N-1 and the SOF of frame N, thus achieving the effect of optimizing long frame 1.

[0215] The following describes the implementation of the solution shown in FIG8 in detail in conjunction with the interaction example of FIG9 , wherein the Nth frame image is taken as the first frame image output using modeB as an example.

[0216] As shown in Figure 9, the solution may include:

[0217] S901: The decision module determines to switch to mode B for drawing output.

[0218] In this example, before the decision module determines to switch to mode B for image output, the sensor can use mode A to output images under the control of the electronic device.

[0219] For example, as shown in FIG9 , in some embodiments, before executing S901 , the camera application may be started after the electronic device receives an operation OP1 input by the user. Thereafter, the various components in the electronic device may cooperate with each other so that the sensor uses mode A to output an image.

[0220] The specific implementation of using modeA to output the image can refer to the example in Figure 3, which will not be repeated here.

[0221] In this example, the decision module may determine the image output mode to be switched based on changes in the current ambient brightness, and / or dynamic range, and / or effective focal length, etc. For example, the decision module may determine that the image output mode needs to be switched to mode B.

[0222] S902: The decision module sends the output image information N to the sensor node.

[0223] Exemplarily, the image output information N may include information indicating that the image is output through mode B.

[0224] In some embodiments, the image output information N may further include an identifier corresponding to the Nth frame, thereby enabling each module in the electronic device to synchronously switch to mode B for image output starting from the Nth frame.

[0225] S903: The sensor node generates control information N.

[0226] Exemplarily, the sensor node may generate control information N according to the received output image information N.

[0227] In conjunction with the description of the control information 34a in FIG3 , in this example, the control information N generated by the sensor node may include relevant parameters instructing the sensor to output an image in the Nth frame.

[0228] In some embodiments, the control information N may include exposure information Expo1 of the Nth frame image. The exposure time corresponding to the exposure information Expo1 may be ExpoT1.

[0229] In other embodiments, the control information N may include other parameters required for outputting the image.

[0230] It is understandable that since the Nth frame image is configured not to be displayed, and after the seamless switching is triggered, all configuration information of the first frame is effective for frame 1. Therefore, when the Nth frame image is output, the parameters configured in RequestN+1 are actually used. This control information N will be overwritten by the parameters configured in RequestN+1. Therefore, the actual configuration in this control information N can be flexibly adjusted as needed. This embodiment of the present application is not limited to this.

[0231] As a specific implementation, each parameter in the control information N can be represented by a register address + value format. For details, please refer to the above description.

[0232] S904: The sensor node sends control information N to the Camera Request Management (CRM).

[0233] For example, the sensor node may send control information N to the CRM during the process of outputting the N-3 frame.

[0234] In some other embodiments, the sensor node may also send the control information N to the CRM before the N-2 frame starts to output the image (that is, before the SOF of the N-2 frame arrives).

[0235] In combination with the above description, in this application, the CRM can send control information to the sensor through sensor driving according to the effectiveness timing of each parameter.

[0236] S905: The camera requests the management to send control information N-1 to the sensor driver.

[0237] For example, if all parameters in the control information N-1 are effective for 2 frames, the CRM can send the control information N-1 to the sensor driver during the image output of the N-3 frame (i.e., between the SOF and EOF of the N-3 frame).

[0238] S906: The sensor driver writes control information N-1 to the sensor.

[0239] Exemplarily, the sensor driver may search for a location in a register of the sensor according to a register address indicated by each parameter in the control information N-1, and write the value of the address indicated by the control information N-1 into the register.

[0240] Thus, after the N-3 frame and the N-2 frame are output, the parameters written into the register can take effect, thereby making the parameters indicated by the control information N-1 effective for 2 frames.

[0241] For example, the parameters effective for the two frames may include exposure information of the first frame of the non-seamless switching, etc. In this way, the control information N-1 may include exposure information when the N-1th frame image is output.

[0242] S907, the N-3 frame is output.

[0243] For example, when the EOF of the N-3th frame arrives, it means that the output of the N-3th frame is completed.

[0244] S908: The sensor transmits the N-3 frame image to the sensor driver.

[0245] For example, the sensor may transmit the RAW image of the N-3th frame image to the sensor driver.

[0246] S909: The sensor driver transmits the N-3 frame image to the camera service.

[0247] For example, in some embodiments, the sensor driver may transmit the N-3rd frame of image to a component in the electronic device that performs preliminary image processing for preliminary processing. For example, the sensor driver may transmit the N-3rd frame of RAW image to a component such as an IFE for processing. The preliminarily processed N-3rd frame of image may be stored in the electronic device for display.

[0248] Correspondingly, the camera service (or SurfaceFlinger configured in the electronic device framework layer) can obtain the data of the N-3 frame image in the display queue according to the arrival of the synchronization signal (Vsync signal) and synthesize it. This completes the process of the sensor driver transmitting the N-3 frame image to the camera service as indicated in S909.

[0249] S910. The camera service obtains the display image of the N-3th frame and transmits it to the camera application.

[0250] S911. The camera application controls the display screen to display the N-3 frame image.

[0251] It should be noted that in other embodiments of the present application, steps S910-S911 may be replaced by: the camera service (or SurfaceFlinger) obtains the display image of the N-3th frame and places the display image into the display queue. In this way, the display screen can obtain the queued display image (e.g., obtain the N-3th frame) from the display queue and display it when the next Vsync signal arrives.

[0252] As shown in FIG9 , after the decision module executes S901 , it may also execute S912 - S913 .

[0253] S912: The decision module marks the Nth frame as not to be displayed.

[0254] S913. The decision module sends a flag indicating that the Nth frame will not be displayed to the camera service.

[0255] Combined with the above description about the possible display abnormality of the first frame after switching modes, in this example, the decision module can determine that when the Nth frame image is output using a new output mode (such as modeB), the Nth frame image can be configured not to be displayed.

[0256] In this example, the decision module may mark the Nth frame image through S912 - S913 . In some implementations, the decision module may mark the Request for transmitting the relevant parameters of the Nth frame as DummyRequest.

[0257] In this way, before the camera service (or SurfaceFlinger) needs to perform synthesis processing on the Nth frame image, the Nth frame image may not be synthesized and displayed according to the flag of not displaying the Nth frame.

[0258] In this way, the N-3 frame image can be output and displayed.

[0259] Afterwards, the electronic device may further output and display subsequent images through the following steps.

[0260] S914: The sensor node generates control information N+1.

[0261] Exemplarily, the control information N+1 may be effective through 1 frame, indicating the output parameters of the Nth frame image.

[0262] In some embodiments, the control information N+1 may include information indicating that mode B is used for outputting the image.

[0263] In some other embodiments, the control information N+1 may further include exposure information Expo3 of the Nth image.

[0264] It should be noted that, in the embodiment of the present application, the exposure information Expo3 may be determined by the sensor node.

[0265] In combination with the above description, in the existing solution, the exposure information carried in RequestN+1 may be exposure information Expo2, and the corresponding exposure duration may be ExpoT2.

[0266] In this example, the exposure information carried in the control information N+1 may be Expo3, and the corresponding exposure duration may be ExpoT3. ExpoT3 may be shorter than ExpoT2.

[0267] In some implementations, ExpoT3 can be greater than or equal to half of ExpoT2 and less than ExpoT2.

[0268] In this way, by exposing the Nth frame according to ExpoT3, the exposure time of the Nth frame can be shortened. In conjunction with the description in FIG8 , by shortening the exposure time of the Nth frame, the time between the SOF of the N-1th frame and the SOF of the Nth frame can be shortened. This also shortens the long frame 1 after the seamless switching is triggered as described above.

[0269] This effectively optimizes the jamming caused by the existence of long frame 1.

[0270] As a possible implementation, the sensor node can determine that the Nth frame is the first frame output using the new output mode after seamless switching before generating control information N+1. Thus, control information N+1 is determined to be effective for one frame and control the output parameters (such as exposure information) of the Nth frame.

[0271] The sensor node can obtain the exposure amount corresponding to the exposure information Expo2, and then determine the information of Expo3 according to the exposure amount corresponding to Expo2.

[0272] Illustratively, in different implementations, the sensor node may obtain the exposure corresponding to Expo2 through different pathways.

[0273] In some embodiments, the sensor node can determine the exposure value indicated by Expo2 based on the current ambient brightness and the AE (auto exposure) policy. For example, the sensor node can obtain Expo2, which includes an exposure duration of ExpoT2 and an exposure gain of ExpoG1. Thus, the sensor node can determine that the exposure value indicated by Expo2 is (ExpoT2 × ExpoG1).

[0274] In other embodiments, the CMAX framework module of the electronic device may be configured with an AE module (not shown in FIG2 ), and the AE module may obtain the exposure amount indicated by Expo2 according to the current ambient brightness and the AE strategy. Correspondingly, the sensor module may obtain the exposure amount indicated by Expo2 from the AE module. Similar to the above description, in this example, the AE module may obtain Expo2 including an exposure duration of ExpoT2 and an exposure gain of ExpoG1. Thus, the AE module may determine that the exposure amount indicated by Expo2 is (ExpoT2×ExpoG1). Correspondingly, the sensor node may obtain the exposure amount indicated by Expo2 from the AE module as (ExpoT2×ExpoG1).

[0275] In this example, the sensor node may also determine information about Expo3 based on the exposure corresponding to Expo2.

[0276] In some embodiments, the sensor node may determine the information of Expo3 based on the maximum exposure gain (eg, Gmax) supported by the target sensor and the exposure corresponding to Expo2.

[0277] For example, the sensor node may determine Expo3 to include: exposure duration is ExpoT3, exposure gain is Gmax, where ExpoT3 = (ExpoT2×ExpoG1) / Gmax.

[0278] It's understandable that ExpoG1 is typically smaller than Gmax. Therefore, ExpoT3 is smaller than ExpoT2. Thus, the exposure duration of the Nth frame exposed according to ExpoT3 is smaller than the exposure duration of the Nth frame exposed according to ExpoT2. This shortens the duration of Long Frame 1.

[0279] In other embodiments, the sensor node may determine the information of Expo3 according to a preset strategy and the exposure amount corresponding to Expo2.

[0280] For example, the sensor node may determine ExpoT3 according to a preset strategy. ExpoT3 is less than ExpoT2. Based on this, the sensor node may also determine that the exposure gain in Expo3 is (ExpoT2×ExpoG1) / ExpoT3.

[0281] It is understood that in each of the above implementations, Expo3 in control information N+1 can be obtained. The exposure duration ExpoT3 in Expo3 is less than ExpoT2, thereby shortening the exposure duration of frame N. Furthermore, the exposure gain in Expo3 can be adjusted accordingly to ExpoT3, ensuring that the exposure remains unchanged before and after the exposure duration adjustment. This ensures exposure performance during the output of frame N+1.

[0282] In this application, Expo2 before modification may be referred to as the default exposure information of frame N. ExpoT2 may be the default exposure duration of frame N. ExpoG1 may be the default exposure gain of frame N.

[0283] It should be noted that in order to ensure the stability of the output of adjacent frames. In some implementations, the exposure time and exposure gain of adjacent frames may be the same. For example, before the exposure time of the Nth frame is modified to ExpoT3 according to the solution provided in the embodiment of the present application, the exposure time of the Nth frame may be ExpoT2. Correspondingly, the exposure time of the N-1th frame may also be ExpoT2. In this way, when the sensor node needs to obtain ExpoT2 of the Nth frame, it can also be achieved by obtaining the exposure time of the adjacent frame (such as the N-1th frame). Similarly, the sensor node can determine the exposure gain ExpoG1 of the default configuration of the Nth frame by obtaining the exposure gain of the N-1th frame.

[0284] S915: The sensor node sends control information N+1 to the camera request management CRM.

[0285] Exemplarily, the control information N+1 may include ExpoT3 of the shortened exposure information (exposure duration).

[0286] S916 , the camera requests the management to send control information N to the sensor driver.

[0287] S917: The sensor driver writes control information N to the sensor.

[0288] Exemplarily, the execution of S916 and the execution of S917 may be completed during the process of outputting the N-2 frame image (ie, between the SOF and EOF of the N-2 frame image).

[0289] S918: The N-2 frame is output.

[0290] S919: The sensor transmits the N-2th frame image to the sensor driver.

[0291] S920: The sensor driver transmits the N-2th frame image to the camera service.

[0292] S921. The camera service obtains the display image of the N-2th frame and transmits it to the camera application.

[0293] S922: The camera application controls the display screen to display the N-2 frame image.

[0294] Thus, the output and display of the N-2 frame image can be completed. The execution of S916-S922 can refer to the description of S905-S911 above, and will not be repeated here.

[0295] Then, the electronic device may continue to output and display the N-1th frame image.

[0296] S923: The sensor node generates control information N+2.

[0297] S924: The sensor node sends control information N+2 to the Camera Request Management (CRM).

[0298] Exemplarily, the control information N+2 may include parameters effective for 2 frames, for indicating the output of the N+2 frame image. For example, the control information N+2 may include exposure information corresponding to the output of the N+2 frame image.

[0299] S925: The camera requests the management to send control information N+1 to the sensor driver.

[0300] S926: The sensor driver writes control information N+1 to the sensor.

[0301] Exemplarily, the exposure information carried in the control information N+1 may be Expo3, and the corresponding exposure duration may be ExpoT3.

[0302] In this example, the CRM receives the exposure information Expo3 during the process of outputting the N-2 frame. Correspondingly, during the process of outputting the N-1 frame, the control information N+1 (including Expo3) can be written into the sensor driver through the sensor driver.

[0303] Among them, since the control information N+1 is the parameter of the first frame after triggering seamless switching, the parameters carried in the control information N+1 (such as exposure information Expo3 and image output mode modeB) are all configured to be effective for 1 frame.

[0304] S927: The N-1 frame is output.

[0305] S928: The sensor transmits the N-1th frame image to the sensor driver.

[0306] S929: The sensor driver transmits the N-1th frame image to the camera service.

[0307] S930: The camera service obtains the display image of the N-1th frame and transmits it to the camera application.

[0308] S931. The camera application controls the display screen to display the N-1th frame image.

[0309] Thus, the output and display of the N-1th frame image can be completed. The execution of S925-S931 can refer to the description of S905-S911 above, and will not be repeated here.

[0310] It can be understood that the N-1th frame image may be the last frame image output using mode A before switching.

[0311] Then, the electronic device may continue to output the Nth frame image, which may be the first frame image output after the seamless switching is triggered.

[0312] S932: The sensor node generates control information N+3.

[0313] S933: The sensor node sends control information N+3 to the Camera Request Management (CRM).

[0314] Exemplarily, the control information N+3 may include parameters effective for 2 frames, for indicating the output of the N+3 frame image. For example, the control information N+3 may include exposure information corresponding to the output of the N+3 frame image.

[0315] S934: The camera requests the management to send control information N+2 to the sensor driver.

[0316] S935: The sensor driver writes control information N+2 to the sensor.

[0317] For example, the process of S934-S935 can be implemented during the process of outputting the image of the Nth frame. In this way, the output parameters of the N+2th frame image can be written into the register so as to take effect after 2 frames.

[0318] S936: The Nth frame is output.

[0319] In combination with the above description, the process of outputting the Nth frame may start with the SOF of the Nth frame and end with the EOF of the Nth frame. During the outputting process of the Nth frame, the effective parameter configuration may be that the output mode in the control information N+1 is modeB.

[0320] Before the SOF of frame N arrives and after the EOF of frame N-1 arrives, the sensor can output an image using a new output mode (e.g., mode B) as indicated by control information N+1, thereby switching the output mode. For example, a reset process can be performed to cause each component to switch to mode B for output. For another example, after the reset process is completed, the exposure of frame N is performed according to the exposure information Expo3 indicated by control information N+1.

[0321] In conjunction with the above description, the exposure duration of the Nth frame image can be ExpoT3, which is shorter than the default Expo2. Therefore, in this example, the duration between the SOF of the Nth frame and the EOF of the N-1th frame can be shorter than the duration between the SOF of the Nth frame and the EOF of the N-1th frame in the existing solution. This achieves the effect of shortening the long frame 1.

[0322] S937: The sensor transmits the Nth frame image to the sensor driver.

[0323] S938: The sensor driver transmits the Nth frame image to the camera service.

[0324] It is understood that after the Nth frame image is output, the camera service may not perform subsequent synthesis or other processing on the Nth frame image received in S938 based on the indication that the Nth frame is not to be displayed received in S913. In this way, the corresponding output processing of the Nth frame can be completed.

[0325] Thus, through the implementation of the above-mentioned S901-S938 solution, the electronic device can adjust the parameter indicating the exposure time in the control information of the Nth frame image (such as adjusting it to ExpoT3), so that the exposure time of the Nth frame image is shortened. This achieves the effect of shortening the long frame 1. In turn, the display jamming caused by the long frame 1 is optimized or avoided. In other words, the display time of the N-1th frame on the display screen is controlled, avoiding display jamming caused by the N-1th frame image being displayed on the display screen for too long.

[0326] In other embodiments of the present application, a solution is provided to optimize long frame 2, so that the display freeze of the first frame image (i.e., the N+1th frame image) displayed after switching to mode B is optimized or the display freeze of this frame image is avoided.

[0327] For example, refer to Figure 10. In the existing solution shown in Solution 1, since the Nth frame is not displayed, the timestamp of the N+1th frame is adjusted to the timestamp of the Nth frame. This allows the display screen to continuously display the image of the N+1th frame during the display interval corresponding to the SOF of the Nth frame to the EOF of the N+1th frame. This results in Long Frame 2, which corresponds to the duration between the SOF of the Nth frame and the EOF of the N+1th frame. This also causes the lag caused by the prolonged display of the N+1th frame.

[0328] Correspondingly, in some embodiments of the present application, the electronic device can control the frame length of the Nth frame and / or the N+1th frame (i.e., the output frame length). For example, the adjusted Nth frame may be smaller than the frame length of the Nth frame before adjustment. For another example, the adjusted frame length of the N+1th frame may be smaller than the frame length of the N+1th frame before adjustment. As a result, the duration between the SOF of the Nth frame and the EOF of the N+1th frame is shortened, thereby shortening the display duration of the N+1th frame on the display screen. That is, the duration of the long frame 2 is shortened, thereby optimizing the jamming caused by the existence of the long frame 2.

[0329] The following describes the implementation of the solution shown in FIG10 in detail in conjunction with the interaction example of FIG11. Here, the Nth frame image is taken as the first frame image output using mode B as an example.

[0330] In addition, in the example shown in FIG11 , the frame lengths of the Nth frame and the N+1th frame are adjusted simultaneously.

[0331] As shown in Figure 11, the solution may include:

[0332] S1101: The decision module determines to switch to mode B for drawing output.

[0333] S1102. The decision module sends output image information N to the sensor node.

[0334] S1103: The sensor node generates control information N.

[0335] S1104 : The sensor node sends control information N to the Camera Request Management (CRM).

[0336] S1105 : The camera requests the management to send control information N-1 to the sensor driver.

[0337] S1106 : The sensor driver writes control information N-1 to the sensor.

[0338] S1107, the N-3 frame is output.

[0339] S1108 : The sensor transmits the N-3 frame image to the sensor driver.

[0340] S1109 : The sensor driver transmits the N-3 frame image to the camera service.

[0341] S1110. The camera service obtains the display image of the N-3th frame and transmits it to the camera application.

[0342] S1111. The camera application controls the display screen to display the N-3 frame image.

[0343] S1112: The decision module marks the Nth frame as not to be displayed.

[0344] S1113. The decision module sends a flag indicating that the Nth frame will not be displayed to the camera service.

[0345] In combination with the solution description in FIG9 , the execution of S1101 to S1113 in FIG11 may refer to the description of S901 to S913 in FIG9 , which will not be repeated here.

[0346] Thus, the N-3 frame image can be output and displayed. The N-3 frame image can be output according to the output mode of modeA.

[0347] S1114. The sensor node generates control information N+1.

[0348] In this example, the control information N+1, as the configuration information for the image output parameters of the switching frame, can be effective for one frame. For example, the image output mode (such as modeB), exposure information, and image output frame length in the control information N+1 can all be effective for one frame. This allows the control information N+1 to take effect during the image output process of the Nth frame. In this way, the parameter configuration for the image output of the Nth frame can be implemented through the control information N+1.

[0349] The switching frame may be the first frame that triggers seamless switching, that is, the Nth frame.

[0350] In this example, the control information N+1 may include information indicating that the image is output using mode B. The control information N+1 may also include exposure information indicating the Nth frame image.

[0351] In some embodiments, the exposure information in the control information N+1 may be the default exposure information Expo2, and the corresponding exposure duration may be the exposure duration ExpoT2.

[0352] In other embodiments, such as the solution example in FIG9 , the exposure information in the control information N+1 may be the adjusted exposure information Expo3, and the corresponding exposure duration may be ExpoT3, which is shorter than ExpoT2.

[0353] In this embodiment, the control information N+1 may also include frame length information.

[0354] For example, the frame length information may include VTS information, which indicates the frame length of the Nth frame when the image is output.

[0355] It is understandable that in existing solutions (such as Solution 1), the frame length information carried in RequestN+1 corresponding to the Nth frame may include the default VTS1. Taking the current output frame rate of 30fps as an example, the VTS1 may indicate that the frame length of the Nth frame is 33ms corresponding to 30fps.

[0356] In this example, the frame length information included in the control information N+1 may be VTS3. The frame length indicated by VTS3 may be smaller than the frame length indicated by VTS1.

[0357] As a possible implementation, when executing S1114 , the sensor node may read the parameter configuration file to obtain the default VTS1 of the Nth frame.

[0358] In an embodiment of the present application, a parameter configuration file may be preset in the electronic device. For example, the parameter configuration file may be a file in XML format. The parameter configuration file may store the default frame lengths corresponding to each frame image. For example, the default frame length of the Nth frame is VTS1. For another example, the default frame length of the N+1th frame is VTS2.

[0359] The sensor node can also determine VTS3 based on VTS1. For example, the sensor node can determine that VTS3 is equal to (VTS1) / 2 based on preconfigured logic. In this way, the frame length indicated by VTS3 can be equal to half of 33ms, that is, 16.5ms.

[0360] For example, a sensor node can determine, based on preconfigured logic, that VTS3 is greater than or equal to (VTS1) / 2 and less than VTS1. The specific implementation can be flexibly adjusted based on actual conditions. This can also ensure that the frame length indicated by VTS3 is less than the default frame length indicated by VTS1.

[0361] In this way, when outputting the image according to VTS3, the frame length of the Nth frame image can be controlled to be less than the default frame length (such as equal to half of the default frame length), thereby saving the output time of the Nth frame image.

[0362] Corresponding to the example in Figure 10, the Nth frame is output based on the frame length including VTS3, which can shorten the duration between the SOF and EOF of the Nth frame image. Compared with the obvious solution, even if the output duration of the N+1th frame remains unchanged, the duration between the SOF of the Nth frame and the EOF of the N+1th frame (i.e., the duration of long frame 2) can also be shortened. In this way, the display time of the N+1th frame image can be shortened, thereby optimizing the freeze caused by the long display of the N+1th frame image.

[0363] S1115 : The sensor node sends control information N+1 to the Camera Request Management (CRM).

[0364] S1116 , the camera requests management to send control information N to the sensor driver.

[0365] S1117 : The sensor driver writes control information N to the sensor.

[0366] S1118. The N-2 frame is output.

[0367] S1119: The sensor transmits the N-2th frame image to the sensor driver.

[0368] S1120: The sensor driver transmits the N-2th frame image to the camera service.

[0369] S1121. The camera service obtains the display image of the N-2th frame image and transmits it to the camera application.

[0370] S1122: The camera application controls the display screen to display the N-2 frame image.

[0371] For the execution steps from S1115 to S1122, reference may be made to the execution process from S915 to S922 in FIG9 , which will not be repeated here.

[0372] This allows the output and display of frame N-2 to be achieved. This frame N-2 can still be output according to mode A. It is understood that during the output of frame N-2, the KMD of the electronic device receives control information N+1. Therefore, during the output of the next frame, this control information N+1 can be written into the sensor register.

[0373] S1123: The sensor node generates control information N+2.

[0374] Exemplarily, the control information N+2 may also include frame length information.

[0375] Combined with the aforementioned explanation of the effectiveness of the frame length information, the frame length information can be effective one frame after being written to the register. In this way, if the control information N+2 is written to the register during the output of the N-1 frame, the control information N+2 can be used to control the output frame length of the N+1 frame.

[0376] In some embodiments, the frame length in the control information N+2 may be VTS4. The mechanism for obtaining VTS4 may refer to the mechanism for obtaining VTS3 in S1114. For example, the sensor node may obtain the default output frame length of the N+1th frame as VTS2 from the parameter configuration file.

[0377] The sensor node can determine the VTS4 actually carried in the control information N+2 based on the VTS2. For example, the VTS4 can be in the range of [(VTS2) / 2, VTS2). For another example, the VTS4 can be equal to (VTS2) / 2.

[0378] In this way, the VTS4 carried in the control information N+2 can be used to control the frame length of the N+1 frame output. This shortens the frame length of the N+1 frame output, thereby shortening the long frame 2. This solves the problem of frame lag caused by the prolonged display of the N+1 frame corresponding to the long frame 2 on the screen.

[0379] It can be understood that, in combination with the description of S1114 and S1123, by simultaneously controlling the VTS in the control information N+1 and the control information N+2, it is possible to control the frame length of the Nth frame and the N+1th frame image output, thereby significantly shortening the duration of the long frame 2.

[0380] In other embodiments, the electronic device can also flexibly select to control the frame length of the Nth frame or the N+1th frame. Thus, when the frame length of the Nth frame or the N+1th frame is shortened, the long frame 2 can be shortened compared to the existing solution. For specific implementation, please refer to S1114 or S1123, and the description of the subsequent corresponding steps. This application will not be repeated here.

[0381] S1124 : The sensor node sends control information N+2 to the Camera Request Management (CRM).

[0382] S1125 . The camera requests the management to send control information N+1 to the sensor driver.

[0383] S1126 : The sensor driver writes control information N+1 to the sensor.

[0384] S1127, the N-1 frame is output.

[0385] S1128. The sensor transmits the N-1th frame image to the sensor driver.

[0386] S1129: The sensor driver transmits the N-1th frame image to the camera service.

[0387] S1130. The camera service obtains the display image of the N-1th frame image and transmits it to the camera application.

[0388] S1131. The camera application controls the display screen to display the N-1th frame image.

[0389] For the execution process of S1124 to S1131 , reference may be made to S924 to S931 in FIG. 9 .

[0390] It is understood that in this example, control information N+1 can be written to the sensor register during the N-1 frame output process. In this way, based on the 1-frame effective mechanism, the output parameters in control information N+1 (such as the frame length of VTS3) can take effect during the N-1 frame output process, thereby achieving the purpose of controlling the output frame length of the N-1 frame image.

[0391] S1132: The sensor node generates control information N+3.

[0392] S1133 : The sensor node sends control information N+3 to the Camera Request Management (CRM).

[0393] S1134 , the camera requests the management to send control information N+2 to the sensor driver.

[0394] S1135 : The sensor driver writes control information N+2 to the sensor.

[0395] S1136: The Nth frame is output.

[0396] S1137 : The sensor transmits the Nth frame image to the sensor driver.

[0397] S1138. The sensor driver transmits the Nth frame image to the camera service.

[0398] The execution of the above steps S1132 to S1138 may refer to S932 to S938 in FIG. 9 .

[0399] In this example, combined with the descriptions from S1114 to S1126, during the output of frame N-1, control information N+1 carrying VTS3 is written into the register. This takes effect after one frame, and during the execution of S1136, this VTS3 takes effect. Thus, the frame length of the output frame N can correspond to the shortened VTS3. As a result, the duration between SOF and EOF during the output of frame N can be effectively shortened, thereby shortening the duration of long frame 2.

[0400] In this process, the image output parameters in the control information N+3 can be used to configure the output of subsequent images.

[0401] It can be understood that when the switching frame is the Nth frame, after the control information of the subsequent frame image is written into the register, the parameters such as exposure information are restored to be effective for 2 frames, and the parameters such as frame length are effective for 1 frame.

[0402] Similarly, the frame length in control information N+2 is effective when 1 frame is used. In this way, when control information N+2 is written into the register during the output of frame N, the frame length in control information N+2 (such as VTS4) can take effect during the output of frame N+1. Take VTS4 as half of the default VTS2 as an example. When the subsequent output of frame N+1 is executed according to VTS4, the duration between SOF and EOF of frame N+1 can be effectively shortened, thereby shortening the duration of long frame 2.

[0403] 9 , in some implementations, if the control information N+1 includes shortened exposure information, the sensor may expose the Nth frame before step S1136 . The exposure duration is shorter than the default exposure duration, thereby shortening the long frame 1.

[0404] Of course, in other implementations, the exposure information in the control information N+1 can also be a default value. In this way, although long frame 1 still exists, long frame 2 is shortened, which can effectively optimize the display freeze after triggering seamless switching.

[0405] S1139. The sensor node generates control information N+4.

[0406] S1140 : The sensor node sends control information N+4 to the Camera Request Management (CRM).

[0407] S1141 , the camera requests management to send control information N+3 to the sensor driver.

[0408] S1142: The sensor driver writes control information N+3 to the sensor.

[0409] S1143, the N+1 frame is output.

[0410] S1144: The sensor transmits the (N+1)th frame image to the sensor driver.

[0411] S1145: The sensor driver transmits the N+1th frame image to the camera service.

[0412] S1146. The camera service obtains the display image of the N+1th frame and transmits it to the camera application.

[0413] S1147. The camera application controls the display screen to display the N+1th frame image.

[0414] The execution process from S1139 to S1147 can realize the display of the N+1 frame and the subsequent sending and writing of control information. The specific execution process can be referred to the above description of the N-2 frame or the N-1 frame.

[0415] It is understood that in this example, the frame length information in the control information N+2 can be written into the register during the Nth frame output process and take effect after 1 frame. Thus, it takes effect during the N+1 frame output process (such as S1143).

[0416] For example, the frame length included in the control information N+2 includes VTS4, and VTS4 is equal to half of the default VTS2.

[0417] In the process of outputting the N+1th frame, the frame length can be shortened to half of the default frame length, thereby shortening the duration between the SOF and EOF of the N+1th frame. This in turn shortens the duration between the SOF of the Nth frame and the EOF of the N+1th frame. The duration between the SOF of the Nth frame and the EOF of the N+1th frame is also the duration of long frame 2, which corresponds to the duration of the N+1th frame displayed on the display screen. In this way, by shortening the duration of long frame 2, the display duration of the N+1th frame on the display screen can be effectively shortened, thereby optimizing the stuttering problem caused by the existence of long frame 2.

[0418] Thus, in this application, as shown in the examples of Figures 8 to 9, by controlling the exposure duration of the Nth frame, the duration of the long frame 1 is effectively shortened, thereby optimizing the display duration of the N-1th frame on the display screen. This optimizes the jamming caused by the presence of the long frame 1. As shown in the examples of Figures 10 to 11, by controlling the output frame length of the Nth frame and / or the N+1th frame, the duration of the long frame 2 is effectively shortened, thereby optimizing the display duration of the N+1th frame on the display screen. This optimizes the jamming caused by the presence of the long frame 2.

[0419] In the specific implementation process, the solutions shown in Figures 8 to 9, or Figures 10 to 11, can be used to optimize Long Frame 1 or Long Frame 2 to optimize the display stuttering problem after triggering seamless switching. Alternatively, the solutions shown in Figures 8 to 11 can be used simultaneously to optimize Long Frame 1 and Long Frame 2, further effectively optimizing the display stuttering problem after triggering seamless switching.

[0420] It should be noted that, in each of the above embodiments, the method of transmitting control information can be implemented through the Request information in the aforementioned examples. For example, the control information is carried in the Request information to realize the transmission between various modules. In addition, in some implementations, each output parameter in the control information (such as output mode, exposure information, output frame length, etc.) can be identified in the form of a register address and a corresponding value. In this way, by carrying the register address and value corresponding to the parameters to be configured in the Request information, each component (such as sensor driver) can realize the writing of the output parameters by writing the corresponding value to the register address carried in the Request information.

[0421] The technical solutions provided in the embodiments of the present application can be applied to the user's electronic devices. The electronic devices may include mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices, or smart city devices. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0422] It is understandable that the electronic device provided in the embodiment of the present application includes a hardware structure and / or software module for performing each function in order to realize the above functions. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.

[0423] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0424] Exemplarily, in some embodiments, referring to FIG12 , a software composition of another electronic device is provided in an embodiment of the present application.

[0425] In this example, similar to the composition in Figure 2, the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0426] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0427] The application layer can include a series of application packages.

[0428] As shown in FIG12 , the application package may include applications such as camera, calendar, map, WLAN, music, SMS, call, navigation, Bluetooth, and video.

[0429] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0430] As shown in FIG12 , the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like.

[0431] The Android runtime consists of the core libraries and the Android runtime. The Android runtime is responsible for converting source code into machine code. It primarily utilizes ahead-of-time (AOT) and just-in-time (JIT) compilation technologies.

[0432] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides an API for users to develop Android applications.

[0433] Native C / C++ libraries can include multiple functional modules, such as surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0434] The hardware abstraction layer runs in user space, encapsulates kernel layer drivers, and provides a calling interface to the upper layer. Exemplarily, the hardware abstraction layer may include a display module, an audio module, a camera module, a Bluetooth module, etc.

[0435] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and Bluetooth driver.

[0436] In other embodiments of the present application, a hardware composition of an electronic device is also provided.

[0437] Exemplarily, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera module, a display, and a subscriber identification module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor 180L, a bone conduction sensor, etc.

[0438] The processor may include one or more processing units, for example, an application processor (AP), a modem processor (Modem), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor (BP or BBP), and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0439] It should be noted that the structures illustrated in the embodiments of this application do not constitute specific limitations on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components, or combine or separate certain components, or arrange the components differently. The above-mentioned components may be implemented in hardware, software, or a combination of software and hardware.

[0440] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of each functional module. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0441] The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0442] For example, Figure 13 shows a schematic diagram of the components of an electronic device 1300. As shown in Figure 13, the electronic device 1300 may include a processor 1301 and a memory 1302. The memory 1302 is used to store computer-executable instructions. For example, in some embodiments, when the processor 1301 executes the instructions stored in the memory 1302, the electronic device 1300 may perform any of the methods described in the above embodiments.

[0443] In some implementations, as shown in FIG13 , the electronic device 1300 may further include a display screen 1303, which may be one or more. The display screen 1303 may be coupled to the processor 1301 and the memory 1302, respectively, so that the display screen 1303 cooperates with the processor 1301 and the memory 1302 to implement the technical solutions provided in the above-mentioned embodiments.

[0444] In some implementations, as shown in FIG13 , the electronic device 1300 may further be configured with a camera 1304, which may be one or more cameras 1304. Each camera 1304 may be configured with a corresponding image sensor. The cameras 1304 may be coupled to the processor 1301 and the memory 1302, respectively, so that the cameras 1304 cooperate with the processor 1301 and the memory 1302 to implement the technical solutions provided in the above-described embodiments.

[0445] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0446] Figure 14 shows a schematic diagram of the composition of a chip system 1400. The chip system 1400 may include: a processor 1401 and a communication interface 1402, which are used to support related devices to implement the functions involved in the above embodiments. In one possible design, the chip system also includes a memory for storing program instructions and data necessary for electronic devices. The chip system can be composed of chips, or it can include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 1402 may also be referred to as an interface circuit.

[0447] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0448] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method flow related to the electronic device in any of the above method embodiments. Specifically, the computer can be the above electronic device.

[0449] The present application also provides a computer program or a computer program product including the computer program, which, when executed on a computer, enables the computer to implement the method flow related to the electronic device in any of the above method embodiments. Specifically, the computer can be the above electronic device.

[0450] The functions, actions, operations, steps, etc. in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0451] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. A data processing method, characterized in that: An electronic device is applied, the electronic device including a camera application and an image sensor, and the method includes: At a first moment, starting the camera application; The image sensor outputs an image based on a first image output mode and a first exposure duration; At a second moment, first control information is acquired, where the first control information includes first image output parameters, where the first image output parameters include a second exposure duration and a second image output mode, wherein the second exposure duration is less than the first exposure duration, the second image output mode is different from the first image output mode, and the second moment is later than the first moment; At a third moment, the image sensor performs exposure based on the second exposure duration, wherein the third moment is later than the second moment; At a fourth moment, the image sensor outputs an image based on the second image output mode, wherein the fourth moment is later than the third moment.

2. The method according to claim 1, characterized in that The image sensor outputs an image based on the second image output mode, including: The image sensor outputs the Nth frame image based on the second image output mode, wherein the first exposure duration is the same as the exposure duration of the N-1th frame, and the Nth frame image is the first frame image output by the image sensor after switching to the second image output mode.

3. The method according to claim 2, characterized in that The second time is earlier than the time corresponding to the SOF (Start of Frame Delimiter) of the N-1th frame image.

4. The method according to claim 2 or 3, characterized in that The third moment is between the moment corresponding to the EOF (End of Frame Delimiter) of the N-1th frame image and the moment corresponding to the SOF of the Nth frame image.

5. The method according to any one of claims 2 to 4, characterized in that The fourth moment is the moment corresponding to the SOF of the Nth frame image.

6. The method according to any one of claims 2 to 5, characterized in that The first control information corresponds to the information of RequestN+1.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: At a fifth time, the first control information is written into a first register, where the first register is a register of the image sensor, wherein the fifth time is later than the second time and earlier than the third time.

8. The method according to claim 7, characterized in that The first control information includes a first address and a first value, wherein the first address is an address of the image sensor storing the exposure duration, and the first value corresponds to the second exposure duration; Writing the first control information into the first register includes: The first value is written to a first address of the first register.

9. The method according to claim 8, characterized in that The fifth moment is between the moment corresponding to the SOF and the moment corresponding to the EOF of the N-1th frame image.

10. The method according to any one of claims 7 to 9, characterized in that The electronic device is equipped with a camera request management module CRM; The acquiring of the first control information includes: The CRM obtains the first control information at the second moment; Writing the first control information into the first register includes: The CRM writes the first control information into the first register at the fifth moment.

11. The method according to any one of claims 1 to 10, characterized in that Before generating the first control information, the method further includes: The second exposure duration is determined.

12. The method according to claim 11, characterized in that Before determining the second exposure duration, the method further includes: Obtaining the first exposure duration; The determining the second exposure duration includes: Based on the first exposure duration, the second exposure duration is determined so that the second exposure duration is shorter than the first exposure duration.

13. The method according to claim 12, characterized in that The method further includes: acquiring a first exposure gain, wherein the first exposure gain is an exposure gain used by the image sensor when outputting an image based on the first output mode; The determining the second exposure duration based on the first exposure duration includes: The second exposure duration is determined based on the first exposure duration and the first exposure gain.

14. The method according to claim 13, wherein: The first control information further includes a second exposure gain, wherein a product of the second exposure gain and the second exposure duration is equal to a product of the first exposure gain and the first exposure duration; The image sensor performs exposure based on the second exposure duration, including: The image sensor performs exposure based on the second exposure duration and the second exposure gain.

15. The method according to claim 13 or 14, characterized in that The second exposure gain is a maximum exposure gain of the image sensor.

16. A data processing method, characterized in that: An electronic device is applied, the electronic device including a camera application and an image sensor, and the method includes: At a first moment, starting the camera application; The image sensor outputs an image based on a first image output mode; At a second moment, first control information is acquired, where the first control information includes first image output parameters, where the first image output parameters include a first frame length and a second image output mode, where the first frame length is less than the second frame length, the second frame length is a frame length configured for the image sensor when outputting an image based on the second image output mode, the second image output mode is different from the first image output mode, and the second moment is later than the first moment; At a fourth moment, the image sensor outputs an image based on the second image output mode and the first frame length, wherein the fourth moment is later than the third moment.

17. The method according to claim 16, characterized in that The fourth moment is the moment corresponding to the SOF of the N-th frame image; the second frame length is the frame length configured for the image sensor when outputting the N-th frame image based on the second image output mode; The image sensor outputs an image based on the second image output mode and the first frame length, including: The image sensor outputs an N-th frame image based on the second image output mode and the first frame length, wherein the N-th frame image is the first frame image output by the image sensor after switching to the second image output mode.

18. The method according to claim 17, characterized in that The second moment is earlier than the moment corresponding to the SOF of the N-1th frame image.

19. The method according to claim 17 or 18, characterized in that The first control information corresponds to the information of RequestN+1.

20. The method according to any one of claims 16 to 19, characterized in that The method further comprises: At a fifth time, the first control information is written into a first register, where the first register is a register of the image sensor, wherein the fifth time is later than the second time and earlier than the third time.

21. The method according to claim 20, characterized in that The first control information includes a second address and a second value, wherein the second address is an address of the image sensor storing the frame length, and the second value corresponds to the first frame length; Writing the first control information into the first register includes: The second value is written to the second address of the first register.

22. The method according to claim 20 or 21, characterized in that The electronic device is equipped with a camera request management module CRM; The acquiring of the first control information includes: The CRM obtains the first control information at the second moment; Writing the first control information into the first register includes: The CRM writes the first control information into the first register at the fifth moment.

23. The method according to any one of claims 20 to 22, characterized in that The fifth moment is between the moment corresponding to the SOF and the moment corresponding to the EOF of the N-1th frame image.

24. The method according to any one of claims 16 to 23, characterized in that The first frame length is smaller than the second frame length, and the first frame length is greater than or equal to half of the second frame length.

25. The method according to any one of claims 16 to 24, characterized in that Before generating the first control information, the method further includes: The first frame length is determined.

26. The method according to claim 25, characterized in that Before determining the first frame length, the method further includes: Obtaining the second frame length; The determining the first frame length includes: The first frame length is determined based on the second frame length.

27. The method according to claim 26, characterized in that The obtaining of the second frame length includes: Read a parameter configuration file to obtain the second frame length, wherein the parameter configuration file is stored in the electronic device.

28. The method according to claim 26 or 27, characterized in that The determining the first frame length based on the second frame length includes: Half of the second frame length is determined as the first frame length.

29. The method according to any one of claims 16 to 28, wherein: The method further comprises: generating second control information at a sixth moment, the second control information including a second image output parameter, the second image output parameter including a third frame length, wherein the third frame length is less than a fourth frame length, the fourth frame length being a frame length configured for the image sensor when outputting the (N+1)th image frame based on the second image output mode, the sixth moment being later than the second moment; At a seventh moment, the image sensor outputs the (N+1)th frame image based on the second image output mode and the third frame length, wherein the seventh moment is later than the fourth moment.

30. The method according to claim 29, wherein The third frame length is greater than or equal to half of the fourth frame length.

31. The method according to claim 29 or 30, characterized in that The sixth moment is earlier than the moment corresponding to the SOF of the Nth frame image.

32. The method according to any one of claims 29 to 31, characterized in that The seventh moment is the moment corresponding to the SOF of the N+1th frame image.

33. The method according to any one of claims 29 to 32, characterized in that The method further comprises: At an eighth moment, the third frame length is written into the register of the image sensor, wherein the eighth moment is between a moment corresponding to the SOF and a moment corresponding to the EOF of the Nth frame image.

34. The method according to any one of claims 29 to 33, wherein: The second control information corresponds to the information of RequestN+2.

35. An electronic device, characterized in that: The electronic device includes: a memory, one or more processors, and one or more cameras; each of the cameras is configured with a corresponding image sensor; The memory is coupled to the processor and coupled to the image sensor; In which, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 15, and / or executes the method as described in any one of claims 16 to 34, and controls the image sensor to output an image.

36. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-15, and / or executes the method as described in any one of claims 16-34.

37. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 15, and / or execute the method according to any one of claims 16 to 34.

38. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 15 and / or execute the method according to any one of claims 16 to 34.

Images