Information processing method and device, system on chip, mainboard, equipment and storage medium

Abstract

The application provides an information processing method and device, a system on chip, a mainboard, equipment and a storage medium. The method comprises the following steps: in the case that it is determined that a first ISP pipeline loses frames, a frame rate of a first image sensor decreases or exposure time increases, a second ISP pipeline loses frames or a number of first requests in a first cache queue is less than a first value, at least one first request cached in the first cache queue is controlled to be used for the operation of a second image sensor and the second ISP pipeline; the first ISP pipeline is used for processing first image data output by the first image sensor; the second ISP pipeline is used for processing second image data output by the second image sensor; and the first request is used for configuring working parameters required for the second image sensor to output the second image data and working parameters required for the second ISP pipeline to process the second image data.

Description

Technical Field

[0001] This application relates to electronic technology, including but not limited to information processing methods and apparatus, system-on-a-chip, motherboard, device, and storage medium. Background Technology

[0002] After years of development, single-camera capabilities have reached a point where significant improvements in user experience are unlikely. Therefore, multi-lens configurations in electronic devices (such as smartphones and tablets) have become the mainstream development trend. For example, an electronic device might include an ultra-wide-angle lens, a wide-angle lens, and a telephoto lens. How to better leverage the capabilities of dual or multi-camera systems is a key area that future imaging solutions need to explore.

[0003] In applications where camera-captured images are previewed, electronic devices typically enable two image data streams simultaneously. The first stream includes data from a first image sensor (e.g., an ultra-wide-angle lens or telephoto lens) and the image data processed by a first ISP pipeline. The second stream includes data from a second image sensor (e.g., a wide-angle lens) and the image data processed by a second ISP pipeline. Generally, the former stream is used for display and preview, while the latter stream provides 3A data (i.e., 3A information) for the first ISP pipeline's 3A calculations, ensuring alignment / synchronization between the 3A data from the first image sensor and the first ISP pipeline and the 3A data from the second image sensor. However, in practical applications, misalignment of the 3A data from the two streams can occur, leading to abnormalities in brightness and white balance, such as jitter and jumps, affecting image quality. Summary of the Invention

[0004] The information processing method, apparatus, system-on-a-chip, motherboard, device, and storage medium provided in this application can ensure the normal operation of the second ISP pipeline, thereby aligning / synchronizing the 3A data of the first image sensor and the first ISP pipeline with the 3A data of the second image sensor.

[0005] In a first aspect, embodiments of this application provide an information processing method, comprising: based on determining that a first ISP pipeline is dropping frames, a first image sensor is experiencing a decrease in frame rate or an increase in exposure time, a second ISP pipeline is dropping frames, or the number of first requests in a first buffer queue is less than a first value, controlling the first buffer queue to cache at least one first request for the operation of a second image sensor and a second ISP pipeline; wherein, the first ISP pipeline is used to process first image data output by the first image sensor; the second ISP pipeline is used to process second image data output by the second image sensor; and the first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

[0006] Secondly, embodiments of this application provide an information processing method, characterized in that the method includes: dispatching a first request to a first buffer queue based on determining that a first ISP pipeline is dropping frames; and / or, not dispatching the first request to the first buffer queue based on determining that a second ISP pipeline is dropping frames; and / or, dispatching the first request to the first buffer queue according to a first ratio based on determining that the number of first requests in the first buffer queue is less than a first value; and / or, performing one of the following actions based on determining that the frame rate of a first image sensor is decreasing or the exposure time is increasing: increasing the exposure time of a second image sensor; adjusting the frame rate of the second image sensor according to the frame rate of the first image sensor; dispatching the first request to the first buffer queue according to the frame rate of the second image sensor; wherein, the first ISP pipeline uses... The first image data output by the first image sensor is processed; the second ISP pipeline is used to process the second image data output by the second image sensor; the first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data; the second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data; the first ratio is greater than the second ratio, and both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than a first value.

[0007] Thirdly, embodiments of this application provide an information processing apparatus, the apparatus comprising: a control module configured to, based on determining that a first ISP pipeline is dropping frames, a first image sensor is experiencing a decrease in frame rate or an increase in exposure time, a second ISP pipeline is dropping frames, or the number of first requests in a first buffer queue is less than a first value, control the first buffer queue to cache at least one first request for the operation of a second image sensor and a second ISP pipeline; wherein, the first ISP pipeline is used to process first image data output by the first image sensor; the second ISP pipeline is used to process second image data output by the second image sensor; and the first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

[0008] Fourthly, embodiments of this application provide an information processing apparatus, characterized in that the apparatus includes: a control module; wherein the control module is configured to: dispatch a first request to a first buffer queue based on determining that a first ISP pipeline has dropped frames; and / or, not dispatch the first request to the first buffer queue based on determining that a second ISP pipeline has dropped frames; and / or, dispatch the first request to the first buffer queue according to a first ratio based on determining that the number of first requests in the first buffer queue is less than a first value; and / or, perform one of the following actions based on determining that the frame rate of a first image sensor has decreased or the exposure time has increased: increase the exposure time of a second image sensor; adjust the frame rate of the second image sensor according to the frame rate of the first image sensor; dispatch the first request to the first buffer queue according to the frame rate of the second image sensor; wherein, the... The first ISP pipeline is used to process the first image data output by the first image sensor; the second ISP pipeline is used to process the second image data output by the second image sensor; the first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data; the second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data; the first ratio is greater than the second ratio, and both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio refers to the dispatch ratio used before determining that the number of first requests in the first buffer queue is less than a first value.

[0009] Fifthly, embodiments of this application provide a system-on-a-chip, including a first processor, a second processor, and a memory; the memory stores a computer program that can run on the processor, the first processor executes the program to implement the method described in the first aspect or the second aspect, and the second processor is used to run one or more ISP pipelines.

[0010] In a sixth aspect, embodiments of this application provide a motherboard including the system-on-a-chip described in the fifth aspect, a first image sensor, and a second image sensor.

[0011] In a seventh aspect, embodiments of this application provide an electronic device, including the motherboard described in the sixth aspect.

[0012] Eighthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0013] In this embodiment, when an abnormal situation is determined to occur, namely, when the first ISP pipeline drops frames, the frame rate of the first image sensor decreases or the exposure time increases, the second ISP pipeline drops frames, or the number of first requests in the first buffer queue is less than a first value, the system controls the first buffer queue to cache at least one of the first requests. This enables the second ISP pipeline to obtain the first request from the first buffer queue, thereby processing the image data output by the second image sensor in a timely manner. This avoids processing delays or frame drops caused by the second ISP pipeline's inability to obtain the first request, and further enables the 3A data of the first image sensor and the first ISP pipeline to be aligned / synchronized with the 3A data of the second image sensor.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0017] Figure 1 This is a schematic diagram of dual-camera synchronization in a synchronous frame rate scenario;

[0018] Figure 2 A schematic diagram of the Pipeline design for related scheme 1;

[0019] Figure 3 A schematic diagram of a lightweight pipeline design for dual-camera synchronization in related scheme 2;

[0020] Figure 4 Schematic diagram of the implementation flow of the information processing method provided in the embodiments of this application Figure 1 ;

[0021] Figure 5 Schematic diagram of the implementation flow of the information processing method provided in the embodiments of this application Figure 2 ;

[0022] Figure 6 Example diagram of the working range of dual-camera synchronization (wide-angle as secondary camera) under the default synchronization frame rate scenario provided in the embodiments of this application;

[0023] Figure 7 Example diagram of the working range of dual-camera synchronous (wide-angle as secondary camera) camera in the default asynchronous frame rate scenario provided in the embodiments of this application;

[0024] Figure 8 A usecase-related pipeline node and control block diagram provided for embodiments of this application;

[0025] Figure 9 Another use case-related pipeline node and control block diagram provided in this application embodiment;

[0026] Figure 10 A schematic diagram illustrating the Request dispatch strategy under the default master-slave synchronization frame rate provided in this application embodiment;

[0027] Figure 11 A schematic diagram of a request dispatch strategy based on frame loss and frame rate fluctuation monitoring under the default master-slave synchronization frame rate provided in this application embodiment;

[0028] Figure 12 A schematic diagram illustrating the implementation flow of the information processing method under the default synchronization frame rate of the main and secondary cameras provided in this application embodiment;

[0029] Figure 13 This is an example diagram of the request dispatching method for dual-camera synchronization (wide-angle lens as secondary camera) in asynchronous frame rate scenarios provided in this application embodiment;

[0030] Figure 14 This is a schematic diagram of the request dispatch strategy based on the number of Aux Request caches and the latency level under the default asynchronous frame rate of the primary and secondary components provided in this application embodiment;

[0031] Figure 15 This is a schematic diagram illustrating the implementation flow of the information processing method under the default synchronization frame rate between primary and secondary components provided in this application embodiment;

[0032] Figure 16 Schematic diagram of the structure of the information processing device provided in the embodiments of this application Figure 1 ;

[0033] Figure 17 Schematic diagram of the structure of the information processing device provided in the embodiments of this application Figure 2 ;

[0034] Figure 18 This is a schematic diagram of the structure of the system-on-a-chip provided in the embodiments of this application;

[0035] Figure 19 This is a schematic diagram of the structure of the motherboard provided in an embodiment of this application;

[0036] Figure 20 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0039] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0040] The descriptions such as "first," "second," and "third" appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They do not indicate any order and do not represent a special limitation on the number of devices in the embodiments of this application. They cannot constitute any limitation on the embodiments of this application.

[0041] First, the relevant technologies involved in this application will be introduced:

[0042] 3A data, also known as 3A information or 3A statistics, is a collective term for Auto White Balance, Auto Exposure, and Auto Focus. 3A data is not image data itself, but rather data (white balance data, exposure data, and focus data) obtained after processing image data using 3A algorithms / 3A operations (auto white balance algorithm, auto exposure algorithm, and auto focus algorithm). 3A data is used to configure the hardware modules of the image sensor and / or image signal processor (ISP). The 3A data processed by the 3A algorithms is returned to update the relevant operating parameters of the image sensor and / or ISP hardware modules.

[0043] An ISP pipeline is a pipeline for image processing within an image signal processor, consisting of multiple image processing functional elements. An ISP can support the parallel operation of one or more ISP pipelines. An ISP pipeline may include, but is not limited to, at least one of the following functional units: Sensor Front End (SFE), Image Signal Processor Front End (IFE), and Image Signal Processor Post End (IPE).

[0044] ISP pipeline frame drops: Normally, image sensors communicating with the ISP pipeline output image data to the ISP pipeline at a frame rate. However, if the ISP pipeline fails to receive or process this image data, or processes the data but does not return the processing result to the HAL layer, all three scenarios constitute frame drops to the ISP pipeline. The HAL layer can determine the corresponding ISP pipeline frame drops based on received frame drop events (such as interrupt signals).

[0045] Request: This refers to either the first request or the second request mentioned below. It consists of operational parameters issued from the application layer to the hardware layer (image sensor and ISP pipeline), also known as frame parameters. The image sensor and ISP pipeline can only function properly and output image data based on this request. For example, the operational parameters carried by this request include, but are not limited to, at least one of the following:

[0046] Configuration parameters of registers on one or more hardware nodes of the ISP;

[0047] Real-time triggered control parameters (such as focus commands / focus operations received from the screen, operations / commands to adjust brightness received from the screen, operations / commands to change other image attributes received from the screen, etc.);

[0048] The buffer size required for image sensor and ISP pipeline output image data.

[0049] After years of development, single-lens camera technology has reached a point where significant improvements in user experience are unlikely. Therefore, multi-lens configurations (such as ultra-wide-angle + wide-angle + telephoto) have become the mainstream in camera development. A hard switch between single lenses can result in noticeable jumps due to parallax and image sensor configuration. Achieving smooth switching between multiple lenses is a key consideration, and better leveraging the capabilities of dual-lens systems is a crucial area for future imaging solutions.

[0050] Currently, dual-camera setups are primarily used for portrait blurring and Spatial Alignment Transform (SAT). In common solutions, the secondary camera is always active in blurring scenarios. In SAT scenarios, dual-camera operation is triggered by either scrolling down the camera or a point-to-point cut, initiating spatial alignment transformation of the two images. To enable seamless switching between preview images on multi-camera devices, smartphones typically use cameras with multiple lenses. Flagship cameras generally have wide-angle, ultra-wide-angle, and telephoto lenses, each with different parameters, primarily differing in field of view (FOV). In preview mode, the user's zoom level needs to be adjusted between different lenses, ensuring a smooth transition between images. While this solves the smooth transition issue, differences in brightness and color still exist between lenses during normal previews and switching, diminishing the user's satisfaction with higher image quality.

[0051] (1) The relevant scheme 1 is described as follows:

[0052] To address the aforementioned issues, the proposed solution enables two image data streams simultaneously during normal, stable previewing. Generally, this is done by distinguishing between a master camera and a slave camera. The master camera outputs its image data stream for display, while the slave camera primarily provides 3A information for the master to perform 3A synchronization. Data synchronization, such as 3A (AE / AF / AWB) information synchronization, is required between the two to ensure consistent brightness and white balance. Typically, the master and slave cameras operate at the same frame rate (e.g., 30fps) using a frame-by-frame alignment method.

[0053] like Figure 1 As shown, in the commonly used dual-camera synchronization (wide-angle lens as secondary camera), the W (wide-angle) lens is always on. During stable previewing in the 1X-3X zoom range, only the W (wide-angle) lens is active as the primary camera. In the 0.6X-1X, 3X, and higher zoom ranges, the W (wide-angle) lens acts as a slave, providing 3A information for the master to perform 3A synchronization; the other lens is active as the master, outputting images for previewing or taking photos. Therefore, the 3A effect in each zoom range is aligned with the W (wide-angle) lens, maintaining consistency in the 3A information of the three cameras (W, UW, and T).

[0054] Regarding solutions for dual-camera synchronization, Figure 2 Here is a schematic diagram of the Pipeline design for related scheme 1, such as... Figure 2 As shown, for Pipeline-1 (i.e., an example of the first ISP pipeline below), the Master still uses the normal Pipeline (data); the Slave's Pipeline-2 (i.e., an example of the second ISP pipeline below) is basically the same as the Master (the image data generated in this path is not used for direct display). Pipeline-2 of the Slave provides 3A information for the Master to perform 3A synchronization; another lens is activated as the Master, and the image output from Pipeline-1 is used for preview or taking photos.

[0055] In related technologies, the main and secondary cameras typically have the same frame rate (e.g., 30fps) and use frame-by-frame alignment, so there are currently no synchronization issues (even if there are occasional frame drops, they are quickly skipped). Since both the Master and Slave lenses operate at 30fps, this poses a significant challenge to power consumption, easily leading to issues like overheating. Furthermore, when the main and secondary cameras have different frame rates, such as changes in brightness causing asynchronous frame drops, synchronization mismatch occurs, introducing problems like fluctuations in 3A synchronization, resulting in abnormal brightness and white balance fluctuations, affecting image quality.

[0056] (2) The relevant scheme 2 is described as follows:

[0057] To address the issues of excessive power consumption leading to overheating and inconsistent image quality due to mismatch between main and secondary cameras at different frame rates in Solution 1, Solution 2 is further developed (e.g., ...). Figure 3 (As shown). It mainly implements the following functions:

[0058] 1. In this technical solution, such as Figure 3As shown, the Slave's Pipeline is trimmed (simplified to Aux Pipeline) through a new software architecture, retaining only the key software and hardware nodes used to generate 3A statistics. The image size can also be reduced, thereby reducing the power consumption of the slave path.

[0059] 2. The new technical solution uses a simplified version of the Aux Pipeline (i.e., an example of the second ISP pipeline below). To further reduce power consumption, it supports lower sensor frame rates, meaning that the frame rates of the Master and Slave can be different. For example, the Master's frame rate is 30fps, and the Slave's frame rate can be 15fps (customizable). Since the Slave's frame rate can be lowered, it means further power consumption optimization. In this application, the Master's frame rate refers to the frame rate of the Master's sensor, and the Slave's frame rate refers to the frame rate of the Slave's sensor.

[0060] To support the aforementioned more flexible functionality, the use cases need to be adjusted. Considering that during preview, the camera system is controlled by the upper-layer FWK (Framework) sending `process_capture_request` (hereinafter referred to as `Request`), and requires a `process_capture_result` (hereinafter referred to as `Result`) carrying cached image data (image buffer), and since the image data generated after cropping by the Aux Pipeline does not need to be processed and stored, this Aux pipeline does not need to return `process_capture_result`. Its control still relies on the `process_capture_request` to configure frame parameters. Therefore, in its design, the dispatching of `Request` in the Aux pipeline depends on the master. Each time the master sends a `process_capture_request`, it will trigger an Aux request according to the corresponding default frame rate ratio.

[0061] Based on the aforementioned solution 1, solution 2 was developed. While solution 2 appears to offer better power efficiency from a design perspective, it may introduce the following stability or performance issues:

[0062] 1. Aux request dispatch is controlled by the Master request distribution. The Master triggers Aux request distribution based on the distributed requests. The Master can distribute requests according to a configured ratio (e.g., 1:1 or 2:1). If the Master fails to dispatch a request due to frame loss or an anomaly, it cannot trigger Aux request dispatch, leading to further consumption and reduction of cached Aux requests. If the Master experiences multiple frame loss or other abnormal requests, the cached Aux requests will be completely exhausted to zero, resulting in delays or frame loss in the processing of 3A Stats statistics in the Aux Pipeline due to insufficient requests.

[0063] 2. When there are changes in brightness, the exposure time changes, and the frame rate of the image sensor will decrease. Because the exposure time of the Master and Slave is not synchronized, the frame rate ratio changes. This will also cause the Aux Request to lose frames due to insufficient dispatch, thus affecting the 3A alignment of the main and secondary cameras and deteriorating the synchronization effect of the two channels.

[0064] To address the shortcomings of the aforementioned solution 2, this application provides an information processing method. Figure 4 Schematic diagram of the implementation flow of the information processing method provided in the embodiments of this application Figure 1 ,like Figure 4 As shown, the method may include the following step 401:

[0065] Step 401: Based on determining that the first ISP pipeline is dropping frames, the frame rate of the first image sensor is decreasing or the exposure time is increasing, the second ISP pipeline is dropping frames, or the number of first requests in the first buffer queue is less than a first value, control the first buffer queue to cache at least one of the first requests for the operation of the second image sensor and the second ISP pipeline; wherein, the first ISP pipeline is used to process the first image data output by the first image sensor; the second ISP pipeline is used to process the second image data output by the second image sensor; the first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

[0066] For example, the first request carries at least one of the following working parameters:

[0067] Configuration parameters of registers on one or more hardware nodes of the ISP;

[0068] Real-time triggered control parameters (such as focus commands / focus operations received from the screen, operations / commands to adjust brightness received from the screen, operations / commands to change other image attributes received from the screen, etc.);

[0069] The buffer size required for the output image data from the second image sensor and the second ISP pipeline.

[0070] In this embodiment, when an abnormal situation is determined to occur, namely, when the first ISP pipeline drops frames, the frame rate of the first image sensor decreases or the exposure time increases, the second ISP pipeline drops frames, or the number of first requests in the first buffer queue is less than a first value, the system controls the first buffer queue to cache at least one of the first requests, so that the second ISP pipeline can obtain the first request from the first buffer queue and can operate normally to process the image data output by the second image sensor, thereby avoiding processing delays or frame drops caused by the second ISP pipeline being unable to obtain the first request.

[0071] Optionally, in some embodiments, the method further includes: if a trigger signal is received indicating the dispatch of a first request, then reading the first request from a first buffer queue to configure operating parameters for the second sensor and / or the second ISP pipeline.

[0072] It is understood that the so-called control of the first cache queue to have at least one first request cached for the operation of the second image sensor and the second ISP pipeline means that whenever the second image sensor and the second ISP pipeline read / read the required first request from the first cache queue at any time, there is at least one first request in the first cache queue for the second image sensor and the second ISP pipeline to read and use (i.e., the first cache queue is not empty), thereby ensuring the operation (i.e., normal operation) of the second image sensor and the second ISP pipeline.

[0073] Optionally, the implementation of controlling the first cache queue to cache at least one of the first requests can be different for different situations described in step 401. For the case of frame loss in the first ISP pipeline, embodiment 1 can be used; for the case of a decrease in the frame rate or an increase in the exposure time of the first image sensor, any one of embodiments 2-4 can be used; for the case of frame loss in the second ISP pipeline, embodiment 5 can be used; for the case where the number of first requests in the first cache queue is less than a first value, embodiment 6 can be used. Controlling the first cache queue to cache at least one of the first requests can include at least one of embodiments 1 to 6.

[0074] Optionally, in Embodiment 1, controlling the first cache queue to cache at least one of the first requests based on determining that the first ISP pipeline has dropped frames includes: dispatching the first request to the first cache queue based on determining that the first ISP pipeline has dropped frames, so as to control the first cache queue to cache at least one of the first requests.

[0075] Furthermore, in some embodiments, the step of dispatching the first request to the first cache queue based on determining that the first ISP pipeline has dropped frames includes: a first thread triggering a second thread to dispatch the first request to the first cache queue based on determining that the first ISP pipeline has dropped frames, so as to control that at least one first request is cached in the first cache queue.

[0076] As mentioned in Scheme 2 above, the dispatch of the slave's first request is controlled by the dispatch of the master's second request; that is, the dispatch of the first request is triggered only when the second request is dispatched. The problem with this is that if the first ISP pipeline fails to dispatch the second request due to frame drops, the dispatch of the first request will not be triggered. This leads to a further depletion of the first request in the first buffer queue due to consumption. Multiple frame drops in the first ISP pipeline will completely exhaust the first request in the first buffer queue, resulting in delays or even frame drops in the second ISP pipeline's 3A operations due to the lack of first requests.

[0077] Based on the above analysis, in this embodiment of the application, based on the determination that the first ISP pipeline has dropped frames, the first request is still dispatched to the first cache queue, thereby ensuring that there are sufficient first requests in the first cache queue for the second ISP pipeline and the second image sensor to use, and thus ensuring the normal operation of the second ISP pipeline and the second image sensor.

[0078] It can be understood that the first thread is a software program that works with the first image sensor and the first ISP pipeline to complete the first image task (such as image preview or shooting), and the second thread is a software program that works with the second image sensor and the second ISP pipeline to complete the second image task (such as statistical / computation of 3A data). The first thread and the second thread can run on the first processor (such as the CPU).

[0079] The first ISP pipeline and the second ISP pipeline can be understood as two different image processing channels. The image processing functional elements included in the first ISP pipeline and the second ISP pipeline can be the same or different. The first ISP pipeline may include SFE, IFE, IPE and / or BPS (Bayer Processing Section). The second ISP pipeline may include SFE, IFE and / or IPE. For example, the first ISP pipeline may include SFE, IFE and IPE, and the second ISP pipeline may include SFE and IPE.

[0080] Optionally, in some embodiments, the first image sensor is the main camera, and the second image sensor is the secondary camera. It can be understood that the first image sensor works with the first ISP pipeline to process one channel of image data, and the second image sensor works with the second ISP pipeline to process another channel of image data. For example, the image data output by the first ISP pipeline is used for display, and the second ISP pipeline is used to perform 3A operations based on the second image data to obtain 3A data. This 3A data is used in the 3A operations of the first ISP pipeline, thereby synchronizing the 3A data (such as AE, AF, and / or AWB data) of the first ISP pipeline and the first image sensor with the second ISP pipeline and the second image sensor, ensuring that the brightness and white balance of these two channels of image data remain consistent.

[0081] In Embodiment 2, controlling the first cache queue to cache at least one of the first requests based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased includes: increasing the exposure time of the second image sensor based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, so as to control the first cache queue to cache at least one of the first requests.

[0082] Optionally, in some embodiments, increasing the exposure time of the second image sensor based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased includes: a first thread increasing the exposure time of the second image sensor based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased.

[0083] As mentioned in Scheme 2 above, the camera system is controlled by Requests issued by the upper-layer FWK, and a new Request will only be issued if a corresponding Result carrying an image buffer is received. In other words, issuing a new second request depends on the Result returned by the first image sensor and the first ISP pipeline. This Result carries image data output by one or more nodes of the first image sensor and the first ISP pipeline based on the read Request. The dispatch of the first request is controlled by the dispatch of the second request; that is, dispatching the second request will trigger the dispatch of the first request. Therefore, the problem is that once the frame rate of the first image sensor decreases (e.g., the exposure time increases), the first ISP pipeline will be slower to return a response (Result) based on the second request read. Consequently, the speed of issuing new second requests will also be slower. If the frame rate of the second image sensor remains unchanged, the first requests in the first buffer queue will be exhausted because it is not possible to trigger the issuance of new first requests as soon as possible. As a result, after the second image sensor outputs image data, the second ISP pipeline will not have any first requests available, which will lead to the second ISP pipeline being unable to process the image data output by the second image sensor. Ultimately, this will cause the second ISP pipeline to experience processing delays or even frame drops.

[0084] Based on the above analysis, in Embodiment 2, based on determining that the frame rate of the first image sensor decreases or the exposure time increases, the exposure time of the second image sensor is increased; thus, the speed at which the second image sensor outputs image data decreases, that is, the frame rate decreases, thereby slowing down the consumption speed of the first request in the first buffer queue, thereby improving the situation of second ISP pipeline processing delay or even frame loss.

[0085] In this embodiment, there is no limit to the maximum exposure time of the second image sensor. Optionally, in some embodiments, increasing the exposure time of the second image sensor includes increasing the exposure time of the second image sensor to a first duration, where the first duration is equal to the exposure time of the first image sensor. This ensures that the frame rate of the second image sensor is synchronized with that of the first image sensor, thereby improving the situation of delay or even frame loss in the second ISP pipeline processing.

[0086] In Embodiment 3, based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, controlling the first buffer queue to cache at least one of the first requests includes: based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, adjusting (e.g., decreasing) the frame rate of the second image sensor according to the frame rate of the first image sensor, so as to control the first buffer queue to cache at least one of the first requests. Compared to Embodiment 2, it is necessary to turn off the second image sensor, then reconfigure the frame rate of the second image sensor, and then turn the second image sensor back on after reconfiguring the frame rate of the second image sensor.

[0087] Optionally, in some embodiments, adjusting the frame rate of the second image sensor according to the frame rate of the first image sensor includes: adjusting the frame rate of the second image sensor to the frame rate of the first image sensor (i.e., the current frame rate).

[0088] In Embodiment 4, based on determining that the frame rate of the first image sensor decreases or the exposure time increases, controlling the first cache queue to cache at least one of the first requests includes: dispatching the first request to the first cache queue according to the frame rate of the second image sensor, so as to control the first cache queue to cache at least one of the first requests.

[0089] Optionally, in some embodiments, dispatching the first request to the first cache queue according to the frame rate of the second image sensor includes: the first thread triggers the second thread to dispatch the first request to the first cache queue based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased; thus, the image processing speed of the second ISP pipeline is avoided from being affected by the slow speed of the first thread issuing new requests.

[0090] Optionally, in some embodiments, dispatching the first request to the first cache queue according to the frame rate of the second image sensor includes: dispatching the first request to the first cache queue at a dispatch rate based on the frame rate of the second image sensor.

[0091] In Embodiment 5, based on determining that the second ISP pipeline has dropped frames, controlling the first cache queue to cache at least one of the first requests includes: based on determining that the second ISP pipeline has dropped frames, not dispatching the first request to the first cache queue, so as to control the first cache queue to cache at least one of the first requests.

[0092] Optionally, in some embodiments, the step of not dispatching the first request to the first cache queue based on determining that the second ISP pipeline has dropped frames includes: the first thread not triggering the second thread to dispatch the first request to the first cache queue based on determining that the second ISP pipeline has dropped frames.

[0093] It is understandable that there may be a situation where the first ISP pipeline does not drop frames and normally dispatches the second request to the second cache queue, but the second ISP pipeline drops frames. In this case, when dispatching the second request to the second cache queue, the dispatch of the first request is not triggered. This can avoid the memory consumption and timeout of flushing the second ISP pipeline caused by redundant dispatch of the first request.

[0094] Here, the second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data. The first image sensor and the first ISP pipeline can operate based on the second request in the second buffer queue.

[0095] It is understood that in some embodiments, if it is indicated that the second image sensor is switched to the main camera, that is, the image data output by the second ISP pipeline is used for display, then the second ISP pipeline needs to be flushed to release the cache resources required by the second ISP pipeline. However, it is necessary to wait for the first request in the first cache queue to be consumed before flushing the second ISP pipeline. Therefore, if the first request is redundantly dispatched, it will not only increase memory overhead, but also increase the time for flushing the second ISP pipeline, thereby increasing the switching latency.

[0096] Optionally, in the specific implementations of Embodiments 1 to 5 above, the initial frame rate of the first image sensor is the same as the initial frame rate of the second image sensor, that is, the initial frame rate of the first image sensor is synchronized with the initial frame rate of the second image sensor.

[0097] In Embodiment 6, controlling the first cache queue to cache at least one first request based on determining that the number of first requests in the first cache queue is less than a first value includes: dispatching the first request to the first cache queue according to a first ratio based on determining that the number of first requests in the first cache queue is less than a first value, so as to control the first cache queue to cache at least one first request.

[0098] Wherein, the first ratio is less than the second ratio, and both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value; the second request is used to configure the working parameters required for the first image sensor to output the first image data and the working parameters required for the first ISP pipeline to process the first image data.

[0099] Optionally, in some embodiments, the step of dispatching the first request to the first cache queue according to a first ratio based on determining that the number of the first request in the first cache queue is less than a first value includes: a first thread triggering a second thread to dispatch the first request to the first cache queue according to a first ratio based on determining that the number of the first request in the first cache queue is less than a first value.

[0100] It's understandable that when dispatching a second request, the dispatch of the first request is triggered according to the default frame rate ratio (i.e., the frame rate ratio between the first and second image sensors). For example, if the frame rate of the first image sensor is 30fps and the frame rate of the second image sensor is 15fps, then the dispatch of the first request is triggered at a 2:1 ratio, meaning two second requests are dispatched before one first request is triggered. Normally, if the frame rates of the first and second image sensors remain constant, and the exposure time or frame rate of the first image sensor remains constant, or if the first ISP pipeline does not drop frames, the second image sensor and the second ISP pipeline can operate smoothly due to sufficient first requests in the first buffer queue. However, once the first ISP pipeline drops frames or the exposure time or frame rate of the first image sensor changes, the dispatch of the first request cannot be triggered according to the agreed-upon ratio. This can lead to the depletion of first requests in the first buffer queue, causing the second ISP pipeline to malfunction due to a lack of first requests.

[0101] Based on the above analysis, in Embodiment 6, if it is determined that the number of first requests in the first cache queue is less than a first value, the dispatch ratio is adjusted to dispatch the first requests at a first ratio that is less than the original ratio (i.e., the second ratio). For example, the second ratio is equal to 2:1, meaning that for every two second requests, one first request is dispatched to the first cache queue. If it is determined that the number of first requests in the first cache queue is less than the first value, the first requests are dispatched according to the first ratio that is less than the second ratio. For example, the first ratio is equal to 1:1, meaning that for every one second request, one first request is dispatched to the first cache queue. This ensures that the number of first requests in the first cache queue is sufficient, thereby ensuring the smooth operation of the second ISP pipeline.

[0102] It is understandable that even if the number of first requests in the first cache queue is improved using Embodiment 6, the number of first requests in the first cache queue may still be insufficient to ensure the smooth operation of the second ISP pipeline. Based on this, in some embodiments, after controlling that at least one first request is cached in the first cache queue, the method further includes: increasing the capacity of the first cache queue based on determining that the number of first requests in the first cache queue is less than a second value, or that the return duration of the 3A data is greater than the frame interval of the second image sensor; wherein, the second value is less than the first value, and the return duration of the 3A data refers to the time length from the start of the second ISP pipeline performing 3A operation on the second image data based on the first request to obtaining the 3A data of the data structure required by the first ISP pipeline to perform 3A operation.

[0103] It is understandable that the second ISP pipeline performs 3A calculations based on the image data output by the second image sensor to obtain 3A data. The 3A calculations of the second ISP pipeline depend on the configuration data Request (i.e., the first request). After the second ISP pipeline generates 3A data, it will be transmitted upward to the Dependancy node in the Kernel layer. The Dependancy parses it into 3A data in the format required by the first ISP pipeline for 3A calculations. However, the operation of the Dependancy node is triggered by the first request. Therefore, the return time of the above 3A data indirectly reflects the speed of sending the first request.

[0104] In this embodiment, the conditions for determining whether the number of first requests in the first cache queue is less than the second value can be varied. For example, it can be determined that the number of first requests in the first cache queue is less than the second value once it is detected that the number of first requests in the first cache queue is less than the second value, and the capacity of the first cache queue can be increased accordingly. Alternatively, it can be determined that the number of first requests in the first cache queue is less than the second value only when the number of times the number of detections of the number of first requests in the first cache queue being less than the second value meets the condition, and the capacity of the first cache queue can be increased accordingly.

[0105] In some embodiments, the condition that the number of times the number of first requests in the first cache queue is less than the second value is satisfied includes: the number of times the number of times the number of first requests in the first cache queue is less than the second value is detected within a third time period is greater than or equal to a number threshold, or the proportion of such number is greater than or equal to a proportion threshold; or, the number of first requests in the first cache queue is detected to be less than the second value N times consecutively; wherein N is a preset value greater than 1.

[0106] Optionally, in the specific implementation of Embodiment 6 above and these embodiments, the initial frame rate of the first image sensor is different from the initial frame rate of the second image sensor, that is, the initial frame rate of the first image sensor is asynchronous with the initial frame rate of the second image sensor.

[0107] This application provides an information processing method. Figure 5 Schematic diagram of the implementation flow of the information processing method provided in the embodiments of this application Figure 2 ,like Figure 5 As shown, the method includes at least one of the following embodiments (1)-(4):

[0108] (1) Based on the determination of frame loss in the first ISP pipeline, dispatch the first request to the first buffer queue;

[0109] (2) Based on the determination that the second ISP pipeline has dropped frames, the first request is not dispatched to the first cache queue;

[0110] (3) Based on the determination that the number of first requests in the first cache queue is less than a first value, the first requests are dispatched to the first cache queue according to a first ratio;

[0111] (4) Based on the determination that the frame rate of the first image sensor has decreased or the exposure time has increased, perform one of the following actions:

[0112] (4)-a: Increase the exposure time of the second image sensor;

[0113] (4)-b: Adjust the frame rate of the second image sensor according to the frame rate of the first image sensor;

[0114] (4)-c: Dispatch the first request to the first cache queue according to the frame rate of the second image sensor; wherein,

[0115] The first ISP pipeline is used to process the first image data output by the first image sensor;

[0116] The second ISP pipeline is used to process the second image data output by the second image sensor;

[0117] The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data;

[0118] The second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data;

[0119] The first ratio is greater than the second ratio. Both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched. The second ratio refers to the dispatch ratio used before it is determined that the number of first requests in the first cache queue is less than the first value.

[0120] Optionally, in some embodiments, for the above embodiments (1), (2) and (4), the initial frame rate of the first image sensor is the same as the initial frame rate of the second image sensor.

[0121] Optionally, in some embodiments, for the above implementation (3), the initial frame rate of the first image sensor is different from the initial frame rate of the second image sensor.

[0122] Optionally, in some embodiments, increasing the exposure time of the second image sensor includes: increasing the exposure time of the second image sensor to a first duration, where the first duration is the exposure time of the first image sensor.

[0123] Optionally, in some embodiments, after dispatching the first request to the first cache queue according to a first ratio, the method further includes: increasing the capacity of the first cache queue based on determining that the number of first requests in the first cache queue is less than a second value, or the return duration of the 3A data is greater than the frame interval of the second image sensor; wherein the second value is less than the first value, and the return duration of the 3A data refers to the time length from the start of the second ISP pipeline performing 3A operation on the second image data based on the first request to obtaining the 3A data of the data structure required by the first ISP pipeline to perform 3A operation.

[0124] Furthermore, in some embodiments, the number of first requests in the first cache queue being less than the second value means that the number of first requests in the first cache queue is less than the second value within a second time period, or that the number of times the number of first requests in the first cache queue being less than the second value is detected meets the condition.

[0125] It should be noted that, regarding the above Figure 5 The information processing method shown, along with related specific implementation methods and additional embodiments, is similar to the description of the method embodiments above. For technical details not disclosed in these embodiments, they can be understood by referring to the description of the method embodiments above.

[0126] Optionally, in all the embodiments described above, the first image sensor is the main camera, and the second image sensor is the secondary camera. Correspondingly, the image data output by the first ISP pipeline is used for display, and the second ISP pipeline is used to perform 3A operations based on the second image data to obtain 3A data. The 3A data is used for the 3A operations of the first ISP pipeline. The operation of the first ISP pipeline depends on the dispatched second request, and the operation of the second ISP pipeline depends on the dispatched first request. Therefore, the second ISP pipeline can only perform 3A operations based on the first request.

[0127] In some embodiments, the method further includes: responding to a switching instruction to flush the second ISP pipeline; wherein flushing the second ISP pipeline includes releasing the cache resources required by the second ISP pipeline without waiting for the first request in the first cache queue to be used up, that is, after receiving the switching instruction, releasing the cache resources occupied by the second ISP pipeline without waiting for the first request in the first cache queue to be read and consumed; wherein the switching instruction is used to instruct the second image sensor to be switched to the main camera.

[0128] It is understandable that in the scenario described above where the first image sensor is the main camera and the second image sensor is the secondary camera, fallback or SAT (i.e., cut-off or cut-off) may be triggered at any time. Fallback or SAT is an example of the switching instruction. Based on this, it is necessary to flush the second ISP pipeline. This flushing does not need to wait for the first request in the first cache queue to be consumed, but directly releases the cache resources required by the first ISP pipeline, thereby shortening the time delay of flushing the second ISP pipeline and also shortening the switching response delay.

[0129] Optionally, in some embodiments, the method further includes: a second thread reading the first request from a first buffer queue in response to a received trigger signal indicating that the first request should be dispatched, for configuring operating parameters for the second sensor and / or the second ISP pipeline.

[0130] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0131] To address the shortcomings of the aforementioned solution 2, this application proposes a low-latency Aux Pipeline processing scheme (i.e., information processing method) that dynamically adjusts the number of Requests dispatched based on path status and performance. The main idea is as follows: (1) In the scenario of synchronous frame rate between the main and secondary cameras, the real-time frame loss events of the main and secondary cameras are incorporated into the control mechanism of Master triggering (Trigger) dispatching Aux Requests to avoid potential problems such as missed dispatching of Aux Requests or redundant dispatching caused by Master frame loss. (2) In the case of asynchronous frame rate between the main and secondary cameras, the number of Requests flowing through the Aux Pipeline and the degree of Aux latency are monitored in real time to temporarily increase the proportion of Master triggering Aux Request dispatching or dynamically adjust the maximum number of Request buffers, thereby reducing the latency of the Aux path.

[0132] Before describing this technical solution, let's first give an overview of the dual-camera synchronization solution based on a cropped Pipeline.

[0133] As mentioned above Figure 3 The aforementioned related scheme 2 is the basic scheme of this technical solution. Figure 6 This is an example diagram of the working range of a dual-camera system (wide-angle lens as secondary camera) in a default synchronized frame rate scenario, as shown below. Figure 6 As shown, the W (wide-angle) lens is always on.

[0134] like Figure 6 As shown, in the 1X to 3X zoom range, during stable preview, only the W (wide-angle) lens is activated as the main camera (30fps, image size depends on scene requirements). In the 0.6X to 1X, 3X and above zoom range, both cameras are activated simultaneously for stable preview. One lens (UW, T, or UT) is activated as the master (default 30fps, image size depends on scene requirements, example in the diagram is 12M), outputting images for preview or photography; the other lens, the W (wide-angle), acts as a slave (default 30fps, can be adjusted according to scene requirements and capabilities, example in the diagram is 15fps), providing 3A information and processing results for the master to perform 3A synchronization. Compared to the aforementioned solution 1, in the aforementioned solution 2, the slave camera uses a cropped version of the pipeline (i.e., an example of a second ISP pipeline), reducing image size and thus lowering power consumption to some extent.

[0135] Figure 7 This is an example diagram of the working range of a dual-camera system (wide-angle lens as secondary camera) in a default asynchronous frame rate scenario, as shown below. Figure 7As shown, the W (wide-angle) lens is always on. During stable preview at 1X to 3X zoom levels, only the W (wide-angle) lens is active as the main camera (30fps, image size depends on scene requirements; the example in the image is 12MB). Figure 6 Compared to the previous usage scenario, when lens W (wide-angle) acts as a slave (with a lower frame rate, which can be determined based on scene requirements and capabilities; the example in the diagram is 15fps), it provides 3A information for the Master to perform 3A synchronization. Compared to related solution 1, the slave camera has a lower frame rate, requiring the use of low-frame-rate slave frames for 3A alignment with high-frame-rate Master frames, increasing the complexity of the solution. However, it can significantly reduce the power consumption of the slave camera.

[0136] The following analysis focuses on a pipeline-related request dispatch mechanism:

[0137] Figure 8 This is a use case-related pipeline node and control block diagram, such as Figure 8 As shown, in the three-camera consistency use case, the Master and Slave each run the same pipeline and are jointly controlled by the same Session. With the help of the Session management mechanism, frame synchronization and request synchronization between the Master and Slave can be achieved.

[0138] When the camera application layer and FWK send a process_capture_request, it is sent to a session shared by both the Master and Slave. The Master and Slave then respectively send requests and receive results. The session controls the dispatch of requests based on the results returned by the Master and Slave. Therefore, under normal circumstances, there is no shortage of requests in the Slave's buffer queue, unless the upper-layer process_capture_request is blocked. In this case, both the Master and Slave will experience pipeline processing delays or image data frame drops due to not receiving requests, but synchronization can be maintained, and the 3A data from the primary and secondary cameras can still be aligned.

[0139] The following analysis examines another pipeline-related request dispatch mechanism:

[0140] Figure 9 For another use case related pipeline nodes and control block diagram, such as Figure 9As shown, due to the support for (Slave) Aux Pipeline pruning and the ability to run at a frame rate asynchronous with the master, it is currently difficult to manage two pipelines using the original single session. Therefore, in the triple-camera consistency use case, the master and slave each run different pipelines and are controlled by different sessions. Sessions are mutually exclusive and can interact through a global metadata (i.e., a global property), but cannot be like... Figure 8 The scheme shown implements master-slave frame synchronization and independent control of each request.

[0141] During preview, the camera system is controlled by the upper-layer FWK's `process_capture_request`, and a `process_capture_result` carrying the buffered image data is required. Since the image data generated by the Aux Sensor (the second image sensor) does not need to be stored and returned after AuxPipeline cropping, this Aux pipeline (the second ISP pipeline) does not need to return `process_capture_result`, therefore the auxsession cannot receive `process_capture_result` from the FWK; however, the Aux pipeline still requires a `Request` to configure frame parameters.

[0142] Therefore, in Figure 9 In the corresponding design, when the Master dispatches a Request, it triggers Aux to send the Request according to the default frame rate ratio. Under normal circumstances, the frame rate ratio of the main and secondary cameras remains unchanged, and the Master does not drop frames, so the above process can run normally. However, once the frame rate of the main and secondary cameras changes, or the Master drops frames, it cannot trigger Aux to send the Request as agreed, thus consuming the LivePendingRequests cached in the Aux Pipeline. When it is exhausted, it will cause the Aux Pipeline to experience delays or frame drops when processing subsequent sensor image data. This not only fails to improve the consistency of the three cameras, but also worsens the consistency effect of the three cameras, resulting in jitter in brightness and color.

[0143] Therefore, the solution needs to be optimized to address its shortcomings. Specific optimization strategies are described below, and different technical solutions are available for different situations.

[0144] The first optimization strategy is a request dispatch strategy based on frame drop and frame rate fluctuation monitoring, under the scenario of default synchronized frame rate between the main and secondary cameras:

[0145] (1) In scenarios where the main and secondary cameras have a default synchronized frame rate, the general request dispatch method is as follows: Figure 10 As shown:

[0146] 1. In the HAL-Layer 1 layer, the Master receives a Request from the Framework side and, according to the agreed ratio of 1:1, triggers the HAL-Layer 1 layer that enables Aux to dispatch a Request (i.e., the first request, which reuses some Master frame parameters, Aux internal buffers, etc.) to the corresponding Request Queue (i.e. the first buffer queue) of the Aux on the HAL-Layer 2 side.

[0147] 2. In the HAL-Layer 2 layer, Aux buffers a maximum of 6 frames of requests as LivePendingRequests for subsequent processing in the Aux Pipeline (an example of a second ISP pipeline). The requests, along with their configuration parameters and resources, are passed to the Kernel layer, which then configures the hardware (Hardware, HW) side image sensor and ISP hardware modules for the normal operation of the hardware and software pipeline.

[0148] 3. As the Master normally sends Requests and returns Results to the FWK, Aux is triggered as the Master Request is dispatched, and so on in a virtuous cycle.

[0149] (2) Request dispatch strategy based on frame drop and frame rate fluctuation monitoring under the default synchronization frame rate of master and slave

[0150] A. In the scenario described above where the main and secondary cameras have a default synchronized frame rate, the potential problems with the Request dispatch method under normal circumstances are as follows (and see also...). Figure 11 ):

[0151] 1. Due to frame drops in the Master, a Result (e.g., corresponding to the M0 Request) is not returned to the Framework. This Request (e.g., M0) and its resources will continue to be used, and the Framework will not issue a new Request. Therefore, HAL-Layer1 cannot trigger Aux to issue a new Request in HAL-Layer1.

[0152] 2. Additionally, the Master's frame rate reduction (sense exposure lengthening) may cause the Result (for example, the M0 Request) to be returned to the Framework more slowly, and the Framework to issue new Requests will also be slower. If the AuxSensor's frame rate is still 30fps, the Master cannot trigger the issuance of Aux Requests in time, and AuxLivingPendingRequests will be exhausted. This will result in insufficient Requests to process the frames (i.e., output image data) from Sensor2 (an example of a second image sensor), causing Aux Pipeline processing delays, and in severe cases, frame drops in the Aux Pipeline.

[0153] B. Response strategies, see [link / reference] Figure 11 :

[0154] 1. A mechanism to detect frame drops in the Master / Slave architecture needs to be added here. If the Master drops frames and fails to send requests, the Aux mechanism should still be used to dispatch requests, thus maintaining the number of Aux requests in rotation. If the Slave drops frames, the Master should not trigger Aux request dispatch when sending requests, to avoid memory consumption and Aux Flush timeouts caused by AuxRequest resource redundancy.

[0155] 2. In low-light environments, longer exposure times cause a drop in the Master frame rate, resulting in frame rate fluctuations between the two. The number of Rrequests sent will also decrease. In this case, the Aux exposure time also needs to be adjusted synchronously to ensure that the two are at the same speed.

[0156] (3) Description of the Request dispatch strategy based on frame drop and frame rate fluctuation monitoring under the default synchronization frame rate of the main and secondary cameras:

[0157] Figure 12 This is a schematic diagram illustrating the implementation flow of the information processing method under the default synchronization frame rate of the main and secondary cameras provided in the embodiments of this application, as follows: Figure 12 As shown, it includes the following steps 1201 to 1210.

[0158] Step 1201: Turn on the camera.

[0159] Step 1202: During initialization, obtain Aux SkipRate = 1 (if it is 1, it means that the master and slave default synchronization frame rate is used, and the following process is followed); where SkipRate represents the skip frame rate;

[0160] Step 1203: Enter the stable preview scene.

[0161] Step 1204: The Master dispatches a Request, triggering Aux to start. This can also be understood as the Master's HAL-Lay1 dispatching a Request to the second cache queue and triggering the start of the second ISP pipeline.

[0162] Step 1205: The Master's HAL-Lay1 detects whether the Master has lost frames. If no frames are lost, the Master's HAL-Lay1 sends a Request and triggers the Aux's HAL-Lay1 to send a Request on a 1:1 basis. If frame loss is detected in the Master, the Master's HAL-Lay1 does not send a Request, but triggers the Aux's HAL-Lay1 to send a Request.

[0163] The method also includes: if frame loss is detected on the Slave, then when the Master sends a Request via HAL-Lay1, it will not trigger the Aux to send a Request via HAL-Lay1.

[0164] Step 1206: In its own 3A processing flow, the Master obtains the Slave's 3A processing result through the Global property and performs its own 3A calculation.

[0165] Step 1207: In the Master 3A processing flow, the algorithm detects a drop in the Master frame rate or an increase in exposure time, and compares it with the Aux frame rate. If the frame rate or exposure time is inconsistent, the exposure time of the Aux is adjusted synchronously to ensure that both are at the same speed; if the frame rates are consistent, then proceed directly to step 1208.

[0166] Step 1208: The Master outputs image data based on the updated 3A parameters.

[0167] Step 1209: If fallback or SAT (slice cut or spot cut) (sensor main / secondary camera switching) is triggered, the Aux pipeline will be shut down (stream off), and the normal pipeline (i.e., the non-cropped pipeline) will be started to output the normal data stream to meet the smooth processing of sensor switching.

[0168] Step 1210: If step 1209 is not triggered, continue previewing or recording images based on the above strategy.

[0169] The second optimization strategy is a request dispatch strategy based on the number of request caches and the degree of latency, under the scenario of default asynchronous frame rates for the main and secondary cameras:

[0170] (1) Under the default asynchronous frame rate of primary and secondary servers, the general Request dispatch method is described as follows:

[0171] Figure 13 This is an example diagram illustrating the request dispatching method for dual-camera synchronization (wide-angle lens as secondary camera) in asynchronous frame rate scenarios, such as... Figure 13 As shown:

[0172] 1. In the HAL-Layer 1 layer, the Master receives a Request from the Framework side and, according to the agreed ratio of 2:1, triggers the HAL-Layer 1 layer that enables Aux to dispatch the Request (reusing some Master frame parameters, Aux internal buffers, etc.) to the corresponding Request Queue (i.e. the first buffer queue) of the Aux on the HAL-Layer 2 side.

[0173] 2. In the HAL-Layer 2 layer, Aux buffers a maximum of 3 frames of requests as LivePendingRequests for subsequent processing in the Aux Pipeline. The requests, along with their configuration parameters and resources, are passed to the Kernel layer, which then configures the image sensor and ISP hardware modules on the HW side for the normal operation of the hardware and software pipeline.

[0174] 3. As the Master normally sends Requests and returns Results to the FWK, Aux is triggered proportionally as Master Requests are dispatched, thus creating a virtuous cycle.

[0175] (2) Under the default asynchronous frame rate of the main and secondary cameras, the Request dispatch strategy based on frame drop and frame rate fluctuation monitoring is described as follows:

[0176] Figure 13 This is an example diagram of the request dispatch strategy for dual-camera synchronization (wide-angle lens as secondary camera) in asynchronous frame rate scenarios.

[0177] A. Figure 13 The potential problems with the corresponding request dispatch strategy are analyzed as follows:

[0178] 1. Due to frame drops by the Master, requests (e.g., M0) that are not returned to the Framework will continue to be used, and the Framework will not issue new requests. Therefore, Aux requests will not be triggered at the HAL-Layer 1 layer.

[0179] 2. Due to the Master reducing the frame rate (straining the sensor exposure), the return of requests (e.g., M0) to the Framework is slower, and the Framework also slows down in sending new requests. If the Aux Sensor's frame rate is still 15fps, and the Master still triggers Aux Requests at a 2:1 ratio, Aux Requests cannot be triggered in time. AuxLivingPendingRequests will be exhausted, resulting in insufficient requests to process the image data output by Sensor2, leading to dropped frames.

[0180] B. The strategies for addressing the above problems are as follows:

[0181] 1. For example Figure 14 As shown, a mechanism to detect the number of Aux LivePendingRequests needs to be added here. Once the number of AuxLivePendingRequests (i.e., Requests in the first cache queue) is less than 3 (i.e., an example of the first value, AuxMaxRequestNum = 3), Aux will be triggered to dispatch Requests at a 1:1 ratio instead of the original 2:1. Each time a Master Request (i.e., the second request) is received, it is retrieved from the Request Queue (i.e., the first cache queue).

[0182] 2. By monitoring the number of LivePendingRequests in the Aux Pipeline, if the number of LivePendingRequests is still frequently at a relatively low value (i.e., an example of the second value, such as 1 or 2) when implementing B-1, it indicates that the Master is experiencing severe frame dropping. Even if the trigger ratio is increased to 1:1, it may still not meet the needs of Aux Request flow. In this case, you can adjust (e.g., increase) AuxMaxRequestNum to make the AuxPipeline have more LinePendingRequests.

[0183] (3) Description of the Request dispatch strategy process based on frame drop and frame rate fluctuation monitoring under the default asynchronous frame rate of primary and secondary servers:

[0184] Figure 15 This is a schematic diagram illustrating the implementation flow of the information processing method under the default synchronization frame rate between primary and secondary components provided in this application embodiment. Figure 15 As shown, it includes the following steps 1501 to 1510.

[0185] Step 1501: Turn on the camera.

[0186] In step 1502, during initialization, obtain the Aux SkipRate. If it is greater than 1, it indicates the default asynchronous frame rate for the main and auxiliary cameras, and the following process is followed.

[0187] In step 1503, enter the stable preview scenario.

[0188] In step 1504, the Master dispatches a Request to trigger the activation of the Aux. It can also be understood that the HAL-Lay1 of the Master dispatches the Request to the second buffer queue and triggers the activation of the second ISP pipeline;

[0189] In step 1505, if Aux LivePendingRequests < AuxMaxRequestNum (default is 3) and based on the numerical size judgment, the Master issues a Request and triggers the Aux to issue a Request on a 1:1 basis. If it is found that the value of LivePendingRequests is relatively low for a long time, AuxMaxRequestNum can be adjusted (such as increased) to enable the AuxPipeline to have more LinePendingRequests.

[0190] In step 1506, if Aux LivePendingRequests = AuxMaxRequestNum, the Master issues a Request according to the original ratio, such as 2:1, to trigger the Aux to issue a Request.

[0191] In step 1507, during the Master 3A processing flow, perform 3Async processing.

[0192] In step 1508, the Master outputs image data based on the updated 3A parameters.

[0193] In step 1509, if fallback or SAT (slicing or dot cutting, the main and auxiliary camera switching of the sensor) is triggered, at this time, the Aux pipeline will be streamed off, and the normal pipeline will be started to output the normal data stream to meet the smooth processing of the sensor switching.

[0194] In step 1510, if step 1509 is not triggered, continue with preview or image recording based on the above strategy.

[0195] It can be understood that in the embodiments of this application:

[0196] (1) Under the default master-slave synchronized frame rate, a request dispatch strategy based on frame drop and frame rate fluctuation monitoring is implemented. By detecting frame drops in either the Master or Slave, if the Master fails to dispatch requests due to frame drops, Aux request dispatch is still triggered, thereby increasing the number of Aux requests dispatched. If the Slave experiences frame drops, Aux request dispatch is not triggered in the next Master request dispatch process to avoid memory consumption and flush timeouts caused by Aux request resource redundancy. In low-light environments, longer exposure times lead to a decrease in the Master's frame rate. In this case, the Aux exposure time also needs to be adjusted synchronously to ensure both are at the same speed. This ensures the normal operation of the Slave pipeline and reduces latency or frame drops.

[0197] (2) Under the default asynchronous frame rate of the master and slave, it is necessary to add a check for Aux LivePendingRequests here. Once it is less than 3 (AuxMaxRequestNum = 3), Aux Request dispatch will not be triggered according to the original trigger ratio, but will be triggered at a 1:1 ratio. If it is still found that LivePendingRequests are often at a relatively low value (such as 1 or 2), it means that the Master is losing frames seriously. You can adjust (e.g., increase) AuxMaxRequestNum so that the Aux Pipeline has more LinePendingRequests to use, thereby ensuring the normal operation of the Slave pipeline and effectively reducing latency or frame loss.

[0198] The effective guarantee of the number of Requets enables the implementation of a low-power solution and increases the system's stability and execution efficiency.

[0199] In this embodiment of the application, under the synchronous frame rate of the main and secondary cameras, the real-time frame loss events of the main and secondary cameras are incorporated into the control mechanism of the MasterTrigger dispatching Aux Requests to avoid potential problems such as the omission of Aux Request dispatch caused by Master frame loss or the redundant dispatch caused by Aux frame loss.

[0200] In this embodiment of the application, under the master-slave asynchronous frame rate, by real-time monitoring of the number of requests flowing through the Aux Pipeline and the degree of Aux latency, the proportion of Aux requests dispatched by the Master Trigger can be temporarily increased or the maximum number of requests can be dynamically adjusted, thereby reducing the latency of the Aux path.

[0201] In this embodiment of the application, under the synchronous frame rate of the main and secondary cameras, such as a 1:1 ratio of main and secondary frame rates, the frame loss situation of the main and secondary paths is monitored: if the main camera loses a frame, then in the next dispatch process of the Master, the Master Request is not dispatched, but the Aux Request is still dispatched, thus avoiding the problem of Aux Request resource shortage caused by not dispatching in the past; if the secondary camera loses a frame, then in the next dispatch process of the Master, the Aux Request is not dispatched to avoid memory consumption and Flush timeout caused by Aux Request resource redundancy.

[0202] In this embodiment of the application, under the master-slave synchronous frame rate, the proportion and dynamics of the Master Trigger dispatching Aux Requests are temporarily increased by monitoring the number of Requests flowing through the Aux Pipeline in real time; if this still cannot improve the situation, the maximum number of Requests is further adjusted based on the Aux latency, thereby achieving the overall goal of reducing the latency of the Aux path.

[0203] What can be expanded upon is:

[0204] (1) For the above solution under asynchronous frame rates of the main and secondary cameras, increasing the value of AuxMaxRequestNum may increase the number of requests, resulting in excessive time spent flushing the Aux Pipeline during system switching, which may prolong the transition time. Considering that Aux does not store or use image data, AuxPipeline Flush does not need to wait for the return of requests like a normal Pipeline, and can directly StreamOff and release related resources in a timely manner.

[0205] (2) Under the synchronized frame rate of the main and secondary cameras, the proportion and dynamics of Aux Requests dispatched by the Master Trigger are temporarily increased by monitoring the number of Requests flowing through the Aux Pipeline in real time. If this still cannot improve the situation, the maximum number of Requests buffers is further adjusted based on the degree of Aux latency. The degree of Aux latency can be measured not only by simply counting the number of times LivePendingRequests are in low digits, but also by comparing the time required for the Stats to return during Slave 3A processing (which depends on Dependancy) with the Slave frame interval. Historical statistical data and probability distribution are used to determine when to adjust the value of LivePendingRequests.

[0206] It should be noted that, in the embodiments of this application, the Framework, HAL-Lay1, HAL-Lay2, and Kernel on the Master side can be understood as an example of a first thread, and the HAL-Lay1, HAL-Lay2, and Kernel on the Aux (i.e., Slave) side can be understood as an example of a second thread. Sensor1 can be understood as a first image sensor, and Sensor2 can be understood as a second image sensor. The SFE, IFE, and IPE on the Master side can be understood as an example of a first ISP pipeline, and the SFE and IFE on the Aux (i.e., Slave) side can be understood as an example of a second ISP pipeline. The Request dispatched by the Master side can be understood as a second request, and the Request dispatched by the Aux (i.e., Slave) side can be understood as a first request. The cache queue used by the Master side to cache Requests can be understood as a second cache queue, and the Requests cached in the second cache queue can also be called LivingPendingRequests. The cache queue used to cache requests on the Aux (i.e., Slave) side can be understood as the first cache queue, and the requests cached in the first cache queue can also be called LivingPendingRequests.

[0207] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0208] Based on the foregoing embodiments, this application provides an information processing device, which includes the included modules and the units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be an AI acceleration engine (such as NPU), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0209] Figure 16 Schematic diagram of the structure of the information processing device provided in the embodiments of this application Figure 1 ,like Figure 16 As shown, the information processing device 160 includes:

[0210] The control module 1601 is configured to control the first buffer queue to cache at least one of the first requests based on the determination that the first ISP pipeline is dropping frames, the frame rate of the first image sensor is decreasing or the exposure time is increasing, the second ISP pipeline is dropping frames, or the number of first requests in the first buffer queue is less than a first value, so as to enable the operation of the second image sensor and the second ISP pipeline; wherein, the first ISP pipeline is used to process the first image data output by the first image sensor;

[0211] The second ISP pipeline is used to process the second image data output by the second image sensor;

[0212] The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

[0213] In some embodiments, the first image sensor is the main camera and the second image sensor is the secondary camera. Accordingly, the image data output by the first ISP pipeline is used for display, and the second ISP pipeline is used to perform 3A operations based on the second image data to obtain 3A data. The 3A data is used for the 3A operations of the first ISP pipeline.

[0214] In some embodiments, controlling the first cache queue to cache at least one of the first requests based on determining that the first ISP pipeline has dropped frames includes: dispatching the first request to the first cache queue based on determining that the first ISP pipeline has dropped frames, so as to control the first cache queue to cache at least one of the first requests.

[0215] In some embodiments, controlling the first cache queue to cache at least one of the first requests based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased includes: based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, performing one of the following actions to control the first cache queue to cache at least one of the first requests: increasing the exposure time of the second image sensor; adjusting the frame rate of the second image sensor according to the frame rate of the first image sensor; and dispatching the first request to the first cache queue according to the frame rate of the second image sensor.

[0216] In some embodiments, increasing the exposure time of the second image sensor includes: increasing the exposure time of the second image sensor to a first duration, wherein the first duration is equal to the exposure time of the first image sensor.

[0217] In some embodiments, controlling the first cache queue to cache at least one of the first requests based on determining that the second ISP pipeline has dropped frames includes: based on determining that the second ISP pipeline has dropped frames, not dispatching the first request to the first cache queue, so as to control the first cache queue to cache at least one of the first requests.

[0218] In some embodiments, in scenarios where the first ISP pipeline drops frames, the frame rate of the first image sensor decreases or the exposure time increases, and the second ISP pipeline drops frames, the initial frame rate of the first image sensor is the same as the initial frame rate of the second image sensor.

[0219] In some embodiments, controlling the first cache queue to cache at least one first request based on determining that the number of first requests in the first cache queue is less than a first value includes: dispatching the first requests to the first cache queue according to a first ratio based on determining that the number of first requests in the first cache queue is less than a first value, so as to control the first cache queue to cache at least one first request; wherein, the first ratio is less than a second ratio, and both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value; the second request is used to configure the operating parameters required for the first image sensor to output first image data and the operating parameters required for the first ISP pipeline to process the first image data.

[0220] In some embodiments, the control module 1601 is further configured to: after dispatching the first request to the first cache queue according to a first ratio, increase the capacity of the first cache queue based on determining that the number of first requests in the first cache queue is less than a second value, or the return time of the 3A data is greater than the frame interval of the second image sensor; wherein, the second value is less than the first value, and the return time of the 3A data refers to the time length from the start of the second ISP pipeline performing 3A operation on the second image data based on the first request to obtaining the 3A data of the data structure required by the first ISP pipeline to perform 3A operation.

[0221] In some embodiments, the number of first requests in the first cache queue being less than the second value means that the number of first requests in the first cache queue is less than the second value within a second time period, or that the number of times the number of first requests in the first cache queue is detected to be less than the second value meets the condition.

[0222] In some embodiments, in scenarios where the number of first requests in the first cache queue is less than a first value, the initial frame rate of the first image sensor is different from the initial frame rate of the second image sensor.

[0223] In some embodiments, the information processing device 160 further includes a flushing module configured to flush the second ISP pipeline in response to a switching command; wherein flushing the second ISP pipeline includes: releasing the cache resources required by the second ISP pipeline without waiting for the first request in the first cache queue to be used up, and the switching command is used to instruct the second image sensor to be switched to the main camera.

[0224] Figure 17 Schematic diagram of the structure of the information processing device provided in the embodiments of this application Figure 2 ,like Figure 17 As shown, the device 170 includes:

[0225] Control module 1701 is configured to: dispatch a first request to a first buffer queue based on the determination of frame loss in the first ISP pipeline;

[0226] And / or, based on the determination that the second ISP pipeline has dropped frames, the first request is not dispatched to the first cache queue;

[0227] And / or, based on the determination that the number of first requests in the first cache queue is less than a first value, the first request is dispatched to the first cache queue according to a first ratio;

[0228] And / or, based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, perform one of the following actions:

[0229] Increase the exposure time of the second image sensor;

[0230] Adjust the frame rate of the second image sensor according to the frame rate of the first image sensor;

[0231] The first request is dispatched to the first cache queue according to the frame rate of the second image sensor; wherein,

[0232] The first ISP pipeline is used to process the first image data output by the first image sensor;

[0233] The second ISP pipeline is used to process the second image data output by the second image sensor;

[0234] The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data;

[0235] The second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data;

[0236] The first ratio is greater than the second ratio, where both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value.

[0237] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0238] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0239] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0240] This application provides a system-on-a-chip (SoC). Figure 18 This is a schematic diagram of the structure of the on-chip system provided in the embodiments of this application, such as... Figure 18 As shown, the system-on-a-chip 180 includes a first processor 1801, a second processor 1802, and a memory 1803; the memory stores a computer program that can run on the processor, and the first processor 1801 executes the program to implement the information processing method described in the embodiments of this application; the second processor is used to run one or more ISP pipelines.

[0241] This application provides a motherboard. Figure 19 This is a schematic diagram of the motherboard structure provided in an embodiment of this application, such as... Figure 19 As shown, the motherboard 190 includes a system-on-a-chip 180, a first image sensor 1901, and a second image sensor 1902.

[0242] This application provides an electronic device. Figure 20 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 20 As shown, the electronic device 200 includes a motherboard 190.

[0243] It should be noted that the memory 1803 is configured to store instructions and applications executable by the first processor 1801, and can also cache data to be processed or already processed by various modules in the first processor 1801 (e.g., image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0244] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0245] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0246] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0247] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0248] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0249] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0251] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0252] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0253] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0254] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0255] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0256] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0257] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0258] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An information processing method, characterized in that, The method includes: Based on the determination that the first ISP pipeline is dropping frames, the frame rate of the first image sensor is decreasing or the exposure time is increasing, the second ISP pipeline is dropping frames, or the number of first requests in the first buffer queue is less than a first value, the system controls the first buffer queue to cache at least one of the first requests for the operation of the second image sensor and the second ISP pipeline; wherein... The first ISP pipeline is used to process the first image data output by the first image sensor; The second ISP pipeline is used to process the second image data output by the second image sensor; The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

2. The method according to claim 1, characterized in that, The first image sensor is the main camera, and the second image sensor is the secondary camera. Accordingly, the image data output by the first ISP pipeline is used for display, and the second ISP pipeline is used to perform 3A operations based on the second image data to obtain 3A data. The 3A data is used for the 3A operations of the first ISP pipeline.

3. The method according to claim 1 or 2, characterized in that, Based on determining that the first ISP pipeline has dropped frames, controlling the first cache queue to cache at least one of the first requests includes: Based on the determination that the first ISP pipeline has dropped frames, the first request is dispatched to the first cache queue to control that at least one of the first requests is cached in the first cache queue.

4. The method according to any one of claims 1-3, characterized in that, Based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, controlling the first cache queue to cache at least one of the first requests includes: Based on the determination that the frame rate of the first image sensor has decreased or the exposure time has increased, one of the following actions is performed to control that at least one of the first requests is cached in the first cache queue: Increase the exposure time of the second image sensor; Adjust the frame rate of the second image sensor according to the frame rate of the first image sensor; The first request is dispatched to the first cache queue based on the frame rate of the second image sensor.

5. The method according to claim 4, characterized in that, The increase in the exposure time of the second image sensor includes: The exposure time of the second image sensor is increased to a first duration, which is equal to the exposure time of the first image sensor.

6. The method according to any one of claims 1-5, characterized in that, Based on determining that the second ISP pipeline has dropped frames, controlling the first cache queue to cache at least one of the first requests includes: Based on the determination that the second ISP pipeline has dropped frames, the first request is not dispatched to the first cache queue, so as to control that at least one of the first requests is cached in the first cache queue.

7. The method according to any one of claims 3-6, characterized in that, The initial frame rate of the first image sensor is the same as the initial frame rate of the second image sensor.

8. The method according to claim 1 or 2, characterized in that, Based on determining that the number of first requests in the first cache queue is less than a first value, controlling the first cache queue to cache at least one of the first requests includes: Based on the determination that the number of first requests in the first cache queue is less than a first value, the first requests are dispatched to the first cache queue according to a first ratio, so as to control that at least one first request is cached in the first cache queue; Wherein, the first ratio is less than the second ratio, and both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched; the second ratio is the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value. The second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data.

9. The method according to claim 8, characterized in that, After distributing the first request to the first cache queue according to the first ratio, the method further includes: Based on the determination that the number of first requests in the first cache queue is less than a second value, or the return time of 3A data is greater than the frame interval of the second image sensor, the capacity of the first cache queue is increased; wherein, the second value is less than the first value, and the return time of 3A data refers to the time length from the start of the second ISP pipeline performing 3A operation on the second image data based on the first request to obtaining the 3A data of the data structure required by the first ISP pipeline to perform 3A operation.

10. The method according to claim 9, characterized in that, The condition that the number of first requests in the first cache queue is less than the second value means that the number of first requests in the first cache queue is less than the second value within the second time period, or that the number of times the number of first requests in the first cache queue is less than the second value meets the condition.

11. The method according to any one of claims 8-10, characterized in that, The initial frame rate of the first image sensor is different from that of the second image sensor.

12. The method according to any one of claims 2-11, characterized in that, The method further includes: In response to a switching command, the second ISP pipeline is flushed; wherein, flushing the second ISP pipeline includes: releasing the cache resources required by the second ISP pipeline without waiting for the first request in the first cache queue to be used up, and the switching command is used to instruct the second image sensor to be switched to the main camera.

13. An information processing method, characterized in that, The method includes: Based on the determination of frame loss in the first ISP pipeline, the first request is dispatched to the first buffer queue; And / or, based on the determination that the second ISP pipeline has dropped frames, the first request is not dispatched to the first cache queue; And / or, based on the determination that the number of first requests in the first cache queue is less than a first value, the first request is dispatched to the first cache queue according to a first ratio; And / or, based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, perform one of the following actions: Increase the exposure time of the second image sensor; Adjust the frame rate of the second image sensor according to the frame rate of the first image sensor; The first request is dispatched to the first cache queue according to the frame rate of the second image sensor; wherein, The first ISP pipeline is used to process the first image data output by the first image sensor; The second ISP pipeline is used to process the second image data output by the second image sensor; The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data; The second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data. The first ratio is less than the second ratio. Both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched. The second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value.

14. An information processing device, characterized in that, The device includes: The control module is configured to, based on determining that the first ISP pipeline is dropping frames, the frame rate of the first image sensor is decreasing or the exposure time is increasing, the second ISP pipeline is dropping frames, or the number of first requests in the first buffer queue is less than a first value, control the first buffer queue to cache at least one of the first requests, so as to enable the operation of the second image sensor and the second ISP pipeline; wherein, The first ISP pipeline is used to process the first image data output by the first image sensor; The second ISP pipeline is used to process the second image data output by the second image sensor; The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data.

15. An information processing device, characterized in that, The device includes: a control module; wherein the control module is configured to: Based on the determination of frame loss in the first ISP pipeline, the first request is dispatched to the first buffer queue; And / or, based on the determination that the second ISP pipeline has dropped frames, the first request is not dispatched to the first cache queue; And / or, based on the determination that the number of first requests in the first cache queue is less than a first value, the first request is dispatched to the first cache queue according to a first ratio; And / or, based on determining that the frame rate of the first image sensor has decreased or the exposure time has increased, perform one of the following actions: Increase the exposure time of the second image sensor; Adjust the frame rate of the second image sensor according to the frame rate of the first image sensor; The first request is dispatched to the first cache queue according to the frame rate of the second image sensor; wherein, The first ISP pipeline is used to process the first image data output by the first image sensor; The second ISP pipeline is used to process the second image data output by the second image sensor; The first request is used to configure the operating parameters required for the second image sensor to output the second image data and the operating parameters required for the second ISP pipeline to process the second image data; The second request is used to configure the operating parameters required for the first image sensor to output the first image data and the operating parameters required for the first ISP pipeline to process the first image data. The first ratio is less than the second ratio. Both the first ratio and the second ratio refer to the ratio of the number of second requests dispatched to the number of first requests dispatched. The second ratio refers to the dispatch ratio used before determining that the number of first requests in the first cache queue is less than the first value.

16. A system-on-a-chip, comprising a first processor, a second processor, and a memory; said memory storing a computer program executable on the processor, characterized in that, The first processor executes the program to implement the method of any one of claims 1 to 12, or the first processor executes the program to implement the method of claim 13; the second processor is used to run one or more ISP pipelines.

17. A motherboard comprising the system-on-a-chip as claimed in claim 16, a first image sensor, and a second image sensor.

18. An electronic device comprising the motherboard of claim 17.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 12, or when the computer program is executed by a processor, it implements the method as described in claim 13.

Images