Method, device and equipment for hibernation wakeup of image sensors in multi-camera

Abstract

The application discloses a kind of image sensor's dormancy wake-up method, device and equipment in multi-purpose camera, it is related to edge data transmission field, including: control all image sensor enters dormant state, pause corresponding image signal processor ISP, and keep register configuration and control communication interface is in receiving state, wake up a preset sensor as current sensor, and connect the current sensor with the video processing pipeline to output video image;The working parameter set of current sensor and target sensor is obtained, and the working parameter set at least includes image output mode and output frame rate;According to the comparison result, corresponding path switching is selected and executed, and the target sensor is updated as new current sensor.The application binds standby / wake-up instruction and frame synchronization signal with sensor group update mechanism, takes effect in vertical blanking period, avoids picture tearing, significantly reduces power consumption while ensuring the smoothness and real-time of switching.

Description

Technical Field

[0001] This invention relates to the field of edge data transmission, and more particularly to a method, apparatus, and device for waking up an image sensor in a multi-camera system from sleep mode. Background Technology

[0002] To meet the combined needs of wide-angle panoramic views and telephoto close-ups in various video conferencing scenarios, conference cameras have evolved from single-camera solutions to multi-camera systems. A typical dual-camera solution uses CMOS image sensors with different focal lengths, switching outputs to meet different viewing angle requirements.

[0003] However, existing dual-camera solutions have significant shortcomings in terms of power consumption and switching response. A common approach is to power on both sensors simultaneously and output image data at full power, with each sensor's image signal processor pipeline running continuously, resulting in system power consumption approximately twice that of a single-sensor mode. In USB-powered conference camera scenarios, power consumption budgets are extremely limited, and excessive power consumption severely restricts the potential for improving sensor resolution and frame rate, and may even lead to insufficient USB power supply and device instability. Another improved approach is to configure an independent hardware power switch for each sensor, using GPIO to completely power off and then power on the sensor during switching. While this approach reduces standby power consumption, each switching requires executing a complete sequence of power-down, discharge, clock stabilization, register initialization, and image stabilization, taking hundreds of milliseconds in total. Users can clearly perceive screen pauses or blackouts during switching. More importantly, even if the operating parameters of the two sensors are exactly the same, each switching still requires rewriting all register configurations and executing the initialization sequence, resulting in unnecessary time overhead. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, and device for waking up the image sensor in a multi-camera system, aiming to solve the technical problems of USB power shortage and excessive power consumption caused by multiple cameras operating at full power simultaneously, the need for hardware restarts to switch sensors resulting in black screens of hundreds of milliseconds, and repeated full initialization even when the operating parameters are exactly the same, which seriously affects the smoothness and interactive experience of real-time video in meetings.

[0005] To achieve the above objectives, this invention proposes a sleep-wake method for image sensors in a multi-view camera, applicable to a multi-view camera containing at least two image sensors and sharing a video processing pipeline. The multi-view camera includes an image signal processor, registers, and a communication interface, comprising: During the system initialization phase, after completing the initial configuration of each image sensor, all image sensors are controlled to enter a sleep state, the corresponding image signal processor (ISP) is paused, and the register configuration and control communication interface are kept in the receiving state. In the sleep state, multiple image sensors are controlled to stop image acquisition. When a video stream needs to be output, a preset sensor is activated as the current sensor, and the current sensor is connected to the video processing pipeline to output the video image. During video stream output, when a sensor switching command is received, the operating parameter set of the current sensor and the target sensor is obtained. The operating parameter set includes at least the image output mode and the output frame rate. The system compares the operating parameter sets of the current sensor and the target sensor, selects and executes the corresponding path switch based on the comparison results, and updates the target sensor to the new current sensor.

[0006] Further, select and execute the path switch, including: First, disconnect the current sensor from the video processing pipeline and put the current sensor into sleep mode; then, When the first switching path is executed, the target sensor is woken up, the image signal processing of the target sensor is restored and its connection with the video processing pipeline is established, and no new operating parameters are written to the target sensor during this process. When the second switching path is executed, the target sensor is woken up, the register configuration corresponding to the output frame rate is written to the target sensor, and then image signal processing is resumed and a connection is established. When the third switching path is executed, the target sensor is woken up, its image output mode is reconfigured, its image signal processor (ISP) is initialized, and then a connection is established.

[0007] Furthermore, the steps to wake up the target sensor include: Sending a software standby command to the currently operating image sensor via the I2C bus to put it into software standby mode, and sending a command to the target image sensor via the I2C bus to exit software standby mode.

[0008] Furthermore, the steps for controlling the current sensor to enter a sleep state and waking up the target sensor include: By leveraging the group update mechanism supported by the image sensor, the write operation of the flow control register containing standby or wake-up instructions is associated with the frame synchronization signal of the image sensor, so that the sleep state switching takes effect during the vertical blanking period.

[0009] Furthermore, the working parameter set also includes at least one of the following: resolution, pixel format, and data bit width; when comparing the working parameter sets, the switching path is further determined based on the differences in resolution and / or pixel format and / or data bit width.

[0010] Furthermore, the multi-view camera outputs video streams via the USB UVC protocol; the determination of whether to output video streams includes monitoring the activity status of the UVC video channels. When the number of active channels changes from zero to non-zero, it is determined that a video stream needs to be output. When the number of active channels changes from non-zero to zero, the current sensor is disconnected and put into sleep mode.

[0011] The present invention also proposes a sleep-wake device for an image sensor in a multi-view camera, comprising: Multi-view camera; At least two image sensors; The video processing pipeline is shared by at least two image sensors; The control module is configured as follows: After system initialization, each image sensor is put into sleep mode. Sleep mode means that the sensor stops image acquisition and output, pauses image signal processing, and keeps register configuration and communication interface active. When a video stream needs to be output, the preset sensor is woken up and its connection to the video processing pipeline is established. During video stream output, in response to sensor switching commands, the operating parameter sets of the current sensor and the target sensor are obtained. The operating parameter sets include at least the image output mode and the output frame rate. If the working parameter sets are completely identical, the first operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor to resume image signal processing and establish a connection; if only the output frame rate is different, the second operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, configure the frame rate parameters, resume image signal processing and establish a connection; if the image output modes are different, the third operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, reconfigure the output mode and initialize the ISP and establish a connection. The control module ensures that at most one sensor is in an active output state at any given time.

[0012] Furthermore, when the control module sends a sleep or wake-up command to the image sensor, it uses the image sensor's group update mechanism to bind the writing of the flow control command to the frame synchronization signal, so that the state switch is completed during the vertical blanking period.

[0013] Furthermore, it also includes a video stream status monitoring module, which monitors the activity status of the video output channels and triggers wake-up when the number of active channels changes from zero to non-zero, and triggers sleep when the number of active channels changes from non-zero to zero.

[0014] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of a sleep-wake method for an image sensor in a multi-view camera.

[0015] This invention puts multiple sensors into software standby and hibernation after initialization, maintaining only register configuration and communication interface listening to eliminate unnecessary power consumption. During video switching, it automatically compares the output mode and frame rate. If the parameters are the same, it only restores the target ISP and pipeline connection, achieving seamless switching within one frame. If only the frame rate is different, it only updates the frame rate register, avoiding full initialization. At the same time, it uses a sensor group update mechanism to bind standby / wake-up commands with frame synchronization signals, which takes effect during the vertical blanking period, avoiding screen tearing. This significantly reduces power consumption while ensuring smooth and real-time switching. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of the sleep-wake method for the image sensor in the multi-view camera of the present invention; Figure 2 This is a schematic diagram of the module structure of the sleep-wake device for the image sensor in the multi-view camera of the present invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of the present invention and are not intended to limit the present invention.

[0021] To better understand the technical solution of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0022] like Figure 1 As shown, Figure 1 This is a flowchart illustrating the sleep-wake method for the image sensor in the multi-view camera of the present invention.

[0023] This invention proposes a sleep-wake method for image sensors in a multi-view camera, applicable to multi-view cameras containing at least two image sensors and sharing a video processing pipeline. The multi-view camera includes an image signal processor, registers, and a communication interface, comprising: S10, during the system initialization phase, after completing the initial configuration of each image sensor, controls all image sensors to enter a sleep state, pauses the corresponding image signal processor (ISP), and keeps the register configuration and control communication interface in a receiving state; wherein, in the sleep state, controls multiple image sensors to stop image acquisition; S20, When a video stream needs to be output, a preset sensor is woken up as the current sensor, and the current sensor is connected to the video processing pipeline to output video images; S30, during the video stream output process, when a sensor switching command is received, the operating parameter set of the current sensor and the target sensor is obtained. The operating parameter set includes at least the image output mode and the output frame rate. S40: Compare the operating parameter sets of the current sensor and the target sensor, select and execute the corresponding path switch based on the comparison result, and update the target sensor to the new current sensor.

[0024] This invention puts multiple sensors into software standby and hibernation after initialization, maintaining only register configuration and communication interface listening to eliminate unnecessary power consumption. During video switching, it automatically compares the output mode and frame rate. If the parameters are the same, it only restores the target ISP and pipeline connection, achieving seamless switching within one frame. If only the frame rate is different, it only updates the frame rate register, avoiding full initialization. At the same time, it uses a sensor group update mechanism to bind standby / wake-up commands with frame synchronization signals, which takes effect during the vertical blanking period, avoiding screen tearing. This significantly reduces power consumption while ensuring smooth and real-time switching.

[0025] This embodiment proposes a sleep-wake method for image sensors in a multi-view camera, applicable to a multi-view camera system containing at least two image sensors and sharing a video processing pipeline. The multi-view camera includes basic hardware such as an image signal processor (ISP), registers, and an I2C communication interface. In this embodiment, a USB conference camera is used as an example. This camera includes two CMOS image sensors of different specifications. The main sensor is a large-format model used for wide-angle, high-definition images, while the secondary sensor is a sensor with optical zoom used for localized video magnification. Both output video streams to the computer via the USB UVC protocol, and the entire device is powered by a Type-C interface. Since the power supply capacity of the USB bus has an upper limit, in traditional solutions, simultaneous full-power operation of both sensors leads to tight power consumption budgets, severely restricting the performance of sensor resolution, frame rate, and other indicators.

[0026] For the aforementioned application scenarios, the method in this embodiment, after completing the initial configuration of each image sensor during the system initialization phase, controls all image sensors to enter a sleep state, simultaneously suspending the operation of the corresponding image signal processor (ISP), while maintaining the register configuration of each sensor unchanged and keeping the I2C communication interface in an active state capable of receiving commands. In this sleep state, all image sensors stop pixel acquisition and image data output. Specifically, the main control chip writes a software standby command to each sensor via the I2C bus, causing the internal analog circuits and pixel arrays of the sensor to stop working, while the digital interface maintains power supply and clock supply. This ensures that register configuration is not lost and the communication interface is always responsive, consuming almost no power. Combining suspending the ISP with stopping image acquisition achieves end-to-end sleep mode from the image data source to the processing pipeline, eliminating the ineffective dynamic power consumption caused by the ISP pipeline idling in traditional solutions.

[0027] When a video stream needs to be output, the system wakes up a preset sensor as the current sensor and connects it to the video processing pipeline to output the video image. In this embodiment, the video stream request is triggered by the host opening the video channel via the USB UVC protocol. After the system detects that the number of active UVC channels changes from zero to non-zero, it determines which sensor should be activated based on the preset sensor number in the configuration file. Then, it sends a command to exit the software standby mode to the sensor via the I2C bus. After the sensor resumes pixel output, the ISP processing instance corresponding to the sensor is restarted, and its video input channel is connected to the subsequent video encoding and UVC output pipeline, completing the construction of a complete data path from the sensor to the host. Here, the woken-up sensor actually inherits all the working parameters preset during the initialization phase, including output mode, frame rate, resolution, exposure parameters, and white balance parameters. Therefore, it only needs to send an exit standby command to resume normal output. The entire process from sleep to image output takes a very short time, much faster than the hundreds of milliseconds required for a complete power-off cold start.

[0028] During video stream output, when the system receives a sensor switching command, it acquires the operating parameter sets of the current sensor and the target sensor. These parameters include at least the image output mode and output frame rate. In actual products, the switching command originates from user operation, such as clicking the wide-angle / telephoto switch button on the camera control interface during a meeting. The system application layer converts this operation into a sensor switching request and sends it to the underlying driver. The specific method for acquiring the parameter set is that the system maintains a global sensor state structure in memory, which records parameters such as the currently configured image output mode, frame rate, and resolution for each sensor. The switching module directly reads the parameters from this structure without accessing hardware registers; the reading process is completed in microseconds. The comparison of parameter sets includes at least two dimensions: the image output mode determines whether the sensor's internal exposure timing and synthesis method is a normal linear mode or an HDR multi-frame synthesis mode; the output frame rate determines the sensor's pixel clock frequency and frame length register value. The degree of change in these two types of parameters directly determines whether the target sensor can directly reuse its existing register configuration. Therefore, choosing these two dimensions as the basis for path decision-making can cover the main scenarios of parameter changes in actual products to the greatest extent.

[0029] After comparing the operating parameter sets of the current sensor and the target sensor, the system selects and executes the corresponding path switch based on the comparison result, and updates the target sensor to the new current sensor after the switch is complete. Specifically, updating the current sensor identifier involves modifying the active sensor number field in the global sensor state structure from the original sensor's index value to the target sensor's index value, while simultaneously updating the corresponding parameter set record. This assignment operation marks the completion of the switch's closed loop, ensuring that the system can correctly identify the currently operating sensor and obtain its accurate operating parameters as a comparison benchmark when the next switch request arrives. This constitutes a sustainable loop state machine that supports any number of sensor switches without state chaos.

[0030] The derivation process from the above technical features to the technical effects is as follows: Immediately after initialization, all sensors are placed into software standby / sleep mode, minimizing system power consumption when there is no video stream output. Although the two sensors are powered on, they consume almost no power, fully releasing the power budget of the USB bus. This allows the system to allocate more resources to support higher resolution or higher frame rate image output when the user starts video. This mechanism directly solves the technical defect in existing technologies where multiple sensors operating at full power simultaneously leads to excessive system power consumption. Maintaining active register configuration and communication interfaces in sleep mode allows sensors to be quickly woken up without re-initialization, a fundamental condition for subsequent rapid switching. Using parameter comparison as the core basis for switching path decisions directly solves the technical problem in existing technologies where blindly performing full initialization even when parameters are identical, resulting in unnecessary time overhead.

[0031] Further, select and execute the path switch, including: First, disconnect the current sensor from the video processing pipeline and put the current sensor into sleep mode; then, When the first switching path is executed, the target sensor is woken up, the image signal processing of the target sensor is restored and its connection with the video processing pipeline is established, and no new operating parameters are written to the target sensor during this process. When the second switching path is executed, the target sensor is woken up, the register configuration corresponding to the output frame rate is written to the target sensor, and then image signal processing is resumed and a connection is established. When executing the third switching path, the target sensor is woken up, its image output mode is reconfigured, and its image signal processor (ISP) is initialized before a connection is established. This embodiment further specifies the steps for selecting and executing the path switching described above. All three switching paths first perform a common preparatory operation: disconnecting the current sensor from the video processing pipeline and putting the current sensor into a sleep state. Specifically, disconnecting the pipeline involves calling the pipeline disconnection interface function of the video processing framework to cut off the data flow path between the current sensor's video input channel and the backend ISP output buffer and video encoder input. This operation essentially removes the mapping relationship of the video buffer in memory, without affecting the sensor's operating state at the hardware level. After disconnection, the system immediately sends a software standby command to the current sensor via the I2C bus, putting it into a sleep state and releasing the ISP processing resources and memory bandwidth it occupies.

[0032] Subsequently, based on the comparison results of the working parameter sets, three paths are executed respectively. When the first switching path is executed, it corresponds to a scene where the image output modes and output frame rates of the two sensors are exactly the same. At this time, the system wakes up the target sensor, resumes the image signal processing of the target sensor, and establishes its connection with the video processing pipeline, without writing any new working parameters to the target sensor during this process. The reason why no new parameters can be written is that the register configuration saved by the target sensor before it entered sleep mode is exactly the same as that of the current sensor. After exiting the software standby mode, the sensor will automatically restore the pixel output according to the original register values, including key parameters such as exposure time, gain, and frame length, which do not need to be modified. From a technical point of view, the switching latency of this path is compressed to within one frame period, which is about 16 milliseconds under the working conditions of 60fps. This time is far below the threshold of screen pause that the human eye can perceive, and users will not feel any screen interruption in actual use.

[0033] When executing the second switching path, it corresponds to a scenario where the two sensors have the same image output mode but different output frame rates. In this case, after waking up the target sensor, the system first writes the register configuration corresponding to the output frame rate to the target sensor, then resumes image signal processing and establishes a connection. The specific operation of configuring the frame rate parameters involves modifying the frame length register inside the sensor. This register controls the total number of lines in each frame, and different frame rates are achieved by adjusting the length of the vertical blanking period. When writing to the frame rate register, it is usually necessary to synchronously adjust the pixel clock frequency or clock division ratio to ensure that the pixel output rate matches the frame rate. Since the two sensors have the same image output mode, their complex configurations such as HDR mode and exposure timing remain consistent. These parameters do not need to be rewritten; only a few frame rate-related registers need to be modified. Therefore, the switching latency of this path is within a few frames, which is also much faster than the hundreds of milliseconds of the complete initialization process.

[0034] When executing the third switching path, it corresponds to a scenario where the two sensors have different image output modes, such as switching from normal mode to HDR high dynamic range mode. In this case, after the system wakes up the target sensor, it needs to reconfigure the target sensor's image output mode and initialize its image signal processor (ISP), and then establish a connection. The reconfiguration of the image output mode involves writing a register sequence corresponding to the new mode to the sensor. For example, in HDR mode, it's necessary to configure the timing parameters for multiple exposures, the gain ratio for each exposure, and the fusion weights for multi-frame synthesis. The number and complexity of these registers are far greater than frame rate adjustments. More importantly, the change in image output mode fundamentally alters the characteristics of the raw image data output by the sensor, such as changing from single-frame linear output to multi-frame alternating output. The ISP processing instance's algorithm pipeline needs to be adjusted accordingly, including the extraction method of 3A statistics, the threshold for bad pixel correction, and the tone mapping curve, all of which need to be reinitialized. Therefore, the destruction and reconstruction process of the ISP instance needs to be performed. Due to the complete restart of the ISP and the rewriting of a large number of registers, the latency of this path is approximately in the hundreds of milliseconds, but this is the necessary reconfiguration overhead for the image output mode change itself, and it represents the optimal switching speed achievable in this scenario.

[0035] The hierarchical design of the three paths described above enables the system to automatically select the optimal recovery strategy based on the degree of parameter change. When the parameters do not change or only change slightly, unnecessary full reconfiguration is avoided. This directly solves the technical problem of unnecessary repeated configuration during sensor switching in the prior art, and achieves a fundamental improvement from "full initialization every time" to "hierarchical recovery on demand".

[0036] Furthermore, the steps to wake up the target sensor include: Sending a software standby command to the currently operating image sensor via the I2C bus to put it into software standby mode, and sending a command to the target image sensor via the I2C bus to exit software standby mode.

[0037] In this embodiment, the steps of controlling the image sensor to enter sleep mode and waking up the target sensor are both implemented via the I2C bus. Specifically, when the current sensor needs to enter sleep mode, the main control chip sends a software standby command to the sensor via the I2C bus, that is, writes an enable value to a specific standby control register inside the sensor. After receiving the command, the sensor's internal state machine enters software standby mode, stops the exposure and readout operations of the pixel array, and shuts off the bias current of some analog circuits, but maintains the power supply to the digital interface and register group, so that the sensor enters a low-power state. When the target sensor needs to be woken up, the main control chip also sends a command to the sensor to exit software standby mode via the I2C bus, that is, writes a disable value to the same standby control register. The sensor's internal state machine exits standby mode, restores the working bias and readout timing of the pixel array, and resumes outputting image data.

[0038] Sleep / wake-up control is achieved via the I2C bus, eliminating the need for additional dedicated hardware pins or power switch circuits. All state transitions are completed simply by reading and writing standard sensor registers. This design fully leverages the hardware characteristic of modern CMOS image sensors, which commonly have built-in software standby functionality, achieving rapid state transition capabilities at extremely low implementation costs. Compared to existing solutions that use GPIO to control an external power switch to power on and off the sensor, the software standby solution reduces state transition time from hundreds of milliseconds to several milliseconds, which is a crucial hardware foundation for the rapid response of the entire transition process.

[0039] Furthermore, the steps for controlling the current sensor to enter a sleep state and waking up the target sensor include: By leveraging the group update mechanism supported by the image sensor, the write operation of the flow control register containing standby or wake-up instructions is associated with the frame synchronization signal of the image sensor, so that the sleep state switching takes effect during the vertical blanking period.

[0040] In the steps of controlling the current sensor to enter a sleep state and waking up the target sensor in this embodiment, the system uses the group update mechanism supported by the image sensor to associate the flow control register write operation containing the standby command or wake-up command with the frame synchronization signal of the image sensor, so that the state switch takes effect strictly within the vertical blanking period.

[0041] Group update mechanism is an advanced feature commonly supported by modern CMOS image sensors. Its working principle is as follows: the main control chip can continuously write multiple register configuration instructions via the I2C bus. These instructions do not take effect immediately but are temporarily stored in the sensor's internal shadow register or instruction buffer. After all the registers that need updating have been written, the main control chip writes another set of update trigger instructions. The sensor will then atomically update all the temporarily stored register values ​​in the buffer to the registers that actually take effect during the vertical blanking period between the completion of the current frame's image data transmission and the start of the next frame. In this way, the timing of the sleep or wake-up instructions is precisely controlled within the interval between two frames, ensuring that the sensor will never suddenly stop or start pixel output in the middle of a frame.

[0042] The direct technical benefit of the aforementioned synchronization mechanism is the avoidance of image anomalies caused by improper timing of flow control state switching. Without frame synchronization, a sleep command might take effect while the sensor is outputting the middle line of a frame, resulting in missing data in the latter half of the frame, manifesting as image tearing at the receiving end. Similarly, if a wake-up command takes effect before the sensor's internal timing is stable, it may cause misalignment of the first frame data or abnormal exposure. By binding state switching to the frame synchronization signal, the current sensor only enters sleep mode after its last frame has been completely output. The first frame of the target sensor after wake-up starts output from the beginning of the complete frame, ensuring the integrity of frame data and image continuity during video stream switching, resulting in a completely smooth image transition perceived by the user.

[0043] Furthermore, the working parameter set also includes at least one of the following: resolution, pixel format, and data bit width; when comparing the working parameter sets, the switching path is further determined based on the differences in resolution and / or pixel format and / or data bit width.

[0044] This embodiment further expands the dimensions of the working parameter set. In addition to the two basic dimensions of image output mode and output frame rate, the working parameter set may also include at least one of resolution, pixel format, and data bit width. When comparing the working parameter sets, the system further determines the switching path based on the differences in these extended dimensions.

[0045] Specifically, resolution refers to the effective pixel width and height of the sensor's output image, such as switching from 1920x1080 to 3840x2160. Changing the resolution affects the pixel merging method or windowing readout area within the sensor, requiring reconfiguration of the sensor's horizontal / vertical windowing registers and pixel skip / merge mode registers. Pixel format refers to the raw image data type output by the sensor, such as different formats like RAW10, RAW12, or YUV422. Changes in pixel format mean changes to the data bit width and color arrangement of each pixel, requiring reconfiguration of the sensor's output data format registers and the data type field of the MIPI CSI interface. Data bit width refers to the bit depth of each pixel component; switching from 10-bit to 12-bit requires adjusting the sensor's analog-to-digital converter precision settings and the data packet format of the MIPI channel.

[0046] The introduction of these extended dimensions enables more refined parameter comparison decisions. In actual products, the system stores all current operating parameters of each sensor in a sensor state structure in memory, which are directly read and compared item by item during switching. If the comparison results of the extended dimensions show differences in resolution, pixel format, or data bit width, but the output mode is the same, the system can decide whether to classify it into the second or third switching path according to preset rules. For example, when the resolution is different, it usually only requires updating a few registers to recover, so it can be classified into the second path; while changes in pixel format involve reconfiguration of the MIPI interface, so it needs to be classified into the third path. This multi-dimensional fine-grained comparison further avoids unnecessary full initialization operations, making the selection of switching paths more closely match the hardware reconfiguration workload caused by actual parameter changes.

[0047] Furthermore, the multi-view camera outputs video streams via the USB UVC protocol; the determination of whether to output video streams includes monitoring the activity status of the UVC video channels. When the number of active channels changes from zero to non-zero, it is determined that a video stream needs to be output. When the number of active channels changes from non-zero to zero, the current sensor is disconnected and put into sleep mode.

[0048] like Figure 2 As shown, Figure 2 This is a schematic diagram of the module structure of the sleep-wake device for the image sensor in the multi-view camera of the present invention.

[0049] The present invention also proposes a sleep-wake device for an image sensor in a multi-view camera, comprising: 10 multi-view cameras; At least two image sensors 20; The video processing pipeline 30 is shared by at least two image sensors 20; Control module 40 is configured as follows: After system initialization, each image sensor 20 is put into sleep mode. Sleep mode means that the sensor stops image acquisition and output, pauses image signal processing, and keeps register configuration and communication interface active. When a video stream needs to be output, the preset sensor is woken up and its connection to the video processing pipeline is established. During video stream output, in response to sensor switching commands, the operating parameter sets of the current sensor and the target sensor are obtained. The operating parameter sets include at least the image output mode and the output frame rate. If the working parameter sets are completely identical, the first operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor to resume image signal processing and establish a connection; if only the output frame rate is different, the second operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, configure the frame rate parameters, resume image signal processing and establish a connection; if the image output modes are different, the third operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, reconfigure the output mode and initialize the ISP and establish a connection. The control module ensures that at most one sensor is in an active output state at any given time.

[0050] This embodiment further defines the determination mechanism for triggering video stream output. Specifically, when a multi-camera outputs a video stream to the host via the USB UVC protocol, the determination process for needing to output the video stream is completed by the video stream status monitoring module in the system. This module continuously monitors the activity status of the UVC video channel, specifically monitoring the active video channel counter maintained by the USB UVC driver layer.

[0051] When the host application opens the camera device and requests video stream data, the USB host sends a probe / submit control request as defined by the UVC protocol to the device. The UVC driver on the device processes this request and increments the active channel count from zero to one. The video stream status monitoring module, triggered by polling or an interrupt, detects the active channel count changing from zero to non-zero and determines that video stream output is needed. It then triggers a callback function to notify the sensor switching control module to execute the process of waking up the preset sensor. This triggering mechanism has a simple enough condition; monitoring only the zero-to-non-zero state transition of a count value is sufficient to accurately capture when the user begins using the camera. Conversely, when the host application closes the video preview or exits a video call, the UVC driver decrements the active channel count from one to zero. The monitoring module detects the active channel count changing from non-zero to zero, determines that the video stream has stopped, triggers a callback to notify the control module to disconnect the current sensor from the video processing pipeline, and puts the sensor into sleep mode.

[0052] By directly binding the start and end determination of the video stream to the channel activity state of the UVC protocol layer, the timing of sleep / wake-up is completely synchronized with the user's actual usage behavior. When the user opens the video, the sensor immediately wakes up and outputs an image; when the user closes the video, the sensor immediately goes into sleep mode to save power, with the response latency of the entire process in the millisecond range. This mechanism solves the power-sensitive application requirements in USB-powered scenarios, ensuring that the system automatically enters a low-power sleep state at any time when the user does not need the video feed, avoiding power waste caused by the sensor continuing to run even when the video stream has stopped.

[0053] Furthermore, the device is the specific hardware and software architecture for implementing the above method. The device includes at least two image sensors, a video processing pipeline shared by these image sensors, and a control module responsible for unified scheduling and decision-making.

[0054] At least two image sensors have different optical characteristics. For example, the main sensor is a large-format CMOS sensor for wide-angle high-definition image acquisition, and the secondary sensor is a sensor with optical zoom for local magnification image acquisition. Each sensor supports software standby mode and group update mechanism, and is connected to the main control chip through an independent I2C bus. Video data is input to the video input interface of the main control chip through an independent MIPI CSI channel.

[0055] The video processing pipeline is shared by at least two image sensors, specifically including an image signal processor (ISP) instance, a video encoder, and a USB UVC output module. Since hardware platforms typically integrate only a limited number of ISP processing engines, two sensors cannot simultaneously use independent ISP pipelines. Therefore, the device design employs a shared approach, with the control module using a mutual exclusion mechanism to determine which sensor currently occupies the ISP resources.

[0056] The control module is the core decision-making and execution unit of the device, configured to perform the following functions: After system initialization, it puts each image sensor into a sleep state, defined as the sensor stopping image acquisition and output, pausing the corresponding image signal processing, while keeping register configuration and communication interface active; when a video stream needs to be output, it wakes up the preset sensor and establishes its connection with the video processing pipeline; during video stream output, in response to sensor switching commands, it acquires the working parameter sets of the current sensor and the target sensor, which include at least the image output mode and output frame rate; it compares the working parameter sets, and if they are completely identical, it performs the first operation, i.e., disconnects the current sensor connection and puts it into sleep mode, wakes up the target sensor to resume image signal processing and establishes a connection; if only the output frame rate is different, it performs the second operation, i.e., disconnects the current sensor connection and puts it into sleep mode, wakes up the target sensor, configures the frame rate parameters, resumes image signal processing and establishes a connection; if the image output modes are different, it performs the third operation, i.e., disconnects the current sensor connection and puts it into sleep mode, wakes up the target sensor, reconfigures the output mode and initializes the ISP and establishes a connection. During all the above operations, the control module maintains a globally active sensor identifier and a mutex lock to ensure that at most one sensor is in an active image output state at any given time.

[0057] In its implementation, the control module can be an embedded software module running on the main control chip, comprising three sub-modules: a sensor management module, an ISP management module, and a switching decision state machine. The sensor management module is responsible for sending standby and wake-up commands to each sensor via the I2C bus and maintaining the current operating parameter set for each sensor. The ISP management module is responsible for creating, destroying, pausing, and resuming ISP processing instances for each sensor. The switching decision state machine is responsible for receiving switching requests, performing parameter comparisons and path selection, and coordinating the first two sub-modules to execute the corresponding path's operation steps sequentially. The three sub-modules work collaboratively under the protection of a mutex lock to ensure state consistency in multi-threaded or interrupt-concurrent scenarios.

[0058] The aforementioned device integrates the sleep mechanism, parameter comparison decision, and hierarchical switching path into the control module in a modular manner, enabling any camera product equipped with this device to achieve low power consumption and fast switching capabilities without the need for additional hardware, thus exhibiting good portability and reusability.

[0059] Furthermore, when the control module sends a sleep or wake-up command to the image sensor, it uses the image sensor's group update mechanism to bind the writing of the flow control command to the frame synchronization signal, so that the state switch is completed during the vertical blanking period.

[0060] In practice, when the sensor management submodule of the control module needs to send a sleep or wake-up command, it does not directly write the final effective value to the sensor's standby control register. Instead, it first temporarily stores the register address and data value corresponding to the standby or wake-up command in the command buffer. Then, the sensor management submodule writes the enable value of the trigger register to the sensor via the I2C bus, specifying the trigger condition as the next vertical blanking period of the frame synchronization signal. After the sensor receives the trigger command, its internal timing controller, after the data transmission of the current frame is completed, utilizes the idle period of the vertical blanking period to activate all the temporarily stored commands at once, completing the state switch.

[0061] The aforementioned approach of using a hardware group update mechanism for frame synchronization binding transforms the asynchronous flow control operation, which could potentially conflict with image frame data output, into a deterministic operation that is completely synchronized with the sensor's own frame rhythm. From a technical perspective, this mechanism ensures that every sleep and wake-up operation occurs within a safe window between image frames, completely eliminating the risk of image tearing and frame data anomalies at the hardware timing level, and providing an underlying guarantee for the smoothness of the switching process.

[0062] Furthermore, the device in this embodiment also includes a video stream status monitoring module. This module is used to monitor the activity status of the video output channels, and triggers a wake-up process when the number of active channels changes from zero to non-zero, and triggers a sleep process when the number of active channels changes from non-zero to zero.

[0063] The video stream status monitoring module, in its implementation, sits between the USB UVC driver layer and the sensor switching control module, acting as an event relay. This module senses channel status changes by registering callback functions with the UVC driver. When the UVC driver processes the host's video stream control request, it calls the registered callback functions, passing the current number of active channels as a parameter to the monitoring module. Internally, the monitoring module maintains a previously recorded value for active channels. Each time a callback is received, it compares this value with the current value. If a rising edge transition from zero to non-zero is detected, a video stream start event is generated and sent to the control module, triggering the wake-up of the preset sensor and pipeline connection. If a falling edge transition from non-zero to zero is detected, a video stream stop event is generated and sent to the control module, triggering the current sensor to sleep and pipeline disconnection.

[0064] This event-driven design decouples video stream status monitoring from specific sensor control logic. The monitoring module is only responsible for status perception and event distribution, while the control module is only responsible for responding to events and executing specific hardware operations. The interface between the two modules is clear, facilitating adaptation between different video output protocols. For example, if the product is changed from USB UVC to HDMI output, only the sensing source of the monitoring module needs to be replaced from the UVC driver callback to the HDMI hot-plug detection signal; the remaining control logic can be reused without any modification.

[0065] The method of this embodiment can also be implemented as an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements all the steps of the sleep-wake method for the image sensor in a multi-view camera described in any of the preceding claims.

[0066] In this embodiment, the processor of the electronic device is the main control chip of the multi-view camera, and the memory is either the chip's built-in static random access memory or an external double-rate synchronous dynamic random access memory. The computer program is burned into a non-volatile storage medium in the form of firmware and loaded into the memory for execution after the device is powered on. When the processor executes the program, it sequentially completes all the steps of sensor initialization and sleep configuration, video stream request response and wake-up operation, sensor switching decision and execution, and sleep recovery after the video stream stops, thereby achieving optimized control of the power consumption and switching performance of the multi-view camera system. By embedding the above method into the electronic device in the form of a computer program, any camera product using the same or compatible hardware platform can obtain the sleep-wake function of this invention simply by burning this program, greatly reducing the deployment threshold and promotion cost of the technical solution.

[0067] The above are only some embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made under the technical concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A sleep-wake method for image sensors in a multi-view camera, applied to a multi-view camera containing at least two image sensors and sharing a video processing pipeline, the multi-view camera including an image signal processor, registers, and a communication interface, characterized in that, include: During the system initialization phase, after completing the initial configuration of each image sensor, all image sensors are controlled to enter a sleep state, the corresponding image signal processor (ISP) is paused, and the register configuration and control communication interface is kept in the receiving state; wherein, in the sleep state, multiple image sensors are controlled to stop image acquisition; When a video stream needs to be output, a preset sensor is activated as the current sensor, and the current sensor is connected to the video processing pipeline to output a video image. During video stream output, when a sensor switching command is received, the operating parameter set of the current sensor and the target sensor is obtained, and the operating parameter set includes at least the image output mode and the output frame rate; The operating parameter sets of the current sensor and the target sensor are compared. Based on the comparison result, the corresponding path switch is selected and executed, and the target sensor is updated to the new current sensor.

2. The sleep-wake method for the image sensor in a multi-view camera according to claim 1, characterized in that, The selection and execution of path switching includes: First, disconnect the current sensor from the video processing pipeline and control the current sensor to enter the sleep state; then, When the first switching path is executed, the target sensor is woken up, the image signal processing of the target sensor is restored and its connection with the video processing pipeline is established, and no new operating parameters are written to the target sensor during this process. When the second switching path is executed, the target sensor is woken up, the register configuration corresponding to the output frame rate is written to the target sensor, and then image signal processing is resumed and a connection is established. When the third switching path is executed, the target sensor is woken up, its image output mode is reconfigured, its image signal processor (ISP) is initialized, and then a connection is established.

3. The sleep-wake method for the image sensor in a multi-view camera according to claim 2, characterized in that, The steps to wake up the target sensor include: Sending a software standby command to the currently operating image sensor via the I2C bus to put it into software standby mode, and sending a command to the target image sensor via the I2C bus to exit software standby mode.

4. The sleep-wake method for the image sensor in a multi-view camera according to claim 2, characterized in that, The steps of controlling the current sensor to enter a sleep state and waking up the target sensor include: By leveraging the group update mechanism supported by the image sensor, the write operation of the flow control register containing standby or wake-up instructions is associated with the frame synchronization signal of the image sensor, so that the sleep state switching takes effect during the vertical blanking period.

5. The sleep-wake method for the image sensor in a multi-view camera according to claim 2, characterized in that, The set of working parameters further includes at least one of the following: resolution, pixel format, and data bit width; when comparing the set of working parameters, the switching path is further determined based on the differences between the resolution and / or pixel format and / or data bit width.

6. The sleep-wake method for the image sensor in a multi-view camera according to claim 2, characterized in that, The multi-view camera outputs a video stream via the USB UVC protocol; the determination of whether to output a video stream includes monitoring the activity status of the UVC video channel, determining that a video stream needs to be output when the number of active channels changes from zero to non-zero, and disconnecting the current sensor and putting it into a sleep state when the number of active channels changes from non-zero to zero.

7. A sleep / wake-up device for an image sensor in a multi-view camera, characterized in that, include: Multi-view camera; At least two image sensors; The video processing pipeline is shared by the at least two image sensors; The control module is configured as follows: After system initialization, each image sensor is put into sleep mode. The sleep mode is characterized by the sensor stopping image acquisition and output, pausing image signal processing, and keeping register configuration and communication interface active. When a video stream needs to be output, the preset sensor is woken up and its connection to the video processing pipeline is established. During video stream output, in response to sensor switching commands, the operating parameter sets of the current sensor and the target sensor are acquired, and the operating parameter sets include at least the image output mode and the output frame rate; If the working parameter sets are completely identical, the first operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor to resume image signal processing and establish a connection; if only the output frame rate is different, the second operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, configure the frame rate parameters, resume image signal processing and establish a connection; if the image output modes are different, the third operation is performed: disconnect the current sensor connection and put it into sleep mode, wake up the target sensor, reconfigure the output mode and initialize the ISP and establish a connection. The control module ensures that at most one sensor is in an active output state at any given time.

8. The apparatus according to claim 7, characterized in that, When the control module sends a sleep or wake-up command to the image sensor, it uses the image sensor's group update mechanism to bind the writing of the flow control command with the frame synchronization signal, so that the state switch is completed during the vertical blanking period.

9. The apparatus according to claim 7, characterized in that, It also includes a video stream status monitoring module, which monitors the activity status of the video output channels and triggers wake-up when the number of active channels changes from zero to non-zero, and triggers sleep when the number of active channels changes from non-zero to zero.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the sleep-wake method for the image sensor in a multi-view camera as described in any one of claims 1 to 6.

Images