Synchronization of optical sensors for reduced power consumption
By learning the frame attributes of the optical sensor and synchronously reading out time periods, and using clock gating and virtual triggers, the synchronization problem of different camera modules in the embedded stereo camera device was solved, and the energy-saving effect of the optical device was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-24
- Publication Date
- 2026-03-10
AI Technical Summary
In embedded stereo camera devices, there is a synchronization problem between different camera modules, which leads to increased power consumption, especially when external triggers cannot activate certain camera modules. How can these modules be effectively synchronized to reduce power consumption?
The system learns the frame attributes of each optical sensor through an automatic algorithm, synchronizes the readout periods of the frames, determines the delay to maximize the overlap of the readout periods, shuts down modules that are not used during the blank periods, and achieves synchronization using clock gating and virtual triggers.
This significantly reduces the power consumption of the optical device without increasing processing time, thus improving the system's energy efficiency.
Smart Images
Figure CN121644980A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates generally to methods for using optical devices, and more particularly, to methods capable of reducing power consumption when operating optical apparatuses comprising a plurality of camera units. BACKGROUND
[0002] A stereo camera apparatus is an apparatus consisting of two camera units, assembled in a stereoscopic module. Stereoscopy is a technique used to create or enhance the illusion of depth in images derived from multiple images of the same scene by stereopsis. In other words, it is the depth impression perceived when a person with normal binocular vision looks at a scene with both eyes, which is responsible for creating two slightly different scene images in the two eyes due to the different positions of the eyes / cameras.
[0003] In embedded stereo camera apparatuses, the power budget is always a limiting factor, so saving power consumption is an important issue for operating such apparatuses.
[0004] In computer vision systems comprising several camera modules of different sensor types (global shutter, rolling shutter), different camera vendors, different frame rates and different shutter exposure times, the required processing of the captured images can differ in processing duration and timing for each camera module.
[0005] The above factors can lead to the fact that the processing of the inputs received by the various camera modules is performed during almost 100% of the system operating time, thus possibly increasing the power consumption of the operating system.
[0006] In order to optimize the processing power consumption in this case, it is common in the art to use external triggers of the camera modules, then to synchronize the operation of these different camera modules based on the general triggers of the camera modules. This solution enables to reduce the "active processing" time of the captured images, thus enabling energy saving.
[0007] However, in certain cases, the various camera modules cannot operate in response to such external triggers, and therefore, the synchronization between such camera modules becomes problematic.
[0008] The problem that the present invention seeks to solve is therefore how to synchronize different camera modules in order to reduce the power consumption of an apparatus comprising these different camera modules, in the case where external triggers cannot be used to activate some of the different camera modules. SUMMARY
[0009] The present disclosure can be summarized by referring to the appended claims.
[0010] It is an object of the present disclosure to provide a method for reducing the power consumption of an operating vision system comprising several optical sensors.
[0011] It is an object of the present disclosure to provide an automatic algorithm capable of learning frame properties associated with each of the different optical sensors comprised in an embedded system and, based on the learned frame properties, synchronizing the readout period of the frames.
[0012] It is an object of the present disclosure to provide a method for determining the delays of the various optical sensors in an optical system to maximize the overlap of the optical sensors readout periods.
[0013] Other objects of the present invention will become apparent from the following description.
[0014] According to a first embodiment of the present disclosure, there is provided a method for operating the energy saving of an optical device comprising a plurality of optical sensors (e.g. a multi-camera module device), the method comprising: determining the frame properties of each of the plurality of optical sensors and, based on the determined frame properties, synchronizing the readout period of the frames; after synchronizing the readout period of the frames, determining the processing period of each of the plurality of optical sensors such that the processing periods at least partially overlap each other; and, obtaining an at least partially overlapping blank period of all of the plurality of optical sensors and wherein, during the at least partially overlapping blank period, the modules comprised within the optical device that are not used during that period are switched off.
[0015] According to another embodiment of the present disclosure, there is provided a method for operating the energy saving of an optical device comprising a plurality of optical sensors, wherein at least one of the plurality of optical sensors is characterized in that it cannot be activated in response to receiving an external trigger signal. The method comprises the steps of:
[0016] (i) providing an optical device comprising a plurality of optical sensors, each of the optical sensors being associated with specific frame properties, the frame properties comprising an exposure period, a readout period and a blank period;
[0017] (ii) acquiring, by at least one processor comprised in the optical device, information related to the specific frame properties of each of the plurality of optical sensors;
[0018] (iii) based on the acquired information, synchronizing, by the at least one processor comprised in the optical device, the readout period of all of the optical sensors;
[0019] (iv) determining, by the at least one processor comprised in the optical device, at least partially overlapping processing time periods of frames associated with at least some of the plurality of optical sensors after synchronizing the readout time periods of all optical sensors;
[0020] (v) determining, by the at least one processor comprised in the optical device, at least partially overlapping blank time periods of at least some of the plurality of optical sensors after determining at least partially overlapping processing time periods of frames associated with at least some of the plurality of optical sensors; and
[0021] (vi) during the at least partially overlapping blank time periods, switching off internal modules within the optical device that are not used during the at least partially overlapping blank time periods, thereby obtaining energy saving in the operation of the optical device.
[0022] According to yet another embodiment, the above step (ii) further comprises the steps of:
[0023] a) defining different combinations of the plurality of optical sensors based on frame properties of the plurality of optical sensors, and defining a virtual trigger for each of the different combinations;
[0024] b) selecting a master optical sensor for each of the different combinations, and defining all other optical sensors belonging to a specific combination as slave optical sensors or as free running optical sensors, slave optical sensors being optical sensors that are triggered with respect to the master optical sensor;
[0025] c) determining, for each of the different combinations, a score based on a mechanism for calculating a frames per second (“FPSs”) score of all optical sensors belonging to each of the different combinations; first
[0026] d) selecting the combination associated with the lowest score obtained.
[0027] Optionally, the mechanism for calculating a frames per second (“FPSs”) score of all optical sensors belonging to each of the different combinations is the greatest common divisor (GCD).
[0028] By yet another embodiment, the above step (iii) further comprises the steps of:
[0029] (i) triggering the plurality of optical sensors according to the virtual trigger defined for the selected combination after selecting the combination associated with the lowest score obtained;
[0030] (ii) collecting timestamps associated with “end of frame” signals received from the plurality of optical sensors; and
[0031] (iii) calculating the relative delay needed to synchronize the end of the frame from the optical sensor with the end of the frame of the master optical sensor.
[0032] According to another embodiment, after applying the calculated relative delay, the method is repeated each time an optical sensor is removed from the optical device or an optical sensor is added to the optical device. BRIEF DESCRIPTION OF DRAWINGS
[0033] For a more complete understanding of the present application, reference is now made to the following detailed description taken in connection with the accompanying drawings in which:
[0034] Figure 1 - shows a flowchart of a method for reducing power consumption in an optical device, built according to an embodiment of the present application;
[0035] Figure 2 - illustrates a method built according to an embodiment of the present application, which reduces the power consumption of a 3D optical device by combining clock gating with the triggering process of the present disclosure;
[0036] Figure 3 - shows an example of the results of the steps performed as part of the method according to an embodiment of the present application, which determines the common frame by implementing the GCD calculation. The frame rates of the three optical sensors included in this example are 10, 35 and 50 frames per second;
[0037] Figure 4 - shows an example of the method for calculating the triggering delay; and
[0038] Figure 5 - shows an example of implementing an embodiment of the present disclosure, which is used to establish the delay for optical sensors with different frame rates. DETAILED DESCRIPTION
[0039] In the present disclosure, the term "comprising" is intended to have an open-ended meaning, thus, when a first element is stated to comprise a second element, the first element can also include one or more other elements, which are not necessarily listed or described herein, nor necessarily recited in the claims.
[0040] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a better understanding of the present application. It should be apparent, however, that the present application can be practiced without these specific details.
[0041] In short, the main idea of the present disclosure can be summarized as follows.
[0042] A typical frame captured by a camera sensor usually comprises an exposure period, a readout period and a blanking period. The solution proposed by the present disclosure is based on implementing an automatic algorithm able to determine the frame properties of each of the different optical sensors included in the embedded system, including the different exposure over time, and based on the determined frame properties, to synchronize the readout period of the frames. Once the readout period is synchronized, the processing period of all the camera modules overlaps to some extent, and therefore their blanking periods also overlap to some extent. Thus, during the overlapping blanking periods, the embedded system is able to turn off the internal modules that are not used and achieve the required energy saving.
[0043] Preferably, the present solution relies on receiving a "frame end" indication. Once the "frame end" indication is received, the timing at which the 3D optical device processor receives a complete frame is marked, which corresponds to the end of readout timing.
[0044] Figure 1 A flowchart of a method for reducing power consumption in an optical device built according to an embodiment of the present invention is shown.
[0045] First, an optical device comprising a plurality of optical sensors (e.g. camera modules) is provided (step 100), wherein at least one of the optical sensors is characterized in that it cannot be activated in response to receiving an external trigger signal, and wherein each of the optical sensors is associated with a specific frame property, the specific frame property comprising an exposure period, a readout period and a blanking period.
[0046] Then the specific frame property of each of the plurality of optical sensors is acquired (step 110), and based on the acquired information, the readout period of all the optical sensors is synchronized (step 120).
[0047] After synchronizing the readout period of all the optical sensors, at least a partial overlapping processing period of frames associated with at least part of the optical sensors is determined (step 130).
[0048] After determining the at least a partial overlapping processing period of frames associated with at least part of the optical sensors, at least a partial overlapping blanking period of the optical sensors is determined (step 140).
[0049] During the at least a partial overlapping blanking period, internal modules within the optical device that are not used during the at least a partial overlapping blanking period are turned off, thus obtaining energy saving in the operation of the optical device (step 150).
[0050] In summary, the method comprises three main stages:
[0051] (i) Definition of the triggering scheme;
[0052] (i) learning and calculation of the relative delays; and
[0053] (iii) synchronization of the related camera modules.
[0054] Phase I - definition of the triggering scheme:
[0055] a) According to the method proposed in the present application, different combinations of the plurality of camera modules are defined based on the properties of the following frames. For each of these different combinations, a virtual trigger is defined. The term "virtual trigger" used in the description and claims refers to a software timer that acts as a trigger.
[0056] b) Next, for each of these different combinations, a "anchor" camera module (master module) is selected and the remaining camera modules belonging to a particular combination are defined by the processor as camera modules that are triggered with respect to the anchor camera module (slave modules) or as camera modules that are free running.
[0057] c) For each of these combinations, a score is determined based on a mechanism such as the greatest common divisor (hereinafter "GCD") of the number of frames per second ("FPSs") of the different participants.
[0058] d) The combination associated with the lowest score is selected.
[0059] Phase II - learning and calculation of the relative delays
[0060] a) After the combination associated with the lowest score has been selected, the processor is configured to trigger the different camera modules (optical sensors) according to the definition of the virtual trigger associated with the selected combination.
[0061] b) Next, the timestamps associated with the "end of frame" signals received from the different optical sensors are collected.
[0062] c) Then, the processor calculates for the different slave camera modules the relative delay required to make the timing of the end of frame of the "slave camera module" the same as the timing of the end of frame of the master camera module (anchor camera module).
[0063] Phase III - synchronization of the related sensors
[0064] The delays calculated at the end of phase III are then applied and the 3D optical device is ready to operate in power saving mode.
[0065] Preferably, each time a channel / optical sensor is removed from the 3D optical device or added to the 3D optical device, the above-described process is repeated.
[0066] Figure 2A method for reducing the power consumption of a 3D optical device constructed according to an embodiment of the present application is exemplified. According to this embodiment, clock gating is combined with the triggering process of the present disclosure.
[0067] By using the above Figure 2 embodiment process, the start of the readout of the optical sensors can be synchronized, thus reducing the time period of the HW clock on, as can be seen from Figure 2 and thus reducing the power used.
[0068] As can be seen from Figure 2 the "on" time is the total duration of all readouts. In the upper part of the figure, the case where the start of the readout of the different optical sensors is not synchronized is shown, thus the duration of the "on" is longer, while in the lower part of the figure, once the start of the readout between all optical sensors is synchronized, their total "on" time is still reduced due to the overlap of the readout times, even though their exposure times are different.
[0069] The synchronization of the operation of the various optical sensors can be performed automatically by tagging all the optical sensors that participate in the establishment of the common virtual trigger process, while other optical sensors can still be manually triggered, while there are other optical sensors that can be excluded from the triggering process.
[0070] Adding / removing channels / optical sensors
[0071] The addition or removal of channels / optical sensors can be performed as follows:
[0072] After determining the possible combinations of the frame properties based on the plurality of camera modules, a common virtual trigger is established for all optical sensors belonging to a certain combination and an anchor optical sensor is selected for the combination, determining whether the selected optical sensor supports the trigger.
[0073] Example 1 - Selecting a virtual trigger for optical sensors included in a 3D optical device.
[0074] Let us consider the example of a device comprising 3 optical sensors, of which 2 do not support the trigger mode, while the third one does. The method of selecting two optical sensor combinations is shown below.
[0075] The first step of the process is to score each possible combination as part of the process of selecting the best combination:
[0076]
[0077] where:
[0078] n - the number of optical sensors that participate in the particular combination for which the score is calculated;
[0079] GCD - greatest common divisor;
[0080] For optical sensors not participating in the combination, their frames are added;
[0081] Next, the combination of optical sensors associated with the lowest score is selected.
[0082] The method described here can also take into account the duration of the readout period and the imaging depth and vision engine (“IDVE”) HW’ used by each optical channel, as well as the power consumption of each of these HW’s.
[0083] Figure 3 An example of the results of the steps performed as part of the method according to an embodiment of the application, which determines the common frame by implementing a GCD calculation, is shown. The frame rates of the three optical sensors included in this example are 10, 35 and 50 frames per second.
[0084] As can be seen from this figure, the GCD result scoring the combination of the 50 and 30 frames per second optical sensors is:
[0085] GCD(50, 30) = 5.
[0086] This results in a common frame of 200 milliseconds being used.
[0087] However, the GCD result scoring the combination of the 50 and 10 frames per second optical sensors is:
[0088] GCD(50, 10) = 10.
[0089] This results in a common frame of 100 milliseconds being used.
[0090] After selecting the combination with the lowest score, there are many options.
[0091] I) First, if the optical sensor selected as the anchor of the selected combination supports trigger mode, a software timer is determined as a virtual trigger for that group of optical sensors.
[0092] In this case, if another optical sensor belonging to the combination supports trigger mode, an additional software timer is determined as a trigger, while in this case, if the other optical sensor does not support trigger mode, this optical sensor is determined as a free-running sensor.
[0093] II) In the case where the optical sensor selected as the anchor of the selected combination does not support trigger mode, the anchor sensor is set as a free-running sensor, while for the other optical sensors, the anchor sensor link unit (“SLU”), which is responsible for synchronizing the input sensor data to the clock domain of the system and converting the pixels to a common format, is selected as the trigger source.
[0094] III) in case none of the optical sensors comprised in the group support a trigger mode, all optical sensors belonging to the group are determined to be free-running sensors.
[0095] When a software timer is used as a trigger source, the application preferably further comprises a step of determining its repetition rate, while each optical sensor will be triggered according to its FPS / LCM, where LCM, least common multiple, also known as the smallest positive integer that is a multiple of two (or more) integers a and b, is the smallest positive integer that is divisible by both.
[0096] Next, the following embodiment provides a method of determining the delays of various optical sensors as acquired from an anchoring optical sensor in order to maximize the overlap existing at the sensor readout period. Preferably, according to this embodiment, the following steps are taken:
[0097] (i) initializing the optical setup operation by using a known (e.g. previously obtained) trigger configuration;
[0098] (ii) collecting timestamps from all optical sensors at the frame start interrupt until a number of (e.g. at least 4) samples are obtained from each optical sensor;
[0099] (iii) calculating the delay between each optical sensor and the known trigger configuration;
[0100] (iv) imposing a forced delay to the optical sensor that is the anchoring optical sensor; and
[0101] (v) restarting the optical setup operation while using the updated forced delay.
[0102] Figure 4 An example of a method for calculating the trigger delay is shown, the method having the following steps: acquiring the timestamps of the anchoring optical sensor, finding the closest timestamp to the anchoring optical sensor by calculating the difference between the respective timestamps and the timestamp of the anchoring optical sensor, and determining the delay equal to the difference between the timestamps.
[0103] Figure 5 An example of implementing the method provided by the present disclosure is shown, wherein the anchoring optical sensor has a rate of 20 frames per second (“FPS”), and the other two optical sensors of the example, optical sensor A and optical sensor B, have a rate of 20 FPS and 10 FPS, respectively. As can be seen from the figure, even though optical sensor A and optical sensor B have different frame rates, by applying the solution provided by the present application, optical sensor A and optical sensor B still have the same delay, which ultimately enables a reduction in the power consumption of the optical setup comprising the exemplary optical sensors.
[0104] In the specification and claims of this application, each of the verbs, "comprise" "contain" and "have," and conjugates thereof, are used to indicate that the object or objects of the verb are not necessarily a complete list of members, components, elements or parts of the subject or subjects of the verb. The application has been described using detailed descriptions of embodiments thereof that are presented in an illustrative manner. Such embodiments are presented for the purposes of illustration and not limitation, and the application is not limited to the embodiments described herein, which include different features and are presented for the purpose of illustration rather than limitation. Some embodiments of the application utilize only some of the features or possible combinations of the features. Variations of those described embodiments and other implementations of the application, including implementations using different combinations of the described features, will occur to persons of ordinary skill in the art. The scope of the application is limited only by the following claims.
Claims
1. A method for operating energy conservation of an optical device, the optical device comprising a plurality of optical sensors, the method comprising: determining a frame property of each of the plurality of optical sensors, and based on the determined frame properties, synchronizing a readout period of the frames; after synchronizing the readout period of the frames, determining a processing period of each of the plurality of optical sensors such that the processing periods at least partially overlap each other; and, obtaining an at least partially overlapping blank period of all of the plurality of optical sensors, and wherein during the at least partially overlapping blank period, internal modules comprised within the optical device that are not used during the at least partially overlapping blank period are switched off.
2. A method for operating energy saving of an optical device, the optical device comprising a plurality of optical sensors, wherein, at least one optical sensor of the plurality of optical sensors is characterized in that the at least one optical sensor is not capable of being activated in response to receiving an external trigger signal, wherein the method comprises the following steps: (i) providing an optical device comprising a plurality of optical sensors, each optical sensor being associated with a specific frame property, the specific frame property comprising an exposure period, a readout period and a blank period; (ii) acquiring, by at least one processor comprised in the optical device, information associated with the specific frame property of each of the plurality of optical sensors; (iii) based on the acquired information, synchronizing, by at least one processor comprised in the optical device, the readout period of all optical sensors; (iv) after synchronizing the readout period of all optical sensors, determining, by at least one processor comprised in the optical device, an at least partially overlapping processing period of frames associated with at least some of the plurality of optical sensors; (v) after determining the at least partially overlapping processing period of frames associated with at least some of the plurality of optical sensors, determining, by at least one processor comprised in the optical device, an at least partially overlapping blank period of at least some of the plurality of optical sensors; and (vi) during the at least partially overlapping blank period, switching off internal modules within the optical device that are not used during the at least partially overlapping blank period, thereby obtaining energy savings in the operation of the optical device.
3. The method of claim 2, wherein, Step (ii) further comprises the following steps: a) defining different combinations of the plurality of optical sensors based on the frame properties of the plurality of optical sensors, and defining a virtual trigger for each of the different combinations; b) selecting a master optical sensor for each of the different combinations, and defining all other optical sensors belonging to a specific combination as slave optical sensors or as free running optical sensors, the slave optical sensors being optical sensors that are triggered with respect to the master optical sensor; c) for each of the different combinations, determining a score based on a mechanism for calculating a frames per second (FPSs) score of all optical sensors belonging to each of the different combinations; and d) selecting the combination associated with the lowest score obtained.
4. The method of claim 3, wherein, The mechanism for calculating a frame per second (FPSs) fraction of all optical sensors belonging to each combination of the different combinations is the greatest common divisor (GCD).
5. The method of claim 3, wherein, Step (iii) further comprises the steps of: (i) upon selection of the combination associated with the lowest fraction obtained, triggering the plurality of optical sensors according to the virtual trigger defined for the selected combination; (ii) collecting time stamps associated with "end of frame” signals received from the plurality of optical sensors; and (iii) calculating a relative delay needed to synchronize the end of frame of the slave optical sensor with the end of frame of the master optical sensor.
6. The method of claim 5, wherein, After applying the calculated relative delay, the method is repeated each time an optical sensor is removed from the optical device or added to the optical device.