Electronic device, method and computer program product

Abstract

An electronic device comprising a circuit configured to perform smoke detection based on a depth image and a confidence image captured by an iToF sensor, and to obtain a smoke detection status.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of time-of-flight imaging, and more particularly to devices, methods and computer programs for time-of-flight imaging. [Background technology]

[0002] A Time-of-Flight (ToF) camera is a range image camera system that determines the range of objects in a scene by measuring the time of flight of a light signal between the camera and the object for each point in the image. A Time-of-Flight camera captures a depth image of the scene. Generally, a Time-of-Flight camera has an illumination section that illuminates an area of ​​interest with modulated light and a pixel array that collects light reflected from the same area of ​​interest. That is, a Time-of-Flight imaging system is used to provide depth sensing or distance measurement.

[0003] In indirect time-of-flight (iToF), an iToF camera captures a depth image and a confidence image of a scene, and each pixel of the iToF camera is attributed with its own depth and confidence measurement. This working principle of iToF measurement is used in many applications related to image processing.

[0004] Although techniques exist for image processing using time-of-flight cameras, it is generally desirable to provide better techniques for image processing using time-of-flight cameras. Summary of the Invention

[0005] According to a first aspect, the present disclosure provides an electronic device comprising a circuit configured to perform smoke detection and obtain a smoke detection status based on a depth image and a confidence image captured by an iToF sensor.

[0006] According to a second aspect, the present disclosure provides a method including performing smoke detection based on a depth image and a confidence image captured by an iToF sensor to obtain a smoke detection situation.

[0007] According to a third aspect, the present disclosure provides a computer program comprising instructions, when the program is executed by a computer, to cause the computer to perform smoke detection based on a depth image and a confidence image captured by an iToF sensor, and to obtain a smoke detection status.

[0008] Further aspects are set out in the dependent claims, the following description and the drawings. [Brief description of the drawings]

[0009] Embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates generally the basic operating principles of a time-of-flight imaging system, which can be used to provide depth sensing or distance measurement. [Diagram 2] 1 illustrates a schematic diagram of an embodiment of an iToF imaging system in an in-vehicle scenario, where images captured by the iToF imaging system are used for smoke detection inside the vehicle. [Diagram 3] 1 illustrates a schematic diagram of an embodiment of an interior imaging system comprising a ToF system used for smoke detection inside a vehicle. [Figure 4] 1 illustrates a schematic diagram of an embodiment of a process of smoke detection based on depth and confidence images. [Figure 5a] 1 illustrates in more detail an embodiment of multiple ROIs defined within a confidence image. [Figure 5b] 1 illustrates in more detail an embodiment of multiple ROIs defined in a depth image. [Figure 6] One embodiment of the process of smoke detection described in FIG. 4 is outlined in more detail. [Figure 7a] 1 shows a confidence image produced by an iToF sensor capturing a scene in an in-car scenario. [Figure 7b] 1 shows a depth image produced by an iToF sensor capturing a scene in an in-car scenario. [Figure 8a] One embodiment of the process of smoke detection described in FIG. 4 is outlined in more detail. [Figure 8b] One embodiment of the process of smoke detection described in FIG. 4 is outlined in more detail. [Figure 9] 1 shows a flow diagram visualizing a method for determining a smoke detection situation. [Figure 10] An embodiment of an iToF device capable of implementing the process of smoke detection and smoke detection situation determination is generally described. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Before proceeding to a detailed description of the illustrative embodiment with reference to Figures 1 to 10, some general remarks will be made.

[0011] The present embodiment discloses an electronic device having a circuit configured to perform smoke detection based on a depth image and a confidence image captured by an iToF sensor and to obtain a smoke detection situation.

[0012] The circuitry of an electronic device includes a processor, which may be, for example, a CPU, memory (RAM, ROM, etc.), memory and / or storage, interfaces, etc. The circuitry may comprise or be connected to input means (mouse, keyboard, camera, etc.), output means (display (e.g. liquid crystal, (organic) light emitting diode, etc.)), (wireless) interfaces, etc., as is commonly known for electronic devices (computers, smartphones, etc.). Additionally, the circuitry may include or be coupled to sensors (e.g., image sensors, camera sensors, video sensors, etc.) for sensing environmental parameters (e.g., radar, humidity, light, temperature), etc., for sensing still and moving image data.

[0013] Smoke detection can be performed in the cabin of a vehicle in an in-car scenario, in a room monitoring scenario for security reasons, etc. In the in-car scenario, the iToF sensor can illuminate a reference area, e.g. the dashboard of the cabin, which is in the field of view of the iToF sensor. The dashboard can be used as a reference. This is because the risk of confusion with detected objects, which are usually made of black and non-reflective material and are also present in the field of view of the iToF sensor, is reduced and therefore false positive results can be prevented.

[0014] In such smoke detection processing, the iToF system including the iToF sensor can detect driver and passenger interaction within a reference area, driver or passenger interaction very close to or in front of the TOF sensor, the presence of large objects that can be placed on the dashboard, smoke puffing by the driver and smoke puffing by the passenger, smoke from e-cigarettes and regular cigarettes based cigarettes, two or more hands interacting on the dashboard or even behind, smoke without a cigarette within the field of view of the iToF sensor, diffuse smoke, well-defined clouds of smoke, etc.

[0015] The smoke detection status may be any smoke detection status, such as a status indicating that smoke has been detected, a status indicating that smoke has not been detected, a status indicating that smoke detection is unreliable, etc. The smoke detection status may be output to a user to inform the user of the incidence of smoke. In an in-car scenario, the smoke detection status may be output to the driver / passenger via the infotainment system, for example, by outputting an appropriate sound from the vehicle's loudspeaker array and / or by outputting text or an image on a display of the infotainment system of the invention vehicle. A smoke detection condition can provide a warning to the driver or can activate a safety-related function whenever smoke is detected within the vehicle cabin.

[0016] The circuit may be configured to define regions of interest ROIs in each of the captured depth image and the captured confidence image, and perform smoke detection based on the ROIs defined in the depth image and the confidence image. The number of defined ROIs may be any integer and positive number suitable for performing smoke detection, such as 1, 2,,,, 6, 7,,, etc. The ROIs may be defined within the captured image such that they are adjacent to one another, such as when multiple ROIs are defined.

[0017] The ROIs defined in the depth and confidence images may be ROIs having any size suitable for smoke detection, for example 20x20 pixels.

[0018] Furthermore, the ROIs defined in the depth and confidence images may be ROIs having any shape suitable for performing smoke detection and object recognition, such as circles, ellipses, polygons, lines, polylines, rectangles, hand-drawn shapes, etc.

[0019] According to some embodiments, the ROI in the depth image may be defined at the same location as the ROI defined in the confidence image. Furthermore, the ROI in the depth image and the confidence image may be defined at a fixed location. The location of the ROI defined in the depth image and the confidence image may be a predefined location or may be a real-time defined location. ROIs may be defined within the captured image such as to form groups of ROIs, in which the ROIs may be adjacent to one another or may be defined one far away from the other.

[0020] According to some embodiments, the circuitry may be configured to estimate a confidence value in the confidence image, which may be estimated based on an in-phase amplitude demodulation component I and on a quadrature amplitude modulation component Q, where both the I and Q components depend on phase measurements associated with respective distances calculated using the depth image.

[0021] Furthermore, the confidence value can be estimated based on the variation of light scattering or light reflection. Smoke may be detected based on the variation of light scattering or light reflection, since the brightness of the confidence image may increase due to light reflection. For example, if the brightness values ​​are almost equal everywhere in the confidence image, there may be no smoke incidence but oversaturation due to an object such as a hand or paper being close to the iToF sensor. Typically, smoke does not appear in the depth image, so the presence of an object close to the iToF sensor may increase the confidence value in the confidence image, but may also increase the depth value in the depth image, thus resulting in a smoke detection condition indicative of no smoke.

[0022] Furthermore, if the number of very bright pixels in the confidence image is high and an object is detected, a smoke detection condition may be obtained indicating that smoke is unlikely to be present when the very bright pixels are outside the detected object.

[0023] The circuitry can be configured to calculate a respective confidence value within each ROI defined in the confidence image and perform smoke detection based on the calculated confidence values. For example, the circuitry can calculate a confidence value for each pixel of the iToF sensor and then calculate an average confidence value of all the confidence values ​​of the pixels within each ROI defined in the confidence image.

[0024] Further, the circuitry may be configured to calculate an average confidence value of all the ROIs based on the confidence values ​​of each of the ROIs, and the average confidence value of all the ROIs may be set as a confidence threshold.

[0025] The circuitry may be configured to obtain a smoke detection status indicating smoke is detected when the confidence threshold is reached by the respective confidence values ​​of each ROI in at least the minimum number of ROIs. Further, the circuitry may be configured to obtain a smoke detection status indicating smoke is not detected when the confidence threshold is not reached in at least the minimum number of ROIs, which is estimated by comparing the respective confidence values ​​of each ROI to the confidence threshold.

[0026] The circuitry can be configured to detect the presence of an object based on object detection performed on the depth image. The object detection can be performed based on any object detection method known to those skilled in the art. The object detected from the object detection method can be any object such as a human hand, a human arm, a piece of paper, a human foot, a pet, etc.

[0027] The circuitry can be configured to detect the presence of an object or hand based on depth variations in the depth image, for example, the presence of an object typically causes an increase in the depth value of the region in the depth image where the object is detected, causing depth variations to be detected in the depth image.

[0028] According to one embodiment, the circuitry may be configured to filter out ROIs covered by the detected object to obtain a plurality of remaining ROIs. The circuitry may be configured to filter out any ROIs covered by the detected object to prevent false positives and erroneous smoke detection results.

[0029] The circuitry can be configured to filter ROIs having high depth variation in the depth image to obtain the remaining ROIs. The circuitry may be configured to filter any ROIs having high depth variation in the depth image such that false positives and erroneous smoke detection results are prevented.

[0030] The circuit may be configured to obtain a smoke detection status indicating that smoke detection is unreliable when the number of remaining ROIs is less than a predefined minimum number of ROIs. For example, if the smoke detection status indicates that smoke detection is unreliable, the smoke detection process is paused or stopped. Or, if the iToF sensor is covered by an object or if the dashboard area (ROI) is covered by an object, the smoke detection process is paused or stopped.

[0031] The circuitry may be configured to perform smoke detection based on the variance of each confidence value within the ROI, which variance may be calculated using, for example, a standard deviation function for each confidence value within the ROI in the confidence image.

[0032] According to the above-mentioned embodiments, smoke detection can be performed in low light conditions, night conditions, etc. Depth measurements in the depth image can provide the desired accuracy for classification, and smoke detection based on confidence values ​​in the confidence image can utilize light reflections in / on the smoke. Thus, iToF smoke detection can be considered as a light independent solution.

[0033] In smoke detection processing, the combination of depth image and confidence image may be considered as a double security process to avoid false positive results by determining the presence of smoke based on a confidence value that is independent of any light conditions, and by using the depth measurements of the depth image to exclude objects that may cause modifications of the confidence value in the confidence image.

[0034] The present embodiment also discloses a method for performing smoke detection based on a depth image and a confidence image captured by an iToF sensor to obtain a smoke detection situation.

[0035] The present embodiment also discloses a computer program including instructions, when executed by a computer, to cause a computer to perform smoke detection and obtain a smoke detection situation based on the depth image and the confidence image captured by the iToF sensor, the computer program being capable of implementing any of the processes and / or operations described above or in the detailed description of the embodiments below.

[0036] Hereinafter, the present embodiment will be described with reference to the drawings.

[0037] (Operation principle of indirect time-of-flight imaging system (iToF)) FIG. 1 shows a schematic of the basic operating principle of a time-of-flight imaging system that can be used to provide depth sensing or distance measurement, where the ToF imaging system 1 is configured as an iToF camera.

[0038] The ToF imaging system 1 captures a three-dimensional (3D) image of a scene 7 by analyzing the time of flight of infrared light emitted from an illumination unit 10 to the scene 7. The ToF imaging system 1 includes an iToF camera, e.g., an imaging sensor 2, and a processor (CPU) 5. The scene 7 is actively illuminated with amplitude-modulated infrared light 8 at a predefined wavelength, using the illumination unit 10, e.g., by several light pulses of at least one predefined modulation frequency generated by a timing generator 6. The amplitude modulated infrared light 8 is reflected from an object in the scene 7. A lens 3 collects the reflected light 9 and forms an image of the object on an imaging sensor 2 having a matrix of pixels of the iToF camera. Depending on the distance of the object from the camera, there is a delay between the emission of the modulated light 8, for example a so-called light pulse, and the reception of the reflected light 9 at each pixel of the camera sensor. The distance between the reflecting object and the camera may be determined as a function of the observed time delay and the speed of the light constant.

[0039] The three-dimensional (3D) image of the scene 7 captured by the iToF camera is also commonly referred to as a "depth map," in which each pixel of the iToF camera is attributed its own depth measurement.

[0040] In indirect time-of-flight (iToF) the phase delay between the modulated light 8 and the reflected light 9 is determined for each pixel by sampling the correlation wave between the demodulated signal 4 generated by the timing generator 6 and the reflected light 9 captured by the imaging sensor 2. This phase delay is proportional to the object distance modulo the wavelength of the modulation frequency. A depth map can thus be determined directly from the phase image, which is the collection of all phase delays determined within the pixels of the iToF camera.

[0041] (In-vehicle iToF imaging system) 2 shows a schematic diagram of an embodiment of an iToF imaging system in an in-vehicle scenario, where images captured by the iToF imaging system are used for smoke detection in a vehicle.

[0042] An iToF imaging system 200, e.g. an iToF camera, is fixed to the ceiling of a vehicle. The iToF imaging system 200 comprises an iToF sensor (see 400 in FIG. 4) that captures a predetermined area inside the vehicle, a field of view 201. For example, the iToF imaging system 200 captures within its field of view 201 a dashboard 202 of a vehicle having an infotainment system, such as the infotainment system 301 shown in FIG. 3 below.

[0043] 1 above, an iToF imaging system 200 emits light pulses of infrared light into a predetermined area inside a vehicle by actively illuminating its field of view 201. Objects included in the field of view 201 of the iToF imaging system 200 reflect the emitted light back to the iToF imaging system 200. The iToF imaging system 200 captures a depth map (e.g., a depth image) of a given area inside the vehicle by analyzing the time of flight of the emitted infrared light. Objects included in the field of view 201 of the iToF sensor of the iToF imaging system 200 may be the vehicle dashboard 202, the driver's / passenger's hands, smoke 204, etc.

[0044] The iToF imaging system 200 captures a depth image (i.e., a depth map) and a confidence image of its field of view 201. Within the depth image and the confidence image are defined pixel regions that correspond to predefined regions of interest 203 within the field of view 201 of the iToF imaging system 200, where the predefined regions of interest 203 are preferably located on a dashboard 202 of a vehicle. The dashboard 202 is made of a dark, non-reflective material and can therefore be used as a reference surface (see 202 in FIG. 3) for the area of ​​interest 203. Light emitted from the iToF imaging system 200 that hits the dark, non-reflective surface of the dashboard 202 will not be reflected back to the iToF sensor, thus preventing erroneous depth results.

[0045] The smoke detection process is performed based on the confidence and depth images provided by the iToF imaging system 200, in particular based on the analysis of image regions that correspond to the predefined regions of interest 203.

[0046] FIG. 3 illustrates a schematic of an embodiment of an interior imaging system comprising a ToF system used for smoke detection inside a vehicle.

[0047] The iToF system 200 generates a depth image (see 401 in FIG. 4) and a confidence image (see 402 in FIG. 4) of the reference plane 202 in its field of view (see 201 in FIG. 2). Based on the obtained depth image and the obtained confidence image, the processor 300 performs smoke detection (see 403 in FIG. 4) and obtains a smoke detection situation (see 404 in FIG. 4), as will be explained in more detail in the following FIGS. 4 to 8. Based on the smoke detection situation, the processor 300 controls the vehicle's infotainment system 301 to notify the vehicle driver / passengers about the occurrence of smoke inside the vehicle. The in-vehicle infotainment system 301 provides a combination of entertainment and information delivery capabilities to the driver and passengers. In-vehicle infotainment systems typically provide entertainment and information to the driver and passengers through displays and loudspeakers. Control elements such as a button panel, a touch screen display, voice commands, etc. are provided for the driver and passengers to interact with the in-vehicle infotainment system 301. The infotainment system 301 may include, for example, an embedded multimedia / navigation system. The infotainment system 301 notifies the driver / passengers of the occurrence of smoke, for example, by outputting an appropriate sound from the vehicle's loudspeaker array and / or by outputting text or an image on a display unit of the in-vehicle infotainment system 301. For example, the infotainment system 301 may notify the driver / user of the smoke occurrence by providing a warning or by activating a safety-related function whenever smoke is detected within the cabin of the vehicle. In this way, the driver may be encouraged to stop smoking if smoke is detected, for example, when a child is present in the car (e.g., as detected by a back seat pressure sensor).

[0048] In an in-car scenario, the smoke detection process may detect smoke generated by the driver and / or smoke generated by passengers. The smoke detection process can detect well-defined smoke clouds, diffuse smoke, smoke generated by cigarettes, smoke coming from the vehicle's engine, etc. In particular, the smoke detection process of the present embodiment can detect that a passenger is smoking without the need to detect a cigarette within the field of view of the iToF sensor.

[0049] In the embodiment of Fig. 3, smoke detection is performed in a vehicle interior scenario. Also, in a room security scenario, smoke detection can be performed in the room. If smoke detection is performed in the room, the iToF sensor may be mounted, for example, on the ceiling of the room, or any suitable location. The region of interest may be defined on any suitable reference surface in the room, such as a wall, a table, etc.

[0050] FIG. 4 illustrates a schematic of one embodiment of a process of smoke detection based on depth and confidence images.

[0051] The iToF sensor 400 uses iToF technology to capture a predefined area in its field of view and obtains a depth image 401 and a confidence image 402 of the field of view (see 202 in FIG. 2). Based on the depth image 401 and the confidence image 402 of the captured area, smoke detection 403 is performed and a smoke detection situation 404 is obtained. One embodiment of the process of smoke detection 403 is described in more detail in relation to FIG. 5 below. The process of smoke detection can be performed, for example, in a vehicle interior scenario, a room monitoring scenario, etc.

[0052] The depth image 401 is an image or images that contain information about the distance of the optical center of the camera (e.g. from the iToF sensor 400) of objects (see 7 in Fig. 1) in the scene. The depth image 401 can for example be determined directly from a phase image, which is a collection of all phase delays determined within the pixels of the iToF sensor 400. The confidence image 402 is an image that contains confidence measures related to the depth information.

[0053] (Region of Interest (ROI)) According to an embodiment described in more detail below, in order to perform the smoke detection process, the confidence image of the iToF sensor (see 402 in FIG. 4) and the depth image of the iToF sensor (see 401 in FIG. 4) are analyzed. A number of predefined regions of interest (ROIs) (see 203 in FIG. 2) are defined in the depth image 401 and in the confidence image 402. These regions of interest (ROIs) correspond to reference planes (see 202 in FIG. 2) in the field of view of the iToF sensor.

[0054] In an in-vehicle scenario, an iToF sensor mounted on, for example, the ceiling of a vehicle cabin captures its field of view (e.g., a predefined area) (see 201 in FIG. 2) and generates a depth image (see 401 in FIG. 4) and a confidence image (see 402 in FIG. 4) of the captured scene. The captured scene is, for example, the dashboard of the vehicle (see 202 in FIG. 2). Therefore, since a vehicle dashboard is typically a dark, non-reflective material, a predetermined number of ROIs 203, i.e., n ROIs 203, are defined within the region of the dashboard in each confidence image and depth image. To improve smoke detection results and prevent false positive smoke detection results, during smoke detection processing 403, the n ROIs have fixed positions in the confidence image 402 (see FIG. 5a) and the depth image 401 (see FIG. 5b). Furthermore, the positions of the n ROIs 203 on the dashboard (see 202 in FIG. 2) defined in the confidence image 402 (see FIG. 5a) are the same as the positions of the n ROIs 203 defined in the depth image 401 (see FIG. 5b).

[0055] 5a and 5b show a schematic representation of one embodiment of a predefined number of ROIs defined in each confidence and depth image.

[0056] FIG. 5a shows in more detail an embodiment of multiple ROIs defined within the confidence image. The confidence image depicts the dashboard with small portions of the dashboard appearing as black in the confidence image and other portions appearing as light grey or white in the confidence image, where black indicates a high confidence value and light grey or white indicates a low confidence value. Black indicates that these portions are located close to the iToF sensor. The indication "False" in the confidence image is the final output of the smoke detector after its entire evaluation is completed, so in this embodiment, no smoke is detected.

[0057] In the embodiment of Fig. 5a, a predetermined number n of ROIs 203 are defined in a confidence image generated by an iToF sensor (see 400 in Fig. 4). The ROIs 203 are represented by rectangular boxes, each of which represents a respective region of interest. The number n of ROIs 203n is an integer, preferably n>1, where the number n of ROIs is equal to 7, i.e. n=7, indicating the number within each rectangular box, which represents each ROI 203. The first six ROIs 203-1 to 203-6 that make up the group of ROIs are adjacent to each other and are defined within the area of ​​the vehicle dashboard 202. A seventh ROI 2037, which is also within the area of ​​the dashboard 202 shown in the confidence image, is defined further away than the first six ROIs 203-1 to 203-6.

[0058] For example, when the driver's hand or head or an object is very close to the iToF sensor, strong reflections occur along with scattering of light effects. This increases the brightness of the entire relatively uniform confidence image. This arrangement of ROIs 203n at different positions in the confidence image (ROIs 203-1 to 203-6 constituting a group of ROIs, and ROI 203-7 further away) makes it possible to distinguish between a uniform increase in brightness from, for example, a hand very close to the iToF sensor and an increase in brightness that has variations and is caused by light reflections from smoke.

[0059] FIG. 5b shows in more detail an embodiment of multiple ROIs defined in the depth image. In the depth image, the dashboard is depicted as in the confidence image for Fig. 5a above, where parts of the dashboard appearing as black in the depth image indicate that this part is located closer to the iToF sensor. The number of ROIs 203 defined in the depth image 401 is the same as the number of ROIs 203 defined in the confidence image 402, as described in Fig. 5a above. That is, the number n of ROIs 203 defined in the depth image 401 is n=7, as shown as the number in each rectangular box, which represents each ROI 203. Furthermore, the ROIs 203 defined in the confidence image are the same as the ROIs 203 defined in the depth image, and in both images, the ROIs 203 have the same fixed position.

[0060] 5a and 5b, the number n of ROIs 203 is equal to 7, i.e., n = 7, without limiting the embodiment in that respect. Alternatively, the number n of ROIs 203 defined in the depth image and the confidence image may be any number suitable for the case.

[0061] In the embodiment of Figures 5a and 5b, the shape of the ROI 203 is rectangular, without limiting the invention in this respect. The shape of the ROI 203 may be any suitable shape, including a circle, an ellipse, a polygon, a line, a polyline, a rectangle, a hand-drawn shape, etc. The size of the ROI 203 may be any size suitable for the desired detection and calculation. For example, the size of each ROI 203 may be 20x20 pixels, which may relate to a length of approximately 1-2 cm on the dashboard. The resolution of the iToF sensor can be any suitable resolution, such as a videographic loudspeaker array resolution, a higher resolution than VGA resolution, or a lower resolution, for example, the resolution may be up to 1.8M pixels, without limiting the present embodiment in this respect.

[0062] Each ROI may be defined, for example, as a rectangular box having a size larger than the pixel size to avoid introducing noise into the values.

[0063] (Smoke detection method) FIG. 6 outlines in more detail one embodiment of the process for smoke detection described in FIG. 4 above.

[0064] In this embodiment, an iToF sensor (see 400 in Fig. 4) illuminates the interior scene (see 201 in Fig. 2) in its field of view and captures a depth image (see 401 in Fig. 4) and a confidence image (see 402 in Fig. 4) of the field of view. A predefined number n of Regions of Interest (ROIs) are defined in each of the confidence image and the depth image. The n ROIs may for example be adjacent to each other and the n ROIs of the confidence image may be defined at the same and fixed position in the depth image (see Fig. 5a, b).

[0065] At 600, a predetermined minimum number m is obtained. This minimum number m indicates the minimum number of valid ROIs that are deemed necessary for meaningful smoke detection. This minimum number m may, for example, be set in advance (at manufacture, system setup, etc.) as a predetermined parameter of the process. At 601, a confidence value C ROI,j is calculated for each ROI j of the n ROIs defined in the confidence and depth images. At 602, object detection is performed in the depth image to detect objects such as hands within the field of view of the iToF camera and ROIs, any covered by the object / hand, are filtered, resulting in a number h of valid ROIs. The filtered ROIs are considered invalid and are not considered further for smoke detection. In 603, the confidence threshold C tot is the confidence value C of each valid ROI defined in the confidence image. ROI,j is calculated for smoke detection based on m. At 604, if the number of valid ROIs h is greater than m, the method proceeds to 605. If the number of valid ROIs h is less than m, the method proceeds to 608 where a smoke detection status is determined indicating that smoke detection is unreliable. In 605, the confidence values ​​C calculated in 601 are calculated for at least m ROIs. ROI,j The confidence threshold C calculated in 603 tothas been reached. If the result at 605 is yes, the method proceeds to 607. At 607, a smoke detection status is determined, indicating that smoke has been detected. If the result at 605 is no, the method proceeds to 606. At 606, a smoke detection status is determined, indicating that smoke has not been detected.

[0066] According to one embodiment, the reliability of a pixel is calculated based on the in-phase amplitude demodulation component and the quadrature amplitude modulation component of the pixel, and is given by Equation 1:

number

[0067] In 601, for each ROI j, C ROI,j The average confidence value in may be calculated, for example, as Equation 2:

number

number

[0068] For all n ROIs in the confidence image, C tot The average confidence value of may be determined at 603 as Equation 4.

number

[0069] The confidence variation in a set of ROIs can be calculated by the standard deviation function as Equation 5.

number

[0070] In the embodiment of Figure 6, object detection is performed in the depth image to detect objects such as hands within the field of view of the iToF camera and ROIs, any covered by the object / hand, are filtered. In an alternative embodiment, alternatively or additionally, the depth variation in each of the n ROIs is determined and those ROIs with too large depth variation are ignored.

[0071] In the embodiment of Fig. 6, object detection such as hand detection is performed and the depth variation of each of the n ROIs is determined. Based on the depth variation in each of the n ROIs, the presence of an object / hand in the captured image is detected. If an object / hand is detected, the respective ROI is excluded from smoke detection (filtering). This can prevent an object or hand from causing variations in light scattering or light reflection in each of the n ROIs that would give a false positive result in smoke detection.

[0072] (Object detection and ROI filtering) An embodiment of object detection performed in process 602 of Figure 6 will now be described in more detail. Object detection can be performed based on any object detection method known to those skilled in the art. As an example of an object detection method, the published paper "Sliding Shape for 3D Object Detection in Depth Images" by Shuran Song and Jianxion Xiao, Proceedings of the 13th European Conference on Computer Vision (ECCV2014), is mentioned.

[0073] FIG. 7a shows a confidence image generated by an iToF sensor capturing a scene in an in-car scenario, and FIG. 7b shows the corresponding depth image. The scene consists of the vehicle dashboard 202, the vehicle driver's right hand 701 and the driver's right leg 702. An object / hand recognition method is preferably performed on the depth image (see Fig. 7b). If a hand-like object is detected, the object detection process provides an activity bounding box 700 for the detected hand in the confidence image 402 (see Fig. 7a). A predetermined number n of ROIs 203n, here n=7, are defined in the confidence image. In Fig. 7a, each ROI is represented by a rectangular box 203-1 to 203-7, such that seven rectangular boxes are shown in Fig. 7a. Six ROIs 203-1 to 203-6 are adjacent to each other to form a group of ROIs, and a seventh ROI 203-7 is defined further away in the confidence image.

[0074] If the active bounding box containing the detected hand overlaps with one or more of the n ROIs 203, these overlapped ROIs 203 are not considered for smoke detection. They are filtered as described in 602 of Fig. 6. If an object / hand is detected and the bounding box 700 covers at least one of the defined ROIs 203, the covered ROI 203 is not considered for further smoke detection or the smoke detection process 403 is paused or stopped. The ROI 203 in the depth image is used to observe the occlusion caused by the detected object / hand, thus avoiding false detections.

[0075] Alternatively, the smoke detection method is stopped as it is deemed unreliable when the active bounding box 700 covers at least one of the n ROIs, or when the active bounding box covers all of the nh ROIs, where h (the remaining ROIs after filtering) is an integer and 1< h < n, or when the active bounding box covers all of the n ROIs.

[0076] As mentioned above, each of the ROIs 203 in the depth image has the same coordinates as in the confidence image. Smoke will not appear in the depth image, as it is considered noise and filtered out. Objects such as hands and fingers will appear in both images. This avoids false detections from detected fingers and hands (shown here in black).

[0077] FIG. 8a illustrates in schematic form in more detail one embodiment of the process of smoke detection described in FIG. In this implementation, the standard deviation function s is used to calculate the variance of confidence values ​​for a set of ROIs.

[0078] The embodiment of FIG. 8a is similar to the embodiment of FIG. 6 above, with the addition of the steps of FIG. 6, in which the confidence value variation s in a set of defined ROIs is calculated to determine the smoke detection status.

[0079] A predetermined minimum number m of valid ROIs required for meaningful smoke detection is given in 600. The confidence value C for each ROI j of the n ROIs is ROI,j and the confidence threshold C tot is the confidence value C ROI,j are calculated in 601 and 603 for smoke detection. The ROIs covered by object detection are excluded and the number h of remaining ROIs is obtained in 602. If the confidence value variation s is greater than a predetermined threshold at 800, the method proceeds to 608 where it is determined that the smoke detection is indicative of unreliable smoke detection. If the confidence value variation s is less than the predetermined threshold at 800, the method proceeds to 604, and if the number of valid ROIs h is greater than m at 604, the method proceeds to 605. If the number of valid ROIs, h, is less than m, the method proceeds to 608 where a smoke detection status is determined indicating that the smoke detection is unreliable. At 605, the respective confidence values ​​C ROI,j is the confidence threshold C tot has been reached. If the outcome at 605 is yes, the method proceeds to 607. At 607, a smoke detection status is determined indicating that smoke has been detected. If the outcome at 605 is no, the method proceeds to 606. At 606, a smoke detection status is determined indicating that smoke has not been detected.

[0080] The confidence value variation s in a set of defined ROIs is calculated by the confidence value C ROI,j Based on,and the confidence threshold C tot It is calculated based on: The confidence value variation s in a set of ROIs can be calculated by the standard deviation function as Equation 6:

number

[0081] During the above-mentioned smoke detection, the smoke detection situation is determined based on the depth variation within the n ROIs to measure the variation s of the light reflection, and based on the (average) confidence value in all the n ROIs in one image (without relying on the depth image), using the above-mentioned standard deviation function s. That is, the average confidence value C totThe variation s in is compared to a smoke detection threshold, which may be any suitable smoke detection threshold.

[0082] FIG. 8b illustrates in more detail and generally one embodiment of the process of smoke detection described in FIG. The embodiment of Figure 8b is similar to the example of Figure 6 above, and in addition to the steps of Figure 6, the number of bright pixels in the confidence image is calculated to determine the smoke detection status. In this example, prior to the steps performed with respect to Figure 6 above, at 801, if the number of bright pixels calculated in the confidence image is greater than a threshold, the method proceeds to 802. At 802, the presence of smoke is deemed unlikely and the smoke detection process is paused or stopped. If at 801 the number of bright pixels calculated in the confidence image is less than a threshold, the method proceeds to 600. At 600, the method proceeds as described in the embodiment of FIG.

[0083] In the embodiment of Fig. 8b, when the number of those calculated bright pixels exceeds a predefined threshold for bright pixels, the determination of the presence of smoke is paused or stopped because the presence of smoke is unlikely or the smoke detection is considered unreliable (see 608 in Fig. 6). If the number of bright pixels exceeds the threshold, there is a risk of false positives due to the detection of an object approaching the iToF sensor and scattering light. This is because light scattering within the ROI is usually detected when an object approaches the iToF sensor.

[0084] (calibration) The iToF system 200 described in the above embodiment is calibrated, for example, by capturing images and performing background subtraction on the captured images. A rectangular shaped ROI 203 can be defined in each of the confidence images and in the depth image based on the subtracted background. The calibration can be performed using any other calibration method known to those skilled in the art.

[0085] Fig. 9 shows a flow chart visualizing the method for determining the smoke detection status. In 900, for example in a car scenario, a depth image (see 401 in FIG. 4) and a confidence image (see 402 in FIG. 4) are acquired by an iToF sensor (see 400 in FIG. 4) capturing a scene (see 202 in FIG. 2) in its field of view (see 201 in FIG. 2). In 901, smoke detection (see 403 in FIG. 4) is performed as described above in FIG. 4 and FIG. 6. At 902, a smoke detection status (see 404 in FIG. 4) is generated based on the smoke detection result obtained at 901. The smoke detection status may be, for example, smoke detection is unreliable, for example, no smoke detected, or, for example, smoke detected, as described in FIG. 5 above.

[0086] (implementation) FIG. 10 illustrates generally one embodiment of an iToF device capable of implementing the process of smoke detection and smoke detection situation determination as described above. The electronic device 1200 includes a CPU 1201 as a processor. The electronic device 1200 further includes an iToF sensor 1206 and a convolutional neural network (CNN) unit 1209 connected to the processor 1201. The processor 1201 may perform, for example, smoke detection 403, which realizes the process described with respect to Figures 3 and 4 in more detail. The CNN 1209 may be, for example, an artificial neural network in hardware, such as a neural network on a GPU, or other hardware dedicated to implementing artificial neural networks. The CNN 1209 may thus be an algorithm accelerator that allows the technology to be used in real time, for example a neural network accelerator. The electronic device 1200 further comprises a user interface 1207 connected to the processor 1201. The user interface 1207 serves as a man-machine interface and allows interaction between an administrator and the electronic system. For example, an administrator can use the user interface 1207 to configure the system. The electronic device 1200 further comprises a Bluetooth interface 1204, a WLAN interface 1205 and an Ethernet interface 1208. These units 1204, 1205 function as input / output interfaces for data communication with external devices. For example, a video camera with an Ethernet, WLAN or Bluetooth connection can be connected to the processor 1201 via these interfaces 1204, 1205 and 1208. The electronic device 1200 further comprises a data storage unit 1202 and a data memory 1203 (here a RAM). The data storage unit 1202 is configured as a long-term memory for storing, for example, algorithm parameters of one or more use cases, for recording iToF sensor data obtained from an iToF sensor 1206 and provided by a CNN 1209, etc. The data memory 1203 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1201 .

[0087] It should be noted that the above descriptions are merely example configurations: alternative configurations may be implemented using additional or other sensors, storage devices, interfaces, etc.

[0088] It should be understood that the above-described embodiments describe methods with exemplary ordering of method steps. However, the particular ordering of method steps is given for illustrative purposes only and should not be construed as binding. For example, step 601 in FIG. 6 may be performed after step 603.

[0089] It should also be noted that the division of the electronic control device of Figure 10 into multiple units is done for illustrative purposes only, and the present disclosure is not limited to any particular division of functionality in particular units. For example, at least some of the circuits may be implemented by respectively programmed processors, field programmable gate arrays (FPGAs), dedicated circuitry, etc.

[0090] All units and entities described in this specification and recited in the accompanying claims may be implemented, for example as integrated circuit logic on a chip, unless otherwise stated, and the functions provided by such units and entities may be implemented by software, unless otherwise stated.

[0091] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and the transmission, storage, or other media on which such computer programs are provided, are contemplated as aspects of the present disclosure.

[0092] The present technology can also be configured as follows. (1) A circuit configured to perform smoke detection (403) based on a depth image (401) and a confidence image (402) captured by an iToF sensor (400) and obtain smoke detection status (404, 606, 607, 608). Electronic devices. (2) the circuitry is configured to define a region of interest (ROI) (203) in each of the captured depth image (401) and the captured confidence image (402), and to perform the smoke detection (403) based on the ROIs defined in the depth image (401) and the confidence image (402). 1. An electronic device according to claim 1. (3) The ROI (203) in the depth image (401) is defined at the same position as the ROI (203) defined in the reliability image (402). An electronic device according to (1) or (2). (4) The ROI (203) in the depth image (401) and the confidence image (402) is defined at a fixed position. An electronic device according to (2) or (3). (5) The circuit calculates the confidence values ​​(C, C) in the confidence image (402). ROI,j , C tot ) is constructed to estimate The electronic device according to any one of (1) to (4). (6) The circuit calculates a respective confidence value (C) in each of the ROIs (203) defined in the confidence image (402). ROI,j ) and calculate the calculated reliability value (C ROI,j ) based on the smoke detection (403). (2) An electronic device according to (1). (7) The circuit calculates the confidence value (C ROI,j ) based on the average confidence value (C tot ) is configured to calculate (6) An electronic device according to (6). (8) Confidence threshold (C tot ) is the average confidence value (C tot ) (7) An electronic device according to (7). (9) The circuit calculates the confidence value (C) for each ROI (203) in at least a minimum number (m) of ROIs. ROI,j ) to calculate the confidence threshold (C tot ), a smoke detection status (607) is obtained indicating that smoke has been detected. (7) An electronic device according to (7). (10) The circuitry determines the confidence threshold (C) for the minimum number (m) of ROIs (203). tot) is not reached, a smoke detection status (606) indicating that smoke is not detected is obtained. (7) An electronic device according to (7). (11) The circuit is configured to detect the presence of an object based on object detection performed on the depth image (401). (2) An electronic device according to (1). (12) The object is a hand. (11) An electronic device according to (11). (13) The circuitry is configured to detect the presence of an object or a hand based on depth variations in the depth image (401). (2) An electronic device according to (1). (14) The circuit is configured to filter the ROIs (203) covered by the detected object to obtain a number (h) of remaining ROIs. (11) An electronic device according to (11). (15) The circuit is configured to filter ROIs (203) having high depth variation in the depth image (401) to obtain a number (h) of remaining ROIs. (2) An electronic device according to (1). (16) The circuitry is configured to obtain a smoke detection status (608) indicating that the smoke detection is unreliable when the number h of the remaining ROIs (203) is less than a predetermined minimum number (m) of ROIs (203). (12) An electronic device according to (12). (17) The circuit calculates the confidence values ​​(C ROI,j ) based on the variation (s) of (6) An electronic device according to (6). (18) A method comprising: performing (901) smoke detection (403) based on a depth image (401) and a confidence image (402) captured by an iToF sensor (400); and obtaining a smoke detection status (404). (19) A computer program comprising instructions that, when the program is executed by a computer, cause the computer to carry out the method according to (18). (20) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a computer, causes the computer to perform the method according to (18).

Claims

1. An electronic device comprising: a circuit configured to perform smoke detection and obtain a smoke detection status based on a depth image and a confidence image captured by an iToF sensor, the electronic device comprising: The circuitry is configured to define a region of interest ROI in each of the captured depth image and the captured confidence image, and to perform the smoke detection based on the ROIs defined in the depth image and the confidence image, and to detect the presence of an object based on object detection performed on the depth image. Electronic devices.

2. The ROI in the depth image is defined at the same position as the ROI defined in the confidence image. The electronic device of claim 1 .

3. The ROI in the depth image is defined at the same position as the ROI defined in the confidence image. The electronic device of claim 1 .

4. The ROIs in the depth image and the confidence image are defined at fixed positions. The electronic device of claim 1 .

5. The circuitry is configured to estimate confidence values ​​in the confidence image. The electronic device of claim 1 .

6. The circuitry is configured to calculate a respective confidence value in each of the ROIs defined in the confidence image and perform the smoke detection based on the calculated confidence values. The electronic device of claim 1 .

7. The circuitry is configured to calculate an average confidence value for all ROIs based on the respective confidence values ​​of the ROIs.

7. The electronic device of claim 6.

8. The confidence threshold is set as the average confidence value of all ROIs.

8. The electronic device of claim 7.

9. The circuitry is configured to obtain a smoke detection status indicating that smoke is detected when a confidence threshold is reached by the respective confidence values ​​of each ROI for at least a minimum number of ROIs.

8. The electronic device of claim 7.

10. The circuitry is configured to obtain a smoke detection status indicating that smoke is not detected when the confidence threshold is not reached for at least a minimum number of ROIs.

9. The electronic device of claim 8.

11. The object is a hand 11. The electronic device of claim 10.

12. The circuitry is configured to detect the presence of an object or a hand based on depth variations in the depth image. The electronic device of claim 1 .

13. The circuitry is configured to filter the ROIs covered by the detected object to obtain a number of remaining ROIs. The electronic device of claim 1 .

14. The circuitry is configured to filter ROIs having high depth variation in the depth image to obtain a number of remaining ROIs. The electronic device of claim 1 .

15. The circuitry is configured to obtain a smoke detection status indicating that the smoke detection is unreliable when a number of remaining ROIs is less than a predetermined minimum number of ROIs.

12. The electronic device of claim 11.

16. The circuitry is configured to perform the smoke detection based on a variation of the respective confidence values ​​within the remaining ROI.

7. The electronic device of claim 6.

17. A method for detecting smoke based on a depth image and a confidence image captured by an iToF sensor to obtain a smoke detection situation, comprising: defining a region of interest (ROI) in each of the captured depth image and the captured confidence image, and performing the smoke detection based on the ROIs defined in the depth image and the confidence image; and detecting the presence of an object based on object detection performed on the depth image.

18. A computer program comprising instructions which, when executed by a computer, cause said computer to carry out the method of claim 17.

Images