Electronic apparatus, mobile device, transparent object detection method, and image correction method

Abstract

The invention relates to an electronic device, a mobile device having the same, a transparent object detection method, and an image correction method. Wherein the electronic device may include circuitry configured to: acquire reflected radiation data received by each pixel from an image captured by a Time of Flight ToF sensor; determining an intensity distribution of pixels in a first region of the image based on the reflected radiation data for each pixel in the first region, the intensity representing a peak of the reflected radiation data for the pixel; and determining whether the image includes a transparent object based on the intensity distribution.

Description

Electronic device, mobile device, transparent object detection method and image correction method

[0001] Citation of Related Applications

[0002] This application claims the benefit of Chinese Patent Application No. 202310041606.9 filed with the State Intellectual Property Office of the People's Republic of China on January 11, 2023, the entire contents of which are hereby incorporated by reference into this document in their entirety. Technical Field

[0003] The present disclosure relates to an electronic device and a mobile device having the electronic device, which can be used for a ToF sensor. The electronic device of the present disclosure can detect whether a transparent object (such as glass, window, etc.) is present in a shooting target of the mobile device. Background Art

[0004] In mobile devices equipped with cameras, it is often necessary to detect the presence of transparent objects in the captured scene. Accurately identifying transparent objects in the scene (to remove or segment them) not only eliminates misunderstandings of the scene caused by transparent objects, but also assists with other visual tasks, such as object detection and image reflection removal. For example, when pedestrians wearing VR helmets or glasses, sweeping or delivery robots, or motor vehicles are moving, if there are transparent obstacles such as large pieces of glass in their path, traditional RGB image sensors, or even the naked eye, will find it difficult to identify them, leading to accidents.

[0005] Furthermore, taking autofocus during photography as an example, mobile devices (such as cameras, mobile phones, and robots and vehicles equipped with photography devices) are equipped with an autofocus system to focus on the main subject in a scene (people, animals, flowers, etc.) when taking a photo. This allows for clear photos to be taken. Currently, photography devices generally use the following different focusing technologies:

[0006] Contrast Detection Auto-Focus (CDAF): This method performs autofocus by searching for the maximum contrast in the image. This is a common method in digital cameras, but finding the optimal state can be a slow process.

[0007] Phase Detection Auto-Focus (PDAF): This method performs autofocus by sensing the phase difference in specific pixels with different microlens orientations. This method is fast, but requires a specific sensor and has difficulty handling some scenes.

[0008] Laser Detected Auto-Focus (LDAF): This method uses a proximity sensor to determine the distance to the target and adjusts the focus accordingly. While this method allows for fast and accurate focusing, it is susceptible to errors when transparent objects (reflective surfaces) are in the path between the proximity sensor and the target. This error can obviously cause the camera to not focus correctly.

[0009] Focusing technology using a depth sensor: This method uses a specific sensor, such as a direct Time of Flight (dToF) sensor or an indirect Time of Flight (iToF) sensor, to obtain distance (depth) data of the main object in the scene to achieve autofocus. This method can achieve focus quickly, but it also requires the use of an external sensor. It can also overcome the shortcomings of PDAF technology in handling certain scenes.

[0010] Furthermore, when using a specific device to detect distance, certain scenarios may be challenging, namely, when the target is behind a transparent object (e.g., glass, window, etc.). For example, when a camera captures a cityscape behind a window, exhibits displayed in glass cases in a museum, and the like, it may be inappropriately focused on the window or glass. Specifically, in a mobile device using a depth sensor, the depth sensor may detect the distance to the transparent object rather than the distance to the target, especially when the distance between the camera and the target is far. This may cause problems when taking photos because the intended focus will be wrong. Therefore, it becomes necessary to detect the presence of a transparent object so that the transparent object can be excluded from the obtained depth data, or to switch to other autofocus methods to prevent this problem.

[0011] If the mobile device only has a depth sensor or the autofocus of the camera relies on depth, it would be beneficial to correct the depth data by removing the depth data of pixels corresponding to transparent objects and replacing them with depth data of background objects.

[0012] Summary of the Invention

[0013] In view of the above problems, the present disclosure is directed to using a depth sensor (specifically, a time-of-flight ToF sensor) to detect the presence of a transparent object (eg, glass, window, etc.).

[0014] To this end, the present disclosure provides an electronic device. According to one embodiment of the present disclosure, the electronic device includes a module that can be implemented as a circuit. The circuit is configured to: obtain reflected radiation data received by each pixel from an image captured by a ToF sensor; determine an intensity distribution of pixels in a first region of the image based on the reflected radiation data of each pixel in the first region, where the intensity represents a peak value of the reflected radiation data received by the pixel; and determine whether the image includes a transparent object based on the intensity distribution.

[0015] The circuit of the electronic device according to the present disclosure may be further configured to: determine a difference in pixel intensity in the first area according to the intensity distribution; and determine whether the image includes the transparent object based on the difference.

[0016] The circuit of the electronic device according to the present disclosure may also be configured to: when the difference is greater than a predetermined threshold, determine that the image includes the transparent object; when the difference is less than or equal to the predetermined threshold, determine that the image does not include the transparent object.

[0017] The circuit of the electronic device according to the present disclosure may be further configured to determine the difference according to a maximum value intensity Max_Intensity and at least one quantile intensity Quantile_Intensity in pixel intensities of the first region.

[0018] The circuit of the electronic device according to the present disclosure may also be configured to determine the difference according to the maximum value intensity Max_Intensity and a quantile intensity Quantile_Intensity in the pixel intensity of the first area. When the ToF sensor is a direct time of flight dToF sensor, the difference may be defined as The calculated ratio.

[0019] When the ToF sensor is an indirect time-of-flight iToF sensor, the difference can be defined as The calculated ratio is: distance represents the shooting distance of the iToF sensor.

[0020] The circuit of the electronic device according to the present disclosure may be further configured to: detect a maximum intensity pixel in the image, and determine a selected area surrounding the maximum intensity pixel as the first area.

[0021] The circuit of the electronic device according to the present disclosure may be further configured to: detect a maximum intensity pixel within a region of interest in the image, and determine a selected region surrounding the maximum intensity pixel as the first region.

[0022] According to the present disclosure, when the ToF sensor is a dToF sensor, a peak value of a histogram of pixels of the dToF sensor corresponds to the intensity, and the circuitry of the electronic device may be configured to: detect one or more pixels in the image having a peak value greater than one in the histogram, select a maximum-intensity pixel from among the one or more pixels, and determine a selected area surrounding the maximum-intensity pixel as the first area. Preferably, the circuitry is further configured to select the maximum-intensity pixel from among pixels within a region of interest in the image having a peak value greater than one.

[0023] The circuit of the electronic device according to the present disclosure may be further configured to: when the result of the judgment indicates that a transparent object is detected, determine a position distribution of pixels with different intensities in the first area, and correct the result of the judgment based on the position distribution.

[0024] The circuit of the electronic device according to the present disclosure may also be configured to: To determine the position distribution, where Ix = Σmry 2 , m represents the intensity of each pixel in the first region, ry represents the distance between the pixel and the first rotation axis y of the first region, and Iy=Σmrx 2 , m represents the intensity of each pixel in the first region, rx represents the distance between the pixel and the second rotation axis x of the first region, and wherein Iy>Ix. When the value is greater than or equal to a predetermined threshold, the circuit is configured to determine that no transparent object is detected and correct the result of the determination.

[0025] The circuit of the electronic device according to the present disclosure may be further configured to pre-process the radiation reflection data of each pixel to remove radiation reflection data representing a cover glass of the ToF sensor before determining the intensity distribution.

[0026] According to the present disclosure, when the ToF sensor is a dToF sensor, the peak of the histogram of the pixels of the dToF sensor corresponds to the intensity, and wherein the circuit of the electronic device may also be configured to: when the result of the judgment shows that a transparent object is detected, obtain all pixels in the image having a peak value greater than one peak value, reorganize all pixels having a peak value greater than one peak value into a plurality of pixel clusters based on the distance data of the pixels in the image, the plurality of pixel clusters respectively corresponding to different distances, select a pixel cluster corresponding to the transparent object from the plurality of pixel clusters as a transparent object pixel cluster, remove the first peak value of the pixels in the transparent object pixel cluster from the image or output the depth corresponding to the first peak value of the pixels in the transparent object pixel cluster as the distance of the transparent object and output the depth corresponding to the second peak value of the pixels in the transparent object pixel cluster as the distance of the target object.

[0027] The electronic device according to the present disclosure may be implemented as the ToF sensor.

[0028] According to another aspect of the present disclosure, an electronic device is provided. The electronic device may include circuitry configured to: obtain a histogram of each pixel from an image captured by a dToF sensor, obtain all pixels having two peaks in the histogram, obtain a first group of pixels from all pixels having two peaks, the first group of pixels being pixels whose first peak of the two peaks is the highest peak, and remove the first peak of the first group of pixels from the image.

[0029] The circuit of the electronic device according to the present disclosure is further configured to pre-process the histogram of each pixel before acquiring all pixels having two peaks, so as to remove the peak representing the cover glass of the dToF sensor from the histogram.

[0030] The circuit of the electronic device according to the present disclosure is further configured to: select pixels corresponding to transparent objects from all pixels having two peaks, and select one or more pixels whose first peak is the highest peak from the pixels corresponding to the transparent objects as the first group of pixels.

[0031] The circuit of the electronic device according to the present disclosure is also configured to: based on the distance data of the pixels in the image, reorganize all the pixels with two peaks into multiple pixel clusters, the multiple pixel clusters respectively corresponding to different distances, select a pixel cluster corresponding to the transparent object from the multiple pixel clusters, and select one or more pixels whose first peak is the highest peak from the pixel cluster corresponding to the transparent object as the first group of pixels.

[0032] According to the electronic device of the present disclosure, the transparent object is determined based on the intensity distribution of pixels in the first area of ​​the image, the intensity representing the peak value of the reflected radiation data received by the pixels. According to another aspect of the present disclosure, a mobile device is also provided, comprising the electronic device described above.

[0033] According to the mobile device of the present disclosure, wherein the mobile device is a photographing device, the autofocus unit of the photographing device is configured to perform autofocus based on the judgment result of the electronic device, and wherein, when the judgment result of the electronic device shows that a transparent object is detected, the autofocus unit is configured to automatically focus on the background of the transparent object.

[0034] According to the mobile device of the present disclosure, the photographing device is a mobile phone.

[0035] According to the mobile device of the present disclosure, the shooting device includes the ToF sensor, the shooting device corrects the depth map generated by the ToF sensor based on the judgment result of the electronic device, and the autofocus unit is configured to perform autofocus based on the corrected depth map.

[0036] According to the mobile device of the present disclosure, the mobile device is a vehicle, virtual reality VR glasses / helmet or a self-moving robot.

[0037] According to another aspect of the present disclosure, a method for detecting transparent objects is provided. The method may include: acquiring reflected radiation data received by each pixel in an image captured by a time-of-flight (ToF) sensor; determining an intensity distribution of pixels in a first region of the image based on the reflected radiation data for each pixel in the first region, the intensity representing a peak value of the reflected radiation data for the pixel; and determining whether the image includes a transparent object based on the intensity distribution.

[0038] According to the transparent object detection method disclosed herein, a difference in pixel intensity in the first region is determined based on the intensity distribution, and based on the difference, whether the image includes the transparent object is determined. According to the transparent object detection method disclosed herein, when the difference is greater than a predetermined threshold, the image is determined to include the transparent object; when the difference is less than or equal to the predetermined threshold, the image is determined to not include the transparent object.

[0039] According to the transparent object detection method disclosed herein, the difference is determined based on a maximum intensity Max_Intensity and at least one quantile intensity Quantile_Intensity among the pixel intensities of the first region. According to the transparent object detection method disclosed herein, the difference is determined based on a maximum intensity Max_Intensity and at least one quantile intensity Quantile_Intensity among the pixel intensities of the first region.

[0040] According to the transparent object detection method disclosed in the present invention, wherein the ToF sensor is a direct time of flight dToF sensor, and the difference is defined as The calculated ratio.

[0041] According to the transparent object detection method disclosed in the present invention, wherein the ToF sensor is an indirect time-of-flight iToF sensor, and the difference is defined as The calculated ratio is: distance represents the shooting distance of the iToF sensor.

[0042] According to the transparent object detection method disclosed herein, the method further comprises: detecting a maximum intensity pixel in the image, and determining a selected area surrounding the maximum intensity pixel as the first area.

[0043] The transparent object detection method according to the present disclosure further includes: detecting a maximum intensity pixel within a region of interest in the image, and determining a selected region surrounding the maximum intensity pixel as the first region.

[0044] According to the transparent object detection method disclosed herein, the ToF sensor is a dToF sensor, the peak of the histogram of the pixels of the dToF sensor corresponds to the intensity, and the method further includes: detecting one or more pixels in the image having more than one peak in the histogram, selecting a maximum intensity pixel from the one or more pixels, and determining the selected area surrounding the maximum intensity pixel as the first area.

[0045] According to the transparent object detection method disclosed herein, the method further comprises: selecting the maximum intensity pixel from pixels with more than one peak value within the target area in the image.

[0046] According to the transparent object detection method disclosed herein, the method further includes: when the judgment result shows that a transparent object is detected, determining the position distribution of pixels with different intensities in the first area, and correcting the judgment result based on the position distribution.

[0047] According to the transparent object detection method disclosed in the present invention, the method further comprises calculating To determine the position distribution, where Ix = Σmry 2 , Iy=Σmrx 2 , m represents the intensity of each pixel in the first region, rx represents the distance between the pixel and the first rotation axis x of the first region, ry represents the distance between the pixel and the second rotation axis y of the first region, and wherein Iy>Ix.

[0048] According to the transparent object detection method disclosed in the present invention, the method further includes: When the value is greater than or equal to a predetermined threshold, it is determined that no transparent object is detected, and the result of the determination is corrected.

[0049] According to the transparent object detection method disclosed herein, the method further comprises pre-processing the reflected radiation data of each pixel to remove the reflected radiation data representing the cover glass of the ToF sensor before determining the intensity distribution.

[0050] According to the transparent object detection method disclosed in the present invention, wherein the ToF sensor is a dToF sensor, the peak of the histogram of the pixels of the dToF sensor corresponds to the intensity, and wherein the method further includes: when the judgment result shows that a transparent object is detected, obtaining all pixels in the image with a peak value greater than one peak value, based on the distance data of the pixels in the image, reorganizing all pixels with a peak value greater than one peak value into multiple pixel clusters, the multiple pixel clusters respectively corresponding to different distances, selecting a pixel cluster corresponding to the transparent object from the multiple pixel clusters as a transparent object pixel cluster, removing the first peak value of the pixels in the transparent object pixel cluster from the image or outputting the depth corresponding to the first peak value of the pixels in the transparent object pixel cluster as the distance of the transparent object and outputting the depth corresponding to the second peak value of the pixels in the transparent object pixel cluster as the distance of the target object.

[0051] According to another aspect of the present disclosure, an image correction method is proposed, which includes: obtaining a histogram of each pixel from an image captured by a dToF sensor, obtaining all pixels having two peaks in the histogram, obtaining a first group of pixels from all the pixels having two peaks, the first group of pixels being pixels whose first peak among the two peaks is the highest peak, and removing the first peak of the first group of pixels from the image.

[0052] According to the image correction method disclosed herein, the method further includes pre-processing the histogram of each pixel before acquiring all pixels having two peaks, so as to remove the peak representing the cover glass of the dToF sensor from the histogram.

[0053] According to the image correction method disclosed herein, it also includes: selecting pixels corresponding to transparent objects from all pixels having two peaks, and selecting one or more pixels whose first peak is the highest peak from the pixels corresponding to the transparent objects as the first group of pixels.

[0054] According to the image correction method disclosed herein, it also includes: based on the distance data of the pixels in the image, reorganizing all the pixels with two peaks into multiple pixel clusters, the multiple pixel clusters corresponding to different distances, selecting a pixel cluster corresponding to the transparent object from the multiple pixel clusters, and selecting one or more pixels whose first peak is the highest peak from the pixel cluster corresponding to the transparent object as the first group of pixels.

[0055] According to the image correction method disclosed herein, the transparent object is determined based on the intensity distribution of pixels in the first area of ​​the image, and the intensity represents the peak value of the reflected radiation data received by the pixel. According to the electronic device disclosed herein, it is possible to detect whether a transparent object exists in an image captured by a ToF sensor. This is very beneficial in a mobile device having a shooting device. The electronic device disclosed herein can enable the shooting device to appropriately perform autofocus based on transparent object detection, thereby obtaining a clear image of the target object. In addition, the electronic device disclosed herein can identify transparent objects in a scene, thereby assisting in removing or segmenting the transparent object from the scene.

[0056] Detailed embodiments of the present disclosure will be described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG. 1 is an example diagram illustrating a histogram of pixels of a dToF sensor.

[0058] FIG. 2 is a schematic diagram illustrating a pixel histogram when a transparent object exists according to an embodiment of the present disclosure.

[0059] FIG3 is a schematic diagram showing a specific embodiment of the second scenario according to C in FIG2 .

[0060] FIG4 is a schematic diagram showing the ranging principle of an iToF sensor;

[0061] FIG5 is another schematic diagram illustrating the ranging principle of an iToF sensor;

[0062] FIG. 6 is a diagram illustrating dToF sensor data for different materials according to an embodiment of the present disclosure.

[0063] FIG. 7 is a graph showing pixel intensity distributions for different materials in the rightmost column of FIG. 6 .

[0064] FIG8 is a schematic diagram illustrating selection of quantile values ​​according to the present disclosure.

[0065] FIG. 9 is a diagram showing how the quantile ratios of different materials in FIG. 6 vary with distance.

[0066] FIG. 10 is a diagram illustrating iToF sensor data for different materials according to an embodiment of the present disclosure.

[0067] FIG. 11 is a diagram showing how the quantile ratios of the different materials in FIG. 10 vary with distance.

[0068] FIG12 is a diagram illustrating dToF depth data for glass and edges according to an embodiment of the present disclosure.

[0069] FIG. 13 is a schematic diagram showing the distribution of pixel intensities of the glass and the edge of FIG. 12 .

[0070] FIG. 14 is a schematic diagram illustrating a depth map obtained by a dToF sensor according to an embodiment of the present disclosure.

[0071] FIG. 15 is a schematic diagram illustrating depth map correction according to an embodiment of the present disclosure.

[0072] FIG. 16 is a schematic diagram illustrating an area of ​​interest according to an embodiment of the present disclosure.

[0073] FIG17 is a schematic block diagram illustrating an electronic device and a mobile device according to one embodiment of the present disclosure.

[0074] FIG18 is a schematic block diagram illustrating a mobile device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Unless otherwise specified, the same reference numerals represent the same or equivalent components.

[0076] [ToF sensor]

[0077] Commonly used depth sensors include dToF sensors and iToF sensors, etc. The present disclosure proposes to detect the presence or absence of transparent objects using radiation data reflected from objects obtained by dToF sensors or iToF sensors. More specifically, transparent object detection is performed using the peak value (i.e., "intensity") of the reflected radiation data received by each pixel in the pixel array of the ToF sensor. The intensity of a pixel corresponds to the maximum value of the amount of light (i.e., the amount of radiation) received by the pixel and returned to the pixel. Since the principles of sensing the depth of the target object by the dToF sensor and the iToF sensor are different, they will be explained separately below.

[0078] dToF sensor

[0079] dToF sensors measure the depth (distance) of a scene by directly capturing the time it takes for light (triggered by an illuminator) to return to the sensor (thus capturing the photon's time of flight). This measurement is performed by counting the number of photons received by each sensor pixel in different time periods (called time bins) and generating a per-pixel histogram. The histogram plots photon counts over time, with time as the horizontal axis and photon counts as the vertical axis. When the target is not transparent, the pixel's histogram has a maximum value (i.e., peak), as shown in Figure 1, A. The horizontal axis value corresponding to this peak represents the time it takes for light from the illuminator to strike the object and return. Using the speed of light, the object's distance can be determined. Thus, a dToF sensor depth map is generated, consisting of the distance (depth) for each pixel. If the target is behind a transparent object, the pixel's histogram will contain more than one local maximum (i.e., more than one peak), depending on transmission and reflection. This includes peaks originating from the transparent object (the first peak, shown in Figure 1, B) and peaks originating from objects behind the transparent object (the second peak, shown in Figure 1, B). The horizontal axis of the histogram in Figure 1 represents time, and the vertical axis represents peak height (also known as "intensity"). The peaks shown in the histograms of Figures 1 and 2 are spike-shaped, not column-shaped. This is because the time bin corresponding to the peak is a time period, and the peak represents the trend of the histogram at countless time points within that time period. Therefore, the histogram of the dToF sensor appears spike-shaped.

[0080] When there are transparent objects and background objects in the scene, the depth of the two can be detected by the dToF sensor. The depth of the two can be detected at the same time. At this time, two peaks may appear in the histogram of a single pixel, representing the transparent object and the background object respectively. The depth map obtained by the dToF sensor is generated by the maximum intensity of each pixel (that is, the maximum peak in the pixel histogram). Therefore, when the peak representing the transparent object is the maximum peak in the pixel histogram, using the depth map obtained by the dToF sensor to perform autofocus will inappropriately focus on the transparent object instead of the photographed object (background object). In addition, when two peaks appear in a single pixel, it will also affect the ranging function of the ToF sensor, that is, it will be impossible to determine the peak corresponding to the distance of the target object. Therefore, it is necessary to detect transparent objects in the scene.

[0081] Taking a camera as an example, when there is a transparent object in the shooting target, it is necessary to consider using the dToF sensor to detect the transparent object in two different situations. One situation is that the transparent object is glass, transparent resin, plastic or window, and the shooting target as the background is too far behind the glass to be detected by the dToF sensor. In this case, it is obviously necessary to avoid using the depth data measured by the dToF sensor to perform autofocus of the camera. Another situation is that the shooting target as the background can be detected by the dToF sensor, but there is depth data representing the distance to the glass or window in the depth map. When this data is used for autofocus, it is possible that the shooting performance of the camera will be seriously reduced. Therefore, the depth map needs to be corrected (that is, the pixel data representing the transparent object is removed) so that the corrected depth map can be used to perform autofocus.

[0082] The present disclosure proposes to use pixel histogram data of a dToF sensor to determine whether a transparent object exists.

[0083] Specifically, when the histogram of a pixel has only one peak (as shown on the left side of A in Figure 2), the peak represents two possible scenarios: one is the true target (① on the right side of A in Figure 2); the other is a transparent object (② on the right side of A in Figure 2), while the true target (i.e., the background object) is too far away to be detected by the dToF sensor (for example, a distant cityscape outside a window).

[0084] When a pixel's histogram has two peaks, with the first peak being the smaller of the two (as shown on the left side of B in Figure 2 ), there are two possible scenarios. One scenario is that the subject does not contain a transparent object, but rather an edge (e.g., a door frame, a wall corner, etc.) as shown on the right side of B in Figure 2 ①. The other scenario is that the subject contains a transparent object (see ② on the right side of B in Figure 2 ②).

[0085] When a pixel's histogram has two peaks, with the first peak being the larger of the two (as shown on the left side of C in Figure 2), there are two possible scenarios. Similar to the above, one scenario involves an edge rather than a transparent object (see ① on the right side of C in Figure 2). The other scenario involves a transparent object (see ② on the right side of C in Figure 2).

[0086] The intensities of the first peaks representing the edges in ① on the right side of B and C in Figure 2 are different, with the peak intensity of the edge in B being lower. This may be because the number of photons hitting the edge and returning in B is smaller, depending on factors such as the angle between the light emitter and the edge and the reflectivity of the edge material.

[0087] The right side ② of B and C in Figure 2 both represent the situation where a transparent object (for example, glass) exists. The intensity of the transparent object in B is low, while the intensity of the transparent object in C is high. This depends at least in part on the transmittance of different transparent objects, and the number of photons returned from glass with high transmittance is correspondingly small. In addition, it is also partially affected by dust on the glass or other substances that may reflect light. At the same time, it may also be affected by the intensity of the light from the illumination source. When the intensity of the light from the illumination source is high, the number of photons returned from the target is correspondingly large, and vice versa. It is worth noting that there will also be a peak in the histogram of the pixel at a position very close to the start time. This is the photon data returned from the cover glass of the camera. According to the technical solution of the present disclosure, the acquired histogram data has been pre-processed to remove the peak very close to the start time to avoid the influence of the cover glass. In addition, in a specific embodiment, a camera with double-layer cover glass is used. At this time, two peaks very close to each other will appear in the histogram of the pixel at a position very close to the start time. In this case, according to the technical solution in the context of the present disclosure, when pre-processing the pixel histogram data, if two peaks are detected at a position very close to the start time and the distance is less than a predetermined threshold (for example, 20 mm), the electronic device of the present disclosure will regard the two peaks as corresponding to one peak of the cover glass, and remove the two peaks from the histogram data before entering subsequent processing. In addition, consider the case where the transparent object is a thicker single layer of glass or consists of two or three layers that are very close to each other. In an embodiment according to the present disclosure, depending on the resolution mode of the ToF sensor (about 6 cm), the above-mentioned transparent object will still have only one peak in the histogram of the pixel.

[0088] In the case shown in A in Figure 2, it is necessary to determine whether there is a transparent object in the photographed object, that is, to determine whether it is the situation on the right side of A②. In this case, the shooting target cannot be detected by the sensor because the distance is too far. At this time, it is necessary to avoid using depth data for autofocus in the camera, and to make the camera adopt other autofocus methods (for example, CDAF). In other mobile devices such as sweeping robots, it is necessary to use a depth sensor and perform avoidance and other actions based on the judgment of detecting a transparent object. The following description will explain in detail how to determine whether there is a transparent object in the case shown in A in Figure 2.

[0089] In the case shown in B in Figure 2, the intensity of the transparent object (i.e., the first peak) is relatively low. As mentioned above, the depth map of the dToF sensor is generated by the maximum intensity of each pixel. Therefore, the presence of the transparent object in this case has a relatively small impact on the autofocus action based on the depth map. However, it is still beneficial to detect the transparent object and remove the pixel depth representing the transparent object from the depth map, and use the corrected depth map for autofocus, etc. In addition, detecting the transparent object enables the dToF sensor to correctly judge the distance to the target object (and the distance to the transparent object).

[0090] In the case shown in Figure 2C, since the intensity of the transparent object (i.e., the first peak) is relatively high, if the depth map is used for autofocus, it may mistakenly focus on the transparent object instead of the background object (i.e., the object being photographed). Therefore, it is necessary to determine whether the situation shown in Figure 2C exists and to correct the depth map.

[0091] FIG3 shows a specific embodiment of the situation shown in C in FIG2 . In FIG3 , the upper side shows a camera 1, a pixel grid 11 of the camera, a transparent object 2, and a background wall 3. The lower side of the figure shows the histogram data of four mutually adjacent pixels in the pixel grid where a transparent object is detected. As shown in the pixel histogram on the lower side of FIG3 , the difference in intensity between the maximum intensity pixel of the transparent object (i.e., the first peak) and the other pixels adjacent to the maximum intensity pixel is large; while the difference in intensity between the maximum intensity pixel of the background wall and the other pixels adjacent to the pixel is small. That is to say, compared with non-transparent objects, the distribution of pixel intensities obtained according to the pixel histogram of transparent objects is different. The inventors discovered and proposed for the first time that it is possible to judge whether a transparent object exists based on this distribution of pixel intensities.

[0092] iToF sensor

[0093] The iToF sensor emits a modulated infrared light signal into the scene, and then the sensor receives the light signal reflected back from the target in the scene to obtain the depth (distance) of the target. For example, as shown in Figure 4, the distance to the target can be estimated by emitting the following two signals to the target and detecting the coherent signals of the received signals returned by these two signals. The two signals are:

[0094] I: "In Phase" signal

[0095] Q: "Quadrant phase" signal, with a 90-degree offset.

[0096] A coherent signal is obtained using a CPAD (Current Assisted Photonic Demodulator). The received light is "collected" in two different "taps", generally referred to as Tap A and Tap B (as shown in the left figure in Figure 5). The signal difference between Tap A and Tap B gives the coherent signal of the signal from the radiator. This coherent signal is the ToF signal of the iToF sensor. That is, TOF signal = Tap A – Tap B. Thus, the TOF signals of the I signal and Q signal can be obtained respectively through the respective tap signals Tap A and Tap B:

[0097] ITOF = Tap A - Tap B (Phase shift = 0)

[0098] QTOF = Tap A - Tap B (Phase shift = 90 degrees)

[0099] The sum of the absolute values ​​of ITOF and QTOF represents the confidence (Confidence, "C"), that is, C = |ITOF| + |QTOF|. The confidence C of each pixel in the pixel array of the iToF sensor represents the amount of active light returning to the pixel. In this disclosure, similar to the dToF sensor, the amount of this active light (i.e., radiation emitted by the radiator) returning to the pixel (i.e., the confidence C) is referred to as the "intensity" of the pixel of the iToF sensor.

[0100] Furthermore, the sum of Tap A and Tap B gives the total amount of light received by the sensor (including ambient and active light), i.e., the global IR (infrared) image, as shown in the right figure of Figure 5. In other words, IR signal = Tap A + Tap B. The IR signal of each pixel in the iToF sensor's pixel array represents the total amount of active and ambient light returning to that pixel. Therefore, it is also called the "intensity" of the pixel.

[0101] The pixel "intensity" represented by the confidence C and the IR signal, as well as the obtained confidence image and infrared image, can be used for transparent object detection as will be described below.

[0102] [Transparent object detection principle]

[0103] Transparent object detection using dToF sensor data

[0104] The inventors analyzed the distribution of pixel intensity (i.e., peak height in the pixel histogram) of different materials (including glass, whiteboard, elevator, marble, display screen, white wall, etc.). Among them, glass represents a transparent object, while other materials (non-transparent objects) are common everyday objects with smooth surfaces. Here, for the sake of convenience, only the results of four materials including glass (transparent object) are selected for illustration. The results are shown in Figure 6. A in Figure 6 represents glass, B represents an elevator (elevator door surface), C represents marble, and D represents a white wall. The dToF sensor data of the surfaces of the above materials are obtained respectively (the image shown in the left column of Figure 6, which is a pixel intensity graph). The histogram data of all pixels in the image is detected to obtain the highest intensity pixel (i.e., the pixel with the highest peak in the histogram). As stated above, before detecting the highest intensity pixel, the histogram of the pixel is pre-processed to remove the peak representing, for example, the camera cover glass. Therefore, the histogram of each pixel has only one peak (corresponding to the surface of the material being photographed).

[0105] The area around the highest intensity pixel is selected as the selected area. For example, as shown in the middle column of Figure 6, with the highest intensity pixel as the center, the area with a radius of R (R is the number of pixels) around the center pixel is selected as the selected area. In the embodiment shown in Figure 6, R is selected as 5 pixels. The intensities of all pixels in the selected area are arranged, for example, in order from low to high, to obtain the intensity distribution of different materials as shown in the right column of Figure 6. The radius R of the selected area is not limited to 5 pixels. However, if the value of R is too small, the pixel intensity distribution of the material cannot be presented, and if the value of R is too large, the intensity distribution of the material may not be properly characterized in the calculation described below. In a specific embodiment of the present disclosure, a dToF sensor with a 24*24 pixel array is used, and the range of R is set to 3 to 8 pixels, preferably 5 pixels. Of course, R is not limited to a specific value or range, but can be adaptively adjusted according to different sensors and test conditions in the calculation of the predetermined threshold as described below.

[0106] Based on the intensity distributions in the right column of Figure 6, it can be intuitively seen that the intensity distribution of glass (A in Figure 6), a transparent object, is significantly different from that of other materials. The same conclusion can be drawn from the graph in Figure 7. Figure 7 corresponds to the intensity distributions in the right column of Figure 6 and displays the intensity distributions of the four different materials shown in the right column of Figure 6 in the form of curves. Specifically, A in Figure 7 represents glass.

[0107] Based on the pixel intensity distributions of different materials shown in Figures 6 and 7, the inventors propose that the intensity distributions of different materials can be characterized to distinguish transparent objects (glass) from other materials. In other words, by characterizing the intensity distributions (curves) shown in Figures 6 or 7, the differences in pixel intensity between different materials can be displayed, allowing glass to be distinguished from other materials.

[0108] The inventors have found that by using a specific ratio that characterizes the intensity distribution, glass (transparent objects) can be successfully distinguished from these materials in a simple and effective manner. For example, the ratio between the maximum intensity and the power value (for example, the power value is 2) of the intensity of the pixels in the selected area is used. Specifically, among the pixel intensities in the above-mentioned selected area, the maximum intensity Max_Intensity and a quantile intensity Quantile_Intensity between the maximum intensity (highest peak) and the minimum intensity (minimum peak) are selected. The quantile value Quantile_Intensity can be selected as, for example, the median quantile value Median_Intensity (A in FIG8 ), the 7th quantile value 7thdecile_Intensity (B in FIG8 ), and so on.

[0109] The choice of the quantile value is obviously not limited to these two values, but can be any quantile value between the maximum intensity and the minimum intensity, as long as it can appropriately characterize the intensity distribution.

[0110] Then, calculate according to the following formula 1:

[0111] For example, using the median quantile value Median_Intensity, more specifically, the 5th decile value 5th decile_Intensity of the decile, the above-mentioned different materials were measured respectively. Specifically, the inventors used dToF sensors to shoot the above-mentioned materials at different distances (for example, 0.1m, 0.2m, 0.3m, 0.4m, 0.5m, 0.6m, 0.7m, 0.8m, 0.9m and 1m shown in Figure 9) to obtain ToF data of materials at different distances. Based on the pixel histogram data in the ToF data, the material intensity distribution at the corresponding distance is obtained according to the method shown in Figure 6. Then, based on the maximum intensity Max_Intensity and the median quantile intensity Median_Intensity, the ratio at the corresponding distance is obtained by the above formula 1. The results are shown in Figure 9.

[0112] As clearly shown in Figure 9, when shooting at different distances, the ratio of glass is significantly higher than that of other materials. Therefore, the inventors propose that a threshold value can be predetermined, and the threshold value can be used to determine whether there is a transparent object in the shooting target. For example, in this particular embodiment, the ratio is calculated based on Formula 1 using the median quantile Median_Intensity, and the threshold value is selected as 0.015. The inventors conducted many different tests and successfully distinguished glass from other materials by calculating the above formula 1 and comparing it with the threshold value. Of course, the threshold value is not limited to a specific value, but as shown in Figure 9, in this particular embodiment, a threshold value or a threshold range can be selected in the range of 0 to 0.02 (excluding 0). The threshold value or threshold range can be pre-stored in an electronic device according to the present disclosure, or stored in a ToF sensor or a mobile device containing the ToF sensor.

[0113] Similarly, the 7th decile value 7thdecile_Intensity was also used to perform calculations for the above-mentioned different materials (results not shown). The results show that when the threshold is selected within the range of 0.002 to 0.008, glass is distinguished from other materials with an even higher success rate. In this particular embodiment, the threshold value of 0.006 is optimal. In summary, in this embodiment, when the 7th decile value 7thdecile_Intensity is used as the Quantile_Intensity in Formula 1, the best result for distinguishing glass is obtained. Therefore, in this particular embodiment, the quantile value intensity Quantile_Intensity in Formula 1 is preferably 7thdecile_Intensity.

[0114] In addition, the power value of the quantile strength in Formula 1 is not limited to 2. The inventors also used 3 as the power value of the quantile strength. However, in this specific embodiment of the present disclosure, a power value of 2 yielded a better result with a smaller amount of computation.

[0115] In addition, regarding the denominator in Formula 1, intensity values ​​other than percentile values ​​can also be used. For example, the maximum intensity in the selected area is removed (to minimize the impact of the maximum intensity on the average and ratio calculations), and then the average is calculated based on the remaining pixel intensities in the selected area. The average value is used as the denominator in Formula 1. It should be understood that the solution using percentile intensity is relatively more preferred because different percentile intensities can be freely selected according to different situations. This case will be used as an example to continue the explanation below.

[0116] The above-described method for representing the intensity distribution selects two pixel intensity values ​​from the pixel intensities within the selected area. Obviously, the intensity distribution representation is not limited to the above method. For example, more than two pixel intensity values ​​can be used to represent the difference in pixel intensity. Furthermore, the pixel distribution represented by a reference curve can be used to detect transparent objects.

[0117] For example, three pixel intensity values ​​may be used: maximum intensity Max_Intensity, median intensity Median_Intensity, and 7th decile intensity 7th decile_Intensity. In this case, the ratio may be calculated using the following formula 2.

[0118] Similarly, four pixel intensity values ​​may be used: maximum intensity (Max_Intensity), median intensity (Median_Intensity), third decile intensity (3rd decile_Intensity), and seventh decile intensity (7th decile_Intensity). In this case, the ratio may be calculated using the following formula 3.

[0119] However, using Formula 1 using two pixel intensity values ​​to determine whether a transparent object exists is the simplest, fastest, and most effective method.

[0120] The threshold values ​​determined using different pixel intensity values ​​and different numbers of pixel intensity values ​​will also be different. These threshold values ​​can be pre-calculated and stored in the electronic device of the present disclosure. Alternatively, they can also be stored in the ToF sensor or a mobile device including a ToF sensor (e.g., a mobile phone, a camera, or a self-propelled robot).

[0121] It is worth noting that the spirit of the present disclosure is to identify transparent objects by characterizing the pixel intensity distribution obtained by the ToF sensor. Therefore, the above calculation formula is only a preferred embodiment and is not intended to limit the present disclosure. The characterization method of the pixel intensity distribution is not limited to the formula mentioned above, and other calculation methods that can characterize the intensity distribution (curve) can also be used. For example, a correlation operation is performed on the predetermined peak value in the intensity distribution in the above-mentioned selected area obtained by the ToF sensor and the predetermined peak value in the intensity distribution in the selected area of ​​a typical transparent material (for example, glass). If the correlation factor obtained is greater than the threshold, it can be considered that the detected intensity distribution is similar to the intensity distribution of glass, and then it is determined that glass is detected. It is also worth noting that the test results may also be different when using different ToF sensors. Therefore, the selection of the above-mentioned specific threshold is not restrictive. When implementing the present disclosure, pre-testing can be performed based on the specific ToF sensor used to obtain threshold data for the ToF sensor.

[0122] Maximum intensity pixel

[0123] In the embodiment of the dToF sensor described above, the surfaces of different materials are directly imaged so that there is only one peak in the pixel histogram. Then, the maximum intensity pixel (called the "global maximum intensity pixel") is selected from the global pixel intensity map of all pixels.

[0124] However, when actually shooting with a dToF sensor, the histogram of the pixel may have more than one peak. If a pixel with more than one peak (for example, B and C in Figure 2) is detected in the pixel histogram, it is assumed that the pixel with more than one peak is a "transparent pixel". All "transparent pixels" are selected from the global pixel intensity map, and the "transparent pixel" with the maximum intensity is detected therefrom. Then, the area around the maximum intensity "transparent pixel" is selected as the selected area (for example, the area with a radius of R centered on the maximum intensity pixel as stated above for the global pixel intensity), and the pixel intensity distribution in the selected area is determined. Subsequently, the intensity distribution based on the maximum intensity "transparent pixel" is characterized in a manner similar to the above-mentioned characterization of the intensity distribution based on the global maximum intensity pixel.

[0125] Furthermore, when using a dToF sensor to capture and detect transparent objects, only the global maximum intensity pixel may be selected when selecting the maximum intensity pixel. If a "transparent pixel" exists, either the global maximum intensity pixel or the maximum intensity "transparent pixel" may be selected, with the maximum intensity "transparent pixel" being preferably selected. Furthermore, preferably, if a "transparent pixel" exists, the maximum intensity pixel is first selected from the global pixel intensity data for all pixels, and a ratio is calculated based on the aforementioned formula representing the intensity distribution, and compared with the corresponding threshold. Then, "transparent pixels" (pixels with more than one peak) are detected. If a "transparent pixel" exists, the maximum intensity "transparent pixel" is selected from all "transparent pixels," and a ratio is calculated based on the aforementioned formula representing the intensity distribution, and compared with the corresponding threshold. If the results of the two comparisons are the same, i.e., both indicate detection or non-detection of a transparent object, detection or non-detection is confirmed. If the results of the two comparisons differ, the result of the second comparison (i.e., the calculation based on the maximum intensity "transparent pixel") prevails.

[0126] Transparent object detection using iToF sensor data

[0127] Similar to the dToF sensor, the inventors used the iToF sensor to shoot and analyze various materials including glass. Here, only the results of four materials are listed as examples. As shown in Figure 10, A represents glass, B represents elevator (elevator door surface), C represents marble, and D represents white wall. The iToF sensor data of the surfaces of the above materials are obtained respectively (such as the IR map shown in the left column of Figure 10). The infrared signal IR (i.e., "intensity") of all pixels in the IR map is obtained, and the pixel with the highest intensity is obtained therefrom. Of course, optionally, the confidence C image in the iToF sensor data can also be used, and the pixel with the highest confidence C (i.e., "intensity") can be obtained therefrom.

[0128] Similar to the dToF sensor, the area around the highest intensity pixel (for example, the area with a radius of R centered on the highest intensity pixel) is selected as the selected area (as shown in the middle column of FIG10 ). In a specific embodiment according to the present disclosure, an iToF sensor with a 640*480 pixel array is used, and the range of R is set to 60 to 160 pixels. The selected R as shown in the middle column of FIG10 is 100 pixels. Of course, R is not limited to a specific value or range, but can be adaptively adjusted according to different sensors and test conditions in the calculation of the predetermined threshold as described below.

[0129] The intensities of all pixels in the selected area shown in the middle column of FIG. 10 are sorted in order from low to high, and the intensity distribution shown in the right column of FIG. 10 is obtained.

[0130] Unlike dToF sensors, the intensity distribution of iToF sensors is characterized by a ratio calculated using the following formula 4:

[0131] Where Max_Intensity represents the maximum intensity in the intensity distribution, Quantile_Intensity represents the intensity per unit, and distance represents the distance between the iToF sensor and the target object. By imaging different materials at different distances, we obtain depth maps and intensity distribution maps corresponding to each distance. Using Equation 4 above, we calculate the ratio at each distance, and the results are shown in Figure 11.

[0132] Figure 11A shows the result of using the median quantile value Median_Intensity as Quantile_Intensity, and Figure 11B shows the result of using the 7th decile value 7th decile_Intensity as Quantile_Intensity. As shown in Figure 11, the ratio of glass is significantly higher than that of other materials at different shooting distances. Therefore, similar to the dToF sensor, a threshold or threshold range can be selected to detect transparent objects using the iToF sensor. When the ratio calculated by the above formula 4 is greater than the selected threshold, it is determined that a transparent object is present; otherwise, a transparent object is not present. In the specific embodiments shown in Figures 10 and 11, the threshold range can be selected, for example, from 0.2 to 0.6 (when using the median quantile value Median_Intensity) or from 0.1 to 0.3 (when using the 7th decile value 7th decile_Intensity).

[0133] Similar to the dToF sensor, more than two quantile intensities can be selected to calculate the ratio. For example, when the maximum value, the middle quantile, and the 7th quantile of the tenth quantile are selected, the calculation is performed using the following formula 5:

[0134] Likewise, the purpose of calculating the ratio using percentile values ​​is to characterize the intensity distribution of the pixels. As long as the intensity distribution shown in the right column of FIG10 can be properly characterized, the present disclosure is not limited to the above-mentioned formula.

[0135] Similar to dToF sensors, different iToF sensors may obtain different threshold values ​​(ranges). This threshold value (range) is pre-tested using the specific iToF sensor used, and the threshold value data obtained for the specific iToF sensor is stored in the electronic device according to the present disclosure, or in the iToF sensor or a mobile device having the iToF sensor.

[0136] [Distinguishing between transparent objects and edges]

[0137] When using a dToF sensor to detect transparent objects using the method described above, in some cases, false detection may occur, that is, the detection of a transparent object is made even though no transparent object exists. This is caused by the presence of edges (for example, door frames, wall corners, etc.).

[0138] As shown in B and C in Figure 2, when a pixel with more than two peaks is detected, there are two possible situations, namely, a transparent object or an edge. Moreover, the inventors found that the difference in pixel intensity of the edge is similar to the difference in pixel intensity of the transparent object to a certain extent. Specifically, A in Figure 12 shows a global pixel intensity map of the transparent object (glass) (upper side in the figure) and a pixel intensity map based on a selected area (radius R = 5 pixels) around the maximum intensity pixel (lower side in the figure). B in Figure 12 shows a global pixel intensity map of the edge (upper side in the figure) and a pixel intensity map of a selected area (radius R = 5 pixels) around the maximum intensity pixel (lower side in the figure). The pixel intensities in the selected areas of the two are arranged in order from low to high, and the result shown in Figure 13 is obtained (A in Figure 13 is glass and B is edge). As shown in Figure 13, the difference in pixel intensity of the edge is similar to the difference in pixel intensity of the transparent object to a certain extent.

[0139] Therefore, during transparent object detection, a situation may arise where the pixel corresponding to a portion of an edge is obtained when determining the global maximum intensity pixel, and when calculating the pixel intensity difference, the result is also greater than the threshold. Consequently, the edge is mistakenly identified as a transparent object.

[0140] In order to solve the above problem, the inventor proposes to use the moment of inertia to distinguish between edges and transparent objects.

[0141] Although the difference in pixel intensity of the edge and glass in the selected area is similar to a certain extent, as shown in Figure 12, the position distribution of pixels of different intensities of the edge and glass in the intensity map (that is, the position distribution of pixels with similar intensities in the intensity map) is obviously different. The different shades of the grayscale map shown in Figure 12 indicate different pixel intensities. This difference can be characterized by the moment of inertia. The moment of inertia comes from rigid body mechanics, which represents the inertia of an object when it rotates around an axis. The moment of inertia can be used to understand the distribution of mass around the main axis. The moment of inertia I is as follows: I = Σmr 2

[0142] Here, m represents the mass of the particle and r represents the distance of the particle from the main axis of rotation.

[0143] The inventors propose that the moment of inertia can be applied to the field of two-dimensional images, that is, the moment of inertia I is used to characterize the position distribution of pixels of different intensities in the two-dimensional pixel intensity map as shown in Figure 12. At this time, m represents the intensity of each pixel. As shown on the lower side of Figure 12, compared with glass, in the intensity map of the edge, pixels with similar intensities are basically distributed along the axis, similar to the mass distribution of a bar or an oblong; while the distribution of pixels with similar intensities of glass is roughly distributed around the central area, roughly similar to the mass distribution of a sphere. Therefore, the inventors propose that the moment of inertia of an oblong can be used to characterize and distinguish the position distribution of pixels of different intensities of the two. The moment of inertia of an oblong depends on three semi-axes.

[0144] The intensity map to be characterized according to the present disclosure is a two-dimensional plane, so the moment of inertia depends on the two semi-axes in the plane, i.e., the two principal axes of rotation, denoted by x and y, where y is the longer axis. The intensity distribution around these two principal axes of rotation is defined by Ix and Iy: Ix = Σmry 2

[0145] m represents the intensity of each pixel, and ry represents the distance between the pixel and the main axis of rotation y.

[0146] Similarly, Iy=Σmrx 2

[0147] m represents the intensity of each pixel, rx represents the distance between the pixel and the main axis of rotation x, and Iy>Ix. The calculated ratio is compared with the inertia threshold to distinguish glass from edges. When it is greater than or equal to the inertia threshold, it is determined that what is detected is an edge rather than a transparent object.

[0148] The inertia threshold can be predetermined. In the specific embodiment shown in FIG12 , an intensity map region with a radius of R (R=5 pixels) centered on the maximum intensity pixel is selected to determine the inertia threshold. The range of the selected region is not restrictive, but, in general, it is consistent with the selected region when transparent object detection is performed using the intensity difference of the selected region. Of course, a region different from the selected region for transparent object detection can also be used, as long as the position distribution of the edge and the transparent object can be properly characterized for distinction. Calculate the ratio of the moment of inertia Iy to Ix of the selected region of the glass and edge shown in the lower side of FIG12 , respectively.

[0149] If the ratio of Iy to Ix Close to 1, indicating that pixels of different intensities are evenly distributed around the center point, similar to the mass distribution of a flat disk. If it is greater than 3 or 4, it indicates that the intensity is distributed along an axis, which can be judged as an edge rather than glass. Therefore, in a specific embodiment, the present disclosure sets the threshold to 3.0 or 4.0. However, this threshold is not restrictive. When implementing the present disclosure, it can be predetermined and stored according to the specific ToF sensor used and / or the specific test situation. Similar to the ratio threshold, the inertia threshold is also stored in the electronic device of the present disclosure. Optionally, it can also be stored in a ToF sensor or a mobile device including a ToF sensor (for example, a mobile phone, a camera, a self-moving robot, a vehicle, VR glasses / helmets, etc.).

[0150] As described above, by utilizing the comparison of the moment of inertia with a threshold value, it is possible to correct erroneous glass detection due to the presence of an edge.

[0151] [Transparent object detection and depth map correction of dToF sensors]

[0152] FIG14 shows a specific embodiment of a depth map obtained by a dToF sensor. When a transparent object is present in the subject or the shooting path (as shown in B and C in FIG2 ), the transparent object will partially appear in the depth map obtained by the ToF sensor. That is, as circled in A in FIG14 , a “floating point” (corresponding to the pixel depth of the transparent object) may appear in the central area of ​​the pixel depth map. If the depth map is used for, for example, autofocus of a camera, it will mistakenly focus on the transparent object instead of the background (i.e., the shooting target). In addition, if the depth map is used for ranging, the distance to the transparent object will be mistakenly obtained instead of the distance to the target as the background.

[0153] Therefore, it is desirable to detect whether there is a pixel depth corresponding to a transparent object in the pixel depth map. Furthermore, if a transparent pixel is detected, it is desirable to remove the depth of the transparent pixel from the depth map to obtain a corrected depth map (as shown in B in FIG. 14 ).

[0154] Transparent objects can be detected through the steps shown in FIG. 15 , and the depth map can be corrected if necessary.

[0155] [Transparent object detection]

[0156] First, in step 101, a dToF sensor is used to acquire an image of the target object, including a depth map (such as the depth map shown in 101) and other ToF data, such as a pixel histogram and a pixel intensity map. Histogram data for all pixels in the target image is obtained. Preferably, a region of interest (ROI) in the target image can be captured, and histogram data for all pixels in the ROI can be acquired. In a specific embodiment of the present disclosure, as shown in FIG16A, when a dToF sensor with a 24*24 pixel array is used, the ROI can be selected as a region 5 pixels away from each edge of the pixel array. In another specific embodiment of the present disclosure, as shown in FIG16B, when an iToF sensor with a 640*480 pixel array is used, the ROI can be selected as a region 100 pixels away from each edge of the pixel array. Obviously, the selection of the ROI is not limited to the above embodiment, but can be selected based on the actual situation of the specific ToF sensor used to avoid the maximum intensity pixel not being from the target located at the center of the image.

[0157] In addition, in step 101, when using a dToF sensor, the acquired pixel histogram data is pre-processed (not shown) to remove peaks in the histogram that are very close to the start time, that is, peaks corresponding to the camera cover glass. In addition, if two peaks appear very close to the start time in the histogram, this may be due to the use of double-layer cover glass in the camera. In this case, if the distance between the two peaks is less than a predetermined threshold (e.g., 20mm), the two peaks are considered to correspond to the cover glass and are removed from the histogram data.

[0158] As shown in 102, if a transparent object 2 (corresponding to C in FIG. 2 ) is present in the capture path, then pixels with a peak value greater than one exist in the pixel array 11 of the dToF sensor. In step 102, all pixels with a peak value greater than one (i.e., assumed "transparent pixels") are detected and acquired. Next, in step 103, a transparent object is detected. The "transparent pixel" with the maximum intensity among the assumed "transparent pixels" is detected. Preferably, the "transparent pixel" with the maximum intensity is detected within the region of interest (ROI) of the target object image. Next, a selected area surrounding the "transparent pixel" with the maximum intensity is selected, for example, an area with a radius R centered around the "transparent pixel" with the maximum intensity as the selected area. In a specific embodiment according to the present disclosure, a 24*24 pixel array is used, and R is set to a range of 3 to 8 pixels. For example, R can be selected as 2, 3, or 5, etc. However, as stated in the transparent object detection principle above, the selection of R is based on the specific ToF sensor used and is not limited to a specific value or range. The intensities of all pixels within the selected area are obtained and sorted to obtain the intensity distribution of the pixels in the selected area. A ratio representing the difference in pixel intensity within the selected area is calculated (e.g., using Formula 1), and the presence of a transparent object is determined based on a comparison of this ratio with a pre-stored predetermined threshold. For more details, see the "Transparent Object Detection Principle" above and will not be repeated here. Furthermore, the R value used by the ToF sensor for transparent object detection is generally consistent with the R value used when determining the predetermined threshold.

[0159] The situation B in Figure 2 is not shown in step 2. However, in the transparent object detection method described above for C in Figure 2 , selecting the maximum-intensity "transparent pixel" (i.e., a pixel with more than one peak) does not limit the selection to the "transparent pixel" with the first peak as the maximum intensity. Therefore, the above transparent object detection method is also applicable to transparent object detection in the situation B in Figure 2 .

[0160] In step 102, the situation in which a transparent object exists as shown in A in Figure 2, but the background object (target object) cannot be detected by the ToF sensor because the distance is too far is not illustrated. In this case, the pre-processed pixel histogram obtained by the ToF sensor will have only one peak. At this time, in step 103, the pixel with the highest peak value in the pixel histogram is used as the maximum intensity pixel. Preferably, the pixel with the highest peak value in the pixel histogram within the region of interest ROI of the target image is used as the maximum intensity pixel. Then, an area around the maximum intensity pixel, for example, an area with the maximum intensity pixel as the center and a radius of R, is selected as the selected area. Then, similar to the selected area centered on the maximum intensity "transparent pixel" mentioned above, the ratio of the difference in pixel intensity within the selected area is calculated by, for example, Formula 1, and based on the comparison of the ratio with a predetermined threshold, it is determined whether there is a transparent object in the image taken by the ToF sensor.

[0161] In summary, in step 101, when using a dToF sensor for transparent object detection, when selecting the maximum intensity pixel, only the global maximum intensity pixel can be selected. If a "transparent pixel" exists, either the global maximum intensity pixel or the maximum intensity "transparent pixel" can be selected, preferably the maximum intensity "transparent pixel".

[0162] Furthermore, preferably, if "transparent pixels" exist, the maximum intensity pixel is first selected from the global pixel intensity data of all pixels in the target image. The selected region is then acquired based on the maximum intensity pixel, and a ratio is calculated based on the formula representing the intensity distribution, which is then compared with a corresponding threshold. Furthermore, "transparent pixels" (pixels with more than one peak value) within the target image are then detected. If "transparent pixels" exist, the maximum intensity "transparent pixel" is selected from all "transparent pixels." The selected region is then acquired based on the maximum intensity "transparent pixel," and a ratio is calculated based on the formula representing the intensity distribution, which is then compared with a corresponding threshold. If the results of the two comparisons are the same, i.e., both indicate detection or non-detection of a transparent object, detection or non-detection is confirmed. If the results of the two comparisons differ, the result of the second comparison (i.e., the calculation based on the maximum intensity "transparent pixel") prevails.

[0163] In addition, in step 103, the moment of inertia as described above can also be used to correct the result of detecting the transparent object to avoid erroneous transparent object detection due to edges. Specifically, when it is determined that there is a transparent object in step 103, the moment of inertia Ix and Iy of the selected area are calculated and compared. If If the value is greater than or equal to the inertia threshold, it is determined that no transparent object exists (instead, an edge is detected), and the transparent object detection result in step 103 is corrected.

[0164] In addition, FIG15 does not show the case of using an iToF sensor to detect a transparent object. When using an iToF sensor, TOF image data obtained by the iToF sensor photographing the target object can be obtained, including an infrared image, a confidence C image, and an intensity map formed by the confidence C or IR signal (i.e., intensity) of each pixel in the pixel array. Based on the intensity map data, the pixel with the highest intensity is determined. Preferably, the pixel with the highest intensity within the region of interest (ROI) of the target image is detected and determined.

[0165] Next, similar to a dToF sensor, an area surrounding the highest-intensity pixel is selected as a selected area, for example, an area with a radius R centered around the highest-intensity pixel. In a specific embodiment according to the present disclosure, an iToF sensor with a 640*480 pixel array is used, and R can be selected within a range of 60 to 160 pixels, for example, R = 100. However, as mentioned in the transparent object detection principle of the iToF sensor described above, the selection of R depends on the specific pixel array of the iToF sensor used and is not limited to a specific value or range. Then, the intensities of all pixels within the selected area are obtained and arranged in order from low to high to obtain the pixel intensity distribution of the selected area. Next, a ratio representing the difference in pixel intensity within the selected area is calculated using, for example, Formula 4, and the presence of a transparent object in the image is determined based on a comparison of this ratio with a pre-stored predetermined threshold. In addition, similar to a dToF sensor, the R value used by the iToF sensor when performing transparent object detection is generally consistent with the R value used when determining the predetermined threshold of the iToF sensor.

[0166] [Depth map correction of dToF sensor]

[0167] When there are pixels with more than one peak in the depth map obtained using the dToF sensor, the transparent object detection result can be used to correct the depth map, that is, to remove the pixel depth representing the transparent object in the depth map.

[0168] Specifically, all pixels in the image captured by the dToF sensor having more than one peak value are selected. When a transparent object is detected in step 103 according to FIG15 , pixels corresponding to the transparent object are selected from all pixels having more than one peak value in step 104. The pixels corresponding to the transparent object may include the case shown in FIG2 C on the right side ②, that is, the peak value of the transparent object, i.e., the first peak value, is the maximum peak value, and / or the case shown in FIG2 B on the right side ②, that is, the peak value of the transparent object, i.e., the first peak value, is not the maximum peak value.

[0169] Then, as shown in 104, one or more pixels whose first peak is the maximum peak are obtained from the pixels corresponding to the transparent object. Next, in step 105, the first peak of the one or more pixels is removed from the image shown in the left figure of 105, and the other peaks of the one or more pixels are retained. Preferably, the second highest peak of the one or more pixels is used as the depth data of the pixel in the depth map. Additionally, preferably, the pixel whose first peak is the highest intensity among the intensities of the one or more pixels can be further selected from the one or more pixels, and the first peak of this pixel is removed from the depth map to obtain a corrected depth map as shown in the right figure of 105.

[0170] In the above manner, the depth map is corrected as shown in the case of ② on the right side of C in Figure 2. This corrected depth map is very beneficial for depth map-based autofocus, because depth map-based autofocus will focus on the pixel with the maximum intensity.

[0171] Optionally, the first peak of pixels corresponding to transparent objects is directly removed from the depth map. This corrects the depth of the transparent object in either of the scenarios shown in Figures B and C (2) on the right. This depth map, after removing the depth of the transparent object, is also beneficial in aspects such as autofocus and ranging. Furthermore, when transparent object pixels are detected, preferably, as shown in step 1031, all pixels in the image with more than one peak at the same or similar distance (corresponding to the depth in the depth map) are reorganized to obtain multiple pixel clusters corresponding to different distances, as shown in 1031. Pixel clusters with the same or similar distances can be determined based on the location of the time bins corresponding to the peaks in the pixel histogram, i.e., pixels within the pixel clusters have peaks at the same or similar time bins. Next, a pixel cluster corresponding to the transparent object is selected from the multiple pixel clusters, as shown in 1032. Next, in step 104, similarly to the above, one or more pixels whose first peak is the maximum peak are obtained from the pixel cluster corresponding to the transparent object. These one or more pixels are the pixels for which depth correction is to be performed, as shown in 104. Then, in step 105, the first peak of one or more pixels to be depth-corrected is removed from the image, that is, other peaks other than the maximum peak of the pixels to be depth-corrected are used in the depth map. Preferably, in a specific embodiment according to the present invention, the second highest peak is used. Additionally preferably, similarly to the above, a pixel having the highest intensity among the intensities of the one or more pixels with the first peak being further selected from the one or more pixels, and the first peak of this pixel is removed from the depth map to obtain a corrected depth map as shown in the right figure of 105. Thus, as shown in 105 in FIG15 , the pixel depth corresponding to the depth / distance of the transparent object is removed from the original image (left figure) to obtain a corrected depth map (right figure).

[0172] The corrected depth map removes the influence of transparent objects.

[0173] [Electronic device according to the present disclosure]

[0174] FIG17 shows an electronic device 150 according to the present disclosure, which can be implemented as an electronic module that can perform the transparent object detection described above. The electronic device 150 can communicate with a ToF sensor device 1511 to receive data acquired by the ToF sensor device 1511. The ToF sensor device 1511 can be a dToF sensor or an iToF sensor and can be included in a mobile device 151. The mobile device 151 is, for example, a mobile device with a camera, a mobile phone, a self-propelled robot, a vehicle, or the like. The electronic device 150 can also be directly implemented as a ToF sensor device. That is, the ToF sensor device 1511 itself can include the electronic device 150 of the present disclosure. In addition, as shown in the dotted box in FIG17 , the electronic device 150 can also be included in the mobile device 151.

[0175] When performing transparent object detection, the circuit module in the electronic device 150 receives TOF data from the ToF sensor device 1511, such as pixel histograms, pixel intensity maps, and depth maps received from a dToF sensor, or confidence images, infrared images, and pixel intensity maps received from an iToF sensor. The circuit module then obtains the intensities of all pixels within a selected region of the image from the received data and, based on the difference in pixel intensities within the selected region, determines whether a transparent object exists within the captured target / path. Based on the determination result of transparent object detection, the circuit module in the electronic device 150 may correct the depth map obtained from the ToF sensor device 1511 in the manner described with respect to FIG. 15 and send the corrected depth map to the processing circuit 1512 for subsequent processing. Alternatively, the electronic device may directly send the determination result to the processing circuit 1512, which then performs subsequent processing based on the determination result. For example, based on the determination result of transparent object detection, the processing circuit 1512 may remove the depth of pixels representing the transparent object from the depth map, thereby correcting the depth map of the target object. Processing circuit 1512 then performs subsequent processing based on the corrected depth. For example, if mobile device 151 is implemented as a camera or mobile phone, when capturing an image, processing circuit 1512 may instruct the camera's or mobile phone's imaging device to perform autofocus based on the determination result received from electronic device 150, as will be described in detail below with reference to FIG. 18 . Mobile device 151 may also be implemented as a vehicle. In this case, processing circuit 1512 may remove the depth representing the distance of the transparent object from the depth map of ToF sensor device 1511 based on the determination result received from electronic device 150 that a transparent object has been detected, and thereby determine the distance between the vehicle and the target object, for example, based on the corrected depth map. If mobile device 151 is implemented as a self-propelled robot, processing circuit 1512 may determine whether a transparent object is in the robot's path based on the determination result received from electronic device 150. If the determination result indicates that a transparent object has been detected, processing circuit 1512 may, for example, plan the route of the self-propelled robot to bypass the transparent object.

[0176] Mobile device 151 may also be implemented as virtual reality glasses / helmets. In this case, processing circuit 1512 may determine that the target object is a transparent object based on the transparent object detection result received from electronic device 150, and, for example, continue walking or operating around the transparent object. Furthermore, processing circuit 1512 may also determine the distance to the actual target object based on this determination result.

[0177] [Mobile phone with ToF sensor]

[0178] 18 shows a mobile device 160 implemented as a mobile phone (e.g., a smartphone), which includes electronic devices implemented as a circuit module (transparent object detection module) 1610, an imaging module 1611, an autofocus AF module 1612, and a distance measurement module 1613. Imaging module 1611 includes imaging devices such as a ToF sensor and an RGB image sensor.

[0179] The transparent object detection module 1610 receives pixel data from the ToF sensor in the imaging module 1611 and performs the transparent object detection function according to the present disclosure as described above. The transparent object detection module 1610 sends the detection result to the autofocus module 1612, and the autofocus module switches the autofocus mode based on the judgment result. For example, if the pixel histogram received by the circuit module 1610 from the ToF sensor has only one peak (note that the peak corresponding to the camera cover glass has been removed by preprocessing, as described above), and the detection result based on the pixel histogram shows that there is a transparent object, it can be determined that there is a transparent object in the shooting target / path and the shooting target (background object) cannot be detected by the dToF sensor because it is too far away. Then, the AF module 1612 will adopt other autofocus methods based on the judgment result, such as CDAF.

[0180] In particular, when the circuit module 1610 detects a pixel having a peak value greater than one from the data received from the ToF sensor, and the detection result indicates the presence of a transparent object, the circuit module 1610 may correct the depth data obtained by the ToF sensor based on the detection result (i.e., remove the depth of the pixel representing the transparent object in the depth map), and send the corrected depth data to the autofocus module 1612. The autofocus module 1612 performs autofocus based on the corrected depth data.

[0181] The transparent object detection module 1610 can also send the detection results to the ranging module 1613, and the ranging module 1613 performs the ranging function based on the detection results. When a transparent object is detected, the distances of the transparent object and the target object can be obtained respectively. In contrast, in the application of ToF ranging in the prior art, if there is a situation where there are 2 peaks in 1 pixel, since it is not known what objects the two peaks are, a depth (distance) can only be determined by fusion, discarding, etc. However, using the method disclosed in the present invention, the distance of the transparent object and the distance of the target object can be clearly determined in the case of multiple peaks, and preferably, it can also distinguish between a transparent object and an edge and determine the distance of the edge and the distance of the target object.

[0182] Based on the detection result of a transparent object, the transparent object detection module 1610 can process the depth map obtained by the ToF sensor to correct the depth map when a transparent object is detected. The transparent object detection module 1610 can send the corrected depth data to the AF module 1612, which performs autofocus based on the corrected depth data. The transparent object detection module 1610 can also send the corrected depth data to the ranging module 1613 for the ranging module 1613 to perform the ranging function.

[0183] As described above, the electronic device of the present disclosure can detect transparent objects based on data obtained from a ToF sensor. Although the above description focuses on transparent object detection in terms of depth map correction and autofocus, the application of the electronic device of the present disclosure is not limited thereto, but can be applied to any computer vision task that requires determining transparent objects in a scene.

[0184] It should be understood by those skilled in the art that various changes, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors without departing from the scope of the spirit of the present disclosure.

Claims

1. An electronic device, comprising a circuit, wherein the circuit is configured to: Acquire reflected radiation data received by each pixel from an image captured by a time-of-flight ToF sensor; determine an intensity distribution of pixels in a first region of the image based on the reflected radiation data of each pixel in the first region, the intensity representing a peak value of the reflected radiation data of the pixel; and Based on the intensity distribution, it is determined whether the image includes a transparent object.

2. The electronic device according to claim 1, wherein: The circuit is also configured to: Determining the difference of pixel intensities in the first area according to the intensity distribution; and Based on the difference, it is determined whether the image includes the transparent object.

3. The electronic device according to claim 2, wherein: The circuit is configured as: When the difference is greater than a predetermined threshold, determining that the image includes the transparent object; When the difference is less than or equal to the predetermined threshold, it is determined that the image does not include the transparent object.

4. The electronic device according to claim 3, wherein: The circuit is configured to determine the difference according to a maximum value intensity Max_Intensity and at least one quantile intensity Quantile_Intensity in pixel intensities of the first region.

5. The electronic device according to claim 3, wherein: The circuit is configured to determine the difference according to a maximum value intensity Max_Intensity and a quantile intensity Quantile_Intensity in pixel intensities of the first region.

6. The electronic device according to claim 5, wherein: The ToF sensor is a direct time-of-flight dToF sensor, and the difference is defined as Calculated ratio.

7. The electronic device according to claim 5, wherein: The ToF sensor is an indirect time-of-flight iToF sensor, and the difference is defined as The calculated ratio is where distance represents the shooting distance of the iToF sensor.

8. The electronic device according to claim 1, wherein: The circuit is also configured to: detecting the maximum intensity pixel in the image, and A selected area surrounding the maximum intensity pixel is determined as the first area.

9. The electronic device according to claim 1, wherein: The circuit is also configured to: detecting the maximum intensity pixel within the region of interest in the image, and A selected area surrounding the maximum intensity pixel is determined as the first area.

10. The electronic device according to claim 1, wherein: The ToF sensor is a dToF sensor, a peak of a histogram of pixels of the dToF sensor corresponds to the intensity, and wherein the circuit is further configured to: detecting one or more pixels in the image having greater than one peak in the histogram, selecting a maximum intensity pixel from the one or more pixels, and A selected area surrounding the maximum intensity pixel is determined as the first area.

11. The electronic device according to claim 10, wherein: The circuit is also configured to: The maximum intensity pixel is selected from pixels having greater than one peak value within a region of interest in the image.

12. The electronic device according to claim 1, wherein: The circuit is also configured to: When the result of the determination indicates that a transparent object is detected, determining the position distribution of pixels with different intensities in the first area, and A result of the determination is corrected based on the position distribution.

13. The electronic device according to claim 12, wherein: The circuit is configured to calculate To determine the position distribution, where I x =Σmr x 2 , I y =Σmr y 2 , m represents the intensity of each pixel in the first region, r x represents the vertical distance between the pixel and the first rotation axis x of the first region, r y represents the vertical distance between the pixel and the second rotation axis y of the first region, and wherein I y >I x .

14. The electronic device according to claim 13, wherein: when When the value is greater than or equal to a predetermined threshold, the circuit is configured to determine that no transparent object is detected and correct the result of the determination.

15. The electronic device according to claim 1, wherein: The circuitry is further configured to pre-process the reflected radiation data for each pixel to remove reflected radiation data representative of a cover glass of the ToF sensor before determining the intensity distribution.

16. The electronic device according to claim 1, wherein: The ToF sensor is a dToF sensor, a peak of a histogram of pixels of the dToF sensor corresponds to the intensity, and wherein the circuit is further configured to: When the result of the determination shows that a transparent object is detected, all pixels in the image having a peak value greater than one are obtained, Based on the same distance, all pixels with more than one peak value are reorganized into multiple pixel clusters, Selecting a pixel cluster corresponding to the transparent object from the plurality of pixel clusters, Selecting one or more pixels whose first peak is the highest peak from the pixel cluster corresponding to the transparent object, and The first peak of the one or more pixels is removed from the image or the depth corresponding to the first peak is output as the distance of the transparent object and the depth corresponding to the second peak of the one or more pixels is output as the distance of the target object.

17. The electronic device according to any one of claims 1 to 16, wherein: The electronic device is implemented as the ToF sensor.

18. An electronic device, comprising a circuit, wherein the circuit is configured to: Get the histogram of each pixel from the image captured by the dToF sensor, Get all pixels that have 2 peaks in the histogram, A first group of pixels is obtained from all pixels having two peak values, wherein the first group of pixels is pixels whose first peak value is the highest peak value among the two peak values, and The first peak of the first group of pixels is removed from the image.

19. The electronic device according to claim 18, wherein: The circuit is also configured to pre-process the histogram of each pixel before acquiring all pixels having 2 peaks to remove the peak representing the cover glass of the dToF sensor from the histogram.

20. The electronic device according to claim 18, wherein: The circuit is also configured to: Selecting pixels corresponding to transparent objects from all pixels having two peaks, and One or more pixels whose first peak value is the highest peak value are selected from the pixels corresponding to the transparent object as the first group of pixels.

21. The electronic device according to claim 18, wherein: The circuit is also configured to: Based on the same distance, all pixels with two peaks are reorganized into multiple pixel clusters. A pixel cluster corresponding to the transparent object is selected from the plurality of pixel clusters, and One or more pixels whose first peak is the highest peak are selected from the pixel cluster corresponding to the transparent object as the first group of pixels.

22. The electronic device according to claim 20 or 21, wherein: The transparency is determined based on an intensity distribution of pixels in a first region of the image, the intensity representing a peak value of reflected radiation data received by the pixel.

23. A mobile device comprising the electronic device according to any one of claims 1 to 22.

24. The mobile device according to claim 23, wherein: The mobile device is a photographing device, and the auto focus unit of the photographing device is configured to perform auto focus based on the determination result of the electronic device, and Wherein, when the determination result of the electronic device shows that a transparent object is detected, the autofocus unit is configured to automatically focus on the background of the transparent object.

25. The mobile device according to claim 24, wherein: The photographing device is a mobile phone.

26. The mobile device according to claim 24, wherein: The photographing device includes the ToF sensor, and the photographing device corrects the depth map generated by the ToF sensor based on the judgment result of the electronic device, and wherein, The auto focus section is configured to perform auto focus based on the corrected depth map.

27. The mobile device according to claim 23, wherein: The mobile device is a vehicle, glasses / helmet or a self-moving robot.

Images