Electronic device and storage medium

Abstract

The electronic device for compensating for ghost reflections in an image captured by an imaging device includes a processing circuit arranged to weight a compensation target image including ghost reflections using a ghost reflection compensation model relating to the intensity distribution in the image of ghost reflections due to light reflections in the imaging device when capturing an image, and to combine the compensation target image and the weighted image to remove ghost reflections in the image.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to image processing, and more particularly to image compensation processing. [Background technology]

[0002] In recent years, object detection / identification / matching / tracking in still images or a series of moving images (e.g., video) has become a common and important application in the fields of image processing, computer vision, and pattern recognition, and plays a key role. The object may be a human body part, such as a face, hand, or body, or another living organism or plant, or any other object to be detected. Object identification is one of the most important computer vision tasks, and its purpose is to identify or verify a specific object from an input photo / video and accurately know the relevant information of the object. In particular, in some application scenarios, when performing object identification based on an object image captured by an imaging device, it is necessary to accurately identify detailed information of the object from the image and further identify the object accurately.

[0003] However, images acquired by current imaging devices often contain various noises, which can deteriorate image quality, lead to inaccurate and even erroneous details, and even affect the imaging and identification of objects.

[0004] Therefore, improved techniques are needed to improve image processing to further suppress noise.

[0005] Unless specifically stated, it should not be assumed that any method described in this section is prior art merely by virtue of its inclusion in this section. Likewise, it should not be assumed that any prior art is recognized by this section with respect to a problem recognized by one or more methods, unless specifically stated. Summary of the Invention

[0006] One object of the present disclosure is to further suppress noise in images, particularly noise related to ghost reflections, and thereby further enhance image quality by improving image processing. In particular, ghosts may exist in the captured image, which may further degrade the image quality. The present disclosure uses a ghost reflection compensation model to compensate the image, effectively removing the ghosts in the image and obtaining a high-quality image.

[0007] According to one aspect, there is provided an electronic device for compensating for ghost reflections in an image captured by an imaging device, the electronic device including a processing circuit arranged to weight an image to be compensated for, the image including ghost reflections, using a ghost reflection compensation model relating to the intensity distribution in the image of ghost reflections due to light reflections in the imaging device when capturing an image, and to combine the image to be compensated for with the weighted image to remove ghost reflections in the image. According to another aspect, there is provided a method for compensating for ghost reflections in an image captured by an imaging device, the method comprising: a calculation step for weighting a compensation target image including ghost reflections using a ghost reflection compensation model relating to the intensity distribution in the image of ghost reflections caused by light reflections in the imaging device when capturing an image; and a compensation step for combining the compensation target image and the weighted image to remove the ghost reflections in the image.

[0008] According to yet another aspect, an apparatus is provided that includes at least one processor and at least one storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method described herein.

[0009] According to yet another aspect, a storage medium is provided having stored thereon instructions that, when executed by a processor, cause the method described herein to be performed. Other features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. [Brief explanation of the drawings]

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. In the accompanying drawings, like reference numerals indicate like items.

[0011] [Figure 1] Figure 1 shows a schematic diagram of the ToF technology. [Figure 2A] FIG. 2A shows light reflection by a close-range object in close-range object imaging. [Figure 2B] FIG. 2B shows the light reflection by the photographic filter. [Figure 3] FIG. 3 is a schematic diagram illustrating the ghost phenomenon in an image. [Figure 4A] FIG. 4A shows an example of ghost reflections in the confidence and depth images. [Figure 4B] FIG. 4B shows an example of ghost reflections in the confidence and depth images. [Figure 4C] FIG. 4C shows an example of ghost reflections in the confidence and depth images. [Figure 5A] FIG. 5A shows the flow of image processing including scattering compensation in the technical solution of the present disclosure. [Figure 5B] FIG. 5B illustrates an exemplary scattering compensation operation in the proposed technique of the present disclosure. [Figure 5C] FIG. 5C shows the result of scattering compensation in the technical solution of the present disclosure. [Figure 6] FIG. 6 shows a flowchart of a ghost reflection compensation method according to an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a block diagram of electronic equipment capable of performing ghost reflection compensation according to an embodiment of the present disclosure. [Figure 8] FIG. 8 shows a model diagram of ghost reflection compensation according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates the extraction of a ghost reflection compensation model according to an embodiment of the present disclosure. [Figure 10A]FIG. 10A is a schematic diagram illustrating an exemplary basic flow for extracting a ghost reflection compensation model from a calibration image according to an embodiment of the present disclosure. [Figure 10B] FIG. 10B is a schematic diagram illustrating an exemplary image rotation operation according to an embodiment of the present disclosure. [Figure 10C] FIG. 10C is a schematic diagram illustrating an exemplary image rotation operation according to an embodiment of the present disclosure. [Figure 11] FIG. 11 shows a flow of image processing including ghost reflection compensation according to an embodiment of the present disclosure. [Figure 12] FIG. 12 shows a flow of image processing including ghost reflection compensation according to an embodiment of the present disclosure. [Figure 13] FIG. 13 shows the results of performing ghost reflection compensation according to an embodiment of the present disclosure. [Figure 14A] FIG. 14A illustrates ghost reflection compensation for a dToF sensor according to an embodiment of the present disclosure. [Figure 14B] FIG. 14B illustrates ghost reflection compensation for a dToF sensor according to an embodiment of the present disclosure. [Figure 15] FIG. 15 illustrates ghost reflection compensation for spot ToF according to an embodiment of the present disclosure. [Figure 16] FIG. 16 illustrates an imaging device according to an embodiment of the present disclosure. [Figure 17] FIG. 17 shows a block diagram of an exemplary hardware configuration of a computer system on which an embodiment of the present invention can be implemented.

[0012] While the embodiments described in this disclosure are susceptible to modification and alternative forms, specific embodiments thereof are shown by way of example in the accompanying drawings and are herein described in detail. It should be understood, however, that the accompanying drawings and detailed description thereto do not limit the embodiments to the particular forms disclosed, but on the contrary, include all modifications, equivalents, and alternatives falling within the spirit and scope of the appended claims. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. For clarity and conciseness, not all features of the embodiments are described in the specification. However, it should be understood that during implementation of the embodiments, numerous implementation-specific settings must be made to achieve the developer's specific goals and to satisfy, for example, device and service constraints, which may vary from embodiment to embodiment. It should further be understood that while the development effort may be very complex and time-consuming, such development effort would be a routine undertaking for those skilled in the art having the benefit of this disclosure.

[0014] It should be further noted here that in order to avoid obscuring the present disclosure with unnecessary details, the attached drawings only show processing steps and / or equipment structures that are closely related to at least the technical solutions of the present disclosure, and other details that are not significantly related to the present disclosure are omitted.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, in which like reference numerals and letters indicate like items, and therefore, once an item is defined in one accompanying drawing, there is no need to repeat the description for the subsequent accompanying drawings.

[0016] In this disclosure, the terms "first", "second", etc. are used only to distinguish elements or steps, and do not denote a chronological order, preference, or importance. In the context of this disclosure, an image may refer to any of a variety of images, such as a color image, a grayscale image, etc. Note that in the context of this specification, the type of image is not specifically limited, as long as it can be processed to extract or detect information. The image may also be the original image or a processed version of the image, such as a version of the image that has undergone rudimentary filtering or pre-processing before performing the operations of this application on the image.

[0017] When an image capture device captures a scene, the resulting image typically contains noise, which may include, for example, scattering and ghost reflection phenomena. While these noise phenomena may enhance the artistic effect of captured images, such as some landscape photos captured by RGB sensors, such noise is often particularly detrimental to all sensors that use light to measure distance (e.g., sensors based on ToF technology, structured light sensors for 3D measurement, etc.) compared to RGB sensors. Hereinafter, we will briefly describe noise issues that arise when capturing images using Time-of-Flight (ToF) technology and ToF sensors with reference to the accompanying drawings. Note that, due to the same principles, these noise issues also exist for image sensors based on other technologies, such as structured light sensors and RGB sensors. For the sake of brevity, this disclosure will not further discuss them individually.

[0018] Time-of-flight technology, as shown in Figure 1, uses a light emitter to illuminate a scene and measure the time it takes for the light to return to the sensor. This time difference, i.e., the time difference between light emission and reception, allows the distance to the scene to be calculated based on the measured time: distance d = ct / 2, where c is the speed of light and t is the measured time. For example, light emission can be accomplished by pulse (direct time-of-flight) or continuous wave (indirect time-of-flight). However, cameras using ToF-based sensors can experience noise phenomena during imaging, including scattering and ghost reflections. These noise phenomena can be caused by light reflections within the camera.

[0019] In particular, when a camera is used to capture an image of a close-range object, the close-range object returns a large amount of active light to the camera, appearing to the camera as a bright light source. This results in a large amount of light reflection and scattering within the camera. This will now be described with reference to the accompanying drawings. As shown in FIG. 2A, an imaging module captures a scene containing three objects. The imaging module includes a lens and a sensor, i.e., an imager. The distances between objects 1, 2, and 3 and the imager are r(1), r(2), and r(3), respectively. Light emitted toward these three objects is reflected by the objects and returns to the corresponding positions of the imager in the imaging module, i.e., imaging positions S(1), S(2), and S(3). Because object 1 is very close to the imaging module, the reflected light intensity is high and bounces around inside the module (e.g., between the lens and the imager). In the captured image, the signal from object 1 is scattered around its position and mixed with the signals from objects 2 and 3. The ToF sensor merges these signals to provide a false depth for objects 2 and 3 (measured depth is between distance r(1) and distance r(2) or r(3)).

[0020] Furthermore, cameras often have a photographic filter installed in front of the lens, which can cause ghost reflections. As shown in Figure 2B, typically, light rays pass through the filter and lens, as indicated by the solid line and the arrow above it, and enter the image point on the sensor. However, some light rays are reflected from the image point toward the lens, as indicated by the arrow pointing in the opposite direction, and then pass through the lens. Because a photographic filter is installed, the signal is reflected by the filter toward the lens, as indicated by the dashed arrow, and then enters the sensor through the lens. This creates a ghost image at a location other than the image point, as if light from a different direction were focused on the sensor.

[0021] This phenomenon can also be seen in RGB sensors. For example, when capturing an image of a close-up or bright object, a ghost image, which is an image with a lighter color than the bright white patch, appears near or at a centrally symmetrical position, as shown in the circled area in Figure 3. The ToF sensor detects this ghost, which means that an incorrect depth is detected in front of the sensor. The following describes the effect of the ghost reflection phenomenon in an image captured by a camera, with reference to the accompanying drawings.

[0022] Figure 4A shows an RGB image of a scene graph captured by a mobile phone. The capture mode is background blur mode, and the integration time is 300 μs. A close-distance object is present on the right side of the image. Figure 4B shows a confidence image of the scene graph. This confidence image indicates the confidence of the depth information in the scene graph. Specifically, each pixel in the confidence image indicates the confidence of the depth provided by each pixel in the scene graph. As can be seen from the figure, the right side indicates close-distance objects, which appear as bright white due to their close distance. As mentioned above, the influence of close-distance objects causes a scattering effect (disorderly white dots) in the center of the image and ghost reflections (disorderly white areas on a dark background) on the left side of the image. Figure 4C shows a depth image of the scene graph. This depth image indicates the depth information of objects in the scene. Each pixel in the depth image indicates the distance from the camera to the object in the scene. As can be seen from the figure, a gray shaded area resembling an object due to ghost reflection appears on the left side of the depth image, which is often mistaken for providing depth information. As can be seen from above, in the captured image, the scattering area is located between the object image and the ghost reflection area. The existence of ghost reflection provides false depth information of this close-range object on the left side of the image, and since the depth is usually very shallow, it becomes impossible to correctly identify the information of the close-range object, especially the depth information.

[0023] As can be seen from the above, when a scene is captured using a camera system including a sensor that performs optical ranging (especially a camera system including a ToF sensor, etc.), such ghost reflection phenomenon is very harmful and leads to erroneous depth detection, and the erroneous depth information has a negative impact on providing high-quality images and many subsequent applications. However, in conventional technologies, processing of captured images does not particularly compensate for ghost reflections, so it is not possible to effectively remove ghost reflections and obtain accurate object detailed information, especially depth information.

[0024] 5A shows the scattering compensation flow in the image processing proposed by the present disclosure, in which scattering compensation is performed on the ToF raw data, and then subsequent data processing is performed on the scattering-compensated data to obtain a confidence image and a depth image. This subsequent data processing may include processing for generating confidence images and depth images that is known in the art and will not be described in further detail here.

[0025] As mentioned above, scattering effects can be caused by light from close-range objects reflecting between the sensor and the lens in an imaging device. This causes blurring around the object, resulting in blurred edges in the image. The characteristics of this blur generated by a point or pixel can be modeled using an appropriate function, which may be, for example, a PSF function (point spread function). Based on the modeling results, an algorithm can be applied to remove the specific blur generated by all points / pixels in the image. This algorithm may be, for example, a deconvolution algorithm. However, other appropriate functions and algorithms known in the art may also be used to model and compensate for scattering. This will not be described in further detail here.

[0026] 5B illustrates an exemplary scattering compensation operation according to an embodiment of the present disclosure. If a bright white card is present in the center of a scene as an imaging target, there may be obvious blurring around the target, as well as blurring of the entire image, as shown in the image on the left. Therefore, a deconvolution algorithm (e.g., inverse transform) corresponding to a modeling function (e.g., a PSF function) is utilized to remove this scattering. As a result, the edges of the white patch become sharper, the blurring in the image is removed, and the scattering effect is compensated, as shown in the image on the right.

[0027] However, scattering compensation cannot effectively remove ghost reflections. Figure 5C shows the results of scattering compensation. This image includes a confidence image and a depth image corresponding to the RGB image shown in Figure 4A. In (a), the top and bottom show the confidence image and depth image corresponding to the original scene graph, respectively, which include scattering and ghost reflections. In (b), the top and bottom show the confidence image and depth image after scattering compensation, respectively. As can be seen, even if scattering noise in the image is removed by scattering compensation, ghost reflections (shaded areas) still exist on the left side of the image, and these ghosts still cause erroneous depth measurements.

[0028] Therefore, one objective of the present disclosure is to effectively remove ghost reflections. In particular, the present disclosure proposes to weight the data / image to be processed using the extracted ghost reflection compensation model, and compensate the data / image to be processed using the weighted data / image, thereby effectively removing ghost reflections.

[0029] As described above, such ghost reflection phenomena are particularly caused by reflections from filters in camera systems. Therefore, the ghost reflection compensation technique according to the present disclosure is particularly advantageous for camera systems that additionally use filters, regardless of whether the sensor type is a ToF sensor, a structured light sensor, an RGB sensor, or other types of sensors.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0031] A flowchart of a method for compensating for ghost reflections in an image captured by an imaging device according to an embodiment of the present disclosure is shown in Figure 6. The method 600 may include a calculating step S601 for weighting a compensation target image including ghost reflections using a ghost reflection compensation model, and a compensating step S602 for combining the compensation target image with the weighted image to remove ghost reflections in the image.

[0032] It should be noted that the imaging device to which the technical solution of the present disclosure can be applied may include various types of optical imaging devices, as long as the imaging device may generate ghost reflections due to optical reflections when capturing an image. For example, the imaging device may include a camera using the aforementioned photographic filter. For example, the imaging device may include a camera used for 3D imaging as described above. For example, the imaging device may include a camera including a sensor based on the aforementioned ToF technology. For example, the imaging device may further correspond to the aforementioned camera used for close-up imaging or similar.

[0033] It should be noted that the compensation target image may be any suitable image, such as an original image obtained by an imaging device, or an image obtained by subjecting the original image to predetermined processing, such as pre-filtering, anti-aliasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that the pre-processing operations may include other types of pre-processing operations known in the art, which will not be described in further detail herein.

[0034] According to an embodiment of the present disclosure, the ghost reflection compensation model substantially reflects the characteristics of the light reflection that causes the ghost reflection in the imaging device, i.e., is a model obtained by modeling the light reflection that causes the ghost reflection. For example, as described above, the light reflection may be caused by a photographic filter and / or a lens, i.e., the light reflection characteristics correspond to the characteristics of the photographic filter and / or the lens. Therefore, the ghost reflection compensation model substantially is a model obtained by modeling the characteristics of the photographic filter and / or the lens. However, this model is not limited thereto. If the imaging device includes other components that cause light reflection that causes the ghost reflection, or if other optical phenomena that cause the ghost reflection exist, the model is equivalent to a model that similarly models the characteristics of such other components or other optical phenomena.

[0035] According to one embodiment, the ghost reflection compensation model may be related to the ghost reflection intensity distribution in the image, for example, the ghost reflection compensation model may be related to the intensity distribution in the image affected by the ghost reflection, such as the intensity distribution over the entire image, or specifically the intensity distribution at the source object and ghost locations.

[0036] According to one embodiment, the ghost reflection compensation model may represent ghost reflection intensity factors in a specific sub-region in the image, where the sub-region includes at least one pixel. For example, the specific sub-region may be a sub-region covering the entire image. For another example, the specific sub-region is a sub-region in the image corresponding to the source object and the ghost reflection location.

[0037] According to an embodiment of the present disclosure, the ghost reflection intensity factor may refer to a factor for removing ghost reflections, which is obtained based on the intensity distribution in an image, preferably the intensity distribution of ghost reflections in the image, and is set so that variation in the compensated scene graph is minimized, in particular, variation between the image intensity at the ghost reflection location and the image intensity in its neighboring region is minimized. For example, the variation may refer to intensity variation, for example, image intensity variation of a ghost reflection region relative to neighboring regions around the ghost reflection. In this case, the factor may be referred to as a ghost reflection compensation factor.

[0038] According to an embodiment of the present disclosure, the ghost reflection compensation model may be expressed in various forms. Hereinafter, the ghost reflection compensation model will be described with reference to the accompanying drawings. Fig. 8 shows an exemplary ghost reflection compensation model according to an embodiment of the present disclosure. (a) shows a flattened representation of the model, and (b) shows a three-dimensional graphical representation of the model. Here, the horizontal and vertical axes indicate the planar size of the model, corresponding to the size of the image, and the height axis indicates the value of the ghost reflection intensity factor of the model.

[0039] According to an embodiment, the model may include parameters for intensity factor, central shift, size, etc. In particular, these parameters should be set so that the model matches as closely as possible to the characteristics of the components in the imaging device, as described above.

[0040] The center shift parameters may include cx and cy, which indicate a shift relative to the image center of a reference position at which the image transformation operation (e.g., including rotation and translation) is performed in the compensation operation, e.g., in the horizontal and vertical directions, respectively. In particular, the positions indicated by the parameters cx and cy correspond to the central axis along which the image rotation is actually performed, meaning that the image is rotated eccentrically. For example, cx and cy may directly indicate the shift amount of the central axis along which the image is rotated in subsequent processing relative to the image center. This allows the central axis to be moved to the image center according to this shift amount before rotation. As another example, cx and cy may correspond to the center point position of the model diagram, e.g., the center point position of the flattened display shown in FIG. 8. In this way, the central axis can be moved from this position to the image center before rotation. According to an embodiment, cx and cy may depend at least on the characteristics of the lens, but may also be related to the characteristics of other components. The purpose of determining this center shift is to properly position the image so that ghosts in the rotated image are aligned with targets in the original image. Specifically, the values ​​of cx and cy can be determined, for example, by experiment or calibration measurement.

[0041] The size parameters may include width and length parameters corresponding to the width and length (in pixels) of the image, respectively. For ease of application, the model should correspond to the image. Therefore, the width and height may refer to the width and length of the model, respectively, as shown in the plane of the three-dimensional illustration in FIG. 8(b). The width and length may depend on the pixel layout of the sensor.

[0042] The parameters related to the intensity factors may include parameters that indicate the intensity factor distribution corresponding to the image. The intensity factor distribution can indicate the ghost reflection intensity factor at each pixel position of the image by being expressed by an appropriate distribution function, as intuitively shown in Figure 8(b).

[0043] According to an embodiment, the intensity factor distribution is set so that the ghost reflection intensity factor in a sub-region close to the image center is higher than the ghost reflection intensity factor in a sub-region close to the image edge. When ghost reflections exist in a captured image, the ghost reflections exhibit different light intensities depending on their location, and in particular, they become weaker from the center toward the edge, and the negative impact on depth measurement also decreases as their intensity decreases. Therefore, by setting the intensity factor as described above, ghost reflections in the image can be appropriately weakened and even removed. For example, the stronger the intensity of a ghost reflection, the greater the factor used to weaken and remove it, thereby providing accurate and effective ghost reflection compensation.

[0044] According to an embodiment, the intensity factor distribution may be determined according to a specific distribution function. According to an embodiment, the intensity factor distribution may include at least one specific distribution function, and each function may have a corresponding weight. For example, the intensity factor may be αf(1)+βf(2)+..., where f(1) and f(2) respectively represent specific functions, and α and β respectively represent the weights of each function. According to an embodiment, the parameters of the distribution function and the weights used in the distribution function may be set according to, for example, the characteristics of the light reflection that causes ghost reflection in the imaging device, particularly the optical characteristics of the components that cause this reflection, so as to match (closely match) these characteristics as much as possible. For example, appropriate parameter values ​​may be determined based on empirical values ​​obtained through prior testing or experiments, or may be adjusted based on the empirical values ​​through further calibration operations.

[0045] Preferably, the intensity factor distribution follows a Gaussian distribution. The Gaussian function is calculated using two parameters. The first parameter is std, i.e., the standard deviation of the Gaussian function, and mu is the mean value of the Gaussian function. Note that these two Gaussian function parameters std and mu may be related to the intensity of reflected light that causes ghost reflections in the imaging device. In particular, the parameters std and mu may depend on the characteristics (e.g., optical characteristics) of components that may cause light reflections in the imaging device, such as the aforementioned lenses, photographic filters, etc. According to an embodiment, the expression of the intensity factor may include a certain number of Gaussian functions, and each Gaussian function may be assigned an appropriate weight.

[0046] For example, this model can be expressed as follows: Center Shift: cx=-2.5,cy=-2.5,width=240,length=180 Strength factor = 1.3×Gaussian(std=38,mu=0)+0.6×Gaussian(std=50,mu=60)

[0047] The parameters of the Gaussian function provided in the intensity factor expression, particularly the parameters of the Gaussian function itself and the weights of the Gaussian function, may be selected, for example, according to the optical characteristic curve of the aforementioned optical component, such as a filter or lens. For example, to remove the influence of light reflection due to this optical characteristic, the intensity factor distribution may be made to better correspond (e.g., inversely match) to the optical characteristic curve. Alternatively, the parameters may be set to initial values ​​based on experience and then adjusted based on the experience through further calibration operations. It should be noted that the intensity factor in the ghost reflection compensation model described above is expressed by a Gaussian function for illustrative purposes only, and other distribution functions may be used in the present disclosure. It is sufficient for this distribution function to enable the model to accurately match the intensity distribution of ghost reflections in an image, and in particular, to match the characteristics of the light reflections that cause ghost reflections in an imaging device, such as the characteristics of components that produce the light reflections that cause ghost reflections. For example, the distribution function may employ other functions having a normal distribution. As another example, it may employ functions having other distributions, such as a Cauchy distribution or a gamma distribution.

[0048] According to one embodiment, the ghost reflection compensation model is extracted from a predetermined number of calibration images. For example, the calibration images are obtained by capturing images of a specific scene for calibration using an imaging device. This predetermined number may be specified by experience or may be the number used in previous calibrations.

[0049] FIG. 9 shows that the model is extracted from multiple images. The left side shows images for extracting the ghost reflection compensation model. These are images obtained for a calibration scene with a white chart. The white chart in each scene is located at different positions, each at four corner positions and a central position. Each image simultaneously contains a bright white patch representing the white chart and a pale color patch representing ghost reflection. Note that the number and arrangement of calibration images are not limited to this, as long as they adequately reflect information about ghost reflections. For example, to more precisely reflect information about ghost reflections in the scene, the white chart may be located at more positions and more calibration images may be obtained. According to embodiments of the present disclosure, various methods may be employed to extract a ghost reflection compensation model from a calibration image. According to one embodiment, the ghost reflection compensation model may be determined so that the intensity variation of a predetermined number of calibration images to which the model is applied satisfies specific requirements. As described above, the intensity variation may refer to the image intensity variation of a ghost reflection region relative to the neighboring region surrounding the ghost reflection, i.e., the image intensity difference between the ghost reflection region and the neighboring region surrounding the ghost reflection. According to another embodiment, optionally or additionally, the ghost reflection compensation model may be determined so that ghost reflections in the image to which the model is applied are eliminated or mitigated. Eliminating or mitigating ghost reflections may refer to the depth / RGB information measured at the location where the ghost reflection exists being consistent with or approaching that of the actual scene. In other words, the ghost reflection compensation model is extracted on the condition that the intensity variation (and, optionally or additionally, the degree of ghost reflection elimination) satisfies specific requirements.

[0050] According to one embodiment, the intensity variation satisfying the requirement may refer to the fact that the statistical value of the variations obtained from all or at least some of the scene graphs, such as the sum or average value of the variations of those scene graphs, satisfies the requirement. For example, the specific requirement may refer to the intensity variation being smaller than a specific threshold, or the specific requirement may refer to the image having minimal variation. Therefore, satisfying the specific requirement means that the intensity of the ghost reflection region and the neighboring regions around the ghost reflection are nearly identical, have small variations, and are smooth and have no boundaries. In this way, the influence of the ghost reflection can be essentially eliminated.

[0051] The model extraction process can be performed in various ways. According to an embodiment, it may be performed in an iterative manner. For example, initial values ​​of each parameter of the ghost reflection compensation model may be set, and the image intensity variation (and, optionally or additionally, the degree of ghost reflection cancellation) described above may be calculated using the set model. It may then be verified whether the image intensity variation (and, optionally or additionally, the degree of ghost reflection cancellation) satisfies a specific requirement. If not, the parameter setting values ​​may be continuously adjusted until the intensity variation meets the specific requirement. The next operation is performed, and the corresponding model at that time is determined as the desired ghost reflection compensation model and used for the subsequent image compensation process. Note that the model parameters that can be determined by the iterative operation may include at least relevant parameters of the distribution function of the model, such as the parameters of the Gaussian function itself and the weights of each Gaussian function when two or more Gaussian functions exist. 10A, the process of determining the intensity variation of one calibration image in one model extraction operation will be described. The calibration image includes a white chart and its ghost reflection, and can be used as an image for model extraction. Note that this determination process may be performed separately for each image used for model extraction.

[0052] First, image transformation is performed based on the center shift parameter of the ghost reflection compensation model. As described above, the center shift parameter indicates the shift amount of the rotation center relative to the image center. Therefore, the image transformation essentially indicates that the image is rotated eccentrically, i.e., rotated around a central axis offset from the image center. Figure 10B shows a case where the image is directly rotated 180 degrees. The position of the cross symbol corresponds to the position of the eccentric axis indicated by the center shift, and the image transformation may indicate that the final image is obtained by directly rotating around this eccentric axis.

[0053] Image transformation may further be performed by a displacement and rotation operation, i.e., a displacement, rotation, and re-displacement operation process, by first displacing the center of rotation based on the parameter values ​​(e.g., moving to the center based on cx and cy), then rotating around the central axis at the center, and then displacing the displaced center in the opposite direction based on the parameter values ​​(i.e., moving the center based on -cx and -cy), as shown in Figure 10C.

[0054] Note that the rotation may be any angle, as long as the source object and ghost reflection positions in the rotated image overlap with the ghost reflection positions and source object positions in the previous image, respectively. In a preferred example, the rotation may be 180°. In one implementation, the displacement and rotation result in the ghost in the displaced and rotated image corresponding to the object position in the original image, and the object in the displaced and rotated image corresponding to the ghost position in the original image. Since the object position in the displaced and rotated image is aligned with the ghost position in the original image, an intensity factor is used to weight the high intensity of the white patch, and the weighted intensity value suppresses the low intensity at the ghost position. This effectively suppresses the ghost intensity and achieves ghost removal. Meanwhile, the weighted intensity value obtained by using the intensity factor to weight other positions in the displaced and rotated image is very small, ensuring that the process of suppressing ghosts in the original image has a relatively small impact on the intensity values ​​of positions other than the ghost position in the original image.

[0055] The transformed image is then multiplied (i.e., weighted) by the ghost reflection intensity factors of the ghost reflection compensation model. In particular, the factor at each location in the ghost reflection compensation model is multiplied by the pixel intensity at the corresponding location in the transformed image to obtain an intensity-scaled image.

[0056] Finally, the original compensated image is subjected to intensity subtraction corresponding to the rotated and intensity-scaled image, for example, by subtracting the intensity at a corresponding region (e.g., a pixel position at a corresponding shifted and rotated position) in the intensity-scaled image from the intensity at a region in the original compensated image, thereby obtaining a compensated image in which intensity variations, particularly intensity variations at ghost reflection positions relative to adjacent regions around the ghost reflection positions (and optionally or additionally, the degree of ghost reflection cancellation), can be calculated. By similarly applying the above determination process to other calibration images, the intensity variation (and optionally or additionally, the degree of ghost reflection elimination) of each image in this model extraction operation can be obtained, and it can be determined whether the statistical data of the intensity variation (and optionally or additionally, the degree of ghost reflection elimination) of these images meets certain conditions.

[0057] For example, if it is a threshold condition, it is determined whether the intensity variation data of the statistical image is smaller than a predetermined threshold, and / or whether the degree of ghost cancellation is greater than a corresponding predetermined threshold. If YES, the currently adopted compensation model is deemed to be desired, and the model extraction operation can be stopped, and this desired model can be used as the ghost reflection compensation model used in the actual imaging process. If NO, the parameters of the model can be adjusted step by step, and the above process can be repeated until the intensity variation statistical data meets the threshold requirement.

[0058] As another example, if the minimization condition is met, and the statistical data of the intensity variation determined in the current extraction operation is not smaller than the previous one, and / or the statistical data of the degree of ghost elimination is not larger than the previous one, it may be assumed that the statistical data of the intensity variation has been minimized and the statistical data of the degree of ghost elimination has been maximized, and the model extraction operation may be stopped, and the compensation model corresponding to the previous operation may be used as the final compensation model.

[0059] The initial values, step sizes, and other parameters of the model parameters in the above iterative operations may be set to any appropriate values ​​as long as they contribute to iterative convergence. In addition, in the iterative operations, all model parameters may be changed simultaneously each time, or only one or more parameters may be changed each time. The former corresponds to the case where all model parameters are determined simultaneously through iteration, while the latter corresponds to the case where one or more parameters are first determined through iteration, and then other parameters are determined through iteration based on the determined parameters.

[0060] According to another implementation, a minimization equation is constructed using the ghost reflection compensation model, and by solving the equation, if a solution is obtained that eliminates ghosts in all scenes to a predetermined degree and minimizes image intensity variation, the desired ghost reflection compensation model is obtained. For example, the equation may be constructed using at least one of cx, cy, and the weights of each Gaussian function as variables.

[0061] For example, the intensity distribution in an image may be represented as a vector or a matrix. Thus, the image-model multiplication described above with reference to FIG. 10A can be represented as a mathematical vector or matrix multiplication. This allows the operation for determining the intensity variations described in FIG. 10A to be represented as a vector or a matrix. This allows the minimization solution to be performed using an appropriate method, such as the least squares method.

[0062] According to an embodiment of the present disclosure, the ghost reflection compensation model can be determined before the imaging device is used by a user, for example, during the manufacturing process, during shipping testing, etc. For example, the ghost reflection compensation model may be determined together with other calibration tasks (e.g., in the case of a ToF camera, temperature compensation, phase gradient, circular error, etc.) during the manufacturing process. In this manner, the ghost reflection compensation model can be constructed in advance and stored in an imaging device such as a camera.

[0063] According to an embodiment of the present disclosure, the ghost reflection compensation model may be determined while the imaging device is being used by a user. For example, the user may be prompted to perform camera calibration when using the imaging device for the first time. This allows the user to capture calibration images in response to operational instructions. The ghost reflection compensation model may then be extracted from the captured images. As another example, the user may be prompted to update the model after capturing a predetermined number of images (e.g., after the shutter has been used a predetermined number of times).

[0064] According to an embodiment of the present disclosure, the ghost reflection compensation model can be updated or pushed during product maintenance service of the imaging device. For example, when an imaging device with a ghost reflection compensation function replaces the photographic filter and / or lens of the imaging device, or when an imaging device without a ghost reflection compensation function performs a software update, the above-described model extraction process may be performed to update or build the model.

[0065] As described above, the ghost reflection compensation model can be equivalent to the characteristics of components, particularly lenses and photographic filters, that characterize components that cause light reflections in an imaging device, resulting in ghost reflections. In a sense, the ghost reflection compensation model corresponds to the lenses and / or photographic filters in the imaging device. According to one embodiment, if the filters and lenses, particularly lenses, included in a camera are fixed, the constructed ghost reflection compensation model may remain relatively fixed and, particularly, unchanged during the imaging process. According to another embodiment, if the filters and / or lenses of a camera are replaceable, the ghost reflection compensation model needs to be updated accordingly when such components of the imaging device are replaced. According to one embodiment, upon component replacement, the ghost reflection compensation model corresponding to the replaced component can be extracted automatically or by prompting the user, as described above. According to another embodiment, the model corresponding to the replaced component can be automatically selected. For example, a camera system may pre-store a set of ghost reflection compensation models corresponding to all filters and / or lenses suitable for the camera system. In this way, after the camera system replaces a filter and / or lens, the ghost reflection compensation model corresponding to the replaced filter and / or lens can be automatically selected and applied from the stored set. According to yet another embodiment, considering that changing filters and / or lenses often causes some changes in optical properties, e.g., lens properties may change and further affect the center shift parameter, when changing filters and / or lenses, even if a corresponding model is pre-stored, it is possible to automatically or prompt the user to extract a new model without automatically selecting it. For example, the system may be pre-set or the user may be prompted to perform this operation. For example, the system may be pre-set to automatically update the model in any case. Or, for example, the user may be prompted to choose whether to calibrate the model or to automatically select the model.

[0066] After determining the ghost reflection compensation model according to the present disclosure as described above, the model can be applied to further optimize the captured image and improve the image quality. According to an embodiment of the present disclosure, in the image weighting operation, intensity scaling is performed for each sub-region in the captured image using the corresponding ghost reflection intensity scaling factor in the ghost reflection compensation model, and the intensity-scaled image can be obtained as a weighted image.

[0067] According to one embodiment, in the image weighting operation, the image to be compensated is rotated and the rotated image is weighted (e.g., multiplied) using a ghost reflection compensation model to obtain a weighted image. According to one embodiment, the compensated image is obtained by subtracting from the image to be compensated pixel intensities at corresponding locations in the weighted image.

[0068] Note that the rotation, multiplication, subtraction, and other operations can be performed in a manner similar to that described with reference to FIG. 10, except that the input image on the left is the captured image to be compensated, and the output image on the right is the compensated captured image in which ghost reflections have been effectively removed. In particular, the image to be compensated is displaced and rotated based on the central parameters cx and cy of the ghost reflection compensation model. That is, the rotation can be performed around an axis that is offset from the center of the image. When performing the multiplication operation, each position of the transformed image is multiplied by the corresponding ghost reflection compensation model factor. For example, intensity scaling can be performed by multiplying the model image and the transformed image after aligning them.

[0069] According to some embodiments, the ghost reflection compensation operation according to the present disclosure may be performed before or after scattering compensation, and essentially the same advantageous effects can be achieved. Fig. 11(a) shows that ghost reflection compensation is performed before scattering compensation, that is, the image to be compensated is the original image obtained by the ToF sensor. Meanwhile, Fig. 11(b) shows that ghost reflection compensation is performed after scattering compensation, that is, the image to be compensated is an image after scattering compensation.

[0070] According to some embodiments of the present disclosure, the method further comprises a scattering compensation step for compensating for scattering in the image. According to some embodiments, the imaging device according to the present disclosure is an imaging device using an imaging filter. According to some embodiments, the imaging device includes a Time of Flight sensor, and the compensated image includes a depth image.

[0071] Although the above examples have mainly been described with respect to a case where one scene has one image to be compensated, the embodiments of the present invention can also be applied to a case where one scene has at least two images to be compensated.

[0072] According to some embodiments, the original image data obtained by capturing a scene may correspond to at least two sub-images, and for each sub-image, the ghost reflection compensation operation according to the present disclosure, including the calculation and compensation steps described above, is performed to obtain at least two compensated sub-images, which are then combined to obtain a final compensated image corresponding to the scene.

[0073] For example, the at least two sub-images include an I-image and a Q-image corresponding to the original image data. Hereinafter, the compensation for the sub-images will be described using the I-image and the Q-image as an example. Figure 12 shows an example including the operation of performing ghost reflection compensation on the I-image and the Q-image.

[0074] To measure distance, an iToF sensor typically needs to capture four components. These components relate to the phase shift between the emitters (lasers) and the light returning to the sensor, if the phase shift is predefined. These four components are recorded for angles of 0, 90, 180, and 270 degrees, respectively. These four components are taken as the ToF raw data.

[0075] From these raw data, I and Q images can be calculated, where I may represent the captured in-phase data. For example, the I image is a combination of the 0-degree component and the 180-degree component. Q may represent the captured quadrature-phase data. For example, the Q image is a combination of the 90-degree component and the 270-degree component. For example, they are calculated as follows:

[0076] I = component (0 degrees) - component (180 degrees) I = component (90 degrees) - component (270 degrees)

[0077] Next, compensation is performed for each of the I and Q images. The specific compensation method is performed in the manner described with reference to Fig. 10, and in particular, the above-mentioned ghost reflection compensation operation may be performed for each of the I and Q images. No further details will be given here.

[0078] Finally, the compensated I and Q images are used to generate confidence and depth images.

[0079] As an example, a confidence image is calculated from I and Q as shown below: Confidence = abs(I) + abs(Q)

[0080] where abs( ) represents the absolute value function, which represents the absolute value of the reliability of each sub-region or pixel point in the I image and the Q image, respectively. For example, the reliability image may also be obtained by other methods known in the art, which will not be described in further detail here.

[0081] By way of example, the depth image may be obtained from at least one of the I and Q images, which may be obtained using methods known in the art and will not be described in further detail here.

[0082] However, the I image and the Q image are merely exemplary, and other sub-images may be used as long as they are captured by an imaging device and can be combined to obtain a confidence image and a depth image.

[0083] FIG. 13 illustrates the beneficial effect of ghost reflection compensation according to an embodiment of the present disclosure. It shows the confidence image and depth image corresponding to the image shown in FIG. 4A. The left section shows, from top to bottom, the confidence image and depth image corresponding to the original scene graph, including scattering and ghost reflections. The middle section shows, from top to bottom, the compensated confidence image and depth image. As can be seen, since scattering compensation is mainly performed, ghost reflections still exist on the left side of the image even after scattering noise is removed by scattering compensation. The right section shows, from top to bottom, the confidence image and depth image compensated according to the method of the present disclosure. As can be seen, the method of the present disclosure can effectively remove ghost reflections in the image and obtain a high-quality output image.

[0084] The above mainly uses the example of an iToF sensor to explain the problem of ghost reflection and the ghost reflection compensation operation. However, as mentioned above, ghost reflection does not depend on the emitter or the image sensor, but mainly on the components that cause light reflection in the imaging device, and in particular mainly on the use of photographic filters and / or lenses in the camera. This means that this phenomenon can also be observed in other 3D measurement systems that use light. Other 3D measurement systems include Indirect ToF sensor using full-field emitter Indirect ToF sensor using spot ToF emitter Direct ToF sensor Structured light sensor Other types of ToF sensors Including, but not limited to:

[0085] Ghost reflection compensation for other types of ToF sensors according to embodiments of the present disclosure will now be described with reference to the accompanying drawings.

[0086] 14A and 14B illustrate ghost reflection compensation when imaging with a direct ToF (dToF) sensor.

[0087] Unlike iTOF, dToF concentrates optical energy over a short period of time. It involves generating photon packets with short pulses of a laser or LED and directly calculating the propagation time of these photons to the target and back. An appropriate technique can then be employed to accumulate multiple events into a histogram, thereby identifying the target peak position above the usually uniformly distributed background noise. For example, this technique may be a technique called time-correlated single photon counting (TCSPC). The TCSPC technique is known in the art and will not be described in further detail here.

[0088] In dToF, ghost reflections may appear as "ghost peaks" in the histogram of the affected pixel. As shown in Figure 14A, the upper histogram corresponds to the histogram of an object with high intensity in the field of view (FoV). The high peaks indicate the corresponding depth. The lower histogram corresponds to the histogram at the location where the ghost reflection appears. The ghost reflections also cause depth peaks, leading to incorrect depth detection.

[0089] According to an embodiment of the present disclosure, ghost reflection compensation can be performed for dToF. In particular, ghost reflection compensation is performed on the pixel histogram obtained from the captured data using the above-described method, for example, by performing operations such as displacement and rotation, multiplication, and subtraction, as shown in FIG. 14B. As can be seen from the output histogram on the right, the ghost reflection compensation of the present disclosure significantly suppresses the histogram corresponding to ghost reflections, and its peak value is much smaller than that of the true object, thereby preventing erroneous depth detection.

[0090] According to one embodiment, for dToF, the ghost reflection compensation model may be generated by the same operation as described with reference to FIG. 10 , except that the input is a pixel histogram obtained from the captured data. Any other operation described above can also be applied to dToF. According to another embodiment, taking into consideration that the ghost reflection compensation model mainly corresponds to components, such as lenses and / or filters, that cause optical reflections in the imaging device, which cause ghost reflections, after obtaining the ghost reflection compensation model for this component using any ToF sensor, this model can be applied to other types of ToF sensors and, in turn, other types of sensors using this component.

[0091] FIG. 15 illustrates ghost reflection compensation when capturing an image using a spot ToF (spotToF) sensor. FIG. 15(a) shows a reliability image captured when no close-range objects are present, while FIG. 15(b) shows a reliability image captured when a close-range object is ideally present. Even if a close-range object is present in the scene, there is no other spot information other than the spot information of the close-range object itself. FIG. 15(c) shows a reliability image with ghost reflections. As can be seen, when a close-range object is present on the right side of the scene, new spots appear on the left side of the scene. These new spots are generated by ghost reflections, which may cause false depth or mix signals with the existing speckle. FIG. 15(d) shows a reliability image after ghost reflection compensation using the technical solution disclosed herein. The new spots caused by ghost reflections are effectively removed, improving image quality.

[0092] According to embodiments of the present disclosure, the ghost reflection compensation function according to the present disclosure may be used automatically or may be selected by the user.

[0093] As an example, the ghost compensation function of the present invention may be implemented automatically. For example, the ghost compensation function may be associated with a specific imaging mode of the camera. When the imaging mode is turned on during imaging, the ghost compensation function is automatically activated. For example, the ghost compensation function is automatically turned on in close-up imaging modes such as macro imaging and portrait imaging modes, but is not automatically turned on in distant-view imaging modes such as landscape imaging modes. As another example, the camera may determine whether to automatically turn on the ghost compensation function depending on the distance to the imaging subject. For example, if the distance to the imaging subject is greater than a predetermined distance threshold, it may be determined that the imaging is distant and that the ghost compensation function does not need to be turned on. On the other hand, if the distance to the imaging subject is less than the predetermined distance threshold, it may be determined that the imaging is close-up and that the ghost compensation function does not need to be turned on.

[0094] For example, the ghost compensation function of the present invention may be set by a user. For example, a prompt may be displayed on the camera's imaging operation screen to prompt the user to turn on the ghost compensation function. When the user selects this function, the ghost compensation function may be turned on to compensate for / remove ghosts during imaging. For example, this function may be selected using a button displayed on a touch-sensitive user operation screen or a button that performs the ghost compensation function on the camera.

[0095] According to embodiments of the present disclosure, the ghost reflection compensation model according to the present disclosure may be stored in various ways. For example, the model may be integrated with an imaging device, particularly a camera lens including a lens and a filter. This allows the model to remain in use even if the camera lens is replaced with another device, eliminating the need for further model extraction. Alternatively, the model may be stored in a device, such as a portable electronic device, that is connected to the imaging device and captures images.

[0096] As described above, ghost reflections are particularly disadvantageous in obtaining depth information, and therefore the technical solution of the present disclosure is particularly suitable for various applications that require obtaining depth information of an object in an imaging scene, such as an imaging device that requires measuring depth information. For example, the technical solution of the present disclosure may be applied to an imaging device using a sensor based on ToF technology, such as iToF, full-field ToF, or spot ToF. For example, since depth / distance information is very important for obtaining good 3D images, the technical solution of the present disclosure may be applied to a 3D imaging device.

[0097] It should be noted that although the effects of ghost reflections are less severe in the case of RGB sensors than in 3D measurement systems, the embodiments of the present disclosure are equally applicable to RGB sensors, particularly when there is only one photographic filter in the system, for example in a portable mobile device, whose camera is equipped with a coverglass, and in this scene the coverglass is realized as a filter.

[0098] In addition, the technical solution of the present disclosure can be applied to certain imaging modes in which ghost reflections may occur. For example, in consideration of the fact that imaging a close-up object may cause a large amount of light reflection, which may further cause ghost reflections, the ghost reflection compensation method of the present disclosure is particularly suitable for modes of an imaging device related to imaging a close-up object, such as a close-up imaging mode and a background blur mode.

[0099] The technical solution of the present disclosure can effectively eliminate the influence of ghost reflections in images. In particular, the technical solution of the present disclosure can accurately determine the depth information, i.e., distance information, of objects in a scene, which can enable accurate focusing during imaging or obtain high-quality images, which is beneficial for subsequent image-based applications.

[0100] For example, in the case of background blurring, the technical solution of the present disclosure can eliminate false depth and obtain a proper object distance. For example, in an autofocus application, even if an object approaches the camera, the object distance can be accurately identified and an image can be captured using a good focus distance. For example, in the case of face ID identification, when an object (e.g., a table) approaches the camera for face identification, the technical solution of the present disclosure can effectively eliminate ghosts in the image and obtain and identify a high-quality image.

[0101] It should be noted that the technical solution of the present disclosure is particularly suitable for cameras in mobile devices, such as cameras in devices such as mobile phones, tablets, etc. The lens and / or camera filter of the camera may be fixed or replaceable.

[0102] An electronic device capable of performing ghost reflection compensation according to the present disclosure is now described. A block diagram of an electronic device capable of performing a ghost reflection compensation method according to an embodiment of the present disclosure is shown in Figure 7. The electronic device 700 includes processing circuitry 720 that may be arranged to weight a compensation target image including ghost reflections using a ghost reflection compensation model, and to combine the compensation target image with the weighted image to remove ghost reflections in the image.

[0103] In the above example of the device configuration, the processing circuit 720 may be in the form of a general-purpose processor or a dedicated processing circuit such as an ASIC. For example, the processing circuit 120 may be configured by a circuit (hardware) or a central processing unit (e.g., a central processing unit (CPU)). Furthermore, a program (software) for operating the circuit (hardware) or the central processing unit may be loaded into the processing circuit 720. This program may be stored in memory (e.g., arranged in memory), stored in an external storage medium connected from the outside, or downloaded via a network (e.g., the Internet).

[0104] According to an embodiment of the present disclosure, the processing circuit 720 may include units for implementing the above functions, such as a calculation unit 722 for weighting a compensation target image including ghost reflections using a ghost reflection compensation model, and a ghost reflection compensation unit 724 for combining the compensation target image and the weighted image to remove ghost reflections in the image. In particular, the processing circuit 720 may further include a scattering compensation unit 726 and a data path processing unit 728. Each unit can perform the operations described above, and therefore will not be described in further detail here.

[0105] The scattering compensation unit 726 and the data path processing unit 728 are depicted with dotted lines to illustrate that these units are not necessarily included in the processing circuitry. For example, these units may be located within the terminal electronics but outside the processing circuitry, or even outside the electronics device 700. Note that although each unit is shown in Figure 7 as a separate unit, one or more of these units may be integrated into a single unit or split into multiple units.

[0106] The above units are merely logic modules divided based on the specific functions they realize, and do not limit the specific implementation manner, and may be realized, for example, in software, hardware, or a combination of software and hardware. When actually realized, the above units may be realized as independent physical entities, or may be realized by a single entity (e.g., a processor (e.g., CPU or DSP), integrated circuit, etc.). Furthermore, the dashed lines in the figures indicate that the units do not actually exist, and that the operations / functions realized by the units may be realized by the processing circuitry itself.

[0107] It should be understood that Figure 7 is merely a schematic structural layout of the terminal side electronic device. The electronic device 700 may further include other possible components (e.g., memory, etc.). Optionally, the terminal side electronic device 700 may further include other components not shown, such as a memory, a network interface, and a controller. A processing circuit may be associated with the memory. For example, the processing circuit may be directly or indirectly connected to the memory (e.g., other components may be connected in between) to access data. The memory can store various information generated by the processing circuitry 720. The memory can be located within the terminal electronics but external to the processing circuitry, or even external to the terminal electronics. The memory can be volatile and / or non-volatile. For example, the memory can include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory.

[0108] An imaging device according to the present disclosure will now be described. Fig. 16 shows a block diagram of an imaging device according to an embodiment of the present disclosure. The imaging device 1600 includes a compensation device 700 that can be used for image compensation processing, particularly ghost reflection compensation, and this compensation device can be realized by electronic equipment, for example, the electronic equipment 700 described above.

[0109] The image capture device 1600 may include a lens portion 1602, which may include various optical lenses known in the art, for optically imaging an object onto a sensor. The imaging device 1600 may include a photographic filter 1604, which may include a variety of photographic filters known in the art, that can be attached to the front of the lens.

[0110] The image capture device 1600 may further include processing circuitry 1606 that may be used to process the captured image. For example, further processing may be performed on the compensated image, or pre-processing may be performed on the compensated image.

[0111] The image capture device 1600 may further include various image sensors, for example, various sensors based on the ToF technology described above, but these sensors may also be located external to the image capture device 1600.

[0112] It should be noted that the photographic filters and processing circuitry are depicted with dashed lines to illustrate that these units are not necessarily included in the image capture device 1600, but may be connected and / or communicate in known manners outside of the image capture device 1600. It should be noted that although each unit is shown as a separate unit in Figure 16, one or more of these units may be integrated into a single unit or may be divided into multiple units.

[0113] In the above example of the device configuration, the processing circuit 1606 may be in the form of a general-purpose processor or a dedicated processing circuit such as an ASIC. For example, the processing circuit 1606 may be configured by a circuit (hardware) or a central processing unit (e.g., a central processing unit (CPU)). Furthermore, a program (software) for operating the circuit (hardware) or the central processing unit may be loaded into the processing circuit 1606. This program may be stored in memory (e.g., arranged in memory), stored in an external storage medium connected from the outside, or downloaded via a network (e.g., the Internet).

[0114] The techniques of the present disclosure can be applied to a variety of products.

[0115] For example, the technology of the present disclosure may be applied to an imaging device itself, for example, built into a camera lens and integrated with the camera lens. In this manner, the technology of the present disclosure may be executed by a processor of the imaging device in the form of a software program, or integrated in the form of an integrated circuit or processor, or used in a device connected to the imaging device, such as a portable mobile device equipped with the imaging device. In this manner, the technology of the present disclosure may be executed by a processor of the imaging device in the form of a software program, or integrated in the form of an integrated circuit or processor, or further integrated into an existing processing circuit, and used to perform ghost reflection compensation in the imaging process.

[0116] The technology of the present disclosure can be applied to various imaging devices, such as lenses mounted on mobile devices, imaging devices for unmanned aerial vehicles, imaging devices in monitoring devices, and the like.

[0117] The present invention may be used in many applications, for example it may be used to monitor, identify and track objects in still or video images captured by a camera, and is particularly advantageous for camera-equipped mobile devices, (camera-based) mobile phones and the like.

[0118] It should also be understood that the above series of processes and devices may be implemented by software and / or firmware. When implemented by software and / or firmware, the programs constituting this software are installed from a storage medium or a network onto a computer having a dedicated hardware configuration, such as a general-purpose PC 1300 shown in FIG. 17, and this computer can execute various functions when various programs are installed. FIG. 17 is a block diagram showing an exemplary configuration of a PC, which is an information processing device that can be employed in an embodiment of the present disclosure. In one example, this PC may correspond to the above exemplary transmitting device or terminal-side electronic device according to the present disclosure.

[0119] 17, a central processing unit (CPU) 1301 executes various processes based on a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage 1308 into a random access memory (RAM) 1303. The RAM 1303 also stores data necessary when the CPU 1301 executes various processes, as needed.

[0120] The CPU 1301, the ROM 1302, and the RAM 1303 are connected to one another via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0121] An input unit 1306 including a keyboard, a mouse, etc., an output unit 1307 including a display such as a cathode ray tube (CRT) or a liquid crystal display (LCD) and a speaker, etc., a storage 1308 including a hard disk, etc., and a communication unit 1309 including a network interface card such as a LAN card, a modem, etc. are connected to the input / output interface 1305. The communication unit 1309 executes communication processing via a network, for example, the Internet.

[0122] If necessary, a drive 1310 is also connected to the input / output interface 1305. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1310 as necessary, and a computer program read from the removable medium 1311 is installed in the storage 1308 as necessary.

[0123] When the above series of processes are realized by software, the programs that make up the software are installed from a network, such as the Internet, or a storage medium, such as the removable medium 1311 .

[0124] Those skilled in the art should understand that such a storage medium is not limited to the removable medium 1311 shown in Figure 17, which stores the program and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disk read-only memories (CD-ROMs) and digital versatile disks (DVDs)), magneto-optical disks (including minidisks (MDs) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1302, a hard disk included in the storage 1308, or the like, which stores the program and is distributed to the user together with the device containing the program.

[0125] It should be noted that the methods and apparatus described herein may be implemented as software, firmware, hardware, or any combination thereof. Some components may be implemented, for example, as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or application specific integrated circuits.

[0126] The method and system of the present invention may be implemented in various ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware, or any combination thereof. The order of the steps of the method described above is used for illustrative purposes only, and unless otherwise specified, the steps of the method of the present invention are not limited to the order specifically described above. In some embodiments, the present invention may be embodied as a program recorded on a recording medium containing machine-readable instructions for implementing the method of the present invention. Therefore, the present invention also includes a recording medium having a program stored thereon for implementing the method of the present invention. Such storage media include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0127] As one skilled in the art will recognize, boundaries between operations described above are for illustrative purposes only. Multiple operations may be combined into a single operation, a single operation may be distributed into additional operations, and operations may be performed with at least partial overlap in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and substitutions are possible. Accordingly, the present specification and accompanying drawings should be considered illustrative only and not restrictive.

[0128] Additionally, embodiments of the present disclosure may further include the following illustrative examples (EE). EE1, an electronic device for compensating for ghost reflections in an image captured by an imaging device, weighting the compensation target image including the ghost reflection using a ghost reflection compensation model relating to the intensity distribution in the image of the ghost reflection caused by light reflection in the imaging device when capturing an image; An electronic device comprising processing circuitry arranged to combine the compensated image and the weighted image to remove ghost reflections in the image. EE2: An electronic device as described in EE1, wherein the ghost reflection compensation model is obtained by training from a predetermined number of calibration images, and the ghost reflection compensation model is trained so that the intensity variation of a ghost reflection area in a calibration image to which the ghost reflection compensation model is applied relative to an adjacent area of ​​the ghost reflection area is smaller than or minimized by a predetermined threshold. EE3,Training the ghost reflection compensation model determines intensity variation by the following operations: The calibration image is displaced in the center based on a preset center displacement parameter of the ghost reflection compensation model to be trained, and rotated around the center after displacement as an axis. Furthermore, the rotated image is displaced in the opposite direction in the center based on the parameter; multiplying the displaced and rotated image by a ghost reflection compensation model having preset intensity factor parameters; The electronic device of EE2, wherein the intensity variation is obtained by subtracting from the calibration image pixel intensities at corresponding locations in the model-multiplied image. EE4. The electronic device of EE1, wherein the ghost reflection compensation model includes a ghost reflection factor corresponding to each subregion in the image, the subregion including at least one pixel. EE5. The electronic device described in EE4, wherein the ghost reflection compensation model is set so that the ghost reflection factor in a sub-region near the center of the image is larger than the ghost reflection factor in a sub-region near the edge of the image. EE6. The electronic device described in EE4, wherein the ghost reflection factor is determined based on a Gaussian distribution. EE7, the processing circuitry An electronic device described in any one of EE4 to EE6, which is arranged to perform intensity scaling for each sub-region in a captured image using a corresponding ghost reflection intensity factor in a ghost reflection compensation model, and obtain an intensity-scaled image as a weighted image. EE8: An electronic device described in EE1, wherein the ghost reflection model relates to the characteristics of a component that causes light reflection that causes ghost reflection when capturing images in the imaging device, and the parameters of the ghost reflection model depend on the characteristics of the component. EE9. The electronic device of EE8, wherein the component includes at least one of a lens and a photographic filter. EE10, the electronic device described in EE8 or EE9, wherein the model of the ghost reflection model includes at least a parameter related to a center shift and related parameters for determining a Gaussian distribution of ghost reflection factors. EE11, the ghost reflection model includes a parameter related to a center shift, and the processing circuit The compensation target image is displaced to its center based on the parameter, rotated around the displaced center as an axis, and then displaced the rotated image in the opposite direction to its center based on the parameter; The electronic device according to EE1, wherein the electronic device is arranged to weight the displaced and rotated image using a ghost reflection compensation model to obtain a weighted image. EE12. The electronic device described in EE11, wherein the displacement and rotation cause ghosts in the displaced and rotated image to correspond to the positions of objects in the original image, and objects in the displaced and rotated image to correspond to the positions of ghosts in the original image. EE13, the processing circuitry The electronic device according to EE1, wherein the electronic device is arranged to obtain a compensated image by subtracting pixel intensities at corresponding locations in the weighted image from the image to be compensated. EE14. The compensated image corresponds to at least two sub-images; The electronic device of EE1, wherein a ghost reflection compensation operation is performed on each sub-image, and a compensated image is obtained by combining at least two compensated sub-images. EE15. The electronic device of EE14, wherein the at least two sub-images include an I image and a Q image obtained by capturing the original image data. EE16. The electronic device according to EE1, wherein the imaging device is an optical imaging device using a photographic filter. EE17: The electronic device according to any one of EE1 to EE16, wherein the imaging device includes a ToF sensor and the image includes a depth image. EE18, a method for compensating for ghost reflections in an image captured by an imaging device, comprising: a calculation step of weighting a compensation target image including ghost reflections using a ghost reflection compensation model relating to the intensity distribution in the image of ghost reflections caused by light reflections in the imaging device when capturing an image; a compensation step for combining the compensated image with the weighted image to remove ghost reflections in the image. EE19. The method of claim EE18, wherein the ghost reflection compensation model is obtained by training from a predetermined number of calibration images, and the ghost reflection compensation model is trained so that the intensity variation of a ghost reflection region in a calibration image to which the ghost reflection compensation model is applied relative to an adjacent region of the ghost reflection region is smaller than or minimized by a predetermined threshold. EE20,Training the ghost reflection compensation model determines intensity variation by the following operations: The calibration image is displaced in the center based on a preset center displacement parameter of the ghost reflection compensation model to be trained, and rotated around the center after displacement as an axis. Furthermore, the rotated image is displaced in the opposite direction in the center based on the parameter; multiplying the displaced and rotated image by a ghost reflection compensation model having preset intensity factor parameters; The method described in EE19, in which intensity variations are obtained by subtracting from the calibration image the pixel intensities at corresponding locations in the model-multiplied image. EE21. The method of EE18, wherein the ghost reflection compensation model includes a ghost reflection factor corresponding to each subregion in the image, the subregion including at least one pixel. EE22. The method of EE20, wherein the ghost reflection compensation model is set so that the ghost reflection factor in a subregion close to the image center is larger than the ghost reflection factor in a subregion close to the image edge. EE23. The method of EE20, wherein the ghost reflection factor is determined based on a Gaussian distribution. EE24, the calculating step The method according to any one of EE20 to EE22, further comprising: performing intensity scaling for each sub-region in the captured image using a corresponding ghost reflection intensity factor in the ghost reflection compensation model, and obtaining an intensity-scaled image as a weighted image. EE25: The method described in EE18, wherein the ghost reflection model relates to the characteristics of a component that produces light reflections that cause ghost reflections when capturing images in the imaging device, and wherein the parameters of the ghost reflection model depend on the characteristics of the component. EE26. The method of EE24, wherein the component includes at least one of a lens and a photographic filter. EE27. The method according to EE24 or 25, wherein the model of the ghost reflection model includes at least a parameter relating to a center shift and related parameters for determining a Gaussian distribution of ghost reflection factors. EE28, the ghost reflection model includes a parameter related to a center shift, and the calculation step Displacing the center of the image to be compensated based on the parameter, rotating the image around the center after displacement as an axis, and further displacing the center of the rotated image in the opposite direction based on the parameter; The method of claim EE18, further comprising weighting the displaced and rotated image with a ghost reflection compensation model to obtain a weighted image. EE29. The method of claim EE28, wherein the displacement and rotation causes ghosts in the displaced and rotated image to correspond to positions of objects in the original image, and objects in the displaced and rotated image to correspond to positions of ghosts in the original image. EE30, the compensation step comprising: A method according to claim EE18, comprising obtaining a compensated image by subtracting pixel intensities at corresponding locations in the weighted image from the image to be compensated. EE31, the compensated image corresponds to at least two sub-images; The method according to claim EE18, wherein a compensated image is obtained by combining at least two compensated sub-images by performing a ghost reflection compensation operation on each sub-image. EE32. The method of EE30, wherein the at least two sub-images include an I image and a Q image obtained by capturing the original image data. EE33. The method of EE18, wherein the imaging device is an optical imaging device using a photographic filter. EE34. The method of any one of EE18 to EE33, wherein the imaging device includes a ToF sensor and the image includes a depth image. EE35, an electronic device for performing ghost reflection compensation on images captured using a direct time-of-flight (dToF) sensor; weighting a pixel histogram to be compensated, which includes ghost reflections and is obtained from the captured raw data, using a ghost reflection compensation model; An electronic device comprising a processing circuit arranged to combine a pixel histogram to be compensated with a weighted pixel histogram to remove ghost reflections. EE36, the ghost reflection model includes a parameter related to a center shift, and the processing circuit The histogram to be compensated is center-shifted based on the parameter, and rotated around the center after the shift, and the center of the rotated histogram is shifted in the opposite direction based on the parameter; The electronic device according to EE35, arranged to weight the displaced and rotated histogram using a ghost reflection compensation model to obtain a weighted histogram. EE37. The electronic device of EE35, wherein the displacement and rotation cause ghost reflection peak values ​​in the displaced and rotated histogram to correspond to target peak values ​​in the original histogram. EE38, the processing circuitry An electronic device according to EE35, arranged to obtain a compensated histogram by subtracting from the compensated histogram values ​​at corresponding positions in the weighted histogram. EE39, a method for performing ghost reflection compensation on an image captured using a direct time-of-flight (dToF) sensor, comprising: a calculating step of weighting a pixel histogram to be compensated, including ghost reflections, obtained from the captured raw data using a ghost reflection compensation model; a compensation step of combining the pixel histogram to be compensated with the weighted pixel histogram to remove ghost reflections. EE40, the ghost reflection model includes a parameter related to a center shift, and the calculation step a histogram to be compensated is centered based on the parameter, rotated around the center after the center shift, and then center-shifted the rotated histogram in the opposite direction based on the parameter; The method of claim EE39, further comprising weighting the displacement and rotation histograms using a ghost reflection compensation model to obtain a weighted histogram. EE41. The method of EE39, wherein the displacement and rotation causes ghost reflection peak values ​​in the displaced and rotated histogram to correspond to target peak values ​​in the original histogram. EE42. The method of EE39, wherein the compensating step further comprises obtaining a compensated histogram by subtracting values ​​at corresponding positions in the weighted histogram from the compensated histogram. EE43, at least one processor; and at least one storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform a method described in any one of EE18-34 and 39-42. EE44, a storage medium storing instructions that, when executed by a processor, cause the method of any one of EE18-34 and 39-42 to be performed.

[0129] Although the present disclosure and its advantages have been described in detail, it should be understood that various modifications, substitutions, and alterations can be made without departing from the spirit and scope of the present disclosure, which is limited by the appended claims. Furthermore, the terms "comprises," "includes," or any other variation thereof in the embodiments of the present disclosure indicate a non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements but also other elements not expressly stated or elements inherent in such process, method, article, or device. Absent further limitations, an element limited by "comprises a..." does not exclude the inclusion of other identical elements in a process, method, article, or device that includes said element.

[0130] Although several specific embodiments of the present disclosure have been described in detail, it should be understood by those skilled in the art that the above embodiments are used for illustrative purposes only and do not limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments can be combined, modified, or substituted without departing from the scope and spirit of the present invention. The scope of the present disclosure is limited by the appended claims.

Claims

1. 1. An electronic device for compensating for ghost reflections in an image captured by an imaging device, comprising: weighting the compensation target image including the ghost reflection using a ghost reflection compensation model relating to the intensity distribution in the image of the ghost reflection caused by light reflection in the imaging device when capturing an image; An electronic device comprising processing circuitry arranged to combine the compensated image and the weighted image to remove ghost reflections in the image.

2. 2. The electronic device of claim 1, wherein the ghost reflection compensation model is obtained by training from a predetermined number of calibration images, and the ghost reflection compensation model is trained so that an intensity variation of a ghost reflection region in a calibration image to which the ghost reflection compensation model is applied relative to an adjacent region of the ghost reflection region is smaller than or minimized by a predetermined threshold.

3. To train the ghost reflection compensation model, the intensity variation is determined by the following operations: The calibration image is displaced in the center based on a preset center displacement parameter of the ghost reflection compensation model to be trained, and rotated around the center after displacement as an axis. Furthermore, the rotated image is displaced in the opposite direction in the center based on the parameter; multiplying the displaced and rotated image by a ghost reflection compensation model having preset intensity factor parameters; The electronic device of claim 2 , wherein the intensity variations are obtained by subtracting from the calibration image pixel intensities at corresponding locations in the model-multiplied image.

4. The electronic device of claim 1 , wherein the ghost reflection compensation model includes a ghost reflection factor corresponding to each subregion in the image, the subregion including at least one pixel.

5. The electronic device of claim 4 , wherein the ghost reflection compensation model is set so that a ghost reflection factor in a sub-region near an image center is larger than a ghost reflection factor in a sub-region near an image edge.

6. The electronic device of claim 4 , wherein the ghost reflection factor is determined based on a Gaussian distribution.

7. The processing circuitry 7. The electronic device according to claim 4, wherein the electronic device is arranged to perform intensity scaling for each sub-region in the captured image using a corresponding ghost reflection intensity factor in the ghost reflection compensation model to obtain an intensity-scaled image as a weighted image.

8. The electronic device according to claim 1 , wherein the ghost reflection model relates to a characteristic of a component that causes light reflection that causes ghost reflection when capturing an image in the imaging device, and parameters of the ghost reflection model depend on the characteristic of the component.

9. The electronic device of claim 8 , wherein the component includes at least one of a lens and a photographic filter.

10. 10. The electronic device according to claim 8, wherein the ghost reflection model includes at least a parameter relating to a center shift and related parameters for determining a Gaussian distribution of ghost reflection factors.

11. The ghost reflection model includes a parameter related to a center shift, and the processing circuitry The compensation target image is displaced to its center based on the parameter, rotated around the displaced center as an axis, and then displaced the rotated image in the opposite direction to the center based on the parameter; 2. An electronic device according to claim 1, arranged to weight the displaced and rotated image using a ghost reflection compensation model to obtain a weighted image.

12. 12. The electronic device of claim 11, wherein the displacement and rotation causes ghosts in the displaced and rotated image to correspond to positions of objects in the original image, and objects in the displaced and rotated image to correspond to positions of ghosts in the original image.

13. The processing circuitry 2. An electronic device according to claim 1, arranged to obtain the compensated image by subtracting from the image to be compensated pixel intensities at corresponding locations in the weighted image.

14. the compensated image corresponds to at least two sub-images; The electronic device of claim 1 , further comprising: performing a ghost reflection compensation operation on each sub-image; and combining at least two compensated sub-images to obtain a compensated image.

15. The electronic device of claim 14 , wherein the at least two sub-images include an I image and a Q image obtained by capturing the original image data.

16. The electronic device according to claim 1 , wherein the imaging device is an optical imaging device using a photographic filter.

17. The electronic device according to claim 1, wherein the imaging device includes a Time of Flight sensor, and the image includes a depth image.

18. 1. A method for compensating for ghost reflections in an image captured by an imaging device, comprising: a calculation step of weighting a compensation target image including ghost reflections using a ghost reflection compensation model relating to the intensity distribution in the image of ghost reflections caused by light reflections in the imaging device when capturing an image; a compensation step of combining the compensated image with the weighted image to remove ghost reflections in the image.

19. 19. The method of claim 18, wherein the ghost reflection compensation model is obtained by training from a predetermined number of calibration images, and the ghost reflection compensation model is trained so that an intensity variation of a ghost reflection region in a calibration image to which the ghost reflection compensation model is applied relative to an adjacent region of the ghost reflection region is smaller than or minimized by a predetermined threshold.

20. To train the ghost reflection compensation model, the intensity variation is determined by the following operations: The calibration image is displaced in the center based on a preset center displacement parameter of the ghost reflection compensation model to be trained, and rotated around the center after displacement as an axis. Furthermore, the rotated image is displaced in the opposite direction in the center based on the parameter; multiplying the displaced and rotated image by a ghost reflection compensation model having preset intensity factor parameters; 20. The method of claim 19, wherein the intensity variations are obtained by subtracting from the calibration image pixel intensities at corresponding locations in the model-multiplied image.

21. 20. The method of claim 18, wherein the ghost reflection compensation model includes a ghost reflection factor corresponding to each subregion in the image, the subregion including at least one pixel.

22. The method of claim 20, wherein the ghost reflection compensation model is set so that a ghost reflection factor in a sub-region near an image center is larger than a ghost reflection factor in a sub-region near an image edge.

23. The method of claim 20 , wherein the ghost reflection factor is determined based on a Gaussian distribution.

24. The calculation step The method according to any one of claims 20 to 22, further comprising: performing intensity scaling for each sub-region in the captured image using a corresponding ghost reflection intensity factor in the ghost reflection compensation model, and obtaining the intensity-scaled image as a weighted image.

25. The method of claim 18 , wherein the ghost reflection model relates to a property of a component that produces a light reflection that causes a ghost reflection during imaging in the imaging device, and wherein parameters of the ghost reflection model depend on the property of the component.

26. 25. The method of claim 24, wherein the component comprises at least one of a lens and a photographic filter.

27. 26. A method according to claim 24 or 25, wherein the model of the ghost reflection model comprises at least a parameter relating to a center shift and related parameters for determining a Gaussian distribution of ghost reflection factors.

28. The ghost reflection model includes a parameter related to a center shift, and the calculating step Displacing the center of the image to be compensated based on the parameter, rotating the image around the center after displacement as an axis, and further displacing the center of the rotated image in the opposite direction based on the parameter; 20. The method of claim 18, further comprising weighting the displaced and rotated images with a ghost reflection compensation model to obtain a weighted image.

29. 29. The method of claim 28, wherein the displacement and rotation causes ghosts in the displaced and rotated image to correspond to positions of objects in the original image, and objects in the displaced and rotated image to correspond to positions of ghosts in the original image.

30. The compensation step includes:

20. The method of claim 18, comprising obtaining the compensated image by subtracting pixel intensities at corresponding locations in the weighted image from the compensated image.

31. the compensated image corresponds to at least two sub-images; 20. The method of claim 18, wherein a ghost reflection compensation operation is performed on each sub-image, and the compensated image is obtained by combining at least two compensated sub-images.

32. 31. The method of claim 30, wherein the at least two sub-images include an I-image and a Q-image obtained by capturing the original image data.

33. 20. The method of claim 18, wherein the imaging device is an optical imaging device using a photographic filter.

34. The method of any one of claims 18 to 33, wherein the imaging device comprises a Time of Flight sensor and the image comprises a depth image.

35. 1. An electronic device for performing ghost reflection compensation on an image captured using a direct time-of-flight (dToF) sensor, comprising: weighting a pixel histogram to be compensated, which includes ghost reflections and is obtained from the captured raw data, using a ghost reflection compensation model; An electronic device comprising a processing circuit arranged to combine a pixel histogram to be compensated with a weighted pixel histogram to remove ghost reflections.

36. The ghost reflection model includes a parameter related to a center shift, and the processing circuitry The histogram to be compensated is center-shifted based on this parameter, and rotated around the center after the shift, and then the center of the rotated histogram is shifted in the opposite direction based on this parameter; 36. An electronic device according to claim 35, arranged to weight the displacement and rotation histograms using a ghost reflection compensation model to obtain a weighted histogram.

37. 36. The electronic device of claim 35, wherein the shifting and rotation causes ghost reflection peak values ​​in the shifted and rotated histogram to correspond to target peak values ​​in the original histogram.

38. The processing circuitry 36. Electronic equipment according to claim 35, arranged to obtain the compensated histogram by subtracting from the compensated histogram values ​​at corresponding positions in the weighted histogram.

39. 1. A method for performing ghost reflection compensation on an image captured using a direct time-of-flight (dToF) sensor, comprising: a calculating step of weighting a pixel histogram to be compensated, including ghost reflections, obtained from the captured raw data using a ghost reflection compensation model; a compensation step of combining the pixel histogram to be compensated with the weighted pixel histogram to remove ghost reflections.

40. The ghost reflection model includes a parameter related to a center shift, and the calculating step a histogram to be compensated is centered based on the parameter, rotated around the center after the center shift, and then center-shifted the rotated histogram in the opposite direction based on the parameter; 40. The method of claim 39, further comprising weighting the displacement and rotation histograms with a ghost reflection compensation model to obtain a weighted histogram.

41. 40. The method of claim 39, wherein the shifting and rotation causes ghost reflection peak values ​​in the shifted and rotated histogram to correspond to target peak values ​​in the original histogram.

42. 40. The method of claim 39, wherein the compensating step further comprises obtaining the compensated histogram by subtracting values ​​at corresponding positions in the weighted histogram from the compensated histogram.

43. at least one processor; and at least one storage device storing instructions that, when executed by said at least one processor, cause said at least one processor to perform a method according to any one of claims 18 to 34 and 39 to 42.

44. A storage medium storing instructions which, when executed by a processor, cause the method of any one of claims 18 to 34 and 39 to 42 to be performed.

Images