Ghost reflection compensation method and apparatus

Abstract

An electronic device for compensating ghost reflections in images taken by an image capturing device, comprising processing circuitry configured to weight an image to be compensated containing ghost reflections by using a ghost reflection compensation model, wherein the ghost reflection compensation model relates to an intensity distribution of ghost reflections in the image caused by light reflections in the image capturing device at the time of taking; and combine the image to be compensated and the weighted image to eliminate ghost reflections in the image.

Description

Technical Field

[0001] This disclosure relates to image processing, and more particularly to image compensation processing. Background Technology

[0002] In recent years, object detection / recognition / comparison / tracking in still images or a series of moving images (such as videos) has been widely and significantly applied in image processing, computer vision, and pattern recognition, playing a crucial role in these fields. Objects can be human body parts, such as faces, hands, and bodies, other living organisms or plants, or any other object desired for detection. Object recognition is one of the most important computer vision tasks, aiming to identify or verify specific objects based on input photographs / videos, thereby accurately obtaining relevant information about the objects. In particular, in some applications, when performing object recognition based on images captured by camera devices, it is necessary to accurately identify the detailed information of the objects from the images, thereby accurately recognizing the objects.

[0003] However, images acquired by current camera devices often contain various noises, which degrade image quality and may result in inaccurate or even incorrect details, thereby affecting the imaging and recognition of objects.

[0004] Therefore, improved techniques are needed to enhance image processing and further suppress noise.

[0005] Unless otherwise stated, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise stated, no problem recognized with respect to one or more methods should be assumed to be recognized in any prior art based on this section. Summary of the Invention

[0006] One object of this disclosure is to improve image processing to further suppress noise in images, especially noise related to phantom reflections, thereby improving image quality.

[0007] In particular, captured images may contain phantoms, resulting in poor image quality. This disclosure utilizes a phantom reflection compensation model to compensate for image artifacts, effectively removing phantoms and obtaining high-quality images.

[0008] In one aspect, an electronic device is provided for compensating for phantom reflections in an image captured by a camera device, comprising processing circuitry configured to: weight an image to be compensated containing phantom reflections using a phantom reflection compensation model, wherein the phantom reflection compensation model relates to the intensity distribution of phantom reflections in the image caused by light reflections in the camera device during the capture; and combine the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0009] On the other hand, a method for compensating for phantom reflections in an image captured by a camera device is provided, comprising the following steps: a calculation step for weighting an image to be compensated containing phantom reflections using a phantom reflection compensation model, wherein the phantom reflection compensation model relates to the intensity distribution of phantom reflections in the image caused by light reflections in the camera device during the capture; and a compensation step for combining the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0010] In another aspect, there is a method comprising at least one processor and at least one storage device, wherein the at least one storage device stores instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform the method described herein.

[0011] On the other hand, a storage medium storing instructions is provided that, when executed by a processor, can cause the methods described herein to be performed.

[0012] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. In the drawings, similar reference numerals indicate similar items.

[0014] Figure 1 An overview diagram of ToF technology is shown.

[0015] Figure 2A This shows light reflection caused by a close-up object in close-up photography.

[0016] Figure 2B The image shows light reflection caused by a photographic filter.

[0017] Figure 3 A schematic diagram of the phantom phenomenon in the image is shown.

[0018] Figures 4A to 4C Examples of phantom reflections in confidence and depth images are shown.

[0019] Figure 5A The image processing flow including scattering compensation in the present disclosure is shown.

[0020] Figure 5B An exemplary scattering compensation operation in the present disclosure is shown.

[0021] Figure 5CThe results of scattering compensation in the scheme of this disclosure are shown.

[0022] Figure 6 A flowchart of a phantom reflection compensation method according to an embodiment of the present disclosure is shown.

[0023] Figure 7 A block diagram of an electronic device capable of phantom reflection compensation according to an embodiment of the present disclosure is shown.

[0024] Figure 8 An illustration of a phantom reflection compensation model according to an embodiment of the present disclosure is shown.

[0025] Figure 9 The extraction of a phantom reflection compensation model according to an embodiment of the present disclosure is illustrated.

[0026] Figure 10A An exemplary basic flow for extracting a phantom reflection compensation model from a calibration image according to an embodiment of the present disclosure is shown. Figure 10B and 10C A schematic diagram of an exemplary image rotation operation according to an embodiment of the present disclosure is shown.

[0027] Figure 11 An image processing flow including phantom reflection compensation according to embodiments of the present disclosure is shown.

[0028] Figure 12 An image processing flow including phantom reflection compensation according to embodiments of the present disclosure is shown.

[0029] Figure 13 The results of performing phantom reflection compensation according to an embodiment of the present disclosure are shown.

[0030] Figure 14A and 14B Phantom reflection compensation for a dToF sensor according to an embodiment of the present disclosure is illustrated.

[0031] Figure 15 The phantom reflection compensation for a point ToF according to an embodiment of the present disclosure is illustrated.

[0032] Figure 16 An imaging apparatus according to an embodiment of the present disclosure is shown.

[0033] Figure 17 A block diagram illustrating an exemplary hardware configuration of a computer system capable of implementing embodiments of the present invention is shown.

[0034] While the embodiments described in this disclosure may be readily modified and alternatively implemented, specific embodiments thereof are shown by way of example in the accompanying drawings and are described in detail herein. However, it should be understood that the drawings and the detailed description thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the claims. Detailed Implementation

[0035] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the embodiments are described in the specification. However, it should be understood that many implementation-specific settings must be made in carrying out the embodiments to achieve the developer's specific goals, such as complying with constraints related to the apparatus and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the present disclosure.

[0036] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only processing steps and / or equipment structures closely related to at least the scheme according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0037] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it need not be discussed again in subsequent drawings.

[0038] In this disclosure, the terms “first,” “second,” etc., are used only to distinguish elements or steps and are not intended to indicate chronological order, preference, or importance.

[0039] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that, in the context of this specification, the type of image is not specifically limited, as long as such an image can be processed for information extraction or detection. Furthermore, an image can be an original image or a processed version of that image, such as a version of an image that has undergone preliminary filtering or preprocessing prior to the operations of this application.

[0040] When capturing a scene using a camera, the resulting image typically contains noise, which may include phenomena such as scattering and ghost reflection. While these noise phenomena can enhance the artistic effect of the image in some cases, such as landscape photos taken with RGB sensors, in many cases, this noise is particularly detrimental to all sensors that use light to measure distance (e.g., Time-to-Flight (ToF) sensors, structured light sensors for 3D measurements, etc.) compared to RGB sensors. The following will briefly describe the noise problems that occur when using Time-to-Flight (ToF) technology and ToF sensors to capture images, with accompanying figures. It should be noted that these noise problems also exist for image sensors based on other technologies, such as structured light sensors and RGB sensors, due to the same principles; for the sake of brevity, they will not be described separately in this disclosure.

[0041] In time-of-flight technology, a light emitter is used to illuminate a scene, and the time it takes for the light to return to the sensor is measured—the time difference between emission and reception. Based on this measured time, the distance to the scene can be calculated as d = ct / 2, where c is the speed of light and t is the measured time. Figure 1 As shown. Light emission can be accomplished, for example, through pulses (direct time-of-flight) or continuous waves (indirect time-of-flight). However, for cameras using ToF-based sensors, noise phenomena may occur during image capture, such as scattering and phantom reflections. Such noise phenomena may be due to light reflections within the camera.

[0042] In particular, when using a camera to photograph close-up objects, the approaching object will reflect a large amount of moving light back towards the camera, acting like a bright light source to the camera, thus causing significant light reflection and scattering within the camera. This will be described below with reference to the accompanying drawings. Figure 2A As shown, a scene containing three objects is photographed using a camera module, which includes a lens and a sensor, i.e., an imager. Objects 1, 2, and 3 are at distances r(1), r(2), and r(3) from the imager, respectively. Light emitted towards these three objects is reflected by the objects and returns to the corresponding positions of the imager in the camera module, i.e., imaging positions S(1), S(2), and S(3). Object 1 is very close to the camera module, resulting in high intensity of reflected light, which bounces inside the module (e.g., between the lens and the imaging device). In the captured image, the signal from object 1 will be scattered around its position and mixed with the signals from objects 2 and 3. The ToF sensor will combine these signals, providing an incorrect depth for objects 2 and 3 (the measured depth is between distance r(1) and distance r(2) or r(3).

[0043] Furthermore, cameras often have a photographic filter placed in front of the lens, which can cause ghosting reflections. For example... Figure 2B As shown, normally, light passes through the filter and lens and is incident on the imaging point of the sensor, as indicated by the solid line and the arrow above it. However, some light is reflected from the imaging point toward the lens and transmitted through the lens, as indicated by the reverse arrow. At this time, due to the presence of the photographic filter, the signal is reflected by the filter toward the lens and then incident on the sensor through the lens, as indicated by the dashed arrow. This is as if light from different directions is forming an image on the sensor, thus producing a phantom image in addition to the imaging point.

[0044] Although this phenomenon is also visible on RGB sensors, for example when photographing close-up or bright objects, ghost images will appear in the vicinity or at a centrally symmetrical position, such as... Figure 3 The area circled in the middle indicates a lighter-colored image compared to a bright white patch, but the ToF sensor detects this phantom, meaning an incorrect depth is detected in front of the sensor. The following will explain the impact of phantom reflections in camera-captured images with reference to the accompanying diagram.

[0045] Figure 4A The image shown is an RGB image of a scene captured by a mobile phone in bokeh mode with an integration time of 300 μs. A near-field object is present on the right side of the image. Figure 4B The confidence image of the scene is shown, indicating the confidence level of the depth information in the scene image. Specifically, each pixel in the confidence image indicates the confidence level of the depth provided by that pixel in the scene image. It can be seen that the right side indicates near objects, which appear bright white due to their proximity. Due to the influence of near objects as described above, a scattering effect (scratchy white dots near the white) is reflected in the middle of the image, while a phantom reflection (scratchy white areas on a dark background) is reflected on the left side of the image. Figure 4C The image shows a depth map of the scene, indicating the depth information of objects in the scene. Each pixel in the depth map indicates the distance from the camera to the objects. It can be seen that a gray shadow resembling an object appears on the left side of the depth map due to phantom reflections, which is often mistakenly interpreted as providing depth information. As can be seen above, in the captured image, the scattering portion lies between the object image and the phantom reflection portion. Due to the presence of phantom reflections, incorrect depth information for the nearby object is provided on the left side of the image; the depth is usually very shallow, leading to the inability to correctly identify information about the nearby object, especially its depth.

[0046] As can be seen above, when using camera systems that include sensors for optical ranging (especially those including ToF sensors) to capture scenes, this phantom reflection phenomenon is very detrimental, leading to the detection of incorrect depth. Incorrect depth information negatively impacts the provision of high-quality images and many subsequent applications. However, current technologies do not specifically compensate for phantom reflections in the processing of captured images, thus failing to effectively eliminate them to obtain accurate object detail information, particularly depth information.

[0047] Figure 5A The scattering compensation process in image processing proposed in this disclosure is illustrated, wherein scattering compensation is performed on the raw ToF data, and then subsequent data processing is performed on the scatter-compensated data to obtain a confidence image and a depth image. This subsequent data processing may include processes known in the art for generating confidence images and depth images, which will not be described in detail here.

[0048] As previously mentioned, scattering effects may result from light from nearby objects reflecting between the sensor and lens in a camera setup. This creates some blurring around the object, making the edges of the image less sharp. This blurring can be modeled using an appropriate function that describes the characteristics of the blur generated by points or pixels, such as the PSF (Point Spread Function). Based on the modeling results, an algorithm can then be applied to eliminate the specific blur generated by all points / pixels in the image. This algorithm could be, for example, a deconvolution algorithm. It should be noted that other suitable functions and algorithms known in the art can be used for modeling and compensating for scattering, which will not be described in detail here.

[0049] Figure 5B An exemplary scattering compensation operation according to an embodiment of this disclosure is illustrated. When a bright white card is present in the center of a scene as the subject of a photograph, significant blurring occurs around the object, and may also occur throughout the entire image, as shown in the left image. This scattering is eliminated by utilizing a deconvolution algorithm corresponding to the modeling function (e.g., the PSF function, or inverse transform), resulting in sharper edges on the white patch, removal of blurring in the image, and compensation for the scattering effect, as shown in the right image.

[0050] However, scattering compensation cannot effectively eliminate phantom reflections. Figure 5C The results of scattering compensation are shown, and the image contains elements corresponding to, for example... Figure 4AThe confidence map and depth map of the RGB image are shown. (a) shows the confidence map and depth map corresponding to the original scene image from top to bottom, which includes scattering and phantom reflections. (b) shows the confidence map and depth map after scattering compensation from top to bottom. It can be seen that even if scattering noise in the image is removed by scattering compensation, phantom reflections (shadow areas) still exist on the left side of the image, and this phantom will still cause incorrect depth measurement.

[0051] Therefore, one objective of this disclosure is to effectively eliminate phantom reflections. In particular, this disclosure proposes to use an extracted phantom reflection compensation model to weight the data / image to be processed, and to effectively eliminate phantom reflections by using the weighted data / image to compensate the data / image to be processed.

[0052] As mentioned above, this phantom reflection phenomenon is particularly caused by reflections caused by filters in the camera system. Therefore, the phantom reflection compensation technique according to this disclosure is especially advantageous for camera systems that additionally use filters, regardless of the type of sensor, whether it is a ToF sensor, a structured light sensor, an RGB sensor, or other types of sensors.

[0053] The embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings.

[0054] Figure 6 A flowchart of a method for compensating for phantom reflections in an image captured by a camera device, according to an embodiment of the present disclosure, is shown. The method 600 may include a calculation step S601 for weighting an image to be compensated containing phantom reflections using a phantom reflection compensation model; and a compensation step S602 for combining the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0055] It should be noted that the camera device to which the technical solutions of this disclosure can be applied can include various types of optical camera devices, as long as the image captured by the camera device may produce phantom reflections due to light reflection. For example, the camera device may include the aforementioned camera using a photographic filter. For example, the camera device may include the aforementioned camera for 3D imaging. For example, the camera device may include the aforementioned camera including a ToF-based sensor. For example, the camera device may also correspond to the aforementioned camera for close-up shooting. And so on.

[0056] It should be noted that the image to be compensated can be any suitable image, such as the original image obtained by a camera device, or an image that has undergone specific processing, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should also be noted that preprocessing operations may include other types of preprocessing operations known in the art, which will not be described in detail here.

[0057] According to embodiments of this disclosure, the phantom reflection compensation model essentially reflects the characteristics of light reflection that causes phantom reflection in the imaging device; that is, it is a model obtained by modeling the light reflection that causes phantom reflection. For example, as mentioned above, this light reflection may be caused by photographic filters and / or lenses, meaning the light reflection characteristics correspond to the characteristics of photographic filters and / or lenses. Therefore, the phantom reflection compensation model is essentially a model obtained by modeling the characteristics of photographic filters and / or lenses. It should be noted that the model is not limited to this. When other components in the imaging device may cause light reflection that causes phantom reflection, or even other optical phenomena that cause phantom reflection, the model is equally equivalent to modeling based on the characteristics of such other components or other optical phenomena.

[0058] According to one embodiment, the phantom reflection compensation model can be related to the intensity distribution of phantom reflections in an image. For example, the phantom reflection compensation model can be related to the intensity distribution in an image affected by phantom reflections, such as the intensity distribution of the entire image, or particularly the intensity distribution at the locations of the source object and the phantom.

[0059] According to one embodiment, the phantom reflection compensation model can indicate the phantom reflection intensity factor at a specific sub-region in the image, wherein the sub-region contains at least one pixel. As an example, the specific sub-region can be various sub-regions covering the entire image. As another example, the specific sub-region is a sub-region in the image corresponding to both the source object and the location of the phantom reflection.

[0060] According to embodiments of this disclosure, the phantom reflection intensity factor can be derived based on the intensity distribution in an image, preferably based on the intensity distribution of phantom reflections in the image, and can refer to a factor configured to minimize the variation in the compensated scene image, particularly minimizing the intensity variation between the image at the phantom reflection location and the images in its neighboring regions, in order to remove phantom reflections. As an example, this variation can refer to intensity variation, such as the image intensity variation between the phantom reflection area and its surrounding neighboring regions. In this case, the factor can be referred to as a phantom reflection compensation factor.

[0061] According to embodiments of this disclosure, the phantom reflection compensation model can be represented in various forms. The phantom reflection compensation model will now be described exemplarily with reference to the accompanying drawings. Figure 8 An exemplary phantom reflection compensation model according to an embodiment of the present disclosure is shown, wherein (a) a planar representation of the model is shown, and (b) a three-dimensional graphical representation of the model is shown, wherein the horizontal and vertical axes indicate the planar size of the model, which corresponds to the size of the image, and the vertical axis indicates the numerical value of the phantom reflection intensity factor of the model.

[0062] According to an embodiment, the model may include parameters such as intensity factor, central shift, and size. Specifically, these parameters should be set to make the model match the characteristics of the components in the imaging device as closely as possible, as described above.

[0063] The parameters related to this center offset may include parameters cx and cy, which indicate the offset of the reference position relative to the image center during the compensation operation (e.g., rotation and shift), for example, in the horizontal and vertical directions, respectively. Specifically, the position indicated by parameters cx and cy actually corresponds to the central axis of the image to be rotated, meaning the image will undergo an off-center rotation. As an example, cx and cy can directly indicate the offset of the central axis used to rotate the image in subsequent processing relative to the image center, so that the central axis can be moved to the image center based on this offset before rotation. As another example, cx and cy can correspond to the center point position of the model illustration, such as... Figure 8 The planar representation shown indicates the center point position, allowing the central axis to be moved from this position to the center of the image before rotation. According to an embodiment, cx and cy may depend at least on the characteristics of the lens, and of course, on the characteristics of other components. The purpose of determining this center offset is to properly position the image so that the phantom in the rotated image aligns with the target in the original image; specifically, the values ​​of cx and cy can be determined, for example, through experimental or calibration measurements.

[0064] Size-related parameters may include `width` and `length`, corresponding to the width and length of the image (in pixels), respectively. Considering that for ease of application, the model illustration should correspond to the image, these width and height can also indicate the width and length of the model illustration, such as... Figure 8 The plane is shown in the 3D illustration in (b). The width and length can depend on the pixel arrangement of the sensor.

[0065] Parameters related to the intensity factor may include parameters indicating the intensity factor distribution corresponding to the image. The intensity factor distribution can be expressed by an appropriate distribution function to indicate, for example, the phantom reflection intensity factor at each pixel location in the image, such as... Figure 8 (b) Visually perceptible.

[0066] According to an embodiment, the intensity factor distribution is set such that the phantom reflection intensity factor in the sub-region near the image center is higher than that in the sub-region near the image edge. When phantom reflections exist in the captured image, the phantom reflection exhibits different light intensities depending on its location, particularly gradually weakening from the center to the edge. The adverse effects on depth measurement also gradually decrease as the intensity decreases. Therefore, by setting the intensity factor as described above, phantom reflections in the image can be appropriately weakened or even eliminated. For example, the greater the intensity of the phantom reflection, the larger the factor used to weaken and eliminate it, thereby providing accurate and effective phantom reflection compensation.

[0067] According to an embodiment, the intensity factor distribution can be determined according to a specific distribution function. According to an embodiment, at least one specific distribution function may be included, and each function may have a corresponding weight. As an example, intensity factor = αf(1) + βf(2) + ..., where f(1) and f(2) respectively indicate the specific function, and α and β respectively indicate the weight of each function. According to an embodiment, the parameters of the distribution function and the weights used for the distribution function can be set, for example, according to the characteristics of light reflection that causes phantom reflection in the imaging device, especially the optical characteristics of the component causing the reflection, in order to match (approximate) the characteristics as closely as possible. For example, the values ​​of the corresponding parameters can be determined based on empirical values ​​obtained from prior testing or experiments, or they can be adjusted based on empirical values ​​through further calibration operations.

[0068] Preferably, the intensity factor distribution follows a Gaussian distribution. A Gaussian function is calculated using two parameters: the first parameter is std, which is the standard deviation of the Gaussian function, and mu is the average value of the Gaussian function. It should be noted that these two Gaussian function parameters, std and mu, can be related to the intensity of reflected light in the imaging device that causes phantom reflections. Specifically, the parameters std and mu can depend on the characteristics (e.g., optical characteristics) of components in the imaging device that may induce light reflections, such as the characteristics of the aforementioned lenses, photographic filters, etc. According to an embodiment, the expression for the intensity factor may contain a specific number of Gaussian functions, each of which can be weighted accordingly.

[0069] As an example, the model can be represented as follows:

[0070] Center offset:

[0071] cx=-2.5, cy=-2.5, width=240, length=180

[0072] Intensity factor = 1.3 * Gaussian (std = 38, mu = 0) + 0.6 * Gaussian (std = 50, mu = 60)

[0073] It should be noted that the parameters of the Gaussian function given in the intensity factor expression, especially the parameters of the Gaussian function itself and the weights of the Gaussian function, can be selected based on the optical characteristic curves of the aforementioned optical components, such as filters and lenses, for example, to make the intensity factor distribution better correspond to (e.g., inverse matching) the optical characteristic curves in order to eliminate the effects of light reflection caused by these optical characteristics. Alternatively, the parameter can be initially set based on experience and then adjusted based on the empirical value through further calibration operations.

[0074] It should be noted that the use of a Gaussian function to represent the intensity factor in the above-described phantom reflection compensation model is merely exemplary. Other distribution functions can also be used in this disclosure, as long as the distribution function enables the model to accurately match the intensity distribution of phantom reflections in the image, particularly matching the characteristics of light reflections in the imaging device that cause phantom reflections, such as the characteristics of the components that trigger the light reflections. As an example, the distribution function can be other functions with a normal distribution. As another example, functions with other distributions can be used, such as Cauchy distribution, gamma distribution, etc.

[0075] According to one embodiment, the phantom reflection compensation model is extracted from a predetermined number of calibration images. As an example, the calibration images are obtained by capturing images of a specific scene used for calibration. This predetermined number can be specified empirically or adopted from a previous calibration.

[0076] Figure 9 The diagram shows that the model was extracted from multiple images. The left side indicates the images used to extract the phantom reflection compensation model. These images were obtained for calibration scenes with white charts in different locations: at the four corners and the center. Each image contains both bright white patches indicating the white charts and pale patches indicating phantom reflections. It should be noted that the number and arrangement of calibration images are not limited to this, as long as they adequately reflect the information about phantom reflections. For example, the white charts could be placed in more locations to obtain more calibration images, thus reflecting the information about phantom reflections in the scene in greater detail.

[0077] According to embodiments of this disclosure, various methods can be used to extract a phantom reflection compensation model from calibration images. According to one embodiment, the phantom reflection compensation model can be determined such that the intensity variation of a specific number of calibration images after applying the model meets a specific requirement. As previously mentioned, the intensity variation can refer to the image intensity variation between the phantom reflection area and its adjacent surrounding areas, i.e., the image intensity difference between the phantom reflection area and its adjacent surrounding areas. According to another embodiment, optionally or additionally, the phantom reflection compensation model can also be determined such that phantom reflections in the images after applying the model are eliminated or mitigated. The elimination or mitigation of phantom reflections can refer to the depth / RGB information measured at the location of the phantom reflection being consistent with or close to the real scene. In other words, the phantom reflection compensation model is extracted based on the condition that the intensity variation (and optionally or additionally, the degree of phantom reflection mitigation) meets a specific requirement.

[0078] According to one embodiment, the requirement for intensity variation to meet the standard may refer to the statistical values ​​of the variations obtained from all or at least some of the scene images, such as the sum or average of the variations of these scene images, meeting the standard. As an example, the specific requirement may refer to the intensity variation being less than a specific threshold, or the specific requirement may refer to the image having minimal variation. Thus, meeting this specific requirement means that the intensity of the phantom reflection area is essentially the same as the intensity of its adjacent surrounding areas, with small, smooth variations and no boundaries, thereby essentially eliminating the influence of phantom reflections.

[0079] The model extraction process can be performed in various ways. According to an embodiment, iterative methods can be used. As an example, initial values ​​for the parameters of the phantom reflection compensation model can be set. The set model is then used to calculate the aforementioned image intensity variation (and optionally, additionally, the degree of phantom reflection reduction), and to verify whether the image intensity variation (and optionally, additionally, the degree of phantom reflection reduction) meets specific requirements. If not, the parameter settings are adjusted, and the next operation is performed until the intensity variation meets the specific requirements, and the corresponding model is determined to be the desired phantom reflection compensation model for subsequent image compensation processing. It should be noted that the model parameters determined through iterative operations can include at least the relevant parameters of the model's distribution function, such as the parameters of the Gaussian function itself and the weights of each Gaussian function in the presence of two or more Gaussian functions.

[0080] The following will refer to Figure 10A This describes the process for determining the intensity variation of a calibration image during a model extraction operation. The calibration image contains a white test card and its phantom reflection, and can be used as the image for model extraction. It should be noted that this determination process can be performed separately for each image used for model extraction.

[0081] First, the image transformation is performed based on the center offset parameter of the phantom reflection compensation model. As mentioned earlier, the center offset parameter indicates the amount of offset of the rotation center relative to the image center. Therefore, the image transformation essentially instructs the image to undergo an off-center rotation, that is, to rotate around a central axis that is off-center from the image center. Figure 10B The example shown is a direct 180-degree rotation, where the position of the cross symbol corresponds to the position of the eccentric axis indicated by the center offset. Image transformation can refer to rotating directly around this eccentric axis to obtain the final image.

[0082] Image transformations can also be performed through shifting and rotation operations, i.e., a process of shifting, rotating, and then shifting again. For example... Figure 10C As shown, the rotation center is first shifted according to the parameter value (e.g., moved to the center according to cx and cy), then rotated around the central axis at the center, and then the shifted center is shifted in the opposite direction according to the parameter value (i.e., moved according to -cx and -cy).

[0083] It should be noted that the rotation can be any angle, as long as the position of the source object and the phantom reflection in the rotated image overlaps with the position of the phantom reflection and the position of the source object in the previous image, respectively. As a preferred example, the rotation can be 180°. According to one implementation, the shift and rotation are such that the position of the phantom in the shifted and rotated image corresponds to the position of the object in the original image, and the position of the object in the shifted and rotated image corresponds to the position of the phantom in the original image. In this way, since the position of the object in the shifted and rotated image is aligned with the position of the phantom in the original image, the high intensity of the white patch can be weighted using an intensity factor, and the low intensity at the phantom position can be suppressed using the weighted intensity value, thereby effectively suppressing the phantom intensity and achieving phantom elimination. On the other hand, the weighted intensity value obtained by weighting other positions in the shifted and rotated image using an intensity factor is very small, ensuring that it has a small impact on the intensity values ​​of other positions in the original image besides the phantom position during the suppression of phantoms in the original image.

[0084] Then, the transformed image is multiplied by the phantom reflection intensity factor of the phantom reflection compensation model (i.e., the weighted above). Specifically, the factor at each position in the phantom reflection compensation model is multiplied by the pixel intensity at the corresponding position in the transformed image to obtain an intensity-scaled image.

[0085] Finally, the original image to be compensated is subtracted from the intensity of the rotated and intensity-scaled image. For example, the intensity of the corresponding region in the intensity-scaled image (e.g., the pixel position after shifting and rotation) is subtracted from the intensity of the region in the original image to be compensated. This yields the compensated image, and allows calculation of the intensity variation within it, particularly the intensity variation between the phantom reflection location and its surrounding neighboring regions (and optionally, additionally, the degree of phantom reflection dissipation).

[0086] By similarly applying the above determination process to other calibration images, the intensity variation (and optionally or additionally, the degree of phantom reflection dissipation) of each image in this model extraction operation can be obtained. Then, it can be determined whether the statistical data of the intensity variation (and optionally or additionally, the degree of phantom reflection dissipation) of these images meet specific conditions.

[0087] As an example, if a threshold condition is applied, the system checks whether the intensity variation data of the statistical image is less than a predetermined threshold, and / or whether the degree of phantom dissipation is greater than the corresponding predetermined threshold. If so, the currently used compensation model can be considered desirable, and the model extraction operation is stopped. This desired model is then used as the phantom reflection compensation model in the actual shooting process. If not, the model parameters can be adjusted incrementally, and the above process is repeated until the statistical data of intensity variation meets the threshold requirement.

[0088] As another example, if the minimization condition is met, then if the statistical data of the intensity variation determined by this extraction operation no longer decreases compared to the previous one, and / or the statistical data of the degree of phantom dissolution no longer increases compared to the previous one, then it can be considered that the statistical data of the intensity variation is minimized and the statistical data of the degree of phantom dissolution is maximized, and the model extraction operation is stopped, and the compensation model corresponding to the previous operation is used as the final compensation model.

[0089] It should be noted that the initial values ​​of the model parameters, the step size, etc., in the above iterative operations can be set to any appropriate values, as long as they contribute to the convergence of the iteration. Furthermore, in the iterative operations, all model parameters can be changed simultaneously each time, or only one or more parameters can be changed each time. The former corresponds to the case where all model parameters are determined simultaneously through iteration, while the latter corresponds to the case where one or more parameters are first determined through iteration, and then other parameters are determined through iteration based on these parameters.

[0090] According to another implementation, a minimization equation can be constructed using a phantom reflection compensation model. The desired phantom reflection compensation model is obtained when the solution that minimizes image intensity variation while eliminating phantoms in all scenes to a predetermined degree is obtained by solving the equation. As an example, at least one of cx, cy, and the weights of each Gaussian function can be used as variables to construct the system of equations.

[0091] As an example, the intensity distribution in an image can be represented as a vector or matrix, as shown above. Figure 10A The multiplication of the described image and the model can be represented mathematically as vector or matrix multiplication, thus allowing us to... Figure 10A The operation describing the determination of intensity variation is represented in vector or matrix form, so that appropriate methods can be applied to minimize it, such as the least squares method.

[0092] According to embodiments of this disclosure, the phantom reflection compensation model can be determined before the imaging device is used by a user, for example, during the manufacturing process, factory testing, etc. As an example, it can be performed during manufacturing along with other calibration work (e.g., for a ToF camera, temperature compensation, phase gradient, cyclic error, etc.). In this way, the phantom reflection compensation model can be pre-built and stored in the imaging device, such as a camera.

[0093] According to embodiments of this disclosure, the phantom reflection compensation model can be determined during user use of the camera device. As an example, upon initial use, the user may be prompted to perform camera calibration. The user can then take calibration images according to the operating instructions, thereby extracting the phantom reflection compensation model from the captured images. As another example, after the user has taken a specific number of images (e.g., the shutter was used a specific number of times), the user may be prompted to update the model.

[0094] According to embodiments of this disclosure, the phantom reflection compensation model can be updated or pushed during product maintenance services for the camera device. As an example, when replacing the camera filter and / or lens of a camera device with existing phantom reflection compensation functionality, or when performing a software upgrade on a camera device without phantom reflection compensation functionality, the aforementioned model extraction process can be performed to update or establish the model.

[0095] As described above, the phantom reflection compensation model can be equivalent to characterizing the properties of components in a camera device that cause light reflections leading to phantom reflections, particularly the properties of lenses and photographic filters. In a sense, the phantom reflection compensation model corresponds to the lenses and / or photographic filters in the camera device. According to one embodiment, if the filters and lenses, especially lenses, included in the camera are fixed, the constructed phantom reflection compensation model can be relatively fixed, especially remaining unchanged during shooting. According to another embodiment, if the camera's filters and / or lenses are replaceable, the phantom reflection compensation model also needs to be updated accordingly when such components are replaced in the camera device. According to one embodiment, when a component is replaced, the phantom reflection compensation model corresponding to the replaced component can be retrieved automatically or prompted to the user, which can be performed as described above. According to another embodiment, the model corresponding to the replaced component can be automatically selected. For example, a set of phantom reflection compensation models corresponding to all filters and / or lenses applicable to the camera system can be pre-stored in the camera system. Thus, after the camera system replaces the filter and / or lens, a phantom reflection compensation model corresponding to the replaced filter and / or lens can be automatically selected from the stored set for application. According to another embodiment, considering that replacing the filter and / or lens often causes a certain degree of change in optical properties, such as changes in lens characteristics, which in turn affect the center offset parameter, when replacing the filter and / or lens, even if the corresponding model is pre-stored, the system can automatically or prompt the user to retrieve a new model instead of automatically selecting one. This can be done through system pre-settings or user prompts. For example, the system can be pre-set to automatically update the model under any circumstances. Or, for example, the user can be prompted whether to perform model calibration or automatically select a model.

[0096] After determining the phantom reflection compensation model according to this disclosure as described above, the model can be applied to further optimize the captured images and improve image quality.

[0097] According to embodiments of this disclosure, in the image weighting operation, for each sub-region in the captured image, intensity scaling can be performed using the corresponding phantom reflection intensity scaling factor in the phantom reflection compensation model, thereby obtaining an intensity-scaled image as a weighted image.

[0098] According to one embodiment, in the image weighting operation, the image to be compensated may be rotated; and the rotated image may be weighted (e.g., multiplied) using a phantom reflection compensation model to obtain a weighted image. According to another embodiment, the compensated image is obtained by subtracting the pixel intensities at corresponding locations in the image to be compensated and the weighted image.

[0099] It should be noted that the rotation, multiplication, and subtraction operations here can be performed in a similar manner to those described above with reference to FIG10, except that the input image on the left is the captured image to be compensated, and the output image on the right is the compensated captured image, in which phantom reflections have been effectively eliminated. Specifically, the image to be compensated is shifted and rotated according to the center parameters cx and cy of the phantom reflection compensation model; that is, the rotation can be performed around an axis off-center from the image center. When performing the multiplication operation, each position of the transformed image is multiplied by the corresponding phantom reflection compensation model factor. For example, multiplication can be performed after aligning the model illustration with the transformed image to perform intensity scaling.

[0100] According to some embodiments, the phantom reflection compensation operation according to this disclosure can be performed before or after scattering compensation, and substantially similar advantageous effects can be achieved. Figure 11 (a) shows phantom reflection compensation performed before scattering compensation, that is, the image to be compensated is the original image obtained from the ToF sensor. Figure 11 (b) shows phantom reflection compensation after scattering compensation, that is, the above-mentioned image to be compensated is an image that has already undergone scattering compensation.

[0101] According to embodiments of this disclosure, the method further includes a scattering compensation step for compensating for scattering in the image. According to some embodiments, the imaging device according to this disclosure is an imaging device using an imaging filter. According to some embodiments, the imaging device includes a ToF sensor, and the image to be compensated includes a depth image.

[0102] The foregoing examples primarily describe the case where the image to be compensated for a scene is a single image. However, the embodiments of the present invention can also be used for cases where there are at least two images to be compensated for a scene.

[0103] According to some embodiments, the raw image data obtained from scene photography may correspond to at least two sub-images, and for each sub-image, a phantom reflection compensation operation according to this disclosure is performed, including the calculation and compensation steps described above, thereby obtaining at least two compensated sub-images. A final compensated image corresponding to the scene can be obtained by combining the at least two compensated sub-images.

[0104] As an example, the at least two sub-images include an I-image and a Q-image to be corresponding to the original image data. The compensation for the sub-images will be illustrated below using the I-image and Q-image as examples. Figure 12 An example of operation involving phantom reflection compensation for I and Q images is shown.

[0105] To measure distance, iToF sensors typically need to capture four components. These components relate to the phase shift between the emitter (laser) and the light returning to the sensor, with a predefined phase shift. These four components are recorded at 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively. These four components are obtained as the raw ToF data.

[0106] Based on this raw data, we can calculate the I and Q images. I indicates the captured in-phase data; for example, the I image is a combination of the components for 0 degrees and 180 degrees. Q indicates the captured quadrature-phase data; for example, the Q image is a combination of the components for 90 degrees and 270 degrees. As an example, their calculation is as follows:

[0107] I=Component(0 degree)-Component(180 degree)

[0108] Q = component (90 degrees) - component (270 degrees)

[0109] Then, compensation is performed on the I and Q images respectively. The specific compensation method can be as described above with reference to Figure 10. In particular, the above-mentioned phantom reflection compensation operation is performed on each of the I and Q images respectively, which will not be described in detail here.

[0110] Finally, the compensated I and Q images are used to generate confidence images and depth images.

[0111] As an example, the confidence image is calculated based on I and Q, as shown below:

[0112] Confidence level = abs(I) + abs(Q)

[0113] Here, abs() indicates the absolute value function, representing the absolute value of the confidence score for each sub-region or pixel in the I-image and Q-image, respectively. As an example, confidence maps can also be obtained using other methods known in the art, which will not be described in detail here.

[0114] As an example, the depth map can be obtained from at least one of the I and Q images, which can be obtained in a manner known in the art and will not be described in detail here.

[0115] However, it should be noted that the I and Q images are merely examples. Other sub-images are also possible, as long as they can be captured by the camera device and combined to obtain the confidence image and depth map.

[0116] Figure 13 The beneficial effects of phantom reflection compensation according to embodiments of the present disclosure are illustrated. Specifically, the effects corresponding to... Figure 4AThe image shown contains a confidence map and a depth map. The left side, from top to bottom, indicates the confidence map and depth map corresponding to the original scene image, which includes scattering and phantom reflections. The middle side, from top to bottom, indicates the confidence map and depth map after compensation processing. It can be seen that, since scattering compensation is primarily performed, even though scattering noise in the image is removed through scattering compensation, phantom reflections still exist on the left side of the image. The right side, from top to bottom, indicates the confidence map and depth map compensated using the method of this disclosure. It can be seen that the method of this disclosure effectively removes phantom reflections in the image, resulting in a high-quality output image.

[0117] The above primarily used an example from an iToF sensor to describe the problem of phantom reflections and phantom reflection compensation operations. However, as mentioned above, phantom reflections do not depend on the emitter or image sensor, but mainly on the components in the imaging device that cause light reflections, especially the use of photographic filters and / or lenses in the camera. This means that this phenomenon can be observed in other 3D measurement systems that use light, including (but not limited to):

[0118] • An indirect ToF sensor using a full-field-of-view emitter.

[0119] • Indirect ToF sensors using point ToF transmitters,

[0120] Direct ToF sensor,

[0121] Structured light sensor,

[0122] Other types of ToF sensors.

[0123] The following description, with reference to the accompanying drawings, describes phantom reflection compensation for other types of ToF sensors according to embodiments of the present disclosure.

[0124] Figure 14A and 14B The image capture using a direct ToF (dToF) sensor is shown to compensate for phantom reflections.

[0125] Unlike iTOF, dTOF concentrates light energy over a short period. It involves generating photon packets using short pulses of laser or LED, and directly calculating the propagation time of these photons to and from the target. Appropriate techniques can then be employed to accumulate multiple events into a histogram to identify the target peak location against a generally uniformly distributed background noise. For example, this technique could be known as Time-Correlated Single Photon Counting (TCSPC), a technique known in the art and not described in detail here.

[0126] For dToF, phantom reflections can be represented as "phantom peaks" in the histogram of the affected pixels. For example... Figure 14A As shown, the upper histogram corresponds to the histogram of objects with high intensity in the field of view (FoV), where the peak values ​​indicate their corresponding depth. The lower histogram corresponds to the histogram at the location where phantom reflections occur, where depth peaks also appear due to the influence of phantom reflections, leading to incorrect depth detection.

[0127] According to embodiments of this disclosure, phantom reflection compensation can be performed for dToF. Specifically, for the pixel histogram obtained from the captured data, phantom reflection compensation is performed using methods similar to those described above, such as shifting and rotating, multiplying, subtracting, etc. Figure 14B As shown in the output histogram on the right, the histogram corresponding to phantom reflections is significantly suppressed through the phantom reflection compensation of this disclosure, and its peak value is much smaller than that of the real object, thus avoiding erroneous depth detection.

[0128] According to one embodiment, for dToF, its phantom reflection compensation model can also be generated as described above with reference to FIG10, except that the input is the pixel histogram obtained from the captured data. Moreover, any other operations described above are also applicable to dToF. According to another embodiment, considering that the phantom reflection compensation model mainly corresponds to the components in the imaging device that cause light reflection leading to phantom reflection, such as lenses and / or filters, after obtaining the phantom reflection compensation model for this component using any ToF sensor, the model can be applied to other types of ToF sensors, or even other types of sensors using this component.

[0129] Figure 15 The image shows phantom reflection compensation when using a spotToF (spotToF) sensor for image capture. Figure 15 (a) shows the confidence map taken when no near-field object is present. (b) shows the confidence map obtained when a near-field object is ideally present, where even if a near-field object is present in the scene, no other point information exists besides the point information of the near-field object itself. (c) shows the confidence map with phantom reflections, where it can be seen that when a near-field object is present on the right side of the scene, new points appear on the left side of the scene. These new points are generated by phantom reflections, which may produce incorrect depth or may mix the signal with the current speckle. (d) shows the confidence map after phantom reflection compensation using the scheme of this disclosure, where the new points caused by phantom reflections are effectively removed, improving image quality.

[0130] According to embodiments of this disclosure, the phantom reflection compensation function of this disclosure can be used automatically or by user selection.

[0131] As an example, the phantom compensation function of the present invention can be implemented automatically. For example, the phantom compensation function can be associated with a specific shooting mode of the camera, and the phantom compensation function will be automatically activated when the shooting mode is turned on during shooting. For example, the phantom compensation function will be automatically activated in close-up shooting modes, such as macro and portrait modes, while it will not be automatically activated in distant shooting modes, such as landscape modes. As another example, the camera can also determine whether to automatically activate the phantom compensation function based on the distance to the subject. For example, when the distance to the subject is greater than a certain distance threshold, it can be considered a distant shot and the phantom compensation function does not need to be activated, while when the distance to the subject is less than the certain distance threshold, it can be considered a close-up shot and the phantom compensation function does not need to be activated.

[0132] As an example, the phantom compensation function of the present invention can be set by the user. For example, a prompt will appear on the camera's shooting interface to indicate whether the phantom compensation function should be enabled. When the user selects the function, the phantom compensation function can be enabled to compensate for / eliminate phantoms when taking pictures. For example, this can be achieved through a button on a touch-screen user interface or a button on the camera that can perform the phantom compensation function.

[0133] According to embodiments of this disclosure, the phantom reflection compensation model can be stored in various ways. As an example, the model can be fixed together with a camera device, particularly a camera lens containing lenses and filters, so that the model can still be used even if the camera lens is replaced with another device, without the need for model retrieval. Alternatively, the model can be stored in a device capable of connecting to the camera device for taking pictures, such as a portable electronic device.

[0134] As mentioned above, phantom reflections are particularly disadvantageous for acquiring depth information. Therefore, the technical solution disclosed herein is especially suitable for various applications that require obtaining depth information of objects in a shooting scene, such as camera devices that need to measure depth information. For example, the technical solution disclosed herein is suitable for camera devices that employ ToF-based sensors, such as iToF, Full-field ToF, and spotToF. For example, the technical solution disclosed herein is suitable for 3D camera devices because depth / distance information is crucial for obtaining good 3D images.

[0135] Note that even though the reflection phantom effect is not as severe in the case of RGB sensors as it is in 3D measurement systems, the embodiments of this disclosure can still be applied to RGB sensors, especially when there is only one photographic filter in the system, such as in portable mobile devices whose cameras are equipped with cover glass, in which case the cover glass is implemented as a filter.

[0136] Furthermore, the solution disclosed herein can be applied to certain specific shooting modes that may produce phantom reflections. For example, given that a large amount of light reflection may occur when shooting close-up objects, which may in turn lead to phantom reflections, the phantom reflection compensation solution according to this disclosure is also particularly suitable for shooting modes of camera devices related to close-up objects, such as close-up shooting modes, background blur modes, etc.

[0137] The solution disclosed herein can effectively eliminate the effects of phantom reflections in images. In particular, the solution disclosed herein can accurately determine the depth information, i.e., distance information, of objects in a scene, thereby enabling accurate focusing during photography or obtaining high-quality images, which will facilitate subsequent image-based applications.

[0138] For example, for background blur effects, the solution disclosed herein can eliminate incorrect depth and obtain appropriate object distance. For example, for autofocus applications, even if an object is close to the camera, the object distance can be accurately identified, and a good focus distance can be used for taking pictures. For example, for face ID recognition, when the subject is close to the camera (e.g., a table) for face recognition, the solution disclosed herein can effectively remove ghosting in the image, thereby obtaining a high-quality image for recognition.

[0139] It should be noted that the technical solutions disclosed herein are particularly applicable to cameras in portable devices, such as cameras in mobile phones, tablets, and other devices. The lens and / or camera filter of the camera can be fixed or replaceable.

[0140] The following describes an electronic device capable of performing phantom reflection compensation according to the present disclosure. Figure 7 A block diagram of an electronic device capable of performing phantom reflection compensation according to an embodiment of the present disclosure is shown. The electronic device 700 includes processing circuitry 720, which is configured to weight an image to be compensated containing phantom reflections using a phantom reflection compensation model; and to combine the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0141] In the structural example of the above-described device, the processing circuit 720 can be in the form of a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the processing circuit 120 can be constructed from circuitry (hardware) or a central processing unit (such as a central processing unit (CPU)). Furthermore, the processing circuit 720 can carry a program (software) for making the circuitry (hardware) or the central processing unit work. This program can be stored in memory (such as arranged in memory) or in an external storage medium connected from the outside, and can be downloaded via a network (such as the Internet).

[0142] According to embodiments of this disclosure, the processing circuit 720 may include various units for implementing the above-described functions, such as a calculation unit 722 for weighting the image to be compensated containing phantom reflections using a phantom reflection compensation model; and a phantom reflection compensation unit 724 for combining the image to be compensated and the weighted image to eliminate phantom reflections in the image. In particular, the processing circuit 720 may also include a scattering compensation unit 726 and a data path processing unit 728. Each unit may operate as described above, and will not be described in detail here.

[0143] The scattering compensation unit 726 and the data path processing unit 728 are drawn with dashed lines to illustrate that this unit is not necessarily included in the processing circuit. As an example, this unit can be in the terminal-side electronic device but outside the processing circuit, or even located outside the electronic device 700. It should be noted that, although Figure 7 The individual units are shown as discrete units, but one or more of these units can be combined into one unit or split into multiple units.

[0144] It should be noted that the above-described units are merely logical modules divided according to their specific functions, and are not intended to limit the specific implementation method. For example, they can be implemented in software, hardware, or a combination of both. In actual implementation, the above-described units can be implemented as independent physical entities, or they can be implemented by a single entity (e.g., a processor (CPU or DSP, etc.), integrated circuit, etc.). Furthermore, the units shown in the accompanying drawings with dashed lines indicate that these units may not actually exist, and the operations / functions they perform can be implemented by the processing circuitry itself.

[0145] It should be understood that Figure 7 This is merely a general structural configuration of the terminal-side electronic device 700; the electronic device 700 may also include other possible components (e.g., memory, etc.). Optionally, the terminal-side electronic device 700 may also include other components not shown, such as memory, network interface, controller, etc. Processing circuitry may be associated with memory. For example, processing circuitry may be directly or indirectly (e.g., with other components possibly connected in between) connected to memory for data access.

[0146] The memory can store various information generated by the processing circuit 720. The memory can also be located within the terminal-side electronic device but outside the processing circuit, or even outside the terminal-side electronic device. The memory can be volatile and / or non-volatile. For example, the memory can include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory.

[0147] The camera device according to this disclosure will now be described. Figure 16 A block diagram of a camera apparatus according to an embodiment of the present disclosure is shown. The camera apparatus 1600 includes a compensation device 700, which can be used for image compensation processing, particularly phantom reflection compensation, and the compensation device can be implemented by an electronic device, such as the electronic device 700 described above.

[0148] The camera device 1600 may include a lens unit 1602, which may include various optical lenses known in the art for imaging objects on a sensor by optical imaging.

[0149] The camera device 1600 may include a photographic filter 1604, which may include various photographic filters known in the art, and may be mounted in front of a lens.

[0150] The camera device 1600 may also include a processing circuit 1606, which can be used to process the acquired image. As an example, the compensated image may be further processed, or the image to be compensated may be preprocessed.

[0151] The camera device 1600 may also include various image sensors, such as the aforementioned ToF-based sensors. However, these sensors may also be located outside the camera device 1600.

[0152] It should be noted that the photographic filter and processing circuitry are drawn with dashed lines to illustrate that this unit is not necessarily included in the imaging device 1600, and can even be connected and / or communicated with outside the imaging device 1600 by known means. It is important to note that, although... Figure 16 The individual units are shown as discrete units, but one or more of these units can be combined into one unit or split into multiple units.

[0153] In the structural example of the above-described device, the processing circuit 1606 can be in the form of a general-purpose processor or a dedicated processing circuit, such as an ASIC. For example, the processing circuit 1606 can be constructed from circuitry (hardware) or a central processing unit (such as a central processing unit (CPU)). Furthermore, the processing circuit 1606 can carry a program (software) for making the circuitry (hardware) or the central processing unit work. This program can be stored in memory (such as arranged in memory) or in an external storage medium connected from the outside, and can be downloaded via a network (such as the Internet).

[0154] The technology disclosed herein can be applied to a variety of products.

[0155] For example, the technology disclosed herein can be applied to the camera device itself, such as being built into or integrated with the camera lens. In this way, the technology disclosed herein can be executed by the processor of the camera device in the form of a software program, or integrated in the form of an integrated circuit or processor. Alternatively, it can be used in a device connected to the camera device, such as a portable mobile device equipped with the camera device. In this way, the technology disclosed herein can be executed by the processor of the camera device in the form of a software program, or integrated in the form of an integrated circuit or processor, or even integrated into an existing processing circuit for phantom reflection compensation during the photography process.

[0156] The technology disclosed herein can be applied to various camera devices, such as lenses mounted on portable devices, camera devices on drones, camera devices in surveillance equipment, and so on.

[0157] This invention can be used in many applications. For example, it can be used to monitor, identify, and track objects in still images or moving videos captured by a camera, and is particularly advantageous for portable devices equipped with cameras, (camera-based) mobile phones, and the like.

[0158] Furthermore, it should be understood that the aforementioned series of processes and devices can also be implemented via software and / or firmware. In the case of implementation via software and / or firmware, data can be transferred from storage media or networks to computers with dedicated hardware architectures, such as… Figure 17 The general-purpose personal computer 1300 shown is equipped with the programs that constitute the software, and the computer is able to perform various functions when various programs are installed. Figure 17 This is a block diagram illustrating an example structure of a personal computer that can be used as an information processing device in embodiments of the present disclosure. In one example, the personal computer may correspond to the exemplary transmitting device or terminal-side electronic device described above according to the present disclosure.

[0159] exist Figure 17In this system, the central processing unit (CPU) 1301 performs various processes based on the program stored in the read-only memory (ROM) 1302 or the program loaded into the random access memory (RAM) 1303 from the storage section 1308. The RAM 1303 also stores, as needed, the data required when the CPU 1301 performs various processes.

[0160] CPU 1301, ROM 1302 and RAM 1303 are connected to each other via bus 1304. Input / output interface 1305 is also connected to bus 1304.

[0161] The following components are connected to the input / output interface 1305: input section 1306, including a keyboard, mouse, etc.; output section 1307, including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 1308, including a hard disk, etc.; and communication section 1309, including a network interface card, such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network, such as the Internet.

[0162] As needed, drive 1310 is also connected to input / output interface 1305. Removable media 1311, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 1310 as needed, so that computer programs read from them can be installed into storage section 1308 as needed.

[0163] When the above series of processes are implemented by software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1311.

[0164] Those skilled in the art will understand that such storage media are not limited to Figure 17 The illustrated removable medium 1311 stores a program and is distributed separately from the device to provide the program to the user. Examples of removable media 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1302, a hard disk included in storage section 1308, etc., containing programs and distributed to the user along with the device containing them.

[0165] It should be noted that the methods and devices described herein can be implemented as software, firmware, hardware, or any combination thereof. Some components may be implemented, for example, as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or application-specific integrated circuits (ASICs).

[0166] Furthermore, the methods and systems of the present invention can be implemented in various ways. For example, the methods and systems of the present invention can be implemented by software, hardware, firmware, or any combination thereof. The order of steps of the method described above is merely illustrative, and unless otherwise specifically stated, the steps of the method of the present invention are not limited to the order specifically described above. In addition, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers recording media storing programs for implementing the method according to the present invention. Such storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0167] Those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial overlap in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. However, other modifications, variations, and substitutions are equally possible. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

[0168] In addition, embodiments of this disclosure may also include the following illustrative examples (EE).

[0169] EE 1. An electronic device for compensating for phantom reflections in images captured by a camera device, comprising processing circuitry configured to:

[0170] An image containing phantom reflections is weighted using a phantom reflection compensation model, wherein the phantom reflection compensation model relates to the intensity distribution of phantom reflections in the image caused by light reflections from the camera device during shooting; and

[0171] Combine the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0172] EE2, the electronic device according to EE1, wherein the phantom reflection compensation model is trained from a predetermined number of calibration images, and the phantom reflection compensation model is trained such that the intensity variation of the phantom reflection region in the calibration image after applying the phantom reflection compensation model is less than a specific threshold or is minimal compared to the neighboring regions of the phantom reflection region.

[0173] EE3, the electronic device according to EE2, wherein the intensity variation is determined during the training of the phantom reflection compensation model by the following operation:

[0174] The calibration image is center-shifted according to the pre-set center shift parameters of the phantom reflection compensation model to be trained, rotated around the shifted center as the axis, and then the rotated image is center-shifted in the reverse direction according to the same parameters; and

[0175] The shifted and rotated image is multiplied by a phantom reflection compensation model with pre-defined intensity factor parameters; and

[0176] The intensity variation is obtained by subtracting the pixel intensity at the corresponding position in the calibrated image and the image obtained after multiplying by the model.

[0177] EE4. The electronic device according to EE 1, wherein the phantom reflection compensation model includes phantom reflection factors corresponding to each sub-region in the image, wherein the sub-region includes at least one pixel.

[0178] EE5. The electronic device according to EE 4, wherein the phantom reflection compensation model is configured such that the phantom reflection factor in a sub-region near the center of the image is greater than the phantom reflection factor in a sub-region near the edge of the image.

[0179] EE6. The electronic device according to EE 4, wherein the phantom reflection factor is determined based on a Gaussian distribution.

[0180] EE7. An electronic device according to any one of EE 4-6, wherein the processing circuit is configured to:

[0181] For each sub-region in the captured image, intensity scaling is performed using the corresponding phantom reflection intensity factor in the phantom reflection compensation model, thereby obtaining an intensity-scaled image as a weighted image.

[0182] EE8. The electronic device according to EE 1, wherein the phantom reflection model is related to the characteristics of a component in the photographing device that causes light reflection resulting in phantom reflection during photographing, and wherein the parameters of the phantom reflection model depend on the characteristics of the component.

[0183] EE9. An electronic device according to EE 8, wherein the component includes at least one of a lens and a photographic filter.

[0184] EE10, an electronic device according to EE 8 or 9, wherein the model of the phantom reflection model includes at least parameters related to the center offset and related parameters for determining the Gaussian distribution of the phantom reflection factor.

[0185] EE11. An electronic device according to EE 1, wherein the phantom reflection model includes parameters related to center offset, and wherein the processing circuitry is configured to:

[0186] The image to be compensated is center-shifted according to this parameter, rotated around the shifted center as an axis, and then the rotated image is center-shifted in the opposite direction according to this parameter; and

[0187] The phantom reflection compensation model is used to weight the shifted and rotated image to obtain a weighted image.

[0188] EE12. The electronic device according to EE 11, wherein the aforementioned shift and rotation cause the phantom in the shifted and rotated image to correspond to the position of the object in the original image, and the object in the shifted and rotated image to correspond to the position of the phantom in the original image.

[0189] EE13. The electronic device according to EE 1, wherein the processing circuit is configured to:

[0190] The compensated image is obtained by subtracting the pixel intensity at corresponding positions in the image to be compensated and the weighted image.

[0191] EE14. The electronic device according to EE 1, wherein the image to be compensated corresponds to at least two sub-images.

[0192] Furthermore, phantom reflection compensation is performed for each sub-image, thereby obtaining a compensated image by combining at least two compensated sub-images.

[0193] EE15, the electronic device according to EE 14, wherein the at least two sub-images include an I-image and a Q-image obtained by capturing raw image data.

[0194] EE16. The electronic device according to EE 1, wherein the camera device is an optical camera device using a photographic filter.

[0195] EE17. An electronic device according to any one of EE 1-16, wherein the camera device includes a ToF sensor and the image includes a depth image.

[0196] EE18. A method for compensating for phantom reflections in images captured by a camera device, comprising the following steps:

[0197] The calculation step is used to weight an image containing phantom reflections by using a phantom reflection compensation model, wherein the phantom reflection compensation model relates to the intensity distribution of phantom reflections in the image caused by light reflections from the camera device during shooting; and

[0198] The compensation step is used to combine the image to be compensated and the weighted image to eliminate phantom reflections in the image.

[0199] EE19. The method according to EE 18, wherein the phantom reflection compensation model is trained from a predetermined number of calibration images, and the phantom reflection compensation model is trained such that the intensity variation of the phantom reflection region in the calibration image after applying the phantom reflection compensation model is less than a specific threshold or minimized compared to the neighboring regions of the phantom reflection region.

[0200] EE20, according to the method described in EE 19, wherein the intensity variation is determined during the training of the phantom reflection compensation model by the following operations:

[0201] The calibration image is center-shifted according to the pre-set center shift parameters of the phantom reflection compensation model to be trained, rotated around the shifted center as the axis, and then the rotated image is center-shifted in the reverse direction according to the same parameters; and

[0202] The shifted and rotated image is multiplied by a phantom reflection compensation model with pre-defined intensity factor parameters; and

[0203] The intensity variation is obtained by subtracting the pixel intensity at the corresponding position in the calibrated image and the image obtained after multiplying by the model.

[0204] EE21, the method according to EE 18, wherein the phantom reflection compensation model includes phantom reflection factors corresponding to each sub-region in the image, wherein the sub-region contains at least one pixel.

[0205] EE22. The method according to EE 20, wherein the phantom reflection compensation model is set such that the phantom reflection factor in a sub-region near the center of the image is greater than the phantom reflection factor in a sub-region near the edge of the image.

[0206] EE23. The method according to EE 20, wherein the phantom reflection factor is determined based on a Gaussian distribution.

[0207] EE24. The method according to any one of EE 20-22, wherein the calculation step further comprises:

[0208] For each sub-region in the captured image, intensity scaling is performed using the corresponding phantom reflection intensity factor in the phantom reflection compensation model, thereby obtaining an intensity-scaled image as a weighted image.

[0209] EE25, the method according to EE 18, wherein the phantom reflection model is related to the characteristics of a component in the photographing device that causes light reflection resulting in phantom reflection during photographing, and wherein the parameters of the phantom reflection model depend on the characteristics of the component.

[0210] EE26. The method according to EE 24, wherein the component includes at least one of a lens and a photographic filter.

[0211] EE27. The method according to EE 24 or 25, wherein the model of the phantom reflection model includes at least parameters related to the center offset and related parameters for determining the Gaussian distribution of the phantom reflection factor.

[0212] EE28, the method according to EE 18, wherein the phantom reflection model includes parameters related to the center offset, and wherein the calculation step further includes:

[0213] The image to be compensated is center-shifted according to this parameter, rotated around the shifted center as an axis, and then the rotated image is center-shifted in the opposite direction according to this parameter; and

[0214] The phantom reflection compensation model is used to weight the shifted and rotated image to obtain a weighted image.

[0215] EE29. The method according to EE 28, wherein the above-mentioned shift and rotation are such that the phantom in the shifted and rotated image corresponds to the position of the object in the original image, and the object in the shifted and rotated image corresponds to the position of the phantom in the original image.

[0216] EE30, the method according to EE 18, wherein the compensation step includes:

[0217] The compensated image is obtained by subtracting the pixel intensity at corresponding positions in the image to be compensated and the weighted image.

[0218] EE31, according to the method of EE 18, wherein the image to be compensated corresponds to at least two sub-images.

[0219] Furthermore, phantom reflection compensation is performed for each sub-image, thereby obtaining a compensated image by combining at least two compensated sub-images.

[0220] EE32, the method according to EE 30, wherein the at least two sub-images include an I-image and a Q-image obtained by capturing raw image data.

[0221] EE33. The method according to EE 18, wherein the camera device is an optical camera device using a photographic filter.

[0222] EE34. The method according to any one of EE 18-33, wherein the camera device includes a ToF sensor and the image includes a depth image.

[0223] EE35, an electronic device for performing phantom reflection compensation for image capture using a direct time-of-flight (dToF) sensor, comprising processing circuitry configured to:

[0224] The pixel histogram to be compensated, which includes phantom reflections, obtained from the raw captured data, is weighted using a phantom reflection compensation model; and

[0225] Combine the pixel histogram to be compensated and the weighted pixel histogram to eliminate phantom reflections.

[0226] EE36. An electronic device according to EE 35, wherein the phantom reflection model includes parameters related to center offset, and wherein the processing circuitry is configured to:

[0227] Based on this parameter, the histogram to be compensated is center-shifted, rotated around the shifted center as the axis, and then the rotated histogram is center-shifted in the opposite direction based on this parameter; and

[0228] The shifted and rotated histograms are weighted using a phantom reflection compensation model to obtain a weighted histogram.

[0229] EE37, the electronic device according to EE 35, wherein the aforementioned shift and rotation cause the phantom reflection peak in the shifted and rotated histogram to correspond to the object peak in the original histogram.

[0230] EE38. An electronic device according to EE 35, wherein the processing circuit is configured to:

[0231] The compensated histogram is obtained by subtracting the values ​​at corresponding positions in the histogram to be compensated and the weighted histogram.

[0232] EE39, A method for phantom reflection compensation in image acquisition using a direct time-of-flight (dToF) sensor, comprising:

[0233] The calculation steps involve weighting the pixel histogram to be compensated, which includes phantom reflections, obtained from the original captured data using a phantom reflection compensation model; and

[0234] The compensation step combines the pixel histogram to be compensated with the weighted pixel histogram to eliminate phantom reflections.

[0235] EE40, the method according to EE 39, wherein the phantom reflection model includes parameters related to the center offset, and wherein the calculation step further includes:

[0236] Based on this parameter, the histogram to be compensated is center-shifted, rotated around the shifted center as the axis, and then the rotated histogram is center-shifted in the opposite direction based on this parameter; and

[0237] The shifted and rotated histograms are weighted using a phantom reflection compensation model to obtain a weighted histogram.

[0238] EE41. The method according to EE 39, wherein the above shift and rotation are such that the phantom reflection peak in the shifted and rotated histogram corresponds to the object peak in the original histogram.

[0239] EE42. The method according to EE 39, wherein the compensation step further comprises: obtaining a compensated histogram by subtracting the values ​​at corresponding positions in the histogram to be compensated and the weighted histogram.

[0240] EE43. A device comprising

[0241] At least one processor; and

[0242] At least one storage device storing instructions thereon, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of EE 18-34 and 39-42.

[0243] EE44. A storage medium for storing instructions that, when executed by a processor, cause the execution of any one of EE18-34 and 39-42.

[0244] While this disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made without departing from the spirit and scope of this disclosure as defined by the appended claims. Furthermore, the terms "comprising," "including," or any other variations thereof used in embodiments of this disclosure are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0245] While some specific embodiments of this disclosure have been described in detail, those skilled in the art should understand that the above embodiments are illustrative only and do not limit the scope of this disclosure. Those skilled in the art should understand that the above embodiments can be combined, modified, or replaced without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. An electronic device for compensating for phantom reflections in images captured by a camera device, comprising processing circuitry configured to: The image to be compensated, which contains phantom reflections, is weighted by using a phantom reflection compensation model, wherein the phantom reflection compensation model is related to the intensity distribution of phantom reflections in the image caused by light reflections in the camera device during shooting; as well as The image to be compensated and the weighted image are combined to eliminate phantom reflections in the image. in, The phantom reflection compensation model includes parameters related to center offset, and the processing circuit is configured to: The image to be compensated is shifted in the center according to this parameter, rotated around the shifted center as the axis, and then the rotated image is shifted in the opposite direction according to this parameter. as well as The phantom reflection compensation model is used to weight the shifted and rotated image to obtain a weighted image.

2. The electronic device according to claim 1, wherein, The phantom reflection compensation model is trained from a predetermined number of calibration images, and the phantom reflection compensation model is trained such that the intensity variation of the phantom reflection region in the calibration image after applying the phantom reflection compensation model is less than a specific threshold or minimized compared to the neighboring regions of the phantom reflection region.

3. The electronic device according to claim 2, wherein, In the training of the phantom reflection compensation model, intensity variation is determined through the following operations: The calibration image is center-shifted according to the pre-set center shift parameters of the phantom reflection compensation model to be trained, rotated around the shifted center as the axis, and then the rotated image is center-shifted in the reverse direction according to the same parameters; and The shifted and rotated image is multiplied by a phantom reflection compensation model with pre-defined intensity factor parameters; and The intensity variation is obtained by subtracting the pixel intensity at the corresponding position in the calibrated image and the image obtained after multiplying by the phantom reflection compensation model.

4. The electronic device according to claim 1, wherein, The phantom reflection compensation model includes phantom reflection factors corresponding to each sub-region in the image to be compensated, wherein the sub-region contains at least one pixel.

5. The electronic device according to claim 4, wherein, The phantom reflection compensation model is set such that the phantom reflection factor in the sub-region near the center of the image is greater than the phantom reflection factor in the sub-region near the edge of the image.

6. The electronic device according to claim 4, wherein, The phantom reflection factor is determined based on a Gaussian distribution.

7. The electronic device according to any one of claims 4-6, wherein, The processing circuit is configured as follows: For each sub-region in the captured image, intensity scaling is performed using the corresponding phantom reflection intensity factor in the phantom reflection compensation model, thereby obtaining an intensity-scaled image as a weighted image.

8. The electronic device of claim 1, wherein the phantom reflection compensation model is related to the characteristics of a component in the imaging device that causes light reflection resulting in phantom reflection during shooting, and wherein, The parameters of the phantom reflection compensation model depend on the characteristics of the component.

9. The electronic device according to claim 8, wherein, The component includes at least one of a lens and a photographic filter.

10. The electronic device according to claim 8 or 9, wherein, The phantom reflection compensation model also includes relevant parameters for determining the Gaussian distribution of the phantom reflection factor.

11. The electronic device according to claim 1, wherein, The aforementioned shifts and rotations cause the phantom in the shifted and rotated image to correspond to the position of the object in the original image, while the object in the shifted and rotated image corresponds to the position of the phantom in the original image.

12. The electronic device according to claim 1, wherein, The processing circuit is configured as follows: The compensated image is obtained by subtracting the pixel intensity at corresponding positions in the image to be compensated and the weighted image.

13. The electronic device according to claim 1, wherein, The image to be compensated corresponds to at least two sub-images. Furthermore, phantom reflection compensation is performed for each sub-image, thereby obtaining a compensated image by combining at least two compensated sub-images.

14. The electronic device according to claim 13, wherein, The at least two sub-images include an I-image and a Q-image obtained by capturing raw image data.

15. The electronic device according to claim 1, wherein, The camera device is an optical camera device that uses a photographic filter.

16. The electronic device according to any one of claims 1-6, wherein, The camera device includes a ToF sensor, and the image to be compensated includes a depth image.

17. A method for compensating for phantom reflections in images captured by a camera device, comprising the following steps: The calculation step is used to weight the image to be compensated, which contains phantom reflections, by using a phantom reflection compensation model, wherein the phantom reflection compensation model is related to the intensity distribution of phantom reflections in the image caused by light reflections in the camera device during shooting; as well as The compensation step combines the image to be compensated and the weighted image to eliminate phantom reflections in the image. The phantom reflection compensation model includes parameters related to center offset, and the calculation steps further include: The image to be compensated is center-shifted according to this parameter, rotated around the shifted center as an axis, and then the rotated image is center-shifted in the opposite direction according to this parameter; and The phantom reflection compensation model is used to weight the shifted and rotated image to obtain a weighted image.

18. The method according to claim 17, wherein, The phantom reflection compensation model is trained from a predetermined number of calibration images, and the phantom reflection compensation model is trained such that the intensity variation of the phantom reflection region in the calibration image after applying the phantom reflection compensation model is less than a specific threshold or minimized compared to the neighboring regions of the phantom reflection region.

19. The method according to claim 18, wherein, In the training of the phantom reflection compensation model, intensity variation is determined through the following operations: The calibration image is center-shifted according to the pre-set center shift parameters of the phantom reflection compensation model to be trained, rotated around the shifted center as the axis, and then the rotated image is center-shifted in the reverse direction according to the same parameters; and The shifted and rotated image is multiplied by a phantom reflection compensation model with pre-defined intensity factor parameters; as well as The intensity variation is obtained by subtracting the pixel intensity at the corresponding position in the calibrated image and the image obtained after multiplying by the phantom reflection compensation model.

20. The method of claim 17, wherein, The phantom reflection compensation model includes phantom reflection factors corresponding to each sub-region in the image to be compensated, wherein the sub-region contains at least one pixel.

21. The method according to claim 19, wherein, The phantom reflection compensation model is set such that the phantom reflection factor in the sub-region near the center of the image is greater than the phantom reflection factor in the sub-region near the edge of the image.

22. The method according to claim 19, wherein, The intensity factor distribution indicated by the intensity factor parameter is determined based on a Gaussian distribution.

23. The method according to any one of claims 19-21, wherein, The calculation steps further include: For each sub-region in the captured image, intensity scaling is performed using the corresponding phantom reflection intensity factor in the phantom reflection compensation model, thereby obtaining an intensity-scaled image as a weighted image.

24. The method of claim 17, wherein the phantom reflection compensation model is related to the characteristics of a component in the imaging device that causes light reflection resulting in phantom reflection during shooting, and wherein, The parameters of the phantom reflection compensation model depend on the characteristics of the component.

25. The method according to claim 24, wherein, The component includes at least one of a lens and a photographic filter.

26. The method according to claim 23, wherein, The phantom reflection compensation model also includes relevant parameters for determining the Gaussian distribution of the phantom reflection factor.

27. The method according to claim 17, wherein, The aforementioned shifts and rotations cause the phantom in the shifted and rotated image to correspond to the position of the object in the original image, while the object in the shifted and rotated image corresponds to the position of the phantom in the original image.

28. The method according to claim 17, wherein, The compensation step also includes: The compensated image is obtained by subtracting the pixel intensity at corresponding positions in the image to be compensated and the weighted image.

29. The method according to claim 17, wherein, The image to be compensated corresponds to at least two sub-images. Furthermore, phantom reflection compensation is performed for each sub-image, thereby obtaining a compensated image by combining at least two compensated sub-images.

30. The method according to claim 29, wherein, The at least two sub-images include an I-image and a Q-image obtained by capturing raw image data.

31. The method according to claim 17, wherein, The camera device is an optical camera device that uses a photographic filter.

32. The method according to any one of claims 17-22, wherein, The camera device includes a ToF sensor, and the image to be compensated includes a depth image.

33. An electronic device for performing phantom reflection compensation for image capture using a direct time-of-flight (dToF) sensor, comprising processing circuitry configured to: The pixel histogram to be compensated, which includes phantom reflections, obtained from the original captured data is weighted by using a phantom reflection compensation model. as well as The pixel histogram to be compensated and the weighted pixel histogram are combined to eliminate phantom reflections. in, The phantom reflection compensation model includes parameters related to center offset, and the processing circuit is configured to: Based on this parameter, the pixel histogram to be compensated is shifted to the center, rotated around the shifted center as the axis, and then the rotated pixel histogram is shifted to the center in the opposite direction based on this parameter. as well as The phantom reflection compensation model is used to weight the shifted and rotated pixel histograms to obtain a weighted pixel histogram.

34. The electronic device according to claim 33, wherein, The aforementioned shift and rotation cause the phantom reflection peak in the shifted and rotated pixel histogram to correspond to the object peak in the original pixel histogram.

35. The electronic device according to claim 33, wherein, The processing circuit is configured as follows: The compensated pixel histogram is obtained by subtracting the values ​​at corresponding positions in the pixel histogram to be compensated and the weighted pixel histogram.

36. A method for phantom reflection compensation in image acquisition using a direct time-of-flight (dToF) sensor, comprising: The calculation step involves weighting the pixel histogram to be compensated, which includes phantom reflections, obtained from the original captured data, using a phantom reflection compensation model. as well as The compensation step combines the pixel histogram to be compensated with the weighted pixel histogram to eliminate phantom reflections. The phantom reflection compensation model includes parameters related to the center offset, and the calculation step further includes: Based on this parameter, the pixel histogram to be compensated is center-shifted, rotated around the shifted center as an axis, and then the rotated pixel histogram is center-shifted in the opposite direction based on this parameter; and The phantom reflection compensation model is used to weight the shifted and rotated pixel histograms to obtain a weighted pixel histogram.

37. The method of claim 36, wherein, The aforementioned shift and rotation cause the phantom reflection peak in the shifted and rotated pixel histogram to correspond to the object peak in the original pixel histogram.

38. The method according to claim 36, wherein, The compensation step further includes: subtracting the values ​​at corresponding positions in the pixel histogram to be compensated and the weighted pixel histogram to obtain the compensated pixel histogram.

39. An apparatus comprising At least one processor; and At least one storage device storing instructions thereon, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 17-32 and 36-38.

40. A storage medium for storing instructions, which, when executed by a processor, cause the method according to any one of claims 17-32 and 36-38 to be performed.

Images