LED Array Optimization Using Artificial Neural Networks

Adaptive LED arrays with segmented LEDs and ANN-based image processing address exposure inconsistencies in LED flash photography, enhancing image quality by dynamically adjusting light distribution.

JP2024542783AActive Publication Date: 2024-11-15LUMILEDS LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024532995
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-02
Filing Date
2022-11-29
Publication Date
2024-11-15
Estimated Expiration
2042-11-29

AI Technical Summary

Technical Problem

Conventional LED arrays face challenges in achieving uniform illumination and addressing over- and under-exposure issues due to varying ambient lighting conditions, leading to suboptimal image quality in flash photography.

Method used

The use of adaptive LED arrays with segmented LEDs and actuators to individually control light emission, combined with artificial neural networks (ANNs) for image processing to optimize illumination based on scene analysis and ambient conditions.

Benefits of technology

Enhances image quality by dynamically adjusting light distribution to achieve uniform brightness and reduce dark bands, improving exposure accuracy and overall image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542783000001_ABST
    Figure 2024542783000001_ABST
Patent Text Reader

Abstract

A lighting arrangement in a mobile device is provided that uses an artificial neural network (ANN) to correct an image of a scene. The mobile device has a lighting device with an LED array that emits light to illuminate a scene. The LED array has LEDs that are separated by a boundary and driven independently. A camera has a sensor for capturing images of the scene. The ANN has a training mode in which the ANN is trained with a set of images captured by the camera to generate parameters. The ANN is trained offline in a cloud network. A processor corrects the image using the trained ANN in an inference mode.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 285,171, filed December 2, 2021, which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to tunable light emitting diode (LED) arrays. [Background technology]

[0003] Efforts are underway to improve various types of lighting systems, and in particular to enable automatic adjustments in lighting configurations to meet image quality metrics. [Brief description of the drawings]

[0004] [Figure 1] 1 illustrates a side view of a lighting configuration, according to some examples. [Diagram 2] 1 shows an LED array, according to some examples. [Diagram 3] 1 illustrates an example of a lighting system, according to some examples. [Figure 4] 1 illustrates an example block diagram of an image enhancement process according to some embodiments. [Figure 5A] 1 illustrates an example of a training process according to some embodiments. [Figure 5B] 1 illustrates an example of an inference process according to some embodiments.

[0005] Corresponding reference characters indicate corresponding parts throughout the several views. Elements in the drawings are not necessarily drawn to scale. The configurations shown in the drawings are merely examples and should not be construed as limiting in any way. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] With the advent of more advanced electronics and increasingly demanding image quality metrics, lighting configurations with advanced capabilities are desired. One such problem is the overexposure and underexposure problem introduced by conventional LED flashes, for example due to varying ambient lighting conditions across an image. To address this, adaptive light sources may be introduced that include multiple segments, each of which may be individually controlled and may have its own settings (e.g., current, pulse width modulation (PWM)) to illuminate a scene. When used as a camera flash, the lighting pattern emitted by the light source may be adapted to the scene. For example, an adaptive light source may provide different amounts of light to different parts of a scene depending on the amount of ambient light those parts of the scene receive and / or depending on the distance of those parts of the scene from the adaptive light source. In one particular example, a light source may provide more light to parts of a scene that are not well lit by ambient light (compared to other parts of the scene) and less light to parts of a scene that are well lit by ambient light or that are closer to the camera (than other parts of the scene). Thus, for a given scene and ambient light conditions, the light source can be adapted to the scene based on algorithmic evaluation to improve certain image quality metrics.

[0007] It is desirable for a mobile device equipped with an LED module to provide sufficient values ​​for flash photography, including values ​​related to material parameters, electrical parameters, optical parameters, power parameters, and design related parameters. Thus, in addition to circuit and design choices, the LED module can use optics (e.g., lenses, etc.) to direct the light emitted by the LED light source. Furthermore, at the system level, other parameters such as the positioning of the LED module with the remaining components (e.g., other sensors) and the form factor of the components add an amount of complexity.

[0008] Flash photography with adaptive LED arrays introduces more challenges than systems using conventional LED arrays. For example, adaptive LED arrays may or may not have a uniform structure, unlike conventional LED arrays. In addition, the optical design for adaptive LED arrays is more complex than for conventional LED arrays, due at least in part to structural issues.

[0009] 1 illustrates a side view of a lighting arrangement 130, according to some examples. The lighting arrangement 130 can be, for example, a mobile device and can include both the lighting device 100 and the camera 102. The camera 102 can capture an image of the scene 104 during an exposure time of the camera 102 without the lighting device 100 or with the lighting device 100 illuminating the scene 104. A processor 140 can be used to control various functions of the lighting arrangement 130. Although only one processor 140 is shown in FIG. 1, in other embodiments, one or more processors can be used to provide the functionality described herein.

[0010] The lighting device 100 may include a light emitting diode (LED) array 106. The LED array 106 may include a number of LEDs 108 capable of generating light 110 during an exposure time of the camera 102. The LED array 106 may include segmented LEDs or micro LEDs in some embodiments.

[0011] In embodiments where the LED array 106 is a micro LED array, the LED array 106 may include thousands to millions of tiny LED pixels that can emit light and can be controlled individually or in groups of pixels (e.g., 5×5 groups of pixels). Micro LEDs are small (e.g., <0.01 mm on a side) and can provide monochromatic or polychromatic light, typically red, green, and blue, using, for example, inorganic semiconductor materials. LED arrays 106 formed from inorganic materials (e.g., binary compounds such as gallium arsenide (GaAs), ternary compounds such as aluminum gallium arsenide (AlGaAs), quaternary compounds such as indium gallium phosphide (InGaAsP), or other suitable materials) are more robust than organic LEDs and allow for use in a wider range of environments. In addition, LED arrays 106 formed from micro LEDs can allow for direct emission and can be more efficient than traditional backlight and liquid crystal display (LCD) combinations.

[0012] In embodiments in which the LED array 106 includes segmented LEDs, the LED array 106 may include one or more non-emitting regions located between adjacent LEDs 108 in the LED array 106, as shown in FIG. 2. The size of the non-emitting regions located between adjacent LEDs 108 (i.e., the distance between adjacent LEDs 108) may be a significant portion (e.g., about 5% to about 10%) of the size of the LEDs 108 in the LED array 106 (i.e., the distance between adjacent sides of the LEDs 108). In some examples, one or more of the non-emitting regions may surround an LED 108 in the LED array 106, causing a dark band to appear in the illumination emitted by the LED array 106.

[0013] The illumination device 100 may include at least one lens 114. The lens 114 may direct the light 110 as illumination 116 toward the scene 104. The illumination device 100 may include one or more actuators 120. In other embodiments, the actuator 120 may not be used. Instead, such a system may have a fixed lens and therefore a fixed aperture. In a system including an actuator 120, the actuator 120 may include individual translators that respectively translate at least one of the LED array 106 or the lens 114 during the exposure time of the camera 102 to blur dark bands in the illumination 116 in the image of the scene 104. In some examples, one of the translators may translate the LED array 106 relative to the lens 114. In some examples, another of the translators may translate the lens 114 relative to the LED array 106. In some examples, the actuator 120 (a single element) may translate both the lens 114 and the LED array 106. In some examples, the lens 114 can define a longitudinal axis that extends from the LED array 106 through the center of the lens 114 to the scene 104 .

[0014] In some examples, the or each actuator 120 may be a one-dimensional actuator capable of translating at least one of the LED array 106 or the lens 114 in an actuation direction angled relative to the longitudinal axis. In some examples, the actuation direction may be generally orthogonal to the longitudinal axis. In some examples, the LED array 106 may be arranged in a two-dimensional pattern having a first array direction and a second array direction orthogonal to the first array direction. In some examples, the actuation direction may be angled relative to the first array direction and angled relative to the second array direction. In some examples, the LED array 106 may be arranged in a one-dimensional pattern extending along the array direction. In some examples, the actuation direction may be non-orthogonal to the array direction. In some examples, the actuation direction may be generally parallel to the array direction. In some examples, the actuator 120 may translate at least one of the LED array 106 or the lens 114 in the actuation direction a distance equal to or greater than the width of one of the one or more non-emitting regions of the LED array 106 during an exposure time of the camera 102. In some examples, the actuator 120 can vibrate at least one of the LED array 106 or the lens 114 in an actuation direction. In some examples, the vibration can have a vibration period that is shorter than an exposure time of the camera 102.

[0015] In some examples, the or each actuator 120 can be a two-dimensional actuator capable of translating at least one of the LED array 106 or the lens in an actuation plane that is angled relative to the longitudinal axis. For example, the actuator 120 can include two motion-generating elements, one element coupled to the LED array 106 and the other motion-generating element coupled to the lens 114. In some examples, the actuation plane can be generally orthogonal to the longitudinal axis.

[0016] In addition to or instead of actuation, multiple LED arrays, including segmented LEDs, may be used to illuminate the scene. In this case, boundaries between LEDs that create dark bands in the illumination can be offset between different LED arrays. The offset helps reduce or eliminate dark bands in the overall illumination in the scene that may be present if only one LED array and lens is used.

[0017] The camera 102 can sense at least the wavelengths of light emitted by the LED array 106. The camera 102 can include an optical system (e.g., at least one camera lens 122) that can collect reflected light 124 reflected from and / or emitted by the scene 104. The camera lens 122 can direct the reflected light 124 onto a multi-pixel sensor 126 to form an image of the scene 104 on the multi-pixel sensor 126. A data signal representing the image of the scene 104 can be received by a controller 128. The controller 128 can optionally further drive the actuator 120 or each translator. The controller 128 can optionally further drive the LEDs 108 in the LED array 106. For example, the controller can optionally control one or more LEDs 108 in the LED array 106 independently of another one or more LEDs 108 in the LED array 106 to illuminate the scene as specified. For example, it may be desirable to provide a first amount of illumination to relatively closer objects in the scene 104 and a second amount of illumination, greater than the first amount, to relatively distant objects in the scene 104 so that they have the same brightness in an image of the scene 104. Other configurations are possible. The camera 102 and illumination device 100 can be disposed within a housing (not shown) that also houses the illumination configuration 130.

[0018] The lighting device 100 may also include an input device, e.g., a user-actuated input device such as a button that is pressed to take a picture. The camera 102 and lighting device 100 may be disposed in a housing that contains the lighting arrangement 130. The transmitter may be used to transmit the illuminated image to a remote processing device, e.g., a server 150, which may be located in a different geographical area (e.g., a city) than the lighting device or provided in a distributed (cloud) network. The image may be communicated to the external processing device via a local network, e.g., WiFi, or a remote network, e.g., a fifth generation (5G) network or some other network.

[0019] FIG. 2 illustrates an LED array according to some examples. The LED array 200 may include LEDs 202 that may be segmented to form corresponding illumination regions in the illumination 116 in the scene 104 shown in FIG. 1. Boundaries 204 may form dark bands in the illumination 116 in the scene 104. In some examples, the dark bands in the first illumination 116 in the scene 104 may correspond to boundaries 204 extending along a first direction and boundaries extending along a second direction orthogonal to the first direction. In some examples, the LEDs 202 may be arranged in a rectilinear array along orthogonal first and second dimensions. In some examples, each boundary 204 may be arranged as an elongated region extending along one of the first dimension or the second dimension. In some examples, at least one boundary 204 (corresponding to a dark band) may extend in a continuous line along the entire extent of the first linear array. In some examples, at least one boundary 204 may include a discontinuity or offset. In some examples, at least one boundary 204 may include parallel segments. In other embodiments in which multiple LED arrays 106 are used, the boundaries of the LED arrays 106 may not coincide and / or may be different (e.g., only one may have a discontinuity).

[0020] 3 illustrates an example of a lighting system according to some embodiments. As above, some of the elements shown in the lighting system 300 may not be present, while other additional elements may be disposed in the lighting system 300. Some or all of these elements may be provided, for example, on a printed circuit board (PCB) or other type of substrate. The lighting system 300 may include, among other items, a controller 310 and a pixel matrix 320. In some embodiments, some or all of the components of the controller 310 may be disposed, for example, on a composite metal oxide semiconductor (CMOS) backplane or other backplane on which semiconductor material may be deposited to form (or otherwise arrange) transistors or related components for controlling individual pixel units 324 of the pixel matrix 320 (including an LED array). The controller 310 may include, among other circuits, a processor 312, a memory 314, and a PWM generator 316 (or current generator).

[0021] The controller 310 may be used, inter alia, to control the driving of pixel units 324 of the pixel matrix 320 using a PWM generator 316 based on image data stored in a memory 314 or received from an external source, for example via internal communication using an internal bus or via a WiFi, 5G, or other network using a transceiver (not shown). The processor 312 may control the PWM duty cycle and / or light intensity, as well as the addressing of the pixel matrix 320, to cause the pixel matrix 320 to generate illumination.

[0022] The pixel matrix 320 includes addressing circuitry 322 in addition to pixel units 324. Other circuitry, such as amplifiers, comparators, and filters, are not shown for convenience. The addressing circuitry 322 may include circuitry for selecting one or more rows and columns with row and column selection signals, respectively, that may be used to select, and thus drive, one or more of the pixel units 324 to emit light. The pixel units 324 may include one or more individual pixels. Each pixel unit 324 may include an LED 326, a switching circuit 328, such as a switch controlled by a PWM signal, and a current source 330. The pixel units 324 may be driven individually based on the processor 312 to provide uniform illumination or may provide illumination tailored to the illuminated scene. For example, if the pixel matrix 320 is formed from micro LEDs, the pixel units 324 may be driven with different offsets due to current limitations. It should be noted that rather than using a PWM generator 316, the lighting system 300 may use a direct current (DC) driver capable of supplying a DC voltage or current (or a slowly varying voltage or current) of appropriate amplitude to generate the desired brightness from a particular LED 326 in the pixel matrix 320.

[0023] In addition to the pixel matrix 320, the controller 310 may also control, for example, the camera 102 shown in Figure 1. In some embodiments, the controller 310 may separately control the illumination provided by each LED 326 such that the LEDs 326 can provide different intensities, colors, correlated color temperatures, Duv values ​​(delta u,v that describes the distance of the color point of the light from the black body curve), color rendering index (CRI) values, etc. This may help the camera 102 capture a desired image quality.

[0024] In some cases, a scene may be illuminated multiple times to meet image quality metrics. In particular, a first amount of illumination may be used to illuminate relatively close objects in the scene, and a second amount of illumination, greater than the first amount of illumination, may be used to illuminate relatively distant objects in the scene to have the same or similar brightness as the relatively close objects in the scene. Different wavelengths of light may be used to determine distance to objects (e.g., IR for calibration and white / visible light for image capture with calibration) and / or other sensors may be present in the illumination configuration.

[0025] The sensor may include a 3D sensor, such as, for example, a Time of Flight (ToF) sensor that can measure the amount of time it takes for light emitted by the LED array and reflected off of objects in the scene to return to the ToF sensor. The time can be used to calculate distances to various elements in the scene. In some embodiments, the 3D sensor may be a structured light sensor that projects a specially designed pattern of light onto the scene.

[0026] A structured light sensor may include one or more cameras to measure the position of each portion of the light pattern reflected from the scene and determine distance by triangulation. In some embodiments, the 3D sensor may be an auxiliary camera or may be multiple auxiliary cameras positioned within a housing at a minimum distance from each other that is sufficient to allow comparison between images captured by the auxiliary cameras. By comparing the positions of objects as seen by the auxiliary cameras, distances to objects in the scene may be determined by triangulation.

[0027] In some embodiments, the 3D sensor may provide an autofocus signal for the main camera in the device. While scanning the focus position of the camera lens, the lighting arrangement may detect which parts of the scene are in focus at which positions. A 3D profile of the scene may then be constructed by converting the corresponding lens positions into distances to objects in focus at those positions. For example, an appropriate autofocus signal may be derived by measuring contrast or by utilizing a phase detection sensor in the camera sensor. If a phase detection sensor is used, in some embodiments, the positions of the individual phase detection sensors may correspond to areas illuminated by separate segments of the LED array 106 in order for the adaptive flash to function optimally.

[0028] As mentioned above, the distance may be measured using the same illumination as emitted by the LED array 106, or may use, for example, IR radiation or radiation of other wavelengths. Thus, in some cases, a first image of the scene may be taken using a first illumination condition, such as uniform illumination where all parts of the scene are illuminated with a known illumination profile. Then, within a short amount of time (e.g., less than a few seconds), a second image of the scene may be taken using a second illumination condition. Based on the first and second images, an optimal or desired brightness of all parts of the scene may be calculated to achieve optimal exposure. In one embodiment, pixel brightness values ​​of the first image may be subtracted from respective pixel brightness values ​​of the second image, and the difference between the pixel brightness values ​​of the first image and the pixel brightness values ​​of the second image may be scaled to determine optimal illumination parameters. A final image may then be taken using the optimal illumination parameters.

[0029] Also, issues affecting illumination other than the difference in distance of objects from the LED array 106 may exist, increasing the complexity in obtaining the desired image quality metric. Because light illuminance decreases according to an inverse square law with distance traveled from a non-coherent light source (such as an LED) or other light source or transducer generating an electromagnetic signal, the 3D profile of the scene may be used to determine the amount of light to distribute to various parts of the scene. However, the algorithm that calculates the desired intensity profile may also take into account the illuminance that each of the objects in the scene receives from ambient light, and the information gathered in the capture of the first image, and may adjust the amount of light accordingly. Objects that are already well illuminated, for example because they are light in color or have a relatively large reflectance (than other objects in the image), may receive less light than could be calculated based solely on the distance from the light source as determined by the 3D profile, and objects that are not well illuminated, for example because they are dark or have a relatively small reflectance, may receive more light.

[0030] In many cases, tuning physical design parameters, such as those described herein, may not be sufficient to meet the image quality metrics desired by the user. Therefore, the lighting configuration is further tuned using image processing algorithms, such as those shown above, to enhance the image captured by the camera 102. To this end, an artificial neural network (ANN)-based approach can be used to optimize or improve the image captured with the segmented LED array structure described above. Due to the computational complexity when multiple images are used to determine multiple variables for the final image settings, the use of an ANN approach may be desirable.

[0031] In particular, at least two parameters can be examined to meet a desired set of image quality metrics, including color variation and illumination pattern of the image. Color variation can be between segments of the LED array 106 and within a particular segment. Color variation can be caused by LED driving factors associated with the LED segments, such as manufacturing variations. Driving factors can include current and PWM settings (e.g., frequency, duty cycle) as well as temperature of the LED array 106. Illumination pattern is related to the design of the segmented LED array 106. Illumination pattern is visible on the scene due to the boundaries 204 (see FIG. 2) between segments, which can degrade image quality.

[0032] In some embodiments, artificial intelligence (AI) / machine learning (ML) may be used to generate conditions for providing lighting. The AI / ML process may include both a training mode to train the AI / ML model (e.g., with a predefined set of images of typical home and / or outdoor objects) and an inference mode to use once the AI / ML model is sufficiently trained. In some embodiments, the AI / ML model may be an ANN. In some embodiments, the external processing device described in connection with FIG. 1 may use object recognition via the AI / ML model to suggest or remotely configure optimal lighting for the object(s). The AI / ML model and storage may be located locally, for example, in the lighting arrangement 130 and / or in the server 150.

[0033] FIG. 4 illustrates an example of a block diagram of an image enhancement process according to some embodiments. Training may be performed on one or more local computing resources. The local computing resources may be provided, for example, in a central processing unit (CPU) and / or a graphics processing unit (GPU) (processor 140) in the lighting arrangement 130. Alternatively or additionally, training may be performed using remote computing resources of a remote processing device, which may be the server 150 or in a distributed (cloud) network. For example, in some cases, initial training may be performed remotely using a variety of images under different lighting conditions, and initial ANN parameters may be transferred to the local computing resources in the lighting arrangement 130 for updating when new images are acquired by the lighting arrangement 130 in inference mode.

[0034] The exemplary method 400 of FIG. 4 includes both initial remote training 410 (offline training) of a model, and local use and online training 420 of the initial remotely trained model. The initial remote training is used to obtain an acceptable model accuracy, which is then used locally and further trained when the initial accuracy is satisfactory. The accuracy can be between about 0% and about 100%, depending on the criticality of the application. For example, the desired accuracy for safety-critical applications may be close to 100%, while an accuracy of over 90% may be acceptable for other applications.

[0035] As shown in the initial remote training 410 of the method 400 of FIG. 4, at operation 402, one or more images may be acquired. The image of the scene may be captured by the camera 102 of FIG. 1. Alternatively, one or more predefined images whose characteristics are known may be acquired from a memory associated with the computing resource or from another device. Since a large number of images may be used for training, there may be various approaches to collecting the images. The image set includes an input image set and an output image set, where each input image has a corresponding output image (the output image is the desired result). The input images may be collected by taking images with one or more imaging devices (e.g., the camera and segmented LEDs described above), and manual correction of the input images may be used to generate the corresponding output images. Alternatively, a simulated environment may be used to artificially generate photorealistic images.

[0036] When the image(s) captured from the camera 102 are used for initial training, a transceiver in the lighting configuration 130 may transmit the image(s) of the illuminated scene to a remote processing device at operation 404. The image(s) may be communicated to the remote processing device over a local network such as WiFi, or a fifth generation (5G) network, or other network. The remote processing device may use image recognition to identify one or more objects in the transmitted image(s) and determine the lighting conditions of those objects.

[0037] After the images are transmitted in operation 404, the ANN may enter a training mode to train the ANN in operation 406. In some cases, the training mode may begin when an image or images are acquired (e.g., captured by camera 102 and transmitted over a network to a remote processing device). In some embodiments, the training mode may be run at predetermined intervals where images of a scene (or a given object) are batched together and executed when processing resources become available.

[0038] In general, model training corresponds to a design space exploration of models that minimize the error between given input data and expected output. Training may therefore take different durations depending, for example, on the amount of input data, the complexity of the ANN, and the available computational power. Thus, input data is fed to the ANN, and the generated output is fed to the ANN as training feedback.

[0039] 5A illustrates a training process according to some embodiments. The ANN 502 is a neural network with multiple layers: an initial (input) layer 502a, intermediate (hidden) layers 502b, ..., 502n-1, and a final (output) layer 502n. Each of the layers 502a, ..., 502n in the ANN 502 includes nodes (neurons) that process data in the ANN 502 through summation and transfer functions. The prediction accuracy of the ANN 502 depends on the number of nodes in the hidden layers. As shown, the ANN 502 receives input data, which is processed through the layers 502a, ..., 502n to generate an output that is fed back to the ANN 502 as training feedback for further changes to the parameters of the ANN 502. The training process 500 can thus be considered as an offline process.

[0040] In some embodiments, the training may be based on a comparison between the captured image and each of several simulated environments in which the image is created without correction. Each of the simulated environments may be under a predefined set of parameters under which the lighting configuration 130 operates. In particular, the parameters may be based on several different aspects of the lighting configuration 130, including the parameters used to drive the lighting device 100 and the camera 102, as well as environmental parameters. In particular, the environment may be based on, among other things, the drive current / PWM specifications (duty cycle, frequency) used to drive each segment of the LED array 106, the optical specifications (e.g., distance from the lens 114 / camera lens 122 from the LED array 106 and position relative to the LED array 106 / from the multi-pixel sensor 126), the sensor specifications (e.g., exposure time, dynamic range of exposure, gain factor of the multi-pixel sensor 126), noise suppression, and ambient light / flicker in the environment based on information from one or more separate sensors. In some cases, the multiple images may be generated with or without the use of the LED array 106, as well as using a conventional LED array to illuminate the scene during or prior to training. These images can be used to generate the parameters.

[0041] Returning to the initial remote training 410 of method 400 of FIG. 4, at operation 408, the ANN can determine if the accuracy of the model is acceptable, with acceptability based at least on the desired accuracy described above. Depending on the model, what is being trained, and the desired accuracy, a relatively large amount of data (images), time, and / or processing power may be used. However, in other embodiments, relatively less data, time, and processing power may be used to achieve acceptable results. As shown in FIG. 4, if the accuracy is determined to be unacceptable, the model returns to operation 406 and the ANN continues to be trained until an acceptable accuracy of the model is achieved based on the predetermined factors described above.

[0042] After the ANN has been trained (and the accuracy determined to be acceptable), local use and training of the initial remotely trained model 420 may begin. Thus, at operation 412, the model may be provided to a local processing device, i.e., the trained parameters for the ANN may be provided to a memory in the lighting arrangement 130 via a transceiver.

[0043] At operation 414, the local processing resource may use the trained ANN as an inference engine and continue to train the ANN to further improve accuracy with new incoming data / images (online training). That is, the local processing resource may use the trained ANN to correct captured images for problems caused by the lighting device 100 and / or image processing, such as color variations and lighting patterns, among others. Thus, the illumination provided by the lighting device 100, optical configuration, and so on, may be adjusted using parameters provided to the ANN in the local processing resource when in inference mode. In inference mode, the results of the ANN are monitored and output data is fed back to a performance analysis of the ANN. As a result of the performance analysis, the input data set and the ANN are updated.

[0044] In particular, if image correction alone is not sufficient to provide the desired response, one or more of the physical parameters of the lighting configuration 130 may be adjusted by a predetermined amount and a new image may be captured by the camera 102. The new image may then be fed into the ANN to determine whether the adjustments have enhanced the image.

[0045] Typical machine learning techniques use thousands or millions of images to accurately train a model. In some cases, all images captured by the camera 102 may be used to train the AI / ML model, while in other embodiments, only a subset of images may be selected from a larger set of available images. That is, in some embodiments, images may be selected for a particular quality or when substantial differences occur in the captured scene. For example, the images selected may be determined to be well focused throughout the image or specific to a particular scene. Images, or portions of images, may be excluded from training according to limitations set by the training algorithm. The images selected may be of any subject or type, provided the above conditions are met. In certain applications, images from similar scene content may be provided, although others may be used. Images may be of any size, and the algorithm may automatically adjust to different image sizes.

[0046] Also, in some cases, the level of accuracy of the parameters determined in the training may depend on the amount of processing resources available, including both computing power and available time. That is, training may provide the same level of accuracy using less computing power and a longer amount of time as more computing power and a shorter amount of time, thereby allowing the selection of an appropriate processing resource. For example, initial training may be performed in the cloud to a relatively high level of accuracy, and the training parameters may be provided to the lighting configuration 130. If further training is used in the lighting configuration 130, the level of accuracy of the parameters may decrease because there are fewer computing resources for training the general ANN in the lighting configuration 130. In some embodiments, the training or inference to provide the corrections may be performed on a dedicated chip in the lighting configuration 130 or may be provided on the main central processing unit. Since calls to the main central processing unit may result in more time being used than if a dedicated chip were used, the selection of the processing resource for training may depend on the amount of time allowed for training.

[0047] In some embodiments, the ANN may be a general model that is first generalized to real-world conditions. The ANN may be used because it may not be possible to tune the LEDs 108 for every scene to correct the lighting provided by the lighting configuration 130 to a level acceptable to a particular user. The user may also provide further feedback during the training of the ANN on the lighting configuration 130 to provide user-specific optimization that may vary from user to user.

[0048] 5B illustrates an inference process 600 according to some embodiments. In the inference process 600, input data is fed to an ANN, which is trained and then deployed. The output of the ANN is monitored, the performance of the ANN is analyzed, and the data used to train the ANN is updated. The updated data is then used to tune the ANN. The inference process 600 can be considered an online process since it is performed on the lighting configuration.

[0049] In addition to being used in mobile devices such as smartphones, laptop computers, or computer tablets, among others, the systems and methods described herein may be applicable in other electronic devices that use adaptive flash. In some embodiments, the training and inference modes of the ANN may be performed within the mobile (or other) device, with training being performed before the device is first delivered to a customer (i.e., during testing after manufacture but before being shipped for commercial / personal use).

[0050] The examples described herein may include or operate on logic or several components, modules, or mechanisms. Modules and components are tangible entities (e.g., hardware) capable of performing specified operations and may be configured or arranged in a particular manner. In an example, a circuit may be arranged (e.g., internally or with respect to an external entity such as another circuit) as specified as a module. In an example, all or part of one or more computer systems (e.g., standalone, client, or server computer systems), or one or more hardware processors, may be configured by firmware or software (e.g., instructions, application parts, or applications) as modules that operate to perform specified operations. In an example, the software may reside on a machine-readable medium. In an example, the software, when executed by the hardware underlying the module, causes the hardware to perform the specified operations.

[0051] Thus, the terms "module" (and "component") are understood to encompass tangible entities, that is physically constructed, specially configured (e.g., hardwired), or temporarily (e.g., transiently) configured (e.g., programmed) to operate as specified or to perform some or all of any of the operations described herein. Considering examples where modules are temporarily configured, it is not necessary that each of the modules is instantiated at any given moment. For example, if the modules include a general-purpose hardware processor configured with software, the general-purpose hardware processor can be configured as different modules at different times. The software may configure the hardware processor accordingly, for example, to configure a particular module at one time and a different module at a different time.

[0052] Memory may include a non-transitory machine-readable medium on which is stored one or more sets of data structures or instructions (e.g., software) that embody or are utilized by any one or more of the techniques or functions described herein. The instructions may also reside, completely or at least partially, in main memory, static memory, and / or in a hardware processor during execution thereof by the device. The term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions.

[0053] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by a device, causing the device to perform any one or more of the techniques of the present disclosure (e.g., training or inferring an ANN), or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices, magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, random access memory (RAM), and CD-ROM and DVD-ROM disks.

[0054] The instructions may further be sent or received over a communications network using a transmission medium via a network interface device utilizing any one of a number of Wireless Local Area Network (WLAN) transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Examples of communications networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a Plain Old Telephone (POTS) network, and a wireless data network. Communications over the network may include one or more different protocols, such as, for example, the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi, the IEEE 802.16 family of standards known as WiMax, the IEEE 802.15.4 family of standards, the Long Term Evolution (LTE) family of standards, the Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, Next Generation (NG) / Fifth Generation (5G) standards, among others. In one example, a network interface device may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas for connecting to a transmission medium.

[0055] The term "processor circuitry" or "processor" as used herein thus refers to, is a part of, or includes circuitry capable of sequentially and automatically performing a sequence of arithmetic or logical operations, or recording, storing, and / or transmitting digital data. The term "processor circuitry" or "processor" may refer to one or more application processors, one or more baseband processors, a physical CPU, a single-core processor or a multi-core processor, and / or any other device capable of executing or otherwise operating computer-executable instructions, e.g., program code, software modules, and / or functional processes.

[0056] example

[0057] Example 1 is a lighting configuration comprising: a lighting device having a light emitting diode (LED) array configured to emit light to illuminate a scene, the LED array having a plurality of LEDs separated by boundaries, the plurality of LEDs configured to be driven independently to provide the light; a camera having a sensor configured to capture images of the scene; and a processor configured to correct the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained with a set of images to generate parameters for correcting the images of the scene, and an inference mode in which the ANN is used on the images captured by the camera using the parameters.

[0058] In Example 2, the subject matter of Example 1 includes wherein the processor is configured to correct the image of the scene using the ANN to correct for color variations among the LEDs, variations within the LEDs, and lighting patterns caused by the boundaries.

[0059] In Example 3, the subject matter of Examples 1-2 includes a transceiver for communicating with a cloud network.

[0060] In Example 4, the subject matter of Example 3 includes wherein the transceiver is configured to transmit the image set to the cloud network, and the ANN is trained offline within the cloud network in the training mode to generate the parameters.

[0061] In Example 5, the subject matter of Example 4 includes wherein the transceiver is configured to receive the parameters from the cloud network, and the ANN is used online by the processor in the inference mode to correct the images of the scene.

[0062] In Example 6, the subject matter of Examples 1-5 includes, in the inference mode, the processor is configured to monitor results of the ANN, analyze performance of the ANN, the performance of the ANN indicating closeness of the image after correction to a simulated image, and update the parameters based on the performance of the ANN.

[0063] In Example 7, the subject matter of Examples 1-6 includes where the ANN is used on a dedicated graphics processing unit (GPU).

[0064] In Example 8, the subject matter of Examples 1-7 includes wherein the processor is configured to train the ANN in the training mode and to correct the image of the scene using the ANN in the inference mode.

[0065] In Example 9, the subject matter of Examples 1-8 includes, in the training mode, the ANN is trained using the same images under different lighting conditions provided by the lighting device.

[0066] In Example 10, the subject matter of Example 9 includes wherein the ANN is trained in the training mode to take into account ambient lighting of the scene.

[0067] In Example 11, the subject matter of Examples 1-10 includes, to correct the image, the processor is configured to adjust at least one setting of a group of settings having at least one setting selected from a current level setting and a pulse width modulation setting for driving each of the plurality of LEDs, an integration time of the sensor, and a gain of the sensor.

[0068] In Example 12, the subject matter of Examples 1-11 includes the processor being further configured to control a mechanical element in the lighting arrangement based on the ANN.

[0069] In Example 13, the subject matter of Examples 1-12 includes, wherein the boundary is a non-emitting region located between adjacent LEDs of the plurality of LEDs that is about 5% to about 10% of the distance between adjacent sides of the adjacent LEDs.

[0070] Example 14 is a mobile device comprising: an illumination device having a light emitting diode (LED) array configured to emit light to illuminate a scene, the LED array having a plurality of LEDs separated by boundaries, the plurality of LEDs configured to be driven independently to provide the light; a camera having a sensor configured to capture images of the scene; and a processor configured to correct the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained offline using a set of the same images under different lighting conditions to generate parameters for correcting the images of the scene, and an inference mode in which the ANN is used online on the images captured by the camera using the parameters.

[0071] In Example 15, the subject matter of Example 14 includes wherein the processor is configured to correct the image of the scene using the ANN to correct for color variations between the LEDs, variations within the LEDs, and lighting patterns caused by the boundaries.

[0072] In Example 16, the subject matter of Examples 14-15 includes the processor having a central processing unit (CPU) and a dedicated graphics processing unit (GPU), and the selection of the ANN used is dependent on available timing conditions for processing the image.

[0073] In Example 17, the subject matter of Examples 14-16 includes wherein the ANN is configured to provide correction of facial recognition in the inference mode.

[0074] Example 18 is a tangible computer-readable storage medium having stored thereon instructions for execution by one or more processors of an electronic device, the one or more processors which, when the instructions are executed, configure the electronic device to perform one or more operations including emitting light to illuminate a scene, the light being emitted from an LED array having a plurality of light emitting diodes (LEDs) separated by boundaries, each of the plurality of LEDs being independently driven to provide the light; capturing images of the scene using a sensor; and correcting the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained using a set of images captured by the sensor to generate parameters for correcting the images of the scene, and an inference mode in which the ANN is used on the images captured by the electronic device using the parameters.

[0075] In Example 19, the subject matter of Example 18 includes the instructions, when executed, further configuring the one or more processors to perform operations including transmitting the image set to a cloud network, the ANN being trained in the training mode to generate the parameters, receiving the parameters from the cloud network, and correcting the images of the scene using the ANN in the inference mode.

[0076] In Example 20, the subject matter of Examples 18-19 includes the instructions, when executed, further configuring the one or more processors to perform operations including: correcting the image of the scene using the ANN to correct for color variations between the plurality of LEDs, variations within the plurality of LEDs, and lighting patterns caused by the boundaries.

[0077] Example 21 is at least one machine-readable medium containing instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-20.

[0078] Example 22 is an apparatus having means for implementing any of Examples 1-20.

[0079] Example 23 is a system that implements any of Examples 1-20.

[0080] Example 24 is a method for implementing any of Examples 1-20.

[0081] While only certain features of the systems and methods have been illustrated and described herein, numerous modifications and changes will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes. Method actions may be performed substantially simultaneously or in differing orders.

Claims

1. an illumination device having a light emitting diode (LED) array configured to emit light to illuminate a scene, the LED array having a plurality of LEDs separated by boundaries, the plurality of LEDs configured to be independently driven to provide the light; a camera having a sensor configured to capture an image of the scene; a processor configured to enhance the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained using a set of images to generate parameters for enhancing the images of the scene, and an inference mode in which the ANN is used on the images captured by the camera using the parameters; A lighting arrangement having

2. 2. The lighting configuration of claim 1, wherein the processor is configured to correct the image of the scene using the ANN to correct for color variations between the LEDs, variations within the LEDs, and illumination patterns caused by the boundaries.

3. 10. The lighting arrangement of claim 1, further comprising a transceiver for communicating with a cloud network.

4. the transceiver is configured to transmit the set of images to the cloud network; The ANN is trained offline in the cloud network in the training mode to generate the parameters.

4. The lighting arrangement of claim 3.

5. the transceiver is configured to receive the parameters from the cloud network; the ANN is used online by the processor in the inference mode to correct the image of the scene.

5. The lighting arrangement of claim 4.

6. In the speculation mode, the processor: monitoring the results of the ANN; Analyzing the performance of the ANN, the performance of the ANN indicating the closeness of the corrected image to a simulated image; updating the parameters based on the performance of the ANN; 2. The lighting arrangement of claim 1, configured so that

7. The lighting configuration of claim 1 , wherein the ANN is implemented on a dedicated graphics processing unit (GPU).

8. The lighting configuration of claim 1 , wherein the processor is configured to train the ANN in the training mode and to use the ANN in the inference mode to correct the image of the scene.

9. The lighting configuration of claim 1 , wherein the ANN is trained in the training mode using the same images under different lighting conditions provided by the lighting device.

10. The lighting configuration of claim 9 , wherein the ANN is trained in the training mode to take into account ambient lighting of the scene.

11. 2. The lighting configuration of claim 1, wherein to correct the image, the processor is configured to adjust at least one setting of a group of settings having at least one setting selected from a current level setting and a pulse width modulation setting for driving each of the plurality of LEDs, an integration time of the sensor, and a gain of the sensor.

12. The lighting arrangement of claim 1 , wherein the processor is further configured to control a mechanical element within the lighting arrangement based on the ANN.

13. 2. The lighting configuration of claim 1, wherein the boundary is a non-emitting region located between adjacent LEDs of the plurality of LEDs that is about 5% to about 10% of a distance between adjacent sides of the adjacent LEDs.

14. an illumination device having a light emitting diode (LED) array configured to emit light to illuminate a scene, the LED array having a plurality of LEDs separated by boundaries, the plurality of LEDs configured to be independently driven to provide the light; a camera having a sensor configured to capture an image of the scene; a processor configured to correct the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained offline using the same set of images under different lighting conditions to generate parameters for correcting the images of the scene, and an inference mode in which the ANN is used online on the images captured by the camera using the parameters; A mobile device having:

15. 15. The mobile device of claim 14, wherein the processor is configured to correct the image of the scene using the ANN to correct for color variations among the LEDs, variations within the LEDs, and illumination patterns caused by the boundaries.

16. 15. The mobile device of claim 14, wherein the processor has a central processing unit (CPU) and a dedicated graphics processing unit (GPU), and the selection of the ANN to be used depends on available timing conditions for processing the image.

17. The mobile device of claim 14 , wherein the ANN is configured to provide correction for face recognition in the inference mode.

18. A tangible computer-readable storage medium storing instructions for execution by one or more processors of an electronic device, the one or more processors performing, when the instructions are executed: emitting light to illuminate the scene, the light being emitted from an LED array having a plurality of light emitting diodes (LEDs) separated by boundaries, each of the plurality of LEDs being independently driven to provide the light; Capturing an image of the scene using a sensor; correcting the images of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained using a set of images captured by the sensor to generate parameters for correcting the images of the scene, and an inference mode in which the ANN is used on the images captured by the electronic device using the parameters.

4. A computer readable storage medium for configuring the electronic device to perform one or more operations including:

19. The instructions, when executed, further comprise: transmitting the image set to a cloud network, and the ANN is trained in the training mode to generate the parameters; receiving the parameters from the cloud network and correcting the image of the scene using the ANN in the inference mode; 20. The tangible computer-readable storage medium of claim 18, further comprising:

20. 20. The tangible computer-readable storage medium of claim 18, wherein the instructions, when executed, further configure the one or more processors to perform operations including: correcting the image of the scene using the ANN to correct for color variations between the plurality of LEDs, variations within the plurality of LEDs, and illumination patterns caused by the boundaries.

Citation Information

Patent Citations

  • Device for identifying neural network pattern

    JP1992355889A

  • Method for determining picture discrimination parameter and picture recognition method

    JP1996138025A

  • Image capturing apparatus and method of controlling the same

    JP2010178288A

  • Color-Error-Corrected Segmented LED Array

    JP2021504892A

  • Multi-hypothesis classification for color constancy

    WO2021093947A1