LED array optimization using artificial neural networks

Abstract

A lighting arrangement in a mobile device is provided that uses an artificial neural network (ANN) to correct an image of a scene. The mobile device has a lighting device with an LED array that emits light to illuminate a scene. The LED array has LEDs that are separated by a boundary and driven independently. A camera has a sensor for capturing images of the scene. The ANN has a training mode in which the ANN is trained with a set of images captured by the camera to generate parameters. The ANN is trained offline in a cloud network. A processor corrects the image using the trained ANN in an inference mode.

Description

Technical Field

[0006] , [Figure 5A] , , [Figure 4] , , [Figure 3] , , [Figure 2] , , , , , , , , , ,

[0005]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 285,171, filed Dec. 2, 2021, which is hereby incorporated by reference in its entirety.

[0002] The present disclosure relates to an adjustable light emitting diode (LED) array.

Background Art

[0003] Efforts have been made to improve various types of lighting systems. In particular, it is desirable to enable automatic adjustment in lighting configurations to meet image quality metrics.

Brief Description of the Drawings

[0004] [Figure 1] A side view of a lighting configuration according to some examples is shown. [Figure 2] An LED array according to some examples is shown. [Figure 3] An example of a lighting system according to some examples is shown. [Figure 4] An example of a block diagram of an image enhancement process according to some embodiments is shown. <000002�> [Figure 5A] An example of a training process according to some embodiments is shown. [Figure 5B] An example of an inference process according to some embodiments is shown.

[0005] Corresponding parts are indicated by corresponding reference numerals throughout several views. The elements in the drawings are not necessarily drawn to scale. The configurations shown in the drawings are merely examples and should not be construed as limiting in any way.

Modes for Carrying Out the Invention

[0007] It is desirable that mobile devices equipped with LED modules provide sufficient values ​​for flash photography, including values ​​related to material parameters, electrical parameters, optical parameters, power parameters, and design-related parameters. Therefore, in addition to circuit and design selection, LED modules can use optics (e.g., lenses) to guide the light emitted by the LED light source. Furthermore, at the system level, other parameters such as the positioning of the LED module relative to other components (e.g., other sensors) and the form factor of the components increase the amount of complexity.

[0008] Flash photography using adaptive LED arrays introduces more challenges compared to systems using conventional LED arrays. For example, unlike conventional LED arrays, adaptive LED arrays may or may not have a uniform structure. In addition, the optical design of adaptive LED arrays is more complex than that of conventional LED arrays, at least in part due to structural issues.

[0009] Figure 1 shows a side view of a lighting configuration according to one example. The lighting configuration 130 can be, for example, a mobile device and may include both a lighting device 100 and a camera 102. The camera 102 can capture an image of the scene 104 during the exposure time of the camera 102, either without the lighting device 100 or with the lighting device 100 illuminating the scene 104. A processor 140 may be used to control various functions of the lighting configuration 130. Although only one processor 140 is shown in Figure 1, in other embodiments one or more processors may be used to provide the functions described herein.

[0010] The lighting device 100 may include a light-emitting diode (LED) array 106. The LED array 106 may include a plurality of LEDs 108 that can generate light 110 during the exposure time of the camera 102. In some embodiments, the LED array 106 may include segmented LEDs or microLEDs.

[0011] In embodiments where the LED array 106 is a micro-LED array, the LED array 106 may include thousands to millions of minute LED pixels that can emit light and be controlled individually or in groups of pixels (e.g., 5x5 groups of pixels). Micro-LEDs are small (e.g., <0.01 mm on each side) and can provide monochromatic or polychromatic light, typically red, green, and blue, using, for example, inorganic semiconductor materials. LED arrays 106 formed from inorganic materials (e.g., binary compounds such as gallium arsenide (GaAs), ternary compounds such as aluminum gallium arsenide (AlGaAs), quaternary compounds such as indium gallium phosphate (InGaAsP), or other suitable materials) are more robust than organic LEDs and enable use in a wider range of environments. In addition, LED arrays 106 formed from micro-LEDs can enable direct emission and can be more efficient than conventional backlight and liquid crystal display (LCD) combinations.

[0012] In embodiments where the LED array 106 includes segmented LEDs, the LED array 106 may include one or more non-emitting regions located between adjacent LEDs 108 within the LED array 106, as shown in Figure 2. The size of the non-emitting regions located between adjacent LEDs 108 (i.e., the distance between adjacent LEDs 108) may be a significant portion (e.g., about 5% to about 10%) of the size of the LEDs 108 within the LED array 106 (i.e., the distance between adjacent sides of the LEDs 108). In some examples, one or more of the non-emitting regions may surround the LEDs 108 within the LED array 106, causing dark bands to appear in the illumination emitted by the LED array 106.

[0013] The illumination device 100 may include at least one lens 114. The lens 114 can direct light 110 towards the scene 104 as illumination 116. The illumination device 100 may include one or more actuators 120. In other embodiments, actuators 120 may not be used. Instead, such a system may have a fixed lens, and therefore a fixed aperture. In a system including actuators 120, the actuators 120 may include individual translators that translate at least one of the LED array 106 or the lens 114, respectively, during the exposure time of the camera 102, so as to blur dark bands in the illumination 116 in the image of the scene 104. In some examples, one of those translators may translate the LED array 106 relative to the lens 114. In some examples, another of those translators may translate the lens 114 relative to the LED array 106. In some examples, the actuator 120 (a single element) may translate both the lens 114 and the LED array 106. In some examples, the lens 114 can define a vertical axis extending from the LED array 106 through the center of the lens 114 to the scene 104.

[0014] In some examples, the actuator 120 or each translator may be a one-dimensional actuator capable of translating at least one of the LED array 106 or the lens 114 in an operating direction angled with respect to the vertical axis. In some examples, the operating direction may be generally orthogonal to the vertical axis. In some examples, the LED array 106 may be arranged in a two-dimensional pattern having a first array direction and a second array direction orthogonal to the first array direction. In some examples, the operating direction may be angled with respect to both the first array direction and the second array direction. In some examples, the LED array 106 may be arranged in a one-dimensional pattern extending along the array direction. In some examples, the operating direction may be non-orthogonal to the array direction. In some examples, the operating direction may be generally parallel to the array direction. In some examples, the actuator 120 can translate at least one of the LED array 106 or the lens 114 in the operating direction by a distance greater than or equal to the width of one of the one or more non-emitting regions of the LED array 106 during the exposure time of the camera 102. In some examples, the actuator 120 can cause at least one of the LED array 106 or the lens 114 to vibrate in the operating direction. In some examples, the vibration may have a period shorter than the exposure time of the camera 102.

[0015] In some examples, the actuator 120 or each translator can be a two-dimensional actuator capable of translating at least one of the LED array 106 or the lens within an operating plane angled with respect to the longitudinal axis. For example, the actuator 120 may include two motion generating elements, one of which is coupled to the LED array 106 and the other motion generating element is coupled to the lens 114. In some examples, the operating plane can be generally orthogonal to the longitudinal axis.

[0016] In addition to or instead of operation, the scene may be illuminated using multiple LED arrays, including segmented LEDs. In this case, the boundaries between LEDs that form dark bands in the illumination can be offset between different LED arrays. This offset helps to reduce or eliminate dark bands in the overall illumination of the scene that may exist when only one LED array and lens are used.

[0017] Camera 102 can sense light of wavelengths emitted by at least the LED array 106. Camera 102 may include an optical system (e.g., at least one camera lens 122) capable of collecting reflected light 124 reflected from and / or emitted by the scene 104. The camera lens 122 directs the reflected light 124 onto the multi-pixel sensor 126, thereby forming an image of the scene 104 on the multi-pixel sensor 126. A data signal representing the image of the scene 104 may be received by the controller 128. The controller 128 may optionally further drive the actuators 120 or each translation unit. The controller 128 may optionally further drive the LEDs 108 in the LED array 106. For example, the controller may optionally control one or more LEDs 108 in the LED array 106 independently of another one or more LEDs 108 in the LED array 106 to illuminate the scene as specified. For example, it may be desirable to provide a first amount of illumination to relatively close objects in scene 104 and a second amount of illumination greater than the first amount to relatively distant objects in scene 104, so that they have the same brightness in the image of scene 104. Other configurations are also possible. The camera 102 and the illumination device 100 can be placed in a housing (not shown) that accommodates the illumination configuration 130.

[0018] The lighting device 100 may also include input devices, such as a user-activated input device, like a button pressed to take a photograph. The camera 102 and the lighting device 100 may be housed in a housing that accommodates the lighting configuration 130. A transmitter may be used to transmit the illuminated image to a remote processing device, such as a server 150, which may be located in a different geographical area (e.g., a city) from the lighting device or located in a distributed (cloud) network. The image may be communicated to the external processing device via a local network, such as Wi-Fi®, or a remote network, such as a fifth-generation (5G) network or any other network.

[0019] Figure 2 shows an LED array according to some examples. The LED array 200 may include LEDs 202 that are segmented and can form corresponding illuminated areas within illumination 116 in scene 104 shown in Figure 1. Boundaries 204 may form dark bands within illumination 116 in scene 104. In some examples, the dark bands within the first illumination 116 in scene 104 may correspond to boundaries 204 extending along a first direction and boundaries extending along a second direction perpendicular to the first direction. In some examples, the LEDs 202 may be arranged in a rectilinear array along the orthogonal first and second dimensions. In some examples, each boundary 204 may be arranged as an elongated region extending along either the first or second dimension. In some examples, at least one boundary 204 (corresponding to a dark band) may extend in a continuous line along the entire extension of the first linear array. In some examples, at least one boundary 204 may include discontinuities or offsets. In some examples, at least one boundary 204 may include multiple parallel segments. In other embodiments where multiple LED arrays 106 are used, the boundaries of those LED arrays 106 do not have to coincide and / or may be different (for example, only one may have a discontinuity).

[0020] Figure 3 shows an example of a lighting system according to some embodiments. Some of the elements shown in the lighting system 300 as above may not be present, while other additional elements may be arranged in the lighting system 300. Some or all of these elements may be provided on, for example, a printed circuit board (PCB) or other type of substrate. The lighting system 300 may include, among many other items, a controller 310 and a pixel matrix 320. In some embodiments, some or all of the components of the controller 310 may be arranged on a composite metal-oxide-semiconductor (CMOS) backplane or other backplane on which semiconductor material can be deposited to form (or otherwise arrange) transistors or related components for controlling individual pixel units 324 of the pixel matrix 320 (including an LED array). The controller 310 may include, among many other circuits, a processor 312, memory 314, and a PWM generator 316 (or current generator).

[0021] The controller 310 may, in particular, be used to control the driving of the pixel units 324 of the pixel matrix 320 using a PWM generator 316 based on image data stored in memory 314, or received from an external source, for example, via internal communication using an internal bus, or via WiFi®, 5G, or other networks using a transceiver (not shown). The processor 312 may control the PWM duty cycle and / or light intensity, as well as the addressing of the pixel matrix 320, to cause the pixel matrix 320 to generate illumination.

[0022] Pixel matrix 320 includes an addressing circuit 322 in addition to pixel units 324. Other circuits such as amplifiers, comparators, and filters are not shown for the sake of simplicity. The addressing circuit 322 can include circuits for selecting one or more rows and columns respectively using row and column selection signals that can be used to select, and thus drive, one or more of the pixel units 324 to emit light. The pixel units 324 can include one or more individual pixels. Each pixel unit 324 can include an LED 326, a switching circuit 328 such as a switch controlled by, for example, a PWM signal, and a current source 330. The pixel units 324 can be individually driven based on the processor 312 to provide uniform illumination or can provide illumination adjusted according to the scene being illuminated. For example, if the pixel matrix 320 is formed of micro LEDs, the pixel units 324 can be driven with different offsets due to current limiting. Instead of using the PWM generator 316, the lighting system 300 can use a direct current (DC) driver that can supply a DC voltage or DC current (or a slowly changing voltage or slowly changing current) of an appropriate amplitude to generate a desired luminance from a particular LED 326 of the pixel matrix 320.

[0023] In addition to the pixel matrix 320, the controller 310 can also control, for example, the camera 102 shown in FIG. 1. In some embodiments, the controller 310 can separately control the illumination provided by each LED 326 such that the LED 326 can provide different intensities, colors, correlated color temperatures, Duv values (delta u, v describing the distance of the color point of light from the blackbody curve), color rendering index (CRI) values, and the like. This can help the camera 102 capture a desired image quality.

[0024] In some cases, to meet the image quality metric, the scene may be illuminated multiple times. In particular, a first illumination amount may be used to illuminate relatively close objects within the scene, and a second illumination amount greater than the first illumination amount may be used to illuminate relatively distant objects within the scene, such that the relatively close objects within the scene have the same or equivalent luminance. To determine the distance to an object, light of different wavelengths may be used (e.g., IR for calibration, and white / visible light for image capture using the calibration), and / or other sensors may be present within the illumination configuration.

[0025] The sensor can include a 3D sensor such as, for example, a time-of-flight (ToF) sensor, which can measure the amount of time it takes for light emitted by an LED array and reflected from an object in the scene to return to the ToF sensor. Using the time, the distances to various elements within the scene can be calculated. In some embodiments, the 3D sensor may be a structured light sensor that projects a specially designed pattern of light onto the scene.

[0026] The structured light sensor can include one or more cameras to measure the position of each part of the light pattern reflected from the scene and determine the distance by triangulation. In some embodiments, the 3D sensor may be one auxiliary camera, or a plurality of auxiliary cameras positioned within the housing at a minimum distance sufficient to enable comparison between images captured by at least those auxiliary cameras. By comparing the positions of the objects seen by those auxiliary cameras, the distance to an object within the scene can be determined by triangulation.

[0027] In some embodiments, the 3D sensor may provide an autofocus signal for the main camera within the device. While scanning the focal position of the camera lens, the lighting configuration can detect which parts of the scene are in focus at which positions. A 3D profile of the scene can then be constructed by converting the corresponding lens positions into distances to objects that are in focus at those positions. For example, an appropriate autofocus signal may be derived by measuring contrast or by utilizing a phase detection sensor within the camera sensor. When a phase detection sensor is used, in some embodiments, the position of each individual phase detection sensor may correspond to areas illuminated by separate segments of the LED array 106, so that the adaptive flash functions optimally.

[0028] As described above, the distance may be measured using the same illumination emitted by the LED array 106, or, for example, IR emission or emission of other wavelengths. Thus, in some cases, a first image of the scene may be taken using first illumination conditions, such as uniform illumination where all parts of the scene are illuminated with a known illumination profile. Then, within a short time (e.g., less than a few seconds), a second image of the scene may be taken using second illumination conditions. To achieve optimal exposure, the optimal or desired luminance for all parts of the scene may be calculated based on the first and second images. In one embodiment, the pixel luminance value of the first image may be subtracted from the respective pixel luminance value of the second image, and the difference between the pixel luminance values ​​of the first and second images may be scaled to determine the optimal illumination parameters. Then, a final image may be taken using the optimal illumination parameters.

[0029] Furthermore, in addition to differences in the distance of objects from the LED array 106, other issues affecting illumination may exist, increasing the complexity in obtaining the desired image quality metric. Since illuminance decreases according to the inverse square law of the distance traveled from a non-coherent light source (e.g., LEDs) or another light source or transducer that generates an electromagnetic signal, the amount of light to distribute to different parts of the scene can be determined using a 3D profile of the scene. However, the algorithm for calculating the desired intensity profile may also take into account the illuminance that each object in the scene receives from ambient light, as well as the information gathered in the first image capture, and adjust the amount of light accordingly. For example, objects that are already well-illuminated, such as being bright in color or having a relatively high reflectivity (compared to other objects in the image), may receive less light than could be calculated based solely on the distance from the light source determined by the 3D profile, while objects that are not well-illuminated, such as being dark or having a relatively low reflectivity, may receive more light.

[0030] In many cases, tuning physical design parameters, such as those described here, may not be sufficient to meet the image quality metrics desired by the user. Therefore, the lighting configuration is further tuned using image processing algorithms, such as those shown above, to enhance (improve the quality of) the images captured by camera 102. For this purpose, an artificial neural network (ANN)-based approach can be used to optimize or improve the images captured by the segmented LED array structure described above. The use of an ANN approach may be desirable due to the computational complexity when multiple images are used to determine multiple variables for the final image settings.

[0031] In particular, to satisfy a desired set of image quality metrics, at least two parameters, including color variation and the illumination pattern of the image, can be examined. Color variation can be between segments of the LED array 106 and within specific segments. Color variation can be caused by LED driving factors related to the LED segments, such as manufacturing variations. Driving factors can include current and PWM settings (e.g., frequency, duty cycle), as well as the temperature of the LED array 106. The illumination pattern relates to the design of the segmented LED array 106. The illumination pattern is visible in the scene due to the boundaries 204 between segments (see Figure 2), which can degrade image quality.

[0032] In some embodiments, artificial intelligence (AI) / machine learning (ML) may be used to generate conditions for providing lighting. The AI / ML process may include both a training mode for training an AI / ML model (e.g., using a predetermined set of images of typical household and / or outdoor objects) and an inference mode for use after the AI / ML model has been sufficiently trained. In some embodiments, the AI / ML model may be an ANN (All Network Angle). In some embodiments, the external processing device described in relation to Figure 1 may use object recognition via the AI / ML model to suggest or remotely configure optimal lighting for (one or more) objects. The AI / ML model and storage may be located locally, for example, within the lighting configuration 130 and / or within the server 150.

[0033] Figure 4 shows an example block diagram of an image enhancement process according to one embodiment. Training may be performed on one or more local computing resources. Local computing resources may be located, for example, in a central processing unit (CPU) and / or graphics processing unit (GPU) (processor 140) within the lighting configuration 130. Alternatively, or in addition, training may be performed using remote computing resources of a teleprocessing device, which may be a server 150 or located in a distributed (cloud) network. For example, in some cases, initial training may be performed remotely using a variety of images under different lighting conditions, and initial ANN parameters may be transferred to local computing resources within the lighting configuration 130 for updating when new images are acquired by the lighting configuration 130 in inference mode.

[0034] The exemplary method 400 in Figure 4 includes both initial remote training of the model 410 (offline training) and local and online training 420 of the initially remotely trained model. The initial remote training is used to obtain an acceptable model accuracy, and then, when the initial accuracy is satisfactory, it is used locally and further trained. The accuracy can be between approximately 0% and approximately 100%, depending on the importance of the application. For example, an accuracy close to 100% may be desired for an application where safety is paramount, while an accuracy of over 90% may be acceptable for other applications.

[0035] As shown in the initial remote training 410 of Method 400 in Figure 4, one or more images may be acquired in Operation 402. Images of the scene may be captured by camera 102 in Figure 1. Alternatively, one or more predetermined images whose characteristics are known may be acquired from memory associated with the computing resource or from another device. Since a large number of images may be used for training, various methods may exist for collecting images. The image set includes an input image set and an output image set, where each input image has a corresponding output image (the output image is the desired result). Input images can be collected by taking images using one or more image devices (e.g., the camera and segmented LEDs mentioned above), and the corresponding output images may be generated using manual correction of the input images. Alternatively, realistic, photographic images may be artificially generated using a simulation environment.

[0036] When one or more images captured from camera 102 are used for initial training, in operation 404, a transceiver in the lighting configuration 130 may transmit images of the illuminated scene to a remote processing device. These images may be transmitted to the remote processing device via a local network such as Wi-Fi®, a fifth-generation (5G) network, or another network. The remote processing device may use image recognition to identify one or more objects in the transmitted images and determine the lighting conditions for those objects.

[0037] After the image is transmitted in operation 404, the ANN may enter training mode in operation 406 and be trained. In some cases, training mode may be initiated when (one or more) images are acquired (e.g., captured by camera 102 and transmitted to a remote processing device over the network). In some embodiments, training mode may be performed in batches of images of a scene (or a given object) at predetermined intervals, and may be executed when processing resources become available.

[0038] Generally, model training corresponds to searching the design space of a model to minimize the error between given input data and the expected output. Therefore, training can take different amounts of time, depending on factors such as the amount of input data, the complexity of the ANN, and the available computing power. Thus, once input data is fed into the ANN, the generated output is fed back into the ANN as training feedback.

[0039] Figure 5A shows a training process according to one embodiment. ANN502 is a neural network having multiple layers: the first (input) layer 502a, intermediate (hidden) layers 502b, ..., 502n-1, and the final (output) layer 502n. Each of the layers 502a, ..., 502n in ANN502 contains nodes (neurons) that process the data in ANN502 through sums and transfer functions. The prediction accuracy of ANN502 depends on the number of nodes in the hidden layers. As shown in the figure, ANN502 receives input data, processes it through layers 502a, ..., 502n, generates an output, which is then fed back to ANN502 as training feedback for further changes to the parameters of ANN502. The training process 500 can therefore be considered an offline process.

[0040] In some embodiments, training may be based on a comparison between captured images and each of several simulated environments in which images are produced without correction. Each simulated environment may be under a predetermined set of parameters under which the lighting configuration 130 operates. In particular, the parameters may be based on several different aspects of the lighting configuration 130, including parameters used to drive the lighting device 100 and camera 102, as well as environmental parameters. In particular, the environment may be based on ambient light / flicker in the environment based, among other things, on information from one or more separate sensors, including the drive current / PWM specifications (duty cycle, frequency) used to drive each segment of the LED array 106, optical specifications (e.g., distance from lens 114 / camera lens 122 to LED array 106, and position relative to LED array 106 / multipixel sensor 126), sensor specifications (e.g., exposure time, dynamic range of exposure, gain coefficient of multipixel sensor 126), noise suppression, and information from one or more separate sensors. In some cases, multiple images may be generated by illuminating the scene with a conventional LED array, not only with or without the use of the LED array 106, but also during or before training. These images can be used to generate parameters.

[0041] Returning to the initial remote training 410 of Method 400 in Figure 4, in operation 408, the ANN can determine whether the model's accuracy is acceptable, which is at least based on the desired accuracy described above. Depending on the model, what is being trained, and the desired accuracy, a relatively large amount of data (images), time, and / or processing power may be used. However, in other embodiments, relatively little data, time, and processing power may be used to obtain an acceptable result. As shown in Figure 4, if the accuracy is determined to be unacceptable, the model returns to operation 406, and the ANN continues to train until an acceptable accuracy for the model is achieved based on the predetermined factors described above.

[0042] After the ANN is trained (and its accuracy is deemed acceptable), local use and training of the initial remotely trained model 420 can begin. Thus, in operation 412, the model can be supplied to the local processing device. That is, the parameters trained for the ANN can be supplied to memory in the illumination configuration 130 via the transceiver.

[0043] In operation 414, the local processing resource may use the trained ANN as an inference engine and continue training the ANN to further improve accuracy with new incoming data / images (online training). That is, the local processing resource may use the trained ANN to correct the captured images for issues caused by the lighting device 100 and / or image processing, such as color variation and lighting patterns. Thus, the lighting provided by the lighting device 100, optical configuration, and others may be adjusted using parameters supplied to the ANN in the local processing resource when in inference mode. In inference mode, the ANN's results are monitored and the output data is fed back into the ANN's performance analysis. As a result of the performance analysis, the input dataset and the ANN are updated.

[0044] In particular, if image correction alone cannot provide the desired response, one or more physical parameters of the illumination configuration 130 may be adjusted by a predetermined amount, and a new image may be captured by the camera 102. The new image is then supplied to the ANN, and it can be determined whether the image has been enhanced by the adjustment.

[0045] Typical machine learning techniques accurately train models using thousands or millions of images. In some cases, all images captured by camera 102 may be used to train the AI / ML model, while in other embodiments, only a subset of images may be selected from a larger set of available images. That is, in some embodiments, images may be selected based on certain quality or when there is a substantial difference in the captured scene. For example, selected images may be determined to be well in focus throughout or to be specific to a particular scene. Images, or parts of images, may be excluded from training according to limitations set by the training algorithm. Selected images may be of any subject or type when the above conditions are met. For a particular application, images from similar scene content may be supplied, but others may be used. Images may be of any size, and the algorithm may automatically adjust to different image sizes.

[0046] Furthermore, in some cases, the level of accuracy of the parameters determined during training may depend on the amount of available processing resources, including both computing power and available time. That is, training can yield the same level of accuracy with less computing power and a longer amount of time as with more computing power and a shorter amount of time, thereby enabling the selection of appropriate processing resources. For example, initial training may be performed in the cloud to a relatively high level of accuracy, and the training parameters may be provided to the illumination configuration 130. If further training is used within the illumination configuration 130, the level of accuracy of the parameters may decrease because there are fewer computing resources available for training a typical ANN within the illumination configuration 130. In some embodiments, training or inference to provide corrections may be performed on a dedicated chip within the illumination configuration 130 or provided on a main central processing unit. Since calls to the main central processing unit may result in longer usage times than when a dedicated chip is used, the selection of processing resources for training may depend on the amount of time available for training.

[0047] In some embodiments, the ANN may be a general model that is first generalized to real-world conditions. The ANN may be used because the lighting provided by the lighting configuration 130 may not be able to be tuned to every scene to compensate to a level acceptable to a particular user. The user may also provide further feedback during training the ANN on the lighting configuration 130 to provide user-specific optimizations that may differ from user to user.

[0048] Figure 5B shows an inference process 600 according to one embodiment. In the inference process 600, input data is supplied to the ANN, the ANN is trained, and then deployed. The output of the ANN is monitored, the performance of the ANN is analyzed, and the data used to train the ANN is updated. The ANN is then tuned using the updated data. Since the inference process 600 takes place on the lighting configuration, it can be considered an online process.

[0049] In particular, in addition to being used in mobile devices such as smartphones, laptop computers, or computer tablets, the systems and methods described herein may be applicable to other electronic devices that use adaptive flash. In some embodiments, the ANN training and inference modes can be performed within a mobile (or other) device, and training is performed before the device is first supplied to the customer (i.e., during testing after manufacturing but before shipment for commercial / personal use).

[0050] The examples described herein may include, or operate on, logic or several components, modules, or mechanisms. Modules and components are tangible entities (e.g., hardware) capable of performing a specified operation and may be configured or arranged in a particular manner. In one example, a circuit may be arranged as designated as a module (e.g., internally or relative to an external entity such as another circuit). In one example, all or part of one or more computer systems (e.g., standalone, client, or server computer systems), or one or more hardware processors, may be configured by firmware or software (e.g., instructions, application parts, or applications) as modules that operate to perform a specified operation. In one example, the software may reside on a machine-readable medium. In one example, the software, when executed by the underlying hardware of the module, causes that hardware to perform a specified operation.

[0051] Therefore, the terms “module” (and “component”) are understood to encompass tangible entities that are physically constructed, specially configured (e.g., hardwired), or temporarily configured (e.g., transiently) (e.g., programmed) to operate as specified, or to perform some or all of the operations described herein. Considering an example where a module is temporarily configured, each module does not need to be instantiated at any given moment. For example, if a module includes a general-purpose hardware processor configured with software, that general-purpose hardware processor can be configured as different modules at different times. Software may, for example, configure the hardware processor so that it constitutes a particular module at one point in time and different modules at different points in time.

[0052] Memory may include a non-temporary, machine-readable medium containing one or more sets of data structures or instructions (e.g., software) that embody or utilize one or more of the technologies or functions described herein. Instructions may also reside, fully or at least partially, in main memory, static memory, and / or hardware processors while they are being executed by the device. The term “machine-readable medium” may include a single or multiple mediums configured to store one or more instructions (e.g., a centralized or distributed database, and / or associated caches and servers).

[0053] The term “machine-readable medium” may include any medium capable of storing, encoding, or carrying instructions for execution by a device, causing the device to execute one or more of the techniques of this disclosure (e.g., training or inference by an ANN), or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable mediums may include solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable mediums may include, for example, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and non-volatile memory such as flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, random-access memory (RAM), and CD-ROM and DVD-ROM disks.

[0054] Instructions may further be transmitted or received over a communication network using a transmission medium via a network interface device that utilizes one of several wireless local area network (WLAN) transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Examples of communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), plain old telephone (POTS) networks, and wireless data networks. Communication over a network may include one or more different protocols, such as, for example, the IEEE 802.11 standard family known as Wi-Fi®, the IEEE 802.16 standard family known as WiMAX®, the IEEE 802.15.4 standard family, the Long-Term Evolution (LTE) standard family, the Universal Mobile Telecommunications System (UMTS) standard family, peer-to-peer (P2P) networks, and next-generation (NG) / fifth-generation (5G) standards. In one example, a network interface device may include one or more physical jacks (e.g., Ethernet®, coaxial, or telephone jacks) or one or more antennas for connecting to a transmission medium.

[0055] As used herein, the terms “processor circuit” or “processor” mean, or include, a circuit capable of sequentially and automatically executing sequences of arithmetic or logical operations, or recording, storing, and / or transferring digital data. The terms “processor circuit” or “processor” may mean one or more application processors, one or more baseband processors, physical CPUs, single-core processors or multi-core processors, and / or any other device capable of executing or otherwise operating computer executable instructions such as program code, software modules, and / or functional processes.

[0056] example

[0057] Example 1 is a lighting device having an array of light-emitting diodes (LEDs) configured to emit light to illuminate a scene, wherein the LED array has a plurality of LEDs separated by a boundary, and the plurality of LEDs are configured to be driven independently to provide the light; a camera having a sensor configured to capture an image of the scene; and a processor configured to correct the image of the scene using an artificial neural network (ANN), wherein the ANN has a training mode in which the ANN is trained using a set of images to generate parameters for correcting the image of the scene, and an inference mode in which the ANN is used on the image captured by the camera using the parameters.

[0058] In Example 2, the subject of Example 1 is configured such that the processor uses the ANN to correct the image of the scene in order to compensate for color variations between the plurality of LEDs, variations within the plurality of LEDs, and lighting patterns caused by the boundaries.

[0059] In Example 3, the subject matter of Examples 1 and 2 includes a transceiver for communicating with a cloud network.

[0060] In Example 4, the subject of Example 3 is configured such that the transceiver transmits the image set to the cloud network, and the ANN is trained offline within the cloud network in the training mode for generating the parameters.

[0061] In Example 5, the subject of Example 4 includes the transceiver being configured to receive the parameters from the cloud network, and the ANN being used online by the processor in the inference mode for correcting the image of the scene.

[0062] In Example 6, the subject matter of Examples 1-5 is configured such that, in the inference mode, the processor monitors the results of the ANN, analyzes the performance of the ANN, the performance of the ANN indicates the closeness of the corrected image to the simulated image, and updates the parameters based on the performance of the ANN.

[0063] In Example 7, the subject matter of Examples 1-6 includes the fact that the ANN is used on a dedicated graphics processing unit (GPU).

[0064] In Example 8, the subject matter of Examples 1-7 is configured such that the processor trains the ANN in the training mode and uses the ANN to correct the image of the scene in the inference mode.

[0065] In Example 9, the subject matter of Examples 1-8 includes the fact that the ANN is trained in the training mode using the same images under different lighting conditions provided by the illumination device.

[0066] In Example 10, the subject of Example 9 is further described by the fact that the ANN is trained to take into account the ambient lighting of the scene in the training mode.

[0067] In Example 11, the subject of Examples 1-10 is configured such that, in order to correct the image, the processor adjusts at least one of a set of settings having current level settings and pulse width modulation settings for driving each of the plurality of LEDs, the integration time of the sensor, and the gain of the sensor.

[0068] In Example 12, the subject matter of Examples 1-11 is further configured to control the mechanical elements in the lighting configuration based on the ANN.

[0069] In Example 13, the subject of Examples 1-12 includes the fact that the boundary is a non-emitting region located between adjacent LEDs of the plurality of LEDs, where the distance between adjacent sides of the adjacent LEDs is approximately 5% to approximately 10%.

[0070] Example 14 is a mobile device comprising: an illumination device having an array of light-emitting diodes (LEDs) configured to emit light to illuminate a scene, wherein the LED array has a plurality of LEDs separated by a boundary, and the plurality of LEDs are configured to be driven independently to provide the light; a camera having a sensor configured to capture an image of the scene; and a processor configured to correct the image of the scene using an artificial neural network (ANN), wherein the ANN has a training mode in which the ANN is trained offline using the same set of images under different lighting conditions to generate parameters for correcting the image of the scene, and an inference mode in which the ANN is used online on the image captured by the camera using the parameters.

[0071] In Example 15, the subject of Example 14 is configured such that the processor uses the ANN to correct the image of the scene in order to correct color variations between the plurality of LEDs, variations within the plurality of LEDs, and lighting patterns caused by the boundaries.

[0072] In Example 16, the subject matter of Examples 14–15 includes the fact that the processor has a central processing unit (CPU) and a dedicated graphics processing unit (GPU), and the selection of the ANN is used depending on the available timing conditions for processing the image.

[0073] In Example 17, the subject of Examples 14–16 includes the fact that the ANN is configured to provide face recognition correction in the inference mode.

[0074] Example 18 is a tangible computer-readable storage medium storing instructions for execution by one or more processors of an electronic device, wherein the one or more processors are configured to perform one or more operations, which include: illuminating a scene by emitting light when the instructions are executed, the light being emitted from an LED array having a plurality of light-emitting diodes (LEDs) separated by a boundary, each of the plurality of LEDs being driven independently to provide the light; capturing an image of the scene using a sensor; and correcting the image of the scene using an artificial neural network (ANN), the ANN having a training mode in which the ANN is trained using a set of images captured by the sensor to generate parameters for correcting the image of the scene; and an inference mode in which the ANN is used on the image captured by the electronic device using the parameters.

[0075] In Example 19, the subject of Example 18 further includes configuring the one or more processors to perform an operation which includes: when the instruction is executed, sending the set of images to a cloud network; the ANN being trained in the training mode to generate the parameters; receiving the parameters from the cloud network; and using the ANN in the inference mode to correct the images of the scene.

[0076] In Example 20, the subject of Examples 18–19 includes configuring one or more processors to perform an operation which, when the instruction is executed, further includes correcting the image of the scene using the ANN to correct for color variations between the plurality of LEDs, variations within the plurality of LEDs, and lighting patterns caused by the boundaries.

[0077] Example 21 is at least one machine-readable medium containing instructions, the instructions being at least one machine-readable medium which, when executed by a processing circuit, causes the processing circuit to perform an operation to implement any of Examples 1 to 20.

[0078] Example 22 is a device having means for implementing any of Examples 1-20.

[0079] Example 23 is a system that implements one of Examples 1-20.

[0080] Example 24 is a way to implement one of Examples 1-20.

[0081] Only specific features of the system and method are illustrated and described herein, but those skilled in the art will notice numerous modifications and variations. It should be understood that, therefore, the attached claims are intended to cover all such modifications and variations. The operation of the method may be performed substantially simultaneously or in different orders.

Claims

1. A lighting device having an array of light-emitting diodes (LEDs) configured to emit light and illuminate a scene, wherein the LED array has a plurality of LEDs separated by a boundary, and the plurality of LEDs are configured to be driven independently to provide the light. A camera having a sensor configured to capture an image of the aforementioned scene, A processor configured to correct the image of the scene using an artificial neural network (ANN), wherein the ANN has a training mode in which the ANN is trained using a set of images to generate parameters for correcting the image of the scene, and an inference mode in which the ANN is used on the image captured by the camera using the parameters, and the image correction includes correcting the lighting pattern including dark bands caused by the boundaries between the plurality of LEDs, A lighting configuration having

2. The lighting configuration according to claim 1, wherein the correction of the image further includes correction of color variations between the plurality of LEDs and manufacturing variations within the plurality of LEDs.

3. The lighting configuration according to claim 1, further comprising a transceiver for communicating with a cloud network.

4. The transceiver is configured to transmit the image set to the cloud network. The ANN is trained offline within the cloud network in the training mode for generating the parameters. The lighting configuration according to claim 3.

5. The transceiver is configured to receive the parameters from the cloud network. The ANN, in the inference mode for correcting the image of the scene, is used online by the processor. The lighting configuration according to claim 4.

6. In the inference mode, the processor: Monitor the results of the aforementioned ANN, The performance of the ANN is analyzed, and the performance of the ANN indicates the closeness of the corrected image to the simulated image. The parameters are updated based on the performance of the ANN. The lighting configuration according to claim 1, configured as described above.

7. The lighting configuration according to claim 1, wherein the ANN is used on a dedicated graphics processing unit (GPU).

8. The lighting configuration according to claim 1, wherein the processor is configured to train the ANN in the training mode and to correct the image of the scene using the ANN in the inference mode.

9. The lighting configuration according to claim 1, wherein the ANN is trained in the training mode using the same images under different lighting conditions provided by the lighting device.

10. The lighting configuration according to claim 9, wherein the ANN is trained to take into account the ambient lighting of the scene in the training mode.

11. The lighting configuration according to claim 1, wherein the processor is further configured to control the mechanical elements in the lighting configuration based on the ANN.

12. The lighting configuration according to claim 1, wherein the boundary is a non-emitting region located between adjacent LEDs among the plurality of LEDs, and is 5% to 10% of the distance between adjacent sides of the adjacent LEDs.

13. A lighting device having an array of light-emitting diodes (LEDs) configured to emit light and illuminate a scene, wherein the LED array has a plurality of LEDs separated by a boundary, and the plurality of LEDs are configured to be driven independently to provide the light. A camera having a sensor configured to capture an image of the aforementioned scene, A processor configured to correct the image of the scene using an artificial neural network (ANN), wherein the ANN has a training mode in which the ANN is trained offline using the same set of images under different lighting conditions to generate parameters for correcting the image of the scene, and an inference mode in which the ANN is used online on the image captured by the camera using the parameters, wherein the image correction includes correcting the lighting pattern, including dark bands caused by the boundaries between the plurality of LEDs. A mobile device that has [a certain feature].

14. The mobile device according to claim 13, wherein the correction of the image further includes correction of color variations between the plurality of LEDs and manufacturing variations within the plurality of LEDs.

15. The mobile device according to claim 13, wherein the processor comprises a central processing unit (CPU) and a dedicated graphics processing unit (GPU), and the selection of the ANN is used depending on available timing conditions for processing the image.

16. A tangible computer-readable storage medium storing instructions for execution by one or more processors of an electronic device, wherein when the instructions are executed, It emits light to illuminate the scene, and the light is emitted from an LED array having a plurality of light-emitting diodes (LEDs) separated by a boundary, and each of the plurality of LEDs is driven independently to provide the light. Using a sensor, capture an image of the scene, An artificial neural network (ANN) is used to correct the image of the scene, the ANN having a training mode in which the ANN is trained using a set of images captured by the sensor to generate parameters for correcting the image of the scene, and an inference mode in which the ANN is used on the image captured by the electronic device using the parameters, the image correction including correction of lighting patterns including dark bands caused by the boundaries between the plurality of LEDs. A tangible computer-readable storage medium configured to perform one or more operations, including the operation of the electronic device.

17. The aforementioned instruction, when executed, further, The set of images is transmitted to the cloud network, and the ANN is trained in the training mode to generate the parameters. The parameters are received from the cloud network, and the image of the scene is corrected using the ANN in the inference mode. A tangible computer-readable storage medium according to claim 16, wherein one or more processors are configured to perform an operation having the action of

18. The tangible computer-readable storage medium according to claim 16, wherein the correction of the image further includes correction of color variations between the plurality of LEDs and manufacturing variations within the plurality of LEDs.

19. The lighting configuration according to claim 1, wherein the set of images used in the training mode includes a subset of images captured by the camera based on substantial differences between the captured images.

20. The lighting configuration according to claim 1, wherein the set of images used in the training mode includes a subset of images captured by the camera, and each specific image in the subset is selected based on at least one parameter selected from parameters including a predetermined focus accuracy over the entire specific image and a predetermined focus accuracy specific to a particular scene within the specific image.

21. The lighting configuration according to claim 1, wherein the ANN is trained using a combination of local training by the processor and remote training by another processor in a cloud network, the remote training providing the initial accuracy of the ANN, and the local training thereafter providing further training.

Images