Detecting camera controller sequences

Abstract

In an exemplary embodiment, a controller for a lighting device and a computer system are provided. In particular, a user may define, by a computer system, a sequence including one or more image capture configurations. Each image acquisition configuration includes a light source identifier to be enabled during image acquisition, a channel for acquiring an image, and an exposure time. Each sequence may also define a custom delay between configurations. The controller starts the stored sequence after receiving the hardware or software trigger, and acquires the image according to the image acquisition configuration (and delay). An artificial intelligence technique may then be applied to the acquired image to identify a suspected defect of the component in the image.

Description

[0001] Priority Statement

[0002] This application claims the priority benefit of U.S. Patent Application No. 17 / 981,380, filed on November 04, 2022, the entire content of which is incorporated herein by reference. Technical Field

[0003] This application generally relates to inspection cameras, and more particularly to a sequence for controlling an inspection camera and an illumination system. Background Art

[0004] Inspection cameras are used in industrial products to assist in detecting defects in manufactured products. For example, if a manufacturer produces metal castings, one or more inspection cameras can be arranged on a production line and / or an assembly line to inspect the produced metal castings or local areas thereof to detect quality control problems. However, during image acquisition, a specific light source may be disadvantageous for imaging products with specific surface characteristics, specific defects, or specific environmental conditions. For example, surface materials or characteristics that affect the light quality of the captured image include reflection characteristics, transparency characteristics, or black / opaque characteristics. For another example, certain types of defects, such as scratches or dirt, may be difficult to detect. In addition, certain environmental conditions, such as complex lighting scenarios, increase the difficulty of product defect detection. Brief Description of the Drawings

[0005] Figure 1 Is a block diagram of an inspection camera system according to an exemplary embodiment.

[0006] Figure 2 Is an example of events sorted by a combination of an initial trigger condition and a timestamp according to an exemplary embodiment.

[0007] Figure 3 And Figure 4 Is a screenshot of a user interface according to an exemplary embodiment.

[0008] Figure 5 Is a flowchart of a method for controlling an illumination device according to an exemplary embodiment.

[0009] Figure 6 Is a block diagram of a mobile device according to an exemplary embodiment.

[0010] Figure 7 Is a block diagram of an exemplary computer system within which executable instructions can cause a machine to implement any one or more of the methods discussed herein. Detailed Description

[0011] By improving the design of the lighting device to enhance the lighting conditions in multiple scenarios, the performance of the detection camera can be optimized. Specifically, unlike a single light source that provides insufficient light when collecting images and cannot obtain images of sufficient quality to determine the presence of surface defects on all surface materials of various components or products, a lighting device with multiple light sources can be used. In addition, a controller can be configured for the lighting device, and the controller can independently control multiple light sources, thereby maximizing the flexibility of the lighting device by flexibly combining lighting modes and sequences, enabling it to provide sufficient light for a large number of different products, components, materials, and environments.

[0012] However, managing multiple different lighting combinations and sequences poses technical challenges. There may be multiple components in a single assembly line or production line, and there may be various potential defects on their surfaces, and there are also various different environmental scenarios, all of which may affect the lighting device's ability to normally generate effective analysis images. In addition, as the number of controllable light sources in the lighting device increases, the technical complexity will increase exponentially.

[0013] In certain application scenarios, especially during high-speed shooting where the component to be detected is in a moving state and multiple images need to be taken, precise timing control and software event sequencing become difficult. For example, if the image processing delay exceeds the time interval between two shootings, the second shooting may be triggered before the first image is fully processed. The machine learning models of the prior art lack an efficient event sequencing function and thus cannot support rapid detection.

[0014] According to an exemplary embodiment, a controller and a computer system for a lighting device are provided to address the above technical challenges. Specifically, the user can define a sequence containing one or more image acquisition configurations through the computer system. Each image acquisition configuration includes: the identification of one or more light sources to be enabled during the image acquisition process, one or more channels for image acquisition, and the exposure time. Each sequence can also define a custom delay time between the image acquisition configurations within the sequence.

[0015] When a trigger signal is received from hardware or software, the controller will execute the stored sequence, thereby triggering the camera to take images based on the stored image acquisition configurations (and delays). Subsequently, artificial intelligence techniques can be applied to the acquired images to identify suspected defects in the components in the images.

[0016] The following describes exemplary systems, methods, techniques, instruction sequences, and computer program products. To understand the various embodiments of the present technical subject matter, a large number of specific details are set forth in the following description for explanation. However, those skilled in the art should understand that the embodiments of the present technical subject matter can be implemented without relying on these specific details. Generally, well-known instruction examples, protocols, structures, and technical details are not elaborated.

[0017] Figure 1 FIG. is a block diagram of a detection camera system 100 according to an exemplary embodiment. The detection camera system 100 may include an illumination device 102, a controller 104, and a computing system 106. The illumination device 102 includes a plurality of different lights, such as light-emitting diode (LED) lights, which can be individually controlled by the controller 104. This means that these lights can be independently turned on or off, so that the illumination device 102 can have some but not all lights on at any given moment. In some embodiments, the brightness of each light can also be independently adjusted, so that in addition to simply turning on or off, each light can also have a custom brightness / darkness grading.

[0018] The illumination device 102 may also include one or more cameras. Each camera can be independently controlled to capture an image when receiving a signal. Variables can also be independently controlled, and these variables may include any software-definable parameters of the imaging system, including but not limited to: the aperture and / or focal length of the lens, the exposure time of the camera, the phase and / or pattern of the diffuser, the wavelength and / or intensity of the light, and so on.

[0019] In an exemplary embodiment, the illumination device 102 is a light dome. The light dome illuminates a target object, such as a metal casting or other product, during use. The light dome includes a housing, in which a plurality of light sources are arranged, which will be described in more detail below. In some examples, these light sources are a plurality of light-emitting diodes (LEDs) or display screens, and their arrangement can be flexibly adjusted to achieve illumination of the target object.

[0020] The above one or more cameras can be mounted on the light dome through a bracket and capture images of the illuminated target object through an opening at the top of the light dome.

[0021] The controller 104 is an electronic component designed to send signals to the illumination device 102 through one or more channels to control multiple lights on the illumination device.

[0022] The computing system 106 includes a variety of software components running on a computing hardware platform. These components include a sequence generation user interface 108. The sequence generation user interface 108 allows a user to create a sequence, which is composed of a plurality of imaging configurations combined in sequence, and a custom delay can be selectively set between the imaging configurations. The created sequence can be stored in the controller 104, such as in the storage component of the controller 104. When receiving an external trigger signal, the controller 104 will call the sequence and execute the sequence operations, which means that the imaging configurations in the sequence will control the illumination device 102 according to the parameters defined in each imaging configuration, and the imaging configurations are separated by a custom delay.

[0023] The external trigger signal can be a hardware trigger signal (e.g., from a programmable logic controller) or a software trigger signal (e.g., from an industrial personal computer). In some exemplary embodiments, one or more trigger signals can be received from the factory computer 109. Once the trigger signal is generated, the sequence is executed, thereby controlling the lighting device 102 to turn on the corresponding lights at the appropriate moment and triggering one or more cameras to capture images at the appropriate time.

[0024] The controller 104 sends the trigger-related information to the image processing component 110 on the computing system 106. When receiving the trigger information, the image processing component 110 uses the system timestamp to record the reception time of the trigger signal. The controller 104 also receives the captured photos from the lighting device 102, adds timestamps to them, and then sends them to the image processing component 110. The image processing component 110 then encodes the photos, acquisition configuration information, timestamps, camera identifiers, and other information into data packets and stores them in the first shared memory 112 in the form of one or more data structures.

[0025] The image analysis component 114 then retrieves and decodes the above data structures from the first shared memory 112. The image acquisition configuration information is used to retrieve the artificial intelligence model corresponding to this acquisition configuration. Each image acquisition configuration has an independent artificial intelligence model, but some acquisition configurations can correspond to multiple models. According to the exemplary embodiments, the artificial intelligence models are not shared among different image acquisition configurations.

[0026] The above artificial intelligence model is used to perform one or more image analysis tasks on the image. For example, these tasks can include: creating a mask for the image to make one or more inferences about one or more defects of the components captured in the image. The artificial intelligence can be achieved by training a machine learning model through machine learning algorithms.

[0027] In the exemplary embodiments, the above machine learning algorithms can be selected from a variety of supervised or unsupervised learning algorithms. Examples of supervised learning algorithms include: artificial neural networks, Bayesian networks, instance-based learning, support vector machines, random forests, linear classifiers, quadratic classifiers, k-nearest neighbor algorithms, decision trees, and hidden Markov models. Examples of unsupervised learning algorithms include: expectation maximization algorithms, vector quantization, and information bottleneck methods.

[0028] The image analysis component 114 encodes the results of the image analysis tasks into another data structure. For example, this data structure can contain an inference mask, network information, defect types, etc. The encoded data structure is stored in the second shared memory 116.

[0029] Then, the central processing component 118 extracts the data structure from the second shared memory 116 and timestamp-sorts the data in the data structure based on the information provided by the programmable logic controller (such as component identification, detection identification, detection ready status, component start / end signals, etc.). The sorted data is packed into data packets and stored in the third shared memory 120. These data packets may contain information such as component identification, detection identification, camera identification, images, inference masks, other post-inference processing results, and error codes.

[0030] Then, the user interface component 122 can access the data packets in the third shared memory 120 and display some content to the user through the graphical user interface. The user can specify the detection mode (such as manual or automatic) and can add customized settings (such as image display parameters, whether and how to upload the images to the cloud environment, etc.).

[0031] It should be noted that in some exemplary embodiments, the user interface component 122 and the user interface 108 can be integrated into a single component. In other words, the sequence definition function of the user interface 108 can be combined with the output and setting functions of the user interface component 122.

[0032] Now looking back at the timestamp sorting operation performed by the central processing component 118, it should be noted that the above-mentioned multiple operations can be executed in parallel. For example, when a new trigger signal is received to capture another image, the detection camera system 100 may still be processing the image captured by the previous trigger. Therefore, external and internal events may continuously arrive and be difficult to track. To effectively manage these events, the central processing component 118 can sort according to the combination of the initial trigger conditions and timestamps of the events. According to one exemplary embodiment, Figure 2 shows an example of events sorted by the combination of the initial trigger conditions and timestamps. In the figure, the "component start" external event 200 with a specific component identification is received at timestamp TS1, indicating that the component start event is received from an external component (such as the factory computer 109) at time 1. Subsequently, the "camera trigger" external event 202 with the same component identification is received at timestamp TS2.0, indicating that the controller 104 triggers the lighting device 102 to start shooting at time 2.0 (according to the sequence being processed by the controller 104).

[0033] Then, the "trigger response" internal event 204 is received at timestamp TS2.1, indicating that the controller returns a response confirming the camera trigger at time 2.1. This event also contains trigger information (such as sequence and imaging configuration) and possible error messages.

[0034] Subsequently, the internal event 206 of "Photo Acquisition and Processing" was received at timestamp TS2.2, indicating that photo shooting and processing were completed at time 2.2. This event contains the photo itself and the AI inference result. Then, the external event 208 of "Camera Trigger Completion" was received at timestamp TS2.3, indicating that the controller 104 confirmed that the camera trigger operation was completed at time 2.3.

[0035] Finally, the external event 210 of "Component Completion" was received at timestamp TS4, indicating that the component completion event was received from an external component (such as the factory computer 109) at time 4.

[0036] It is worth noting that all events 200 - 210 have been sorted. They are all grouped by component identifier, and within the event group corresponding to that component identifier, all events are arranged in chronological order of the timestamps. This sorting logic is not affected by the actual reception order of the events. Due to parallel processing and latency, the actual reception and processing times of some events may not match the times indicated by the timestamps. For example, due to communication or processing latency, event 206 may be received earlier than event 204. The timestamp sorting mechanism can correct such problems and at the same time organize the events based on the component identifier (to handle the interleaved reception of events from different components caused by parallel processing). Events 202 - 208 can be used to obtain the information required to create the data packet 212 for the corresponding component identifier.

[0037] As previously mentioned, the user interface 108 provides a mechanism for the user to define imaging configurations (including lighting combinations). Figure 3 and Figure 4 shows a screenshot of this mechanism. Specifically, first refer to Figure 3 , the user interface 300 displays a screen on which the user can name / select a sequence name 302 here, and can also name / create multiple imaging configurations 304A - 304H. Although not shown in the figure, the user can also specify a custom delay time between one or more of the imaging configurations 304A - 304H. Each sequence contains multiple imaging configurations processed in sequence. Thus, in this example, the imaging configuration 304A is executed before the imaging configuration 304B, and so on.

[0038] Refer to Figure 4, the user interface 300 also displays a screen where a single imaging configuration can be modified. Here, the user can select an imaging configuration through the drop-down menu 402, and its name will be displayed in the field 404. The exposure time can be set through the slider 406. The drop-down menu 408 is used to select an editing mode: when the "ring" editing mode is selected and any light source pattern is clicked, the lights in the entire ring area will be turned on or off; if other editing modes are selected, such as the "single light source" mode, only a single light source will be turned on or off when clicked. As shown in the figure, after the user selects the "ring" editing mode, the LEDs 412A - 412H in the light source layout diagram 410 are shown as the "lit" state, while other LEDs (such as LED 412I) are shown as the "unlit" state.

[0039] The user can also select different individual LEDs or combinations as the lit light sources in the imaging configuration. In the ring mode, the system defaults that when the user selects any LED on the ring, all LEDs on that ring are automatically selected. In this embodiment, there are 5 selectable ring areas, and when the user selects any LED within a certain ring, the entire ring is triggered to be selected. For example, after the user selects LED 412B, all LEDs 412A - 412H within that ring are selected and lit. The user can choose any combination or number of ring areas to be lit simultaneously.

[0040] In addition, in some exemplary embodiments (not shown in the figure), the user interface 300 can integrate the real-time viewfinder image of the corresponding camera. For example, when the user places the component to be tested in the lighting device 102, the user can observe the lighting effect on the component to be tested in real time according to the changes they make to the light combination in the user interface 300.

[0041] The following is the data that can be forwarded for each image:

[0042] {

[0043] "part id": <string>,

[0044] "image_timestamp": <number>,

[0045] "ok": <boolean>,

[0046] "result": <string>,

[0047] "ng_types": [string],

[0048] "sequence": <string>,

[0049] "capture config": <string>,

[0050] "capture_index": <number>,

[0051] "camera_id": <string>,

[0052] "net\:vork_results":{

[0053] (network name):{

[0054] #Defect network only

[0055] "ng_stats":[{

[0056] "name": <string>,

[0057] "color": <string>,

[0058] "ng": <boolean>,

[0059] "result": <string>,

[0060] "count": <number>,

[0061] "segments":[{"

[0062] "ng": <boolean>,IItrue ifis NG

[0063] "result": <string>, one of OK, NG, LIMIT

[0064] "x": <number>,II x coordinate of the centroid of the defect in the original image space

[0065] "y": <number>, IIy coordinate of the centroid of the defect in the original image space

[0066] "vh width": <number>,II vertical / horizontal width of the defect in the original image space

[0067] "vh_height": <number>,II vertical / horizontal height of the defect 111 the original image space

[0068] "mr_width": <number>,II minimum bounding box width of the defect in the original image space

[0069] "mr_length": <number>,II minimum bounding box length of the defect in the original image space

[0070] "area": <number>,II area of the defect in the original image space

[0071] "circumference": <number>II circumference of the defect in the original image space

[0072] }],

[0073] }],

[0074] #Location network only,

[0075] "location_results":[{

[0076] (location nan1e) <string>:{

[0077] "points":

[0078] {

[0079] "x": number,

[0080] }

[0081] }

[0082] }

[0083] }]

[0084] r,

[0085] "y": number,

[0086] ,

[0087] "lines":

[0088] {

[0089] }

[0090] ,

[0091] "pointl": {

[0092] },

[0093] "x": number,

[0094] "y": number,

[0095] "point2": {

[0096] },

[0097] "x": number,

[0098] "y": number,

[0099] "circles":

[0100] {

[0101] }

[0102] "center": {

[0103] },

[0104] "x": number,

[0105] "y": number,

[0106] "radius": number,

[0107] The following data can be forwarded for each component:

[0108] {

[0109] "part id": <string>,

[0110] "part timestamp": <number>,

[0111] "camera_id": <string>,

[0112] "ok": <boolean>,

[0113] "result": <string>,

[0114] "not_enough_image_results": <boolean>,

[0115] "image_results":{

[0116] (image_timestamp):{

[0117] "ok": <boolean>,

[0118] "result": <string>,

[0119] "ng_types": [string],

[0120] "sequence": <string>,

[0121] "capture config": <string>,

[0122] "capture_index": <number>,

[0123] "camera_id": <string>,

[0124] "network_results":{

[0125] (network name):{

[0126] #Defect network only

[0127] "ng_stats":[{

[0128] "name": <string>,

[0129] "color": <string>,

[0130] "ng": <boolean>,

[0131] "result": <string>,

[0132] "count": <number>,

[0133] "segments":[{

[0134] "ng": <boolean>, / / true ifis NG

[0135] "result": <string>, one of OK, NG, LIMIT

[0136] "x": <number>, / / x coordinate ofthe centroid ofthe defect in theoriginal image space

[0137] "y": <number>, / / y coordinate ofthe centroid ofthe defect in theoriginal image space

[0138] "vh_width": <number>, / / vertical / horizontal width ofthe defect in theoriginal image space

[0139] "vh_height": <number>, / / vertical / horizontal height ofthe defect in theoriginal image space

[0140] "mr_width": <number>, / / minimum bounding box width ofthe defect in theoriginal image space

[0141] "mr_length": <number>, / / minimum bounding box length ofthe defect inthe original image space

[0142] "area": <number>, II area of the defect in the original image space

[0143] "circumference": <number> / / circumference of the defect in the original image space

[0144] }

[0145] }],

[0146] }],

[0147] #Location network only,

[0148] "location_results":[{"

[0149] (location name) <string>:{

[0150] "points":

[0151] },

[0152] }]

[0153] {

[0154] }

[0155] ,

[0156] "x": number,

[0157] "y": number,

[0158] "lines":

[0159] {

[0160] }

[0161] ,

[0162] "pointl": {

[0163] },

[0164] "x": number,

[0165] "y": number,

[0166] "point2": {

[0167] L

[0168] "x": number,

[0169] "y": number,

[0170] "circles":

[0171] {

[0172] }

[0173] "center": {

[0174] L

[0175] "x": number,

[0176] "y": number,

[0177] "radius": number,

[0178] }

[0179] }

[0180] }

[0181] }

[0182] The following data can be forwarded for each camera:

[0183] t

[0184] "part_id": <string>,

[0185] "ok": <boolean>,

[0186] "not enough image results": <boolean>,

[0187] "results":[{"

[0188] }]

[0189] "Image_timestamp": <number>,

[0190] "ok": <boolean>,

[0191] "ng_types": [string],

[0192] "sequence": <string>,

[0193] "capture_config": <string>,

[0194] "capture_index": <number>,

[0195] "net\vork": <string>,

[0196] "camera_id": <string>,

[0197] "ng_stats": [{"

[0198] "name": <string>,

[0199] "color": <string>,

[0200] "ng": <boolean>,

[0201] "count": <number>,

[0202] "segments": [{"ng": <boolean> ,

[0203] "x": <nun1ber> ,

[0204] "y": <number>,

[0205] "vh_width": <number>,

[0206] "vh_height": <number>,

[0207] "mr width": <number>,

[0208] "mr_length": <number>,

[0209] "area": <number>,

[0210] "circumference": <number>,

[0211] }],

[0213] [[ID=8}]

[0214] #Location network only.

[0215] "points":{

[0216] (point name) <string>:

[0217] "x": number,

[0218] "y": number,

[0219] }]

[0220] }

[0221] Component - level result example (json)

[0222] {

[0223] "part id": <string>,

[0224] "part timestamp": <number>,

[0225] "ok": <boolean>,

[0226] "result": <string>,

[0227] "not_enough_image_results": <boolean>,

[0228] "image_results":{

[0229] (image_timestamp):{

[0230] "ok": <boolean>,

[0231] "result": <string>,

[0232] "ng_types": [string],

[0233] "sequence": <string>,

[0234] "capture config": <string>,

[0235] "capture_index": <number>,

[0236] "camera_id": <string>,

[0237] "network_results":{

[0238] (network name):{

[0239] #Defect network only

[0240] "ng_stats":[{

[0241] "name": <string>,

[0242] "color": <string>,

[0243] "ng": <boolean>,

[0244] "result": <string>,

[0245] "count": <number>,

[0246] "segments": [{"

[0247] "ng": <boolean>, / / true ifis NG

[0248] "result": <string>, one of OK, NG, LIMIT

[0249] "x": <number>,II x coordinate of the centroid of the defect in the original image space

[0250] "y": <number>, IIy coordinate of the centroid of the defect in the original image space

[0251] "vh_width": <number>,II vertical / horizontal width of the defect in the original image space

[0252] "vh_height": <number>,II vertical / horizontal height of the defect in the original image space

[0253] "mr_width": <number>,II minimum bounding box width of the defect in the original image space

[0254] "mr_length": <number>,II minimum bounding box length of the defect in the original image space

[0255] "area": <number>,II area of the defect in the original image space

[0256] "circumference": <number>II circumference of the defect in the original image space

[0257] }],

[0258] }],

[0259] #Location net\vork only,

[0260] "location_results":[{

[0261] (location name) <string>:{

[0262] "points":

[0263] {

[0264] }

[0265] ,

[0266] "x": number,

[0267] "y": number,

[0268] "lines":

[0269] {

[0270] "pointl": {

[0271] },

[0272] "x": number,

[0273] "y": number,

[0274] "point2": {

[0275] },

[0276] "x": number,

[0277] "y": number,

[0278] }

[0279] L

[0280] "circles":

[0281] }

[0282] }

[0283] }

[0284] }

[0285] }

[0287] },

[0288] }]

[0289] {

[0290] "center": {

[0291] },

[0292] "x": number,

[0293] "y": number,​

[0294] "radius": number,

[0295] Example of image-level results (csv)

[0296] time, part-id, auto_ok_ng, human_ok_ng, image_count_mismatch, ng_types

[0297] 07 / 18 / 2021 - 00:01:58, 0000000000000000001223Y178P223ll2000000000000, OK,

[0298] OK,

[0299] False,

[0300] 07 / 18 / 2021 - 00:02:03, 0000000000000000001223Y178P223ll1000000000000, NG,

[0301] NG,

[0302] False, Defect Type 1Defect Type 2

[0303] Figure 5 is a flowchart showing a method 500 for controlling a lighting device according to an exemplary embodiment. In one exemplary embodiment, the method may be executed by a computer system coupled to a controller connected to the lighting device.

[0304] In operation 502, a sequence is accessed. The sequence includes one or more imaging configurations, each of which defines an exposure time and an indication of which independently controllable light sources need to be lit during the image acquisition process. The sequence can be defined through the user interface of the computer system or obtained from other devices. The sequence can be organized according to the trigger order of the imaging configurations, and when transitioning from one imaging configuration to another, a custom delay time can also be selectively specified for each transition.

[0305] In operation 504, the sequence is stored in a memory accessible by the controller. In an exemplary embodiment, the memory is an integral part of the controller itself.

[0306] In operation 506, an image acquired by the controller when executing a certain imaging configuration in the sequence is received. When the lighting device lights some of the multiple independently controllable light sources according to one or more imaging configurations, at least one camera of the lighting device will capture an image.

[0307] At operation 508, one or more events related to performing an imaging configuration are received, each event including a component identifier and a timestamp. It should be noted that operation 508 can occur at any stage of method 500.

[0308] At operation 510, an artificial intelligence model corresponding to one of the multiple imaging configurations is retrieved. At operation 512, an image is input into the retrieved artificial intelligence model to generate one or more inferences about the image, and these inferences are related to potential defects in the components captured in the image.

[0309] At operation 514, the events are grouped by component identifier. At operation 516, the events are sorted by timestamp within each group.

[0310] At operation 518, the image and its inferences are encapsulated into a data packet.

[0311] At operation 520, based on the inference results in the data packet, an alert about component defects is sent to the user through a user interface.

[0312] Figure 6 FIG. 600 is a block diagram showing a software architecture 602 that can be installed in the above device. Figure 6 For non-limiting examples only, the actual architecture can implement the functions described herein. The software architecture 602 is implemented by the hardware of the machine 700 (including a processor 710, a memory 730, and I / O components 750) shown. This architecture can be regarded as a layer stack, including an operating system 604, libraries 606, frameworks 608, and applications 610. At runtime, the application 610 calls the API 612 through the software stack and receives a response message 614. Figure 7 The operating system 604 manages hardware resources and provides common services, including a kernel 620 (responsible for hardware abstraction layer functions such as memory management and processor scheduling), services 622 (system services), and drivers 624 (controlling hardware interfaces such as display drivers, camera drivers,

[0313] or or low-power consumption drivers, flash drivers, serial communication drivers (such as USB drivers), drivers, audio drivers, power management drivers, etc.).

[0314] In some embodiments, the library 606 provides underlying general infrastructure for the application 610. The library 606 may include a system library 630 (such as the C standard library), which provides functions such as memory allocation functions, string operation functions, and mathematical functions. In addition, the library 606 may also include an API library 632, such as: a media library (supporting the rendering and operation of various media formats, such as MPEG4, H.264 / AVC, MP3, AAC, AMR audio codec, JPEG / JPG, PNG, etc.); a graphics library (such as the OpenGL framework for rendering 2D / 3D in the graphics context of a display); a database library (such as SQLite that provides relational database functions); a web library (such as WebKit that provides web browsing functions). The library 606 may also include a variety of other libraries 634 to provide more APIs to the application 610.

[0315] The framework 608 provides high-level general infrastructure available for the application 610. For example, the framework 608 provides graphical user interface functions, advanced resource management, advanced location services, etc. The framework 608 may provide a variety of other APIs that the application 610 can call, and some of these APIs may be specific to a certain operating system 604 or platform.

[0316] In an exemplary embodiment, the application 610 includes a home screen application 650, a contacts application 652, a browser application 654, an e-book reader application 656, a location application 658, a media application 660, a messaging application 662, a game application 664, and various other applications such as a third-party application 666. The application 610 is a program that executes predefined functions and can be built using an object-oriented programming language (such as Objective-C, Java, or C++) or a procedural programming language (such as C or assembly language). In a specific example, the third-party application 666 (such as an application developed by a non-platform vendor through the Android TM or iOS TM SDK) can be mobile software running on a mobile operating system (such as iOS TM 、Android TM 、 Phone, etc.). In this case, the third-party application 666 can call the API calls 612 provided by the operating system 604 to implement the functions described herein.

[0317] Figure 7 A schematic diagram of a computer system of a machine 700 is shown, which executes instructions 716 to implement the methods described herein. Specifically, Figure 7 Taking the computer system as an example, the structure of the machine 700 is shown, where the instructions 716 (such as software, programs, applications, applets, or other executable code) cause the machine 700 to execute the described methods. For example, the instructions 716 may cause the machine 700 to execute Figure 5 The method, or implementation Figures 1-5 The function shown. Instruction 716 converts the general-purpose non-programmed machine 700 into a programmed machine 700 that performs a specific function. In other embodiments, the machine 700 can operate as a stand-alone device or be coupled to other machines through a network. In a network deployment, the machine 700 can act as a server or a client in a server-client network, or as a peer machine in a peer-to-peer (or distributed) network. The machine 700 includes, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smart phones, mobile devices, wearable devices (such as smart watches), smart home devices (such as smart appliances), other smart devices, network devices (routers / switches / bridges), or any machine that can execute the instruction 716 to perform the specified operations. Although only a single machine 700 is shown, the term "machine" also includes a collection of machines that jointly execute the instruction 716 to implement the method described above.

[0318] The machine 700 may include a processor 710, a memory 730, and an input / output (I / O) component 750 interconnected by a bus 702. In an exemplary embodiment, the processor 710 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), other processors, or a combination thereof) may include a processor 712 and a processor 714 that execute the instruction 716. The term "processor" includes a multi-core processor (i.e., a processor that includes two or more independent processor cores that can concurrently execute the instruction 716). Although Figure 7 Multiple processors 710 are shown, the machine 700 may also include: a single-core single-processor 712, a multi-core single-processor 712, multiple single-core processors 712 / 714, multiple multi-core processors 712 / 714, or any combination thereof.

[0319] The memory 730 may include a main memory 732, a static memory 734, and a storage unit 736, all accessible to the processor 710 through the bus 702. The main memory 732, the static memory 734, and the storage unit 736 store the instruction 716 that implements the method or function described herein. The instruction 716 may reside entirely or partially in the main memory 732, the static memory 734, the storage unit 736, inside the processor 710 (such as a processor cache), or a combination thereof during execution.

[0320] The I / O component 750 includes a variety of components for receiving input, providing output, transmitting information, or capturing measurements. The specific type of I / O component 750 depends on the type of machine: for example, a portable device such as a mobile phone may include a touch input device, while a headless server may not include such a device. It should be noted that Figure 7 All components of the I / O component 750 are not shown. For simplicity of description, the I / O component 750 is grouped by function (this grouping method is non-limiting) and includes an output component 752 and an input component 754: Output component 752: Visual components (such as plasma display panel (PDP), light-emitting diode (LED) display, liquid crystal display (LCD), projector, cathode ray tube (CRT)); Acoustic components (such as speakers); Haptic components (such as vibration motors, resistance mechanisms); Other signal generators, etc. Input component 754: Character input components (such as keyboards, touchscreens that support character input, optical keyboards); Pointing input components (such as mice, touchpads, trackballs, joysticks, motion sensors); Tactile input components (such as physical buttons, touchscreens that provide touch position / force); Audio input components (such as microphones), etc.

[0321] In other embodiments, the I / O component 750 may further include: Biometric component 756: Detect gestures / facial expressions / voice / body postures / eye movements; Measure biometric signals (blood pressure, heart rate, body temperature, sweat, brain waves); Identity recognition (voiceprint, retina, face, fingerprint, electroencephalogram recognition), etc. Motion component 758: Acceleration sensors (such as accelerometers), gravity sensors, rotational sensors (such as gyroscopes), etc. Environmental component 760: Light sensors (such as photometers), temperature sensors (detect ambient temperature), humidity sensors, pressure sensors (such as barometers), acoustic sensors (microphones that detect background noise), proximity sensors (such as infrared sensors), gas sensors (detect the concentration of hazardous gases or atmospheric pollutants), etc. Location component 762: Position sensors (such as GPS receivers), altitude sensors (barometers / altimeters that calculate altitude through air pressure), direction sensors (such as magnetometers), etc.

[0322] Communication can be achieved through a variety of technologies. The I / O component 750 includes a communication component 764, which can connect the machine 700 to the network 780 or the device 770 through the couplings 782 and 772 respectively. For example, the communication component 764 may include a network interface component or other devices adapted to the network 780. In a further example, the communication component 764 may include: wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, components (such as low power consumption), components and other communication modality components. The device 770 can be other machines or a variety of peripheral devices (such as coupled through USB).

[0323] In addition, the communication component 764 can detect an identifier or include an identifier detection component. For example: a radio frequency identification (RFID) tag reading component, an NFC smart tag detection component, an optical reading component (such as an optical sensor for detecting one-dimensional barcodes (such as UPC codes), two-dimensional barcodes (such as QR codes, Aztec codes, Data Matrix, Dataglyph, Maxi Code, PDF417, Ultra Code, UCC RSS-2D codes)), an acoustic detection component (such as a microphone for identifying a marked audio signal). Through the communication component 764, various information can also be obtained, such as: geolocation based on the IP protocol, positioning based on signal triangulation, a specific location detected through an NFC beacon signal, etc.

[0324] The class of memories (i.e., the memories of 730 / 732 / 734 and / or the processor 710) and the storage unit 736 can store one or more instruction sets 716 and data structures (such as software) that embody or are used to implement the methods herein. When these instructions (such as instruction 716) are executed by the processor 710, the disclosed embodiments are implemented.

[0325] In this document, "machine storage medium", "device storage medium", and "computer storage medium" are synonymous terms and can be used interchangeably. It refers to a single or multiple storage devices and / or media (such as a centralized / distributed database and associated caches and servers) that store executable instructions and / or data, specifically including but not limited to: solid-state memory, magneto-optical media (including internal and external memories of the processor). Specific examples of machine storage medium / computer storage medium / device storage medium include: non-volatile memory (such as semiconductor memories: EPROM, EEPROM, FPGA, flash devices), magnetic disks (internal hard disks, removable disks), magneto-optical disks, CD-ROM / DVD-ROM optical disks. The above terms explicitly exclude carrier waves, modulated data signals, and other media belonging to the "signal medium" category below.

[0326] The network 780 or a part thereof can be: an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), the Internet or a part thereof, a public switched telephone network (PSTN) or a part thereof, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, or a combination of the aforementioned networks. For example, network 780 or a portion thereof may be a wireless / cellular network, and coupling 782 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other cellular / wireless connection. At this time, coupling 782 may implement a variety of data transmission technologies, such as: Single-Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data Rate for GSM Evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP) technology (including 8G), Fourth Generation Wireless Network (4G), Universal Mobile Telecommunications System (UMTS), High-Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, protocols defined by other standard organizations, remote protocols, and other data transmission technologies.

[0327] Instruction 716 may be sent or received over network 880 via a network interface device (such as the network interface component included in communication component 764) using a transmission medium and utilizing a variety of well-known transmission protocols (such as the Hypertext Transfer Protocol HTTP). Similarly, instruction 716 may also be sent or received to device 770 via coupling 772 (such as a point-to-point coupling) using a transmission medium. In the present disclosure, "transmission medium" and "signal medium" have the same meaning and are used interchangeably. "Transmission medium" and "signal medium" should be understood to include any non-transitory medium capable of storing, encoding, or carrying instruction 716 executed by machine 700, including digital or analog communication signals or other non-transitory media that facilitate such software communication. Thus, "transmission medium" and "signal medium" should be understood to include terms such as any form of modulated data signal, carrier wave, etc. A "modulated data signal" refers to a signal generated by setting or changing one or more of its characteristics in a manner that encodes signal information.

[0328] In the present disclosure, "machine-readable medium", "computer-readable medium", and "device-readable medium" have the same meaning and are used interchangeably. These terms are defined to include both machine storage media and transmission media. Thus, the terms include both storage devices / media and carrier waves / modulated data signals.< / string> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / string> < / boolean> < / number> < / string> < / boolean> < / string> < / string> < / string> < / number> < / string> < / string> < / string> < / boolean> < / boolean> < / string> < / boolean> < / number> < / string> < / string> ​< / number> < / number> < / number> < / number> < / number> < / number> < / number> < / nun1ber> < / boolean> < / number> < / boolean> < / string> < / string> < / string> < / string> < / number> < / string> < / string> < / boolean> < / number> < / boolean> < / boolean> < / string> < / string> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / string> < / boolean> < / number> < / string> < / boolean> < / string> < / string> < / string> < / number> < / string> < / string> < / string> < / boolean> < / boolean> < / string> < / boolean> < / string> < / number> < / string> < / string> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / number> < / string> < / boolean> < / number> < / string> < / boolean> < / string> < / string> < / string> < / number> < / string> < / string> < / string> < / boolean> < / number> < / string>

Claims

1. A system, comprising: A lighting device, which includes a plurality of independently controllable light sources and at least one camera; A controller; And A computer system, which includes at least one hardware processor and a non-transitory computer-readable medium storing instructions, the non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform the following operations: Access a sequence containing one or more imaging configurations, each of the one or more imaging configurations defining an exposure time and an indication of which independently controllable light sources need to be lit during image acquisition; Store the sequence in a memory accessible by the controller; Receive, from the controller, an image acquired during the execution of one or more of the imaging configurations in the sequence, the image being acquired by at least one camera of the lighting device when the lighting device lights a portion of the independently controllable light sources according to the one or more imaging configurations; Access an artificial intelligence model corresponding to the imaging configuration; Input the image into the retrieved artificial intelligence model to generate one or more inferences about the image, the inferences being related to potential defects in the components captured in the image; And Package the image and the one or more inferences into a data packet.

2. The system according to claim 1, wherein The operations further include: Based on one or more inferences in the data packet, issue an alert to the user through a user interface that there are defects in the components.

3. The system according to claim 1, wherein The operations further include: Receive one or more events related to the execution of the one or more imaging configurations, each event containing a component identifier and a timestamp; Group the events by component identifier; and Sort the events within each group based on the timestamp.

4. The system according to claim 3, wherein The packaging uses the sorted events in the group corresponding to the component.

5. The system according to claim 1, wherein Each of the imaging configurations has at least one corresponding artificial intelligence model that is not shared with other imaging configurations.

6. The system according to claim 3, wherein The operations further include: Encode the image and the imaging configuration into a first data structure; and Store the first data structure in a first shared memory shared by an image processing component that performs image reception and an image analysis component that performs retrieval of the artificial intelligence model and image transfer.

7. The system according to claim 6, wherein The operations further include: Encode the one or more inferences into a second data structure; and Store the second data structure in a second shared memory shared by the image analysis component and a central processing component that performs grouping and sorting.

8. The system according to claim 7, wherein The operations further include: Store the data packet in a third shared memory shared by the central processing component and a user interface that can display component defects based on the one or more inferences.

9. A method, comprising: Access a sequence containing one or more imaging configurations, each of the one or more imaging configurations defining an exposure time and an indication of which independently controllable light sources need to be lit during image acquisition; Store the sequence in a memory accessible by the controller; Receive, from a controller, an image acquired during execution of one or more imaging configurations in the sequence, the image being acquired by at least one camera of the illumination device while the illumination device illuminates portions of independently controllable light sources according to the one or more imaging configurations; Access an artificial intelligence model corresponding to the imaging configuration; Input the image into the retrieved artificial intelligence model to generate one or more inferences about the image, the inferences being related to potential defects in components captured in the image; And Encapsulate the image and the one or more inferences into a data packet.

10. The method according to claim 9, further comprising: Based on one or more inferences in the data packet, issue an alert to a user through a user interface that a defect exists in the component.

11. The method according to claim 9, further comprising: Receive one or more events related to execution of the one or more imaging configurations, each event including a component identifier and a timestamp; Group the events by component identifier; And Sort the events within each group based on the timestamp.

12. The method according to claim 11, wherein The encapsulation uses the sorted events in the group corresponding to the component.

13. The method according to claim 9, wherein Each of the imaging configurations has at least one corresponding artificial intelligence model that is not shared with other imaging configurations.

14. The method according to claim 11, further comprising: Encode the image and the imaging configuration into a first data structure; And Store the first data structure in a first shared memory shared by an image processing component that receives the image and an image analysis component that retrieves the artificial intelligence model and transfers the image.

15. The method according to claim 14, further comprising: Encode the one or more inferences into a second data structure; And Store the second data structure in a second shared memory shared by the image analysis component and a central processing component that performs grouping and sorting.

16. The method according to claim 15, further comprising: Store the data packet in a third shared memory shared by the central processing component and a user interface that can display component defects based on the one or more inferences.

17. A non-transitory computer-readable medium that contains instructions executable by one or more machines to perform the following operations: Access a sequence that includes one or more imaging configurations, each of the one or more imaging configurations defining an exposure time and an indication of which independently controllable light sources need to be illuminated during image acquisition; Store the sequence in a memory accessible by a controller; Receive, from a controller, an image acquired during execution of one or more imaging configurations in the sequence, the image being acquired by at least one camera of the illumination device while the illumination device illuminates portions of independently controllable light sources according to the one or more imaging configurations; Access an artificial intelligence model corresponding to the imaging configuration; Input the image into the retrieved artificial intelligence model to generate one or more inferences about the image, the inferences being related to potential defects in components captured in the image; And Encapsulate the image and the one or more inferences into a data packet.

18. The non-transitory computer-readable medium according to claim 17, wherein The operation further includes: Based on one or more inferences in the data packet, issuing an alert to the user through the user interface that the component is defective.

19. The non-transitory machine-readable storage medium according to claim 17, wherein The operation further includes: Receiving one or more events related to the execution of the one or more imaging configurations, each event including a component identifier and a timestamp; Grouping the events by component identifier; and Sorting the events within each group based on the timestamp.

20. The non-transitory machine-readable storage medium according to claim 19, wherein The encapsulation uses the sorted events in the group corresponding to the component.

Images