Real-time visualization of state managed asynchronous runtime

Abstract

To provide a system, a device, and a method for configuring hardware resources to process a plurality of image processing cycles in a plurality of parallel pipelines.SOLUTION: A method for processing image data by an electronic system includes creating each image processing cycle of a plurality of implemented image processing cycles associated with a temporal sequence of triggers in response to one or more respective trigger events, acquiring a plurality of input / output (I / O) signals of the plurality of image processing cycles, and generating a plurality of cycle status signals. Each cycle status signal is associated with a sequence of time stamps having a temporal resolution higher than a predetermined resolution and indicates the progress of each image processing cycle based on the I / O signals. The method also includes visualizing the plurality of cycle status signals concurrently with respect to at least a first temporal axis on a user interface dynamically and in real time while executing the image processing cycles.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] (Related Applications) This application is a continuation-in-part of and claims priority to U.S. patent application Ser. No. 18 / 217,417, entitled "State Management Asynchronous Runtime," filed Jun. 20, 2023, which is incorporated herein by reference in its entirety.

[0002] The disclosed embodiments relate generally to electronic circuits, and more particularly to systems, apparatus, and methods for configuring hardware resources to process multiple image processing cycles in multiple parallel pipelines. [Background technology]

[0003] Computer processors implement parallelism, pipelining, software task overlap, and dedicated hardware acceleration to enhance their processing capabilities. These processing-enhancing techniques are often limited to highly repeatable cycles that perform functionally identical or similar jobs with little or no variation from cycle to cycle. Computer processors are configured using a register model, consistently applying a single configuration to drive runtime execution. For example, processor pipelines often have a fixed number of stages. Multiple stages are staggered, with stages in a current cycle often executing simultaneously with distinct stages in a previous cycle. It is beneficial to develop flexible and highly efficient parallel processing mechanisms while fully utilizing existing computer resources. Summary of the Invention

[0004] Various embodiments of the present application are directed to a core high-performance runtime engine configured to process image data in parallel pipelines for heavy-duty industrial machine vision applications. Machine vision applications are applied to electronic devices (e.g., imaging devices including one or more cameras) and perform functions including, but not limited to, image processing, code reading, product information extraction, and print quality verification. Specifically, a state-managed asynchronous pipelined architecture is applied to the image processing electronics of these machine vision applications. Multiple image processing cycles are executed in parallel in multiple pipelines, leveraging hardware capabilities without bottlenecks. Each image processing cycle includes multiple time-critical stages and operations that are arbitrarily triggered and executed alternately in multiple threads. In some embodiments, a trigger event is received while a previous image processing cycle is still in progress and corresponds to an entirely different image processing cycle assigned to a different pipeline that is executed independently of the previous image processing cycle. In some prior art implementations, a stage of an image processing job uses available hardware capacity and may cause other image processing jobs or other stages of the same image processing job to wait in a queue. Conversely, in this application, a state-managed, asynchronous pipeline architecture leverages available hardware capacity and delivers performance gains by executing multiple image processing cycles in independent parallel pipelines, with the performance gains scaling linearly with the number of processor cores in the imaging device.

[0005] In one aspect, an image processing method is implemented in an electronic device (e.g., an imaging device) having a memory and one or more processors. The method includes identifying a plurality of image processing cycles associated with a temporal sequence of triggers. Each image processing cycle is created in response to one or more respective trigger events. The method further includes assigning the plurality of image processing cycles to a plurality of parallel pipelines (e.g., each image processing cycle is assigned to a respective parallel pipeline) and, for each parallel pipeline, retrieving a respective cycle data container from a cycle data pool. The method further includes processing a first image processing cycle in a first parallel pipeline independently of the processing of the remaining image processing cycles in each remaining parallel pipeline to generate first report data. The method further includes providing the first report data to a client device, for example, for further processing (e.g., storage, classification, analysis, and / or visualization).

[0006] In some embodiments, the cycle data pool includes a predetermined number of cycle data containers before identifying the multiple image processing cycles. Once assigned to a respective image processing cycle, each parallel pipeline directly uses one of the predetermined number of cycle data containers without waiting for the creation of a new cycle data container. In some circumstances, the predetermined number of cycle data containers are fully used by different pipelines, and no cycle data containers can be pulled for a new pipeline. In some embodiments, processing the first image processing cycle further includes inputting image data associated with one or more respective trigger events into the first cycle data container, generating first cycle data including first report data, and storing the first cycle data in the first cycle data container.

[0007] In some embodiments, the imaging device is communicatively coupled to a client device (e.g., a laptop computer, desktop computer, or mobile phone) via a wired or wireless communications link. A user interface for the machine vision application enables the client device to present report data collected from the imaging device in a user-friendly manner. In some embodiments, the client device is located remotely from the imaging device and is coupled to the imaging device via a server. A web-based user interface enables the client device to remotely review report data reported by the imaging device via the server.

[0008] Some implementations of this application target performance optimization and analysis of complex, multi-process applications, particularly those found in sectors such as factory automation and quality control. Some implementations focus on systems requiring high-precision event logging and interactive tuning, such as machine vision systems, barcode reading, and label verification systems. Advanced systems are provided for real-time measurement, data integration, and interactive analysis of various event types, including time, digital, analog, and image data. Some implementations incorporate the use of intuitive visual interfaces and potential machine learning techniques for real-time system parameter adjustment to improve system performance and reliability.

[0009] More specifically, in another aspect, a method is implemented in an electronic system having a memory and one or more processors for tracking image processing cycles in real time. The method includes performing a plurality of image processing cycles associated with a temporal sequence of triggers, each image processing cycle being generated in response to one or more respective trigger events. The method further includes acquiring a plurality of input / output (I / O) signals for the plurality of image processing cycles and generating a plurality of cycle status signals based on the I / O signals. Each cycle status signal is associated with a sequence of timestamps indicating progress of the respective image processing cycle. The sequence of timestamps has a temporal resolution greater than a predetermined resolution. The method further includes dynamically and simultaneously visualizing the plurality of cycle status signals in real time with respect to at least a first time axis in a user interface while the image processing cycles are being performed.

[0010] According to some embodiments, an electronic system includes one or more processors, a memory, and one or more programs stored in the memory, the programs configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described herein.

[0011] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by an electronic device having one or more processors and a memory, the one or more programs including instructions for performing any of the methods described herein.

[0012] Thus, a method, system and apparatus are disclosed that enables optimal design, execution and performance of a barcode scanner.

[0013] The various embodiments described above can be combined with other embodiments described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, particularly upon consideration of the drawings, specification, and claims. Furthermore, it should be noted that the language used herein has been selected primarily for ease of reading and explanation, and may not have been selected to define or encompass the entire inventive subject matter. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a perspective view of an electronic device (eg, a scanning device) according to some embodiments. [Figure 2] FIG. 2 is a block diagram of an exemplary electronic device, according to some embodiments. [Figure 3] FIG. 3 is an example of an image processing environment including an electronic device that processes image data using parallel pipelines, according to some embodiments. [Figure 4] FIG. 4 illustrates a process for processing image data in multiple parallel pipelines according to some embodiments. [Figure 5] FIG. 5 illustrates another example of a process for processing image data in multiple parallel pipelines, according to some embodiments. [Figure 6] FIG. 6 illustrates another example of a process for processing image data in multiple parallel pipelines, according to some embodiments. [Figure 7] FIG. 7 is a diagram illustrating multiple image processing cycles performed in multiple parallel pipelines, according to some embodiments. [Figure 8] FIG. 8 is a flow diagram of a method for processing image data, according to some embodiments. [Figure 9] FIG. 9 is a diagram of a data structure for signals and data applied in an image processing environment, according to some embodiments. [Figure 10]FIG. 10 is a timing diagram of an image processing cycle performed in one of multiple parallel pipelines, according to some embodiments. [Figure 11A] FIG. 11A is an exemplary time diagram of multiple image processing cycles, according to some embodiments. [Figure 11B] FIG. 11B is an exemplary time diagram of multiple image processing cycles, according to some embodiments. [Figure 11C] FIG. 11C is an exemplary time diagram of multiple image processing cycles, according to some embodiments. [Figure 11D] FIG. 11D is an exemplary time diagram of multiple image processing cycles, according to some embodiments. [Figure 12] FIG. 12 is an example of a collapsed graphical user interface (GUI) visualizing signals for multiple imaging cycles 404 over a duration of time, according to some embodiments. [Figure 13] FIG. 13 is another exemplary user interface 1300 for visualizing signals from multiple imaging cycles, according to some embodiments. [Figure 14A] FIG. 14A is a portion of an exemplary user interface visualizing cycle status signals and I / O signals for three image processing cycles, according to some embodiments. [Figure 14B] FIG. 14B is a portion of another example user interface displaying a histogram of capture times for multiple imaging cycles, according to some embodiments. [Figure 15] 15 is a flow diagram of a method for tracking image processing cycles in real time, according to some embodiments. Reference is made to the embodiments illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. DETAILED DESCRIPTION OF THE INVENTION

[0015] Various embodiments of the present application are directed to a core high-performance runtime engine configured to process image data in parallel pipelines for heavy-duty industrial machine vision applications. A state-managed, asynchronous pipeline architecture is applied to an electronic device (e.g., an imaging device) for image processing in these machine vision applications. Specifically, the electronic device identifies multiple image processing cycles associated with a temporal sequence of triggers, with each image processing cycle being created in response to one or more respective trigger events (e.g., image acquisition). The electronic device assigns the multiple image processing cycles to multiple parallel pipelines. For each parallel pipeline, an existing cycle data container is drawn from a cycle data pool. A first image processing cycle is processed in the first parallel pipeline to generate first report data independently of the remaining image processing cycles in each remaining parallel pipeline. The first report data is provided to a client device. In some embodiments, each image processing cycle includes multiple time-critical operations and stages that are optionally triggered and executed alternately in multiple threads of a corresponding pipeline. Such a state-managed, asynchronous pipeline architecture leverages available hardware capacity and independently executes multiple image processing cycles in parallel pipelines, resulting in significant performance gains that scale linearly with the number of processor cores (e.g., CPU, FPGA, NPU, and / or GPU) in the electronic device.

[0016] The primary goal of the state-managed, asynchronous pipeline architecture is to fully utilize hardware resources, including but not limited to processors, acquisition sources, input / output, communication channels, processing threads, and processes, to achieve maximum performance for the image processing system. Furthermore, this architecture significantly simplifies user intervention for runtime control, configuration, and monitoring, improving the user experience. In some embodiments, the processing pipeline and associated hardware and software settings may be modified at any time without generating significant overhead. In some embodiments, deep instrumentation enables self-analysis and tuning of the architecture automatically, enabling improvements in image processing performance with little or no user intervention.

[0017] Some implementations of this application target performance optimization and analysis of complex, multi-process applications, particularly those found in sectors such as factory automation and quality control. Some implementations focus on systems requiring high-precision event logging and interactive tuning, such as machine vision systems, barcode reading, and label verification systems. Advanced systems are provided for real-time measurement, data integration, and interactive analysis of various event types, including time, digital, analog, and image data. Some implementations incorporate the use of intuitive visual interfaces and potential machine learning techniques for real-time system parameter adjustment to improve system performance and reliability.

[0018] FIG. 1 is a perspective view of an electronic device 100 (e.g., a scanning device) according to some embodiments. In some embodiments, the electronic device 100 may also be referred to as a code reader, barcode scanner, label scanner, optical scanner, or image capturing system. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). The electronic device 100 includes a housing 110 (e.g., a body or exterior case) for protecting components located inside the electronic device 100. In some embodiments, the housing 110 includes integrated fittings or brackets for holding the internal components in place. In some embodiments, the electronic device 100 includes a top cover 102 located on an upper side of the electronic device 100. In some embodiments, the top cover 102 is transparent or partially transparent.

[0019] In some embodiments, the electronic device 100 includes one or more distance sensors 104 located within the electronic device 100 (e.g., internal distance sensors). For example, referring to FIG. 1 , the distance sensors 104 are located inside the electronic device 100 (e.g., adjacent the top cover 102) and face the front end of the electronic device 100. In some embodiments, the distance sensors 104 are included in an aiming module. The distance sensors 104 project a beam of light onto a target when the target is placed on the electronic device 100 to assist in visual alignment of the target. This helps align the camera with the imaging plane or center point of the field of view. In some embodiments, each distance sensor 104 is a time-of-flight (TOF) sensor, an ultrasonic sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, or an infrared (IR) distance sensor. In some embodiments, the electronic device 100 includes two or more distance sensors 104, each of the same type (e.g., each of the two or more distance sensors is a TOF sensor). In some embodiments, electronic device 100 includes two or more distance sensors, at least two of which are of distinguishable types (e.g., electronic device 100 includes a TOF distance sensor and a radar sensor). In some embodiments, electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) the presence or absence of an object within a sensing area. The sensing area is designed for the proximity sensors to operate. In some embodiments, electronic device 100 determines the distance between a target and electronic device 100 using distance measurement techniques, such as an image focus finder, an analog-to-digital conversion (ADC) circuit, and / or a digital-to-analog conversion (DAC) circuit.

[0020] More specifically, in some embodiments, the distance sensor 104 is a TOF sensor. A TOF sensor measures the time elapsed between when a signal (e.g., a wave pulse, an LED pulse, a laser pulse, or infrared light) emitted from the sensor reflects off an object and returns to the sensor. Distance is then calculated using the speed of light in air and the time between the transmission and reception of the signal. In some embodiments, the distance sensor 104 is an ultrasonic sensor. An ultrasonic sensor or sonar sensor detects the distance to an object by emitting high-frequency sound waves. The ultrasonic sensor emits high-frequency sound waves toward the object and a timer is started. The object reflects the sound waves back toward the sensor. A receiver picks up the reflected waves and stops the timer. The time it takes for the sound waves to return is calculated against the speed of sound to determine the distance traveled. In some embodiments, the distance sensor 104 is a radar sensor. A radar sensor (e.g., a radar distance sensor) calculates the distance to an object by transmitting high-frequency radio waves (e.g., microwaves) and measuring the reflection of the waves from the object. In some embodiments, the radar sensor is configured to determine the distance, angle, and radial velocity of an object relative to the position of the electronic device 100. In some embodiments, the distance sensor 104 is a LiDAR sensor, which measures the range of an object through light waves from a laser (e.g., instead of radio or sound waves). In some embodiments, the distance sensor 104 is an infrared (IR) distance sensor. An IR distance sensor operates on the principle of triangulation and measures distance based on the angle of a reflected beam.

[0021] In some embodiments, electronic device 100 further includes a plurality of light sources 106 (e.g., eight light-emitting diodes (LEDs) in FIG. 1 ) mounted on a printed circuit board (PCB) 108. Light source 106 is also referred to as a lighting source, illumination source, or illuminator. In some embodiments, light source 106 is part of an illumination system of electronic device 100, which also includes illuminators (e.g., bright field illuminators and dark field illuminators), a reflector, and a lighting module. Details of the illumination system are described in U.S. Patent Application No. 14 / 298,659, entitled "Combination Dark Field and Bright Field Illuminator," filed June 6, 2014 (now U.S. Patent No. 8,989,569, issued March 24, 2015, which is incorporated herein by reference in its entirety).

[0022] In some embodiments, the light sources 106 have one or more lighting types. Examples of lighting types include, but are not limited to, LED light sources, laser light sources, and liquid crystal display (LCD) lights. Each lighting type has its own lighting characteristics, such as color (e.g., blue, red, or green) and / or intensity. The light sources 106 are mounted (e.g., soldered) to a PCB 108 located within the electronic device 100 (e.g., behind the top cover 102). The PCB 108 includes a front surface that faces the top cover 102 of the electronic device 100. In some embodiments, the light sources mounted to the front surface of the PCB 108 include both long-distance light sources and low-angle light sources.

[0023] In some embodiments, the electronic device 100 includes a camera 112. The lens of the camera 112 is exposed through an opening in the PCB 108 and is physically surrounded by light sources 106. The light sources 106 are grouped into multiple illumination units (e.g., a first illumination unit and a second illumination unit). Each illumination unit is configured to be independently controlled to illuminate a different region of the field of view of the camera 112. In one example, two light sources 106 near the corners of the top cover 102 are grouped together to form an illumination unit. The four illumination units are independently controlled to illuminate respective regions of the field of view of the camera 112, either sequentially or simultaneously.

[0024] In some embodiments, electronic device 100 further includes one or more indicators 114. Each indicator 114 is located on an edge of top cover 102 of electronic device 100 and is configured to be illuminated according to a light pattern in which a single color or different colors are displayed for a sequence of temporal durations determined based on a frequency. In some situations, the light pattern represents a message containing data or status of electronic device 100. For example, indicator 114 may continuously brighten red in response to detecting the presence of a product on or near the top cover, and may turn and remain green for a short period of time in response to successfully scanning a barcode displayed on the product. In some embodiments, each indicator 114 includes one or more LEDs from which light is emitted, the light being displayed on indicator 114 in a substantially uniform and homogeneous manner.

[0025] 2 is a block diagram of an exemplary electronic device 100, according to some embodiments. The electronic device 100 includes one or more distance sensors 104, as described above with respect to FIG. 1. In some embodiments, the one or more distance sensors 104 include one or more of a time-of-flight sensor, an ultrasonic sensor, a radar sensor, or a LiDAR sensor. In some embodiments, the electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) whether an object is present within a sensing area. The sensing area is designed for the proximity sensors to operate. In some embodiments, the electronic device 100 determines the distance between an object and the electronic device 100 using distance measurement techniques, such as an image focus finder, analog-to-digital conversion (ADC), and / or digital-to-analog conversion (DAC).

[0026] The electronic device 100 includes a light source 106. In some embodiments, the light source 106 includes a long-distance light source 262, a low-angle light source 264, and / or a dome light source 266, as described in FIG. 3 and in U.S. patent application Ser. No. 14 / 298,659, filed June 6, 2014, and entitled "Combination Dark Field and Bright Field Illuminator" (now U.S. Pat. No. 8,989,569, issued March 24, 2015, which is incorporated by reference herein in its entirety). In some embodiments, the light source 106 provides illumination with visible light. In some embodiments, the light source 106 provides illumination with invisible light (e.g., infrared or ultraviolet light).

[0027] In some embodiments, electronic device 100 includes a decoder 212 for decoding data contained in the barcode and transmitting the data to a computing device. In some embodiments, decoder 212 is part of software application 230. Details of the decoder 212 are described in U.S. patent application Ser. No. 14 / 298,659, filed Jun. 6, 2014, entitled "Combination Dark Field and Bright Field Illuminator" (now U.S. Pat. No. 8,989,569, issued Mar. 24, 2015, the entire contents of which are incorporated herein by reference).

[0028] In some embodiments, electronic device 100 includes one or more input interfaces 210 to facilitate user input. In some embodiments, electronic device 100 is a battery-operated device and includes a rechargeable battery. In this example, input interface 210 may include a charging port for charging the battery.

[0029] In some embodiments, the electronic device 100 includes a camera 112, which includes an image sensor 216 and a lens 218. The lens 218 directs the path of light rays and focuses them onto the image sensor 216 to recreate an image on the image sensor as accurately as possible. The image sensor 216 converts light (e.g., photons) into electrical signals that the electronic device 100 can interpret. In some embodiments, the lens 218 is an optical lens made from glass or other transparent material. In some embodiments, the lens 218 is a liquid lens composed of an optical liquid material, whose shape, focal length, and / or working distance change when a current or voltage is applied to the liquid lens. In some embodiments, the electronic device 100 (e.g., via the one or more processors 202) uses distance information obtained by the distance sensor 104 to determine an optimal current or voltage to apply to the liquid lens 218 so that it has an optimal focal length for decoding barcode data contained in the image. In some embodiments, the camera 112 is configured to capture images in color. In some embodiments, the camera 112 is configured to capture images in black and white.

[0030] The electronic device 100 also includes one or more processors (e.g., CPUs) 202, one or more communication interfaces 204 (e.g., network interfaces), memory 206, and one or more communication buses 208 (sometimes called chipsets) for interconnecting these components.

[0031] In some embodiments, electronic device 100 includes radios 220. Radios 220 enable one or more communication networks, allowing electronic device 100 to communicate with other devices, such as computing devices or servers. In some embodiments, radios 220 are capable of data communication using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, Ultrawide Band (UWB), and / or software defined radio (SDR)), custom or standard wired protocols (e.g., Ethernet or HomePlug), and / or any other suitable communication protocols, including communication protocols not yet developed as of the filing date of this application.

[0032] Memory 206 includes high-speed random-access memory such as DRAM, SRAM, DDRRAM, or other random-access solid-state memory devices. In some embodiments, memory includes non-volatile memory such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. In some embodiments, memory 206 includes one or more storage devices located remotely from one or more processors 202. Memory 206, or alternatively, non-volatile memory within memory 206, includes a non-transitory computer-readable storage medium. In some embodiments, memory 206, or the non-transitory computer-readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof: Operating logic 222, which includes procedures for handling various basic system services and for performing hardware-dependent tasks. A communications module 224 (e.g., a wireless communications module) that connects to and communicates with other network devices (e.g., local networks such as routers providing Internet connectivity, network-attached storage devices, network routing devices, server systems, computing devices, and / or other connected devices) coupled to one or more communications networks via one or more communications interfaces 204 (e.g., wired or wireless). An application 230 that acquires an image including a label (e.g., a barcode), decodes the label, and controls one or more components of electronic device 100 and / or other connected devices according to the determined state. In some embodiments, application 230 includes: An illumination module 232 that selects and deploys a sequence of one or more light sources 106 and / or illumination patterns 234 for the current read cycle (e.g., based on distance measurements, such as direct or indirect measurements from the distance sensor 104). A distance module 236 that determines (eg, selects) a sequence of focal lengths to be employed during the current read cycle based on distance measurements from one or more distance sensors 104 . An exposure and gain module 238 that samples the image 244 captured by the camera 112. An image acquisition and processing module 240 that acquires and processes images, for example, according to the processes shown in any of FIGS. A decoder 212 for decoding the data contained in the barcode and transmitting it to a computing device. Data 242 of the electronic device 100. Data 242 includes, but is not limited to: Image data 244 (e.g., camera data). Symbology data 246 (e.g., type of code, such as barcode). Device settings 248 of the electronic device 100, such as default options, image acquisition settings (e.g., exposure and gain settings) and recommended user settings. 250 user settings, such as the recommended shade of lenses (e.g., photochromic lenses). Sensor data 252 obtained (e.g., measured) from the distance sensor 104 and / or other sensors included in the electronic device 100.

[0033] In some embodiments, the distance sensor 104 is monitored by the illumination module 232. When a user initiates a current read cycle, the distance sensor 104 identifies a distance field (e.g., near field, mid-field, or far field) corresponding to the object's position. The illumination module 232 selects an illumination sequence corresponding to the distance field to execute. If a good read was achieved in the previous read cycle (e.g., a good read from the third illumination pattern in the near-field illumination sequence) and the current read cycle has the same distance field as the previous read cycle, the application 230 starts the current read cycle using the values ​​of the previous good read (e.g., the third illumination pattern in the near-field illumination pattern, the previous focus position, exposure, and / or gain) before starting the illumination sequence from the beginning. Because users typically read many similar parts, the device can achieve a good read more quickly if it starts with known good settings from a previous decode operation. If the previous settings did not result in a good read, the illumination sequence for the current distance field starts from the beginning, repeating capture-after-capture with each sequence.

[0034] In some embodiments, the exposure and gain module 238 rejects images that do not fall within predetermined attribute ranges for "brightness" and / or "sharpness" (e.g., rejected images are not processed by the image acquisition and processing module 240). In some embodiments, the exposure and gain module 238 updates the image acquisition settings (e.g., exposure and gain) for the next image capture to provide optimal "brightness" for image processing.

[0035] In some embodiments, after an image is captured (e.g., using camera 112), electronic device 100 (e.g., via application 230) evaluates the quality of the captured image. For example, electronic device 100 reads (e.g., determines) the image's sharpness value, average light mean value, and / or average dark mean value and determines whether to accept or reject the image. If the result does not meet or exceed predetermined target values, the image is rejected and another image is recaptured. If the result meets or exceeds predetermined target values, the image is processed (e.g., by image acquisition and processing module 240).

[0036] As an example, in some embodiments, a good quality image is an image sample having a light average value between 100 and 170 (out of a range of 0 to 255), a dark average value between 20 and 80 (out of a range of 0 to 255), and a sharpness value above 6000 (out of a range of 0 to approximately 12,000).

[0037] In some embodiments, data collected during image sampling (eg, evaluation) is captured and added (eg, as data 242).

[0038] In some embodiments, after qualifying an image, electronic device 100 (e.g., via application 230) determines whether to adjust exposure or gain settings (e.g., using a light mean correction path or a dark mean correction path) for the next image. If it decides to do so, electronic device 100 collects target light and dark mean values ​​for comparison, develops a proportional-integral (PI) controller transfer function, and calculates the necessary changes in exposure to obtain an ideal exposure for the next image.

[0039] In some embodiments, once an image is successfully decoded, the exposure, gain, and focus values ​​are fed back to the application 230. On the next read cycle, the application 230 checks whether these decode settings are pending. If so, the electronic device 100 attempts to load the camera settings and any previous settings rather than calculating the next setting configuration. If the previous decode settings are used, the application 230 samples the image for data but does not adjust the values ​​of the feedback controllers.

[0040] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned memory devices and correspond to sets of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 206 stores a subset of the modules and data structures identified above. Additionally, memory 206 can store additional modules or data structures not listed above. In some embodiments, a subset of the programs, modules, and / or data stored in memory 206 is stored in and / or executed by a server system and / or an external device (e.g., a computing device).

[0041] FIG. 3 illustrates an exemplary image processing environment 300 including an electronic device 100 that processes image data using a parallel pipeline, according to some embodiments. In some embodiments, the electronic device 100 is an imaging device. In some embodiments, the electronic device 100 is a code reader, a barcode scanner, a label scanner, or an optical scanner. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). In some embodiments, the electronic device 100 is configured to acquire image data including one or more images, process the image data using a parallel pipeline, and provide report data 312 generated from the image data to one or more client devices 302 (e.g., device 302A, device 302B, device 302C, or device 302D). The one or more client devices 302 may be, for example, a desktop computer, a tablet computer, a mobile phone, or an intelligent, multi-sensing, network-connected home device (e.g., a display assistant device). Each client device 302 can collect report data 312 from electronic device 100, receive user input, execute user applications, and present report data 312 or other information in its user interface. In some embodiments, the user application includes an interactive user application. The user interface of the interactive user application is displayed on the client device for receiving user input related to electronic device 100 and for visualizing report data 312 generated by electronic device 100.

[0042] The electronic device 100 is configured to enable multiple parallel pipelines. The electronic device 100 identifies multiple image processing cycles associated with a temporal sequence of triggers, with each image processing cycle being created in response to one or more respective trigger events (e.g., one or more image capture operations). The multiple image processing cycles are assigned to the multiple parallel pipelines. Existing cycle data containers are drawn directly from each parallel pipeline's cycle data pool. The electronic device processes the multiple image processing cycles of the multiple parallel pipelines and generates respective report data independently of each other. In some embodiments, the temporal sequence of triggers corresponds to an ordered sequence of images processed during the image processing cycle. The image processing cycle report data 312 is generated separately from the multiple parallel pipelines, independent of the order of the sequence of images. In some circumstances, the report data of the image processing cycles is provided to the client device 302 organized according to the order of the sequence of the corresponding images.

[0043] In some embodiments, a user application implemented on the client device 302 is driven by a first programming language, and multiple image processing cycles are executed on the electronic device 100 by a second programming language different from the first programming language. The multiple image processing cycles are configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first and second programming languages. For example, the intermediate data representation is implemented in JSON (JavaScript Object Notation). The user application includes a web-based user interface, and the first programming language includes JavaScript. A runtime associated with the image processing cycles uses a second language (e.g., C++). JSON is the native format of the runtime by building JSON support into almost all runtime C++ objects, allowing JavaScript programs to understand workflow or report formats automatically used by the runtime.

[0044] In some embodiments, the report data 312 or user input is processed locally at the client device 302 and / or remotely at the server 304. The one or more servers 304 provide system data (e.g., boot files, operating system images, and user applications) to the client device 302 and, in some embodiments, process the report data 312 and user input received from one or more client devices 302 when the user applications execute on the client device 302. In some embodiments, the data processing environment 300 further includes a storage device 306 for storing data related to the server 304, the client device 302, the electronic device 100, and the user applications executing on the client device 302. For example, the storage device 306 can store video content, static visual content, and a product database.

[0045] The one or more servers 304, the one or more client devices 302, the electronic device 100, and the storage device 306 are communicatively coupled to each other via one or more communications networks 308. The communications network 308 is the medium used to provide a communications link between these devices and computers connected together within the data processing environment 300. The one or more communications networks 308 may include connections, such as wired, wireless communications links, or fiber optic cables. Examples of the one or more communications networks 308 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. The one or more communication networks 308 are optionally implemented using any known network protocol, including various wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. Connection to the one or more communication networks 308 may be established directly (e.g., using a 3G / 4G connection to a wireless carrier), through a network interface 310 (e.g., a router, switch, gateway, hub, or intelligent, dedicated whole-home control node), or through any combination thereof.Thus, one or more communications networks 308 represent the Internet, a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational, and other electronic systems that route data and messages.

[0046] In some embodiments, electronic device 100 is communicatively coupled to client device 302 (302A) via wired communication link 312. In some embodiments, electronic device 100 is communicatively coupled to client device 302 (302A) via a local communication network 314 enabled by network interface 310. Both electronic device 100 and client device 302 (e.g., 302A) are located at the same location (e.g., a warehouse or factory). In some embodiments, electronic device 100 is remote from client device 302 and is communicatively coupled to client device 302 (e.g., device 302B, device 302C, and / or device 302D) via at least a WAN.

[0047] FIG. 4 illustrates a process 400 for processing image data in multiple parallel pipelines 402 (e.g., 402A, 402B, and 402C) according to some embodiments. The process 400 is implemented by the electronic device 100 having the memory 206 and one or more processors 202. For example, the process 400 is executed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 identifies multiple image processing cycles 404 associated with a temporal sequence of triggers (e.g., image acquisition). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is generated in response to one or more respective trigger events (e.g., capture of a single image or capture of a set of images). The multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from a cycle data pool 408. In some embodiments, the cycle data pool 408 includes a predetermined number of cycle data containers 406 prior to identifying the multiple image processing cycles 404. When assigned to a respective image processing cycle 404, each parallel pipeline 402 directly uses one of the predetermined number of cycle data containers 406 without waiting for the generation of a new cycle data container 406. This expedites the start and execution of each parallel pipeline 402. In some situations, a cycle data container 406 may be completely used by a different pipeline 402 and may not be available for the assigned pipeline 402 in a new image processing cycle 404.

[0048] After the multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402, the multiple image processing cycles 404 are processed independently of one another in the multiple parallel pipelines 402, thereby independently generating report data 312 (e.g., 312A, 312B, and 312C). The report data 312 is provided to the client device 302 for further processing (e.g., storage, classification, analysis, and / or visualization). In some embodiments, the client device 302 is coupled to the electronic device 100 via a local communications network and executes an interactive user application 412. For example, the client device 302 and the electronic device 100 are located in a factory automation network without Internet access. The interactive user application 412 includes a user interface configured to locally configure, adjust, and debug the electronic device 100 and associated workflows and jobs. In some embodiments, the interactive user application 412 is web-based and implemented on the client device 302, which is coupled to a WAN (e.g., the Internet). For example, the web-based user interface is enabled by a browser-based user application 412. The browser-based user application 412 is served directly from the client device 302 and communicates with the electronic device 100 using HTTP and web sockets. A web-based user interface is presented to allow the client device 302 to remotely configure, tune, and debug the electronic device 100 and its associated workflows and jobs.

[0049] In some embodiments, process 400 enables a highly interactive browser-based user experience by exchanging data structures between a web-based user interface of a user application 412 and the runtime of multiple parallel pipelines 402. In some embodiments, user application 412 is implemented based on Hypertext Transfer Protocol (HTTP), and client devices 302 running user application 412 include Internet of Things (IOT) devices. In one example, configuration of user application 412 and pipelines 402 is via a data exchange scheme using JavaScript Object Notation (JSON), allowing a JavaScript-driven web-based user interface to natively interact with a core runtime implemented in C++ or other non-dynamic computer language.

[0050] In some embodiments, the first trigger event of the first image processing cycle 404A precedes the second trigger event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Also, in some embodiments, whichever of the first report data 312A or the second report data 312B becomes available first is reported to the client device 302 first, regardless of the temporal order of the first and second trigger events. If the second report data 312B for the second image processing cycle 404B is available before the first report data 312A for the first image processing cycle 404A, the second report data 312B is provided to the client device 302 before the first report data 312A. If the second report data 312B is available after the first report data 312A, the second report data 312B is provided to the client device 302 after the first report data 312A.

[0051] For each parallel pipeline 402, an existing cycle data container 406 is drawn from the cycle data pool 408 and used to store cycle data 414 associated with the corresponding image processing cycle 404. For example, a first image processing cycle 404A is implemented in the first parallel pipeline 402A. The first cycle data 414A includes image data associated with one or more respective trigger events of the first image processing cycle 404A and intermediate data generated by and passed between stages of the first image processing cycle 404A. In some embodiments, the first cycle data 414A includes first report data 312A generated from the intermediate data and reported to the client device 302. In some embodiments, all of the image data, intermediate data, and report data of the first image processing cycle 404A are reported to the client device 302. In some embodiments, only the report data 312A of the first image processing cycle 404A is reported to the client device 302.

[0052] In one example, the electronic device 100 is configured to recognize a barcode in an image. In response to a first trigger event 602 in which an image including a barcode is captured, a first image processing cycle 404A is generated. First cycle data 414 associated with the first image processing cycle 404 includes a serial number corresponding to the barcode. First report data 312A generated from the first image processing cycle 404 includes a product name and a price of the product.

[0053] In some prior art implementations, each pipeline has a fixed number of stages, with the stages executing alternately, such that the first stage of a current cycle executes simultaneously with the second stage of the previous cycle preceding the current cycle. In contrast, in various embodiments herein, parallel pipelines 402 have no such restriction, and each parallel pipeline 402 is switched on for a corresponding image processing cycle 404 without overhead. While the stages of each cycle execute within the same pipeline 402, the pipeline 402 contains a managed number of threads and is associated with a cycle data container 406 drawn from a cycle data pool 408.

[0054] FIG. 5 illustrates another exemplary process 500 for processing image data in multiple parallel pipelines 402, according to some embodiments. The process 500 is performed by the electronic device 100, which has the memory 206 and one or more processors 202 (FIG. 2). For example, the process 500 is performed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 provides an acquisition service 502 that identifies multiple image processing cycles 404 associated with a temporal sequence of triggers (e.g., image acquisition). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more respective trigger events (e.g., capturing a single image, capturing a set of images). Based on the temporal order of the triggers, the multiple image processing cycles are placed in a runnable cycle queue 504. The multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from the cycle data pool 408. Cycle data 414 is associated with or generated for each image processing cycle 404 and stored in the corresponding cycle data container 406 in the course of managing and executing the image processing cycle 404.

[0055] In some embodiments, the first image processing cycle 404A is implemented in a first parallel pipeline 402A and inspects an input image, e.g., to detect stains on a drug bottle label. The first image processing cycle 404A includes multiple stages, e.g., an inspection stage 506, a post-inspection stage 508, a report generation stage 510, and a data return stage 512. The input image is inspected in the inspection stage 506 to generate intermediate data that is stored in the cycle data container 406A along with the cycle data 414. The intermediate data is further processed in the post-inspection stage 508 to provide additional information. The additional information is used to generate first report data 312A in the report generation stage 510. The first report data 312A is placed in a queue of pending data 514 and provided to the client device 302 in the data return stage 512. In some embodiments, the report data 312 is organized in the queue of pending data 514 based on the chronological order of associated trigger events. Also, in some embodiments, report data 312 is organized into queues of pending data 514 based on the chronological order of its own availability, regardless of the chronological order of the associated triggering events.

[0056] Each stage of the image processing cycle 404A is not limited to a fixed hardware allocation but has flexible access to the hardware resources allocated to the corresponding parallel pipeline 402A. In other words, each stage of the image processing cycle 404A can be accelerated using all of the allocated pipeline hardware resources or can be executed in parallel with one or more of the remaining stages of the image processing cycle 404A.

[0057] In some embodiments, the plurality of image processing cycles 404 assigned to the plurality of pipelines 402 include a plurality of first image processing cycles 404-1. The runnable cycle queue 504 further includes a plurality of second image processing cycles 404-2 following the first image processing cycles 404-1. When the plurality of pipelines 402 are fully occupied by the plurality of first image processing cycles 404-1, a status signal 516 (e.g., wait for idle) is sent to the runnable cycle queue 504 to request that the runnable cycle queue 504 hold the plurality of second image processing cycles 404-2 until the plurality of pipelines 402 have completed processing the plurality of first image processing cycles 404-1.

[0058] High-precision instrumentation is built in to provide a deep, intuitive understanding of the effects of adjusting various performance parameters, including thread, process, and core loads, and hardware utilization. This functionality is designed to enable a real-time performance visualizer in a web-based user interface. It also enables on-the-fly deep analysis, with the potential to automatically adjust parameters for maximum performance without user intervention. In some embodiments, while processing the first image processing cycles 404-1 in the parallel pipelines 402, the electronic device 100 dynamically adjusts one or more of the following parameters: the number of pipelines, thread pool size, thread priority, delay, throttling parameters, core allocation, acquisition parameters, lighting, and camera focus. The second image processing cycles 404-2 are assigned to the parallel pipelines 402 and processed in the parallel pipelines 402 based on the one or more adjusted parameters. Specifically, in one example, the electronic device 100 monitors the utilization of central processing unit (CPU) cores. Based on the CPU core utilization, the electronic device 100 dynamically adjusts the computational resources allocated to each of the parallel pipelines. In this manner, parameters controlling hardware usage limits (e.g., CPU core utilization) are dynamically adjustable and can be changed during runtime. By measuring and automatically adjusting these parameters, it is possible to tailor the runtime for best performance depending on the situation.

[0059] FIG. 6 illustrates another exemplary process 600 for processing image data in multiple parallel pipelines 402, according to some embodiments. The process 600 is implemented by the electronic device 100 having the memory 206 and one or more processors 202 (FIG. 2). For example, the process 600 is executed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 provides an acquisition service 502 that identifies multiple image processing cycles 404 associated with a temporal sequence of triggers 602 (e.g., image acquisition) for an engine queue thread 604. Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more respective trigger events 602 (e.g., capturing a single image, capturing a set of images). In some embodiments, the electronic device 100 further includes a configuration service 606 that provides a job configuration that is applied to define (operation 608) the image processing cycles 404 for processing image data captured by the temporal sequence of triggers 602. Based on the temporal order of the triggers 602, the image processing cycles 404 are placed in the runnable cycle queue 504. The image processing cycles 404 are assigned to the parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from the cycle data pool 408. In the course of managing and executing the image processing cycles 404, cycle data 414 is associated with or generated for each image processing cycle 404 and stored in the corresponding cycle data container 406.

[0060] A trigger event 602 includes the acquisition of a single image, which is processed in a corresponding image processing cycle 404. The single image is optionally processed through multiple stages 610 (e.g., 610A and 610B) managed by a single thread 612 or multiple threads 612 of the image processing cycle 404. In other words, in some situations not shown, a first stage 610A is performed at least partially in parallel with a second stage 610B. In some embodiments, multiple trigger events 602 include the acquisition of multiple images. In response to multiple trigger events 602, the image processing cycle 404 executes to process multiple images associated with the multiple trigger events 602 in multiple stages 610. The multiple stages 610 are optionally processed by a single thread 612 or multiple threads 612 of the image processing cycle 404. One of the multiple images is optionally processed through multiple stages managed by a subset of the associated single thread 612 or multiple threads 612 of the image processing cycle 404. In one example, the image processing stage 610B is implemented in a subset of the parallel threads 612. In some embodiments, the electronic device 100 implements each of the successive stages of the image processing cycle 404 sequentially using all or a portion of the first hardware resources allocated to the corresponding parallel pipelines 402.

[0061] Note that multiple images are input to the parallel pipelines 402, with corresponding cycle data containers 406 being provided to the parallel pipelines 402. The cycle data containers 406 optionally store raw image data for multiple images or pre-processed image data for multiple images (e.g., filtered images, corrected images, associated image features). In some embodiments, an image signal processor (ISP) is configured to filter or correct the raw image data captured by the camera. Each image processing cycle 404 is configured to process the raw image data or pre-processed image data.

[0062] In some embodiments, the first image processing cycle 404 includes a first thread 612A and a second thread 612B. The second thread 612B optionally starts before or after the first thread 612. The first thread 612A generates a first subset of first cycle data 414. The second thread 612B starts before the generation of the first subset of first cycle data 414 and is suspended to wait until the first subset of first cycle data 414 has been generated and provided by the first thread 612A.

[0063] In some embodiments, a workflow template is loaded into a first parallel pipeline 402A. The stages 610 of the first image processing cycle 404A are performed uninterrupted by the remaining image processing cycles (e.g., 402B and 402C of FIG. 4). The stages 610 of a different image processing cycle 404 may be performed in a second parallel pipeline 402B using the same or a different workflow template. While these two image processing cycles are performed separately in the two pipelines 402, there is no overhead of switching between workflow templates.

[0064] In some embodiments, the stages 610 of an image processing cycle 404 are forced to follow a temporal ordering with the previous image processing cycle. This forces certain operations (e.g., transmission of result data) to occur sequentially between cycles. For example, both image processing cycles 404A and 404B are processed by the first parallel pipeline 402A, and the second image processing cycle 404B follows the first image processing cycle 404A. The stages 610 of the second image processing cycle 402B are forced to follow a temporal ordering with the first image processing cycle 402A. The first report data 312A generated by the first image processing cycle 402A is optionally used by the second image processing cycle 402B.

[0065] The electronic device 100 includes a state-management asynchronous runtime engine 614 configured to keep the runtime in a known state. Instead of individual commands to change parameters or having to set a mode to perform an operation, the runtime state is always consistent and can only be changed to a new state (or not changed if the new state is not acceptable). This eliminates much of the complexity of managing the user experience (e.g., undo is trivial). In some embodiments, the state used by the image processing cycle 404 is contained in cycle data 414 stored in a cycle data container 406. The cycle data 414 flows through the stages and operations of the image processing cycle 404 implemented in the parallel pipeline 402. The application of cycle data in the cycle data container 406 maximizes parallelism with other services, such as image acquisition. The cycle data container 406 contains all the information necessary to complete the purpose of the corresponding image processing cycle 404 and can be completely replaced for the new purpose of the next image processing cycle 404.

[0066] In some embodiments, cycle data 414 further includes an input queue for data acquired or generated during execution of parallel pipelines 402. For example, image acquisition is optionally overlapped with other processing, and images are pushed to a queue in cycle data 414 when stage 610 needs image data and may be blocked if the data has not yet arrived.

[0067] FIG. 7 is a temporal diagram 700 of multiple image processing cycles 404 implemented in multiple parallel pipelines 402, according to some embodiments. Each of the multiple image processing cycles 404 is processed in a respective parallel pipeline 402 independently of the remaining image processing cycles in the respective remaining parallel pipelines. Diagram 700 includes ten signal lanes, each corresponding to an individual pipeline and displaying a respective cycle status signal 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, or 722. In each signal lane, a respective one of the cycle status signals 702-722 is associated with a sequence of timestamps indicating the progress of a respective image processing cycle implemented by a respective parallel pipeline 402. Each of the ten cycle status signals 702-722 is displayed synchronously across the ten signal lanes of diagram 700. In some embodiments, each of the cycle status signals 702-722 includes multiple line segments or bars corresponding to multiple imaging stages (e.g., trigger, acquisition, configuration, decode, image processing). The multiple line segments or bars are optionally coded with different colors. The different cycle status signals 702-722 can be sorted, grouped, and filtered to show the relationship between successive barcode reading operations and to show how different imaging events overlap and interact with each other. In some embodiments, a cursor can be moved in a corresponding user interface to hover over one of the different bars of the cycle status signals 702-722 to highlight the relationship between the traces of the ten cycle status signals 702-722.

[0068] 8 is a flow diagram of a method 800 for processing image data, according to some embodiments. The method is implemented in electronic device 100, and in particular in image acquisition and processing module 240 of electronic device 100. In some embodiments, electronic device 100 includes one or more cameras 112 configured to capture images. In some embodiments, electronic device 100 is coupled to a camera or a different electronic device having a camera, and configured to acquire image data including a series of images.

[0069] The electronic device 100 identifies (802) a plurality of image processing cycles 404 associated with the temporal sequence of triggers 602. Each image processing cycle 404 is created in response to one or more respective trigger events. Each trigger event includes the acquisition of a respective image. The electronic device 100 assigns (804) the plurality of image processing cycles 404 to a plurality of parallel pipelines 402. For each parallel pipeline 402, a respective cycle data container 406 is drawn from a cycle data pool 408 (806). The electronic device 100 processes (808) the first image processing cycle 404A of the first parallel pipeline 402A independently of the processing of the remaining image processing cycles 404 (e.g., 404B and 404C of FIG. 4 ) in each remaining parallel pipeline 402, and generates first report data. The first report data is provided (810) to the client device 302 (e.g., 302A of FIG. 3 ), for example, for display on a web-based user interface of the client device 302.

[0070] In some embodiments, a trigger event includes the acquisition of a single image, and the single image is processed in a corresponding image processing cycle 404. Also, in some embodiments, multiple trigger events include the acquisition of multiple images, and in response to the multiple trigger events, the image processing cycle 404 is executed to process the multiple images associated with the multiple trigger events. In a first processing cycle 404A, multiple trigger events are processed by multiple threads. In some situations, a first thread 612A (FIG. 6) generates a first subset of first cycle data 414A that is applied by a second thread 612. The second thread 612 is started and suspended to wait until the first subset of first cycle data 414A is generated and available to the first thread.

[0071] In some embodiments, the cycle data pool 408 includes a predetermined number of cycle data containers 406 before identifying the multiple image processing cycles 404. When assigned to a respective image processing cycle 404, each parallel pipeline 402 directly uses one of the predetermined number of cycle data containers 406 without waiting for the creation of a cycle data container 406. This expedites the start and execution of each parallel pipeline 402. In some situations, the predetermined number of cycle data containers are fully used by different pipelines 402, and a cycle data container 406 cannot be retrieved for the pipeline 402 assigned to the new image processing cycle 404.

[0072] In some embodiments, processing the first image processing cycle 404A further includes inputting (812) image data associated with one or more respective trigger events associated with the first image processing cycle 404A into the first cycle data container 406A, generating (814) first cycle data 414A including the first report data 312A, and storing (816) the first cycle data 414A in the first cycle data container 406A. In some embodiments, after the first image processing cycle 404A is completed, the data stored in the first cycle data container 406A is purged and the first cycle data container 406A is returned to the cycle data pool 408 to await retrieval and use by the next parallel pipeline 402. In some embodiments, the report data 312A reported to the client device 302 for the first image processing cycle 404A includes all of the first cycle data 414A and the image data. The first cycle data includes all intermediate or final data generated by the first parallel pipeline 402A. The client device 302, for example, stores the report data 312 in a database (e.g., storage 306), visualizes the report data 312, and controls the report data 312.

[0073] In some embodiments, the first image processing cycle 404A includes a first thread 612A and a second thread 612B. The first image processing cycle 404A is processed in the first parallel pipeline 402A by, in sequence, generating a first subset of first cycle data by the first thread 612A, starting the second thread 612B before generating the first subset of first cycle data, pausing the second thread 612B, and waiting until the first subset of first cycle data has been generated by the first thread 612A.

[0074] In one example, the first image processing cycle 404A is generated in response to a first trigger event. The first trigger event includes capturing an image including a barcode. The first cycle data includes a serial number corresponding to the barcode. The first report data includes a product name and a price. In another example, the first image processing cycle 404A is generated in response to a first trigger event. The first trigger event includes capturing an image of a label affixed to a medication bottle. In some embodiments, the first report data includes an image of the label. The client device 302 receives the image of the label, determines that the label in the image is damaged, and displays the image of the label with a message indicating that the medication bottle label is damaged. In some embodiments, the electronic device 100 determines that the label in the image is damaged and generates first report data 312A including a message indicating that the medication bottle label is damaged. The client device 302 receives the message and displays the message without the image of the label.

[0075] In some embodiments, the first cycle data 312A includes intermediate data generated by and passed between multiple stages of the first image processing cycle 404A and used to generate the first report data, with multiple data items of the intermediate data forming a queue and stored sequentially in the first cycle data container 406A.

[0076] In some embodiments, the temporal sequence of triggers corresponds to an ordered sequence of images processed during multiple image processing cycles 404 allocated to multiple parallel pipelines 402. A first image is acquired before a second image, and report data 312 corresponding to the first image is provided to the client device before report data 312 corresponding to the second image.

[0077] In some embodiments, the first image processing cycle 404A is generated in response to multiple trigger events, each of which includes the acquisition of a respective image. In some circumstances, the pipeline 402 begins processing a trigger event that precedes the first image processing cycle 404A while waiting for a later trigger event.

[0078] In some embodiments, the first image processing cycle 404A includes a first stage 610A and a second stage 610B (FIG. 6). The first image processing cycle 404A in the first parallel pipeline 402A is processed by performing a portion of the first stage 610A and a portion of the second stage 610B in parallel.

[0079] In some embodiments, the first image processing cycle 404A includes a series of stages (e.g., 506-512 in FIG. 5) that are processed in the first parallel pipeline 402A by sequentially performing each of the series of stages using first hardware resources allocated to the first parallel pipeline 402A.

[0080] In some embodiments, following the first image processing cycle 404A, the electronic device processes a second image processing cycle 404A in a second parallel pipeline 402B, which is different from the plurality of image processing cycles 404. After providing the first report data 312A to the client device, the electronic device 100 provides second report data to the client device 302. In some embodiments, the second report data is visualized on a user interface of the client device 302. In some embodiments, the first trigger event of the first image processing cycle 404A precedes the second trigger event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Also, in some embodiments, regardless of the temporal order of the first and second trigger events, the first report data 312A or the second report data 312B that becomes available first is reported first to the client device 302. If the second report data 312B of the second image processing cycle 404B is available before the first report data 312A of the first image processing cycle 404A, the second report data 312B is provided to the client device 302 before the first report data 312A. If the second report data 312B is available after the first report data 312A, the second report data 312B is provided to the client device 302 after the first report data 312A.

[0081] In some embodiments, the first image processing cycle 404A is performed in the first parallel pipeline 402A by loading a workflow template into the first parallel pipeline 402A and staging multiple stages of the first image processing cycle 404A according to the workflow template without being interrupted by the remaining image processing cycles 404.

[0082] In some embodiments, the electronic device 100 monitors the utilization of a central processing unit (CPU) core. Based on the utilization of the CPU core, the respective computational resources distributed among the multiple parallel pipelines 402 are dynamically adjusted. This is done automatically without user intervention. High-precision instrumentation is built in, allowing deep and intuitive understanding of the effects of adjusting various performance parameters, including thread, process, and core loads, and hardware utilization. In some embodiments, computer performance is visualized in a user interface on the client device 302. Computer performance is analyzed in real time during execution, potentially automatically adjusting parameters to improve performance without user intervention.

[0083] In some embodiments, the plurality of image processing cycles 404 includes a plurality of first image processing cycles 404-1 (FIGS. 5 and 6). While processing the plurality of first image processing cycles 404-1 in the plurality of parallel pipelines 402, the electronic device 100 dynamically adjusts one or more parameters of the number of pipelines, thread pool size, thread priority, delay, throttling parameters, core allocation, acquisition parameters, lighting, and camera focus. A plurality of second image processing cycles 404-2 follow the plurality of first image processing cycles 404-1. The electronic device allocates the plurality of second image processing cycles 404 to the plurality of parallel pipelines 402. Based on the one or more adjusted parameters, the plurality of second image processing cycles 404-2 are processed in the plurality of parallel pipelines 402.

[0084] In some embodiments, the client device 302 executes an interactive user application. The electronic device 200 enables display of a user interface and displays the first report data on the user interface of the client device 302. Furthermore, in some embodiments, the user application is driven by a first programming language, and the plurality of image processing cycles 404 are executed by a second programming language different from the first programming language. The plurality of image processing cycles 404 are configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first and second programming languages. In some embodiments, the interactive user application is implemented locally on the client device 302, which is coupled to the electronic device via a wired communication link or a local communication network. In some embodiments, the interactive user application is web-based and implemented on the client device 302, which is coupled to the electronic device via a WAN (e.g., the Internet). For example, the web-based user interface is enabled in a browser-based application served directly from the client device 302 and communicating with the electronic device 100 using HTTP and web sockets. The user interface is useful for defining, configuring, adjusting, and debugging the electronic device 100 and associated workflows and jobs.

[0085] In some embodiments, the first image processing cycle 404A is implemented in an electronic device 100 that is coupled to a client device via a local communications network.

[0086] In one aspect of the present application, a multi-process optimization application is implemented in factory automation. Quality control requires monitoring and iterative tuning of multiple system settings (e.g., including camera settings, image processing settings, and processor settings). Examples of critical system settings include, but are not limited to, image capture times for multiple image processing cycles, where a series of images must be captured at precise times or precise intervals. Some embodiments include high-precision instrumentation and interactive analysis of complex multi-process and multi-core applications used to set up embedded devices in real time in factory automation scenarios. In some embodiments, multiple image processing cycles generate a wide variety of event-type data, including I / O timing, signal data, performance metrics, image data, and related data changes at one or more levels of time precision. The data generated by the image processing cycles is dynamically integrated to maintain insights over time. An interactive oscilloscope-style user interface enables real-time visualization, manipulation, and intuitive tuning of system settings, improving the iterative trial-and-error process. This oscilloscope-style user interface facilitates tuning of software application 230 (FIG. 2) that acquires images containing labels (e.g., barcodes) and decodes the labels, providing users with unprecedented insight into the behaviors of electronic device 100 in image processing environment 300.

[0087] FIG. 9 is a diagram of a data structure 900 for signals and data applied in an image processing environment 300, according to some embodiments. The electronic system includes an electronic device 100 including a camera and a computing device 302 for visualizing signals and data. The electronic system implements multiple image processing cycles 404 ( FIG. 4 ) associated with a temporal sequence of triggers 602, with each image processing cycle 404 being created in response to one or more respective trigger events 602 ( FIG. 6 ). The electronic system includes a system configuration 902 used to set up the multiple image processing cycles 404. During an image processing cycle 404, the electronic system acquires multiple input / output (I / O) signals 904, optionally available at an input or output interface of the electronic system. Based on the multiple I / O signals 904, the electronic system generates multiple cycle status signals 906, each associated with a sequence of timestamps 914 and indicating the progress of a respective image processing cycle 404. In some embodiments, the cycle status signal 906 is generated with high precision, and the sequence of timestamps 914 has a time resolution higher than a predetermined resolution (e.g., every two consecutive timestamps having a time difference of less than 20 nanoseconds (ns). For example, the predetermined resolution is 20 ns, and the time resolution is optionally 2 ns, 5 ns, 10 ns, and 13 ns.

[0088] In some embodiments, the system settings 902 include a set of job configurations, and the electronic system includes a configuration service 606 ( FIG. 6 ) to which the set of job configurations are applied to define image processing cycles 404 that process captured image data in response to a temporal sequence of triggers 602. In some embodiments, the system settings 902 (e.g., job configurations) of the electronic system include, but are not limited to, one or more of camera settings 902A, image processing settings 902B, and processor settings 902C. Examples of camera settings 902A include, but are not limited to, camera exposure time, focal length, and gain. Examples of image processing settings 902B include, but are not limited to, image filtering options, lighting parameters, window of interest (WOI), color conversion, linear and non-linear calibration settings, contrast correction, and histogram normalization. Examples of processor settings 902C include, but are not limited to, thread pool size, thread priority, thread affinity (core assignment), trigger delay, stack size, timeout, throttling, and heartbeat frequency. In some embodiments, system settings 902 further include application settings for software application 230 (FIG. 2). Application settings include user-configurable settings (e.g., symbology type, thresholds, region of interest (ROI) size) and internal system settings (e.g., thread priority, delays, timeouts, network usage). Application settings optionally overlap with image processing settings 902B or processor settings 902C.

[0089] In some embodiments, the plurality of I / O signals 904 are associated with timestamps 914 obtained from two or more sources and correlated according to a precision time protocol. Each source may include an internal or external source. One example of a precision time protocol is IEEE 1588. In some embodiments, the plurality of I / O signals 904 includes one or more of a digital signal 904A, an event signal 904B, and a blip signal 904C. In one example, the plurality of I / O signals 904 includes one or more images captured in response to each trigger 603, a trigger signal for each trigger 602, and barcode information extracted from the one or more images. In some embodiments, the digital signal 904A has a low-to-high transition edge and a high-to-low transition edge, one or both of which correspond to a virtual or physical I / O event. In some embodiments, the event signal 904B is obtained for one or more of an error or exception, a mode change, receipt of an external command, a process or trigger overrun, a data mismatch, an unexpected serial number, a stop event, a start event, an external device connection, and / or an external device disconnection. In some embodiments, the blip signal 904C indicates time durations associated with the electronic system, the parallel pipeline 402 (FIG. 4), and the threads 612 (FIG. 6). Examples of the blip signal 904 include, but are not limited to, start and end times for image acquisition, image processing, pre-processing, post-processing, formatting, report generation, network transfer, and parsing. Additionally, in some embodiments, the blip signal 904 is stored with identifiers for image thumbnails and full-resolution images.

[0090] Additionally, in some embodiments, the electronic system further includes one or more metric signals 908 indicative of overall system performance associated with the plurality of image processing cycles 404. The one or more metric signals 908 are optionally measured directly by the electronic system or generated based on the I / O signals 904. In some embodiments, the one or more metric signals 908 include a hardware performance signal 908A and a signal quality signal 908B. Examples of the hardware performance signal 908A include, but are not limited to, image acquisition time, CPU load, core temperature, memory usage level, number of threads, network traffic, and parameter changes. In some embodiments, a subset of the metric signals 908 includes secondary signals generated based on real-time analysis of the I / O signals 904. For example, the duty cycle, period (e.g., pulse width), high-level time, low-time level, and duration between transitions of the digital signal 904 are measured to determine the frequency, corresponding to the secondary signal of the digital signal 904, which acts as an analog signal for further processing and rendering.

[0091] Additionally, in some embodiments, the electronic system generates one or more statistical signals 910 based on a subset of the system settings 902, the cycle status signals 906, the I / O signals 904, and the one or more metric signals 908. For example, the one or more statistical signals 910 include one or more of the mean, median, minimum, maximum, variance, histogram, and outliers of the image capture time set in response to the trigger event 602. In some circumstances, the statistical signals 910 are visualized as a histogram, trend chart, scatter plot, etc.

[0092] In some embodiments, events are identified based on the system settings 902 and signals 904-910, and a subset of the system settings 902 and signals 904-910 are organized according to the events as event data 912 for further data storage and / or visualization. In some embodiments, the events correspond to event signals 904B that are directly measured related to the events. Also, in some embodiments, the events are detected in real time based on the system settings 902 and signals 904-910. Examples of events include, but are not limited to, physical input / output events, topic publication events, and topic subscription events.

[0093] Some embodiments of the present application are directed to easy and efficient event recording. In some embodiments, the image processing environment 300 is configured to record different types of events with adaptive timestamp precision and adaptive data formatting. Event data 912 includes digital signals, metric signals, errors or exceptions (including data such as stack traces), execution timing, context information, image capture times and thumbnails, counts, etc. These means make data recording simple to use and implement with controlled overhead, allowing electronic systems to receive data from multiple sources, from shared memory to networks compliant with User Datagram Protocol (UDP), Transmission Control Protocol (TCP), or WebSockets.

[0094] 10 is a time diagram 1000 of an image processing cycle 404 implemented in one of a plurality of parallel pipelines 402, according to some embodiments. The electronic system implements a plurality of image processing cycles 404 (FIG. 4) associated with a temporal sequence of triggers 602, with each image processing cycle 404 being created in response to one or more respective trigger events 602 (FIG. 6). During an image processing cycle 404, the electronic system obtains a plurality of input / output (I / O) signals 904, optionally available at an input or output interface of the electronic system. Based on the plurality of I / O signals 904, the electronic system generates one or more cycle status signals 906 for each image processing cycle 404, each associated with a sequence of timestamps 914 and indicating the progress of the respective image processing cycle 404. In some embodiments, the cycle status signal 906 is generated with high precision, and the sequence of timestamps 914 has a time resolution higher than a predetermined resolution (e.g., every two consecutive timestamps having a time difference of less than 20 nanoseconds (ns)).

[0095] In some embodiments, the time diagram 1000 corresponds to one of a plurality of signal lanes. In each signal lane, one or more cycle status signals 906 indicate the progress of a respective image processing cycle 404 implemented by a respective parallel pipeline 402. For example, the cycle status signals 906 include an image acquisition status signal 906A, a pipeline thread status signal 906B, and an image processing thread status signal 906C. The image acquisition status signal 906A includes a trigger event indicator 1002 representing the trigger event 602 ( FIG. 6 ) and an image acquisition status bar 1004. The pipeline thread status signal 906B includes an extended line having multiple segments 1006, each segment representing a respective stage of the pipeline thread. The image processing thread status signal 906C includes an extended line having multiple segments 1008 and a circular indicator 1010 indicating the detection of a barcode in an image captured in response to the trigger event 602. The cycle status signal 906 includes multiple segments 1006, 1008 corresponding to multiple imaging stages (e.g., trigger, acquire, setup, decode, image processing), optionally coded with different colors.

[0096] 11A-11D are four exemplary time diagrams 1100, 1120, 1140, and 1160 of multiple image processing cycles 404, according to some embodiments. Each time diagram 1100, 1120, 1140, and 1160 visualizes the cycle status signals 906 of multiple image processing cycles 404 in multiple parallel signal lanes. The multiple parallel signal lanes are stacked, and the cycle status signals 906 of different signal lanes are displayed synchronously with respect to the same time axis 1110. In other words, each cycle status signal 906 is associated with a sequence of timestamps 914, and the cycle status signals 906 of the multiple image processing cycles 404 are synchronized with each other. In each signal lane, one or more cycle status signals 906 indicate the progress of each image processing cycle 404 performed by a respective image processing pipeline 402. In some embodiments, each signal lane includes a cycle identification 1102 and a cycle time 1104 indicating how recently the corresponding image was captured.

[0097] 11A and 11B, in some embodiments, the cycle status signal 906 for each image processing cycle 404 includes at least an image acquisition status signal 906A and a pipeline thread status signal 906B. The image acquisition status signal 906A includes a trigger event indicator 1002 representing the trigger event 602 (FIG. 6) and an image acquisition status bar 1004. The pipeline thread status signal 906B includes an expanded bar with multiple segments 1006, each representing a respective stage of the pipeline thread (e.g., pipeline setup, decode). Furthermore, in some situations (FIG. 11A), multiple image processing cycles 404 are implemented by a single processor core (e.g., by a single image processing pipeline 402). After a previous image is fully processed in a previous image processing cycle 404P in a single image processing pipeline 402, the trigger event 602 for the next image processing cycle 404N occurs, and the next image is captured and processed in the next image processing cycle 404N in the next image processing pipeline 402N. Also, in some situations (FIG. 11B), multiple image processing cycles 404 are implemented in multiple parallel pipelines 402. While a previous image is being processed in a previous image processing cycle 404P in a previous image processing pipeline 402P, a trigger event 602 for a next image processing cycle 404N occurs, and the next image is captured and processed in the next image processing cycle 404N in the next image processing pipeline 402N. The overlap between the next image processing pipeline 402N and the previous image processing pipeline improves the image processing rate.

[0098] 11C and 11D, in some embodiments, the cycle status signal 906 for each image processing cycle 404 includes an image acquisition status signal 906A, a pipeline thread state signal 906B, and multiple image processing thread state signals 906C. The pipeline thread state signal 906B and the image processing thread state signal 906C are arranged parallel to one another and are at least partially aligned to the same portion of the time axis 1110. The image processing thread state signal 906C includes multiple thread indicators 1152 that indicate the progress of the multiple processing threads 612 (FIG. 6). In some situations (FIG. 11C), the trigger event 602 for the next image processing cycle 404N occurs only after the previous image has been completely processed in the previous image processing cycle 404P, and the next image is captured and processed in the next image processing cycle 404N by the multiple parallel threads 1152 of the single image processing pipeline 402. Also, in some situations (FIG. 11D), while a previous image is being processed in a previous image processing cycle 404P of a previous image processing pipeline 402P, a trigger event 602 for a next image processing cycle 404N occurs, and the next image is captured and processed in the next image processing cycle 404N within the next image processing pipeline 402N by multiple parallel threads 1152. Furthermore, in some embodiments, the combination of multiple image processing pipelines 402 and multiple processing threads 612 results in a processing rate that is equal to or faster than the image frame rate at which a sequence of images is captured.

[0099] 12 is an example of a collapsed graphical user interface (GUI) 1200 visualizing signals of multiple image processing cycles 404 over an extended duration, according to some embodiments. In some embodiments, a computing device (e.g., client device 302 of FIG. 3 ) executes a dedicated digital oscilloscope application (e.g., software application 230 of FIG. 2 ) to enable display of user interface 1200 on the computing device's screen. Also, in some embodiments, the computing device (e.g., client device 302 of FIG. 3 ) enables display of user interface 200 on its screen via a general-purpose browser application. The progress of the multiple image processing cycles 404 is tracked by multiple cycle status signals 906 visualized in user interface 1200. Dynamically and in real time, while the image processing cycles 404 are being executed by the electronic system, user interface 1200 simultaneously visualizes multiple cycle status signals 906 at least with respect to time axis 1100. In some embodiments, a subset of the plurality of I / O signals 904 is visualized simultaneously with the plurality of cycle status signals 906, and each of the subsets of the plurality of I / O signals 904-1, 904-2 is displayed with respect to a time axis. In some embodiments, one or more metric signals 908 indicate overall system performance associated with the plurality of image processing cycles 404 and are visualized simultaneously with the plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to a time axis 1100.

[0100] In some circumstances, given the extended duration, each of the plurality of I / O signals 904-1, 904-2 and the plurality of cycle status signals 906 collapses along the time axis 1100, and details of the signals 904-1, 904-2, 906 are not discernible in FIG. 12 . Further, in some circumstances, a zoom operation is performed in FIG. 12 to expand the time axis 1100 and display details of at least the signals 904-1, 904-2, 906 in FIG. 13 . In some embodiments, despite the loss of details of the signals 904-1, 904-2, 906, the signal density shown in FIG. 12 is applied to detect an anomalous event 1202 in the cycle status signal 906. Furthermore, in some embodiments, the extended duration facilitates detection of another anomalous event 1204 (also referred to as an outlier 1204) in the metric signal 908 of the user interface 1200. Data associated with the anomalous events 1202, 1204 is extracted and stored for further review and analysis. Such anomalous events 1202, 1204 are optionally reported in real time to a user of the user interface 1200 by displaying an alert on the user interface 1200 or by including the detection of the anomalous event in an output report.

[0101] Stated differently, in some embodiments, an outlier 1204 of the at least one metric signal 908 is detected based on the outlier criteria. In response to detecting the outlier 1204, a portion of the at least one metric signal 908 associated with the outlier 1204 is highlighted in the user interface 1200. In some circumstances, in response to detecting the outlier 1204, additional coincidence data is recorded to facilitate further review and analysis of the outlier 1204. Furthermore, in some embodiments, the at least one metric signal 908 includes two or more metric signals 908. The electronic system monitors a correlation of the two or more metric signals 908, and the outlier criteria is associated with a variation in the correlation. Furthermore, in some embodiments, the at least one metric signal 908 includes one or more of a CPU usage level, a CPU operating time, a core temperature, memory usage, and a network bandwidth. In response to detecting the outlier 1204, an action is initiated to control CPU usage by a subset of the plurality of image processing cycles 404. By these means, abnormal events are monitored and adapted to adjust the allocation of computational, storage and / or network resources between image processing cycles 404, allowing resources to be used more efficiently.

[0102] The electronic system is configured for dynamic data summarization and consolidation with outlier detection. In some embodiments, the electronic system dynamically consolidates recorded data (e.g., I / O signals 904) based on the age of the data. The most recent data is stored with greater precision, while older data is gradually summarized to maintain key statistics while ensuring the continued availability of historical information, or to downscale image resolution. Additionally, in some embodiments, an outlier condition is specified. Following a determination that a subset of the settings 902 and signals 904-912 meets the outlier condition (i.e., in response to detecting the outlier condition), corresponding event data 912, including the subset of the settings 902 and signals 904-912, is stored for further review and analysis.

[0103] FIG. 13 illustrates another exemplary user interface 1300 for visualizing signals of multiple image processing cycles 404, according to some embodiments. A computing device (e.g., client device 302 of FIG. 3 ) may run a dedicated digital oscilloscope application (e.g., software application 230 of FIG. 2 ) or a general-purpose browser application, enabling the user interface 1300 to be displayed on the computing device's screen. Compared to user interface 1200 of FIG. 12 , user interface 1300 corresponds to a temporal window 1206 ( FIG. 12 ) having a shorter duration and provides more signal detail. The progress of the multiple image processing cycles 404 is tracked by multiple cycle status signals 906 visualized in user interface 1300. While the image processing cycles 404 are being executed by the electronic system, the user interface 1300 simultaneously visualizes the multiple cycle status signals 906 dynamically and in real time, at least with respect to the time axis 1100.

[0104] In some embodiments, the time axis 1100 includes a single time axis. For each cycle status signal 906, an active portion and an associated active duration of the respective status signal 906 are identified. According to the associated active duration, the user interface 1300 displays the active portions of the respective cycle status signals 906 in rows 1320 on the single time axis. For example, each cycle status signal 906 is cropped into an image processing thread status signal 906C including multiple thread indicators 1152. The thread indicators 1152 of the multiple cycle status signals 906 are shifted to the same row 1320. Furthermore, in some embodiments, it is determined that at least one of the first image processing cycle and the second image processing cycle has encountered an abnormal condition according to a determination that the active portion of the first cycle status signal 906-1 of the first image processing cycle and the active portion of the second cycle status signal 906-2 of the second image processing cycle overlap each other on the single time axis. In some embodiments, following a determination that the active portion of the first cycle status signal 906-1 of the first image processing cycle and the active portion of the second cycle status signal 906-2 of the second image processing cycle overlap each other on a single time axis 1110, the electronic system automatically extends the inter-image time gap between the two triggers 602 configured to enable the capture of two consecutive images.

[0105] In some embodiments, in response to detecting an overlap between the two cycle status signals 906-1, 906-2, the user interface 1300 is displayed such that the overlapping portion of each of the two cycle status signals is highlighted. Further, in some embodiments, in response to detecting an overlap between the two cycle status signals 906-1, 906-2, the signals 902, 904, and 908 are monitored to identify an outlier 1322 corresponding to an anomalous event, and the outlier 1322 is optionally highlighted in the user interface 1300. Further, in some embodiments, in response to detecting an overlap between the two cycle status signals 906-1, 906-2, the signals 902, 904, 908 are monitored to identify an outlier 1322 corresponding to an anomalous event, and the outlier 1322 is optionally highlighted in the user interface 1300.

[0106] In some embodiments, a subset of the plurality of I / O signals 904 (e.g., blip signal 904C) is visualized simultaneously with the plurality of cycle status signals 906, with each subset of the plurality of I / O signals 904 being displayed with respect to the time axis 1100. In some embodiments not shown, one or more metric signals 908 indicate overall system performance associated with the plurality of image processing cycles 404 and are visualized simultaneously with the plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to the time axis 1100.

[0107] In some embodiments, one or more statistical signals 910 are generated based on a subset of the plurality of system settings 902, the plurality of cycle status signals 906, the plurality of I / O signals 904, and the one or more metric signals 908. The one or more statistical signals 910 are visualized simultaneously with the plurality of cycle status signals 906 and on an axis different from the time axis 1110. Referring to FIG. 13 , in some embodiments, the statistical signal 910 of image capture times 1302 is determined over an extended period of time (e.g., for 177 image processing cycles 404). A histogram 1304 of the statistical signal of image capture times 1302 is visualized in the user interface 1300, representing the distribution of the image capture times 1302 of the 177 image processing cycles 404. A median 1306, a mean 1308, a 2x standard deviation value 1310, and a 3x standard deviation value 1312 are marked on the histogram 1304. One or more outliers 1314 are detected above three times the normal value 1312 and correlated with external and internal events. In some embodiments, the event data 912 associated with the outliers 1314 is analyzed after the anomaly occurs.

[0108] In some embodiments, the electronic system enables display of a user interface 1300 on the screen of the computing device 302 via a browser application. The user interface 1300 enables a sophisticated browser-based environment and resembles the user interface of a digital oscilloscope. The user interface 1300 navigates over a wide range of time scales and renders appropriate signals, events, or data, including I / O signals 904, cycle status signals 906, metric signals 908, timing durations, statistical signals 910, image thumbnails, and parameter information. The user interface 1300 provides an intuitive visual representation of the operation of the software application 230 (FIG. 2) over time. In some embodiments, the user interface 1300 enables one or more of the following features: sorting, filtering, grouping, and recording.

[0109] In some embodiments, the electronic system enables interactive data analysis. The user interface 1300 allows a user to identify signal interactions, such as correlations between high CPU usage and operation duration, for example, by providing a feature that tracks and displays such relationships. These signal interactions result in outlier detection and action. For example, if an outlier condition occurs that requires further review, the event data 912, including associated time logs and statistics, is permanently stored in the database 242 (FIG. 2) for further review and analysis.

[0110] In some embodiments, the electronic system enables image-based data rendering. The electronic system renders image-based data and reveals overlaps between images. As an example, images of a long box moving on a conveyor belt are captured from a fixed camera position. The images overlap for comprehensive surface inspection. The electronic system allows a user to visualize these overlaps on a time axis in the user interface 1300. In some embodiments, the user interface 1300 allows adjustment of inter-image timing based on one or more factors. Furthermore, in some situations, inter-image timing is automatically inferred based on a subset of the signals 904-912.

[0111] In some embodiments, the image processing environment 300 is adjusted or optimized in real time. The user interface 1300 enables analysis of timing data for real-time system optimization. Based on the user interface 1300, a user can interactively adjust the system settings 902 in real time while observing the impact of the system settings 902 on system performance as reflected by the signals 904-912. In some embodiments, the signals 904-912 are analyzed using machine learning or other automated methods. Based on the analysis results, the electronic system automatically, dynamically, and in real time adjusts the system settings 902.

[0112] Referring to FIG. 13 , in some embodiments, the user interface 1300 further displays operational status signals different from the cycle status signal 906. For example, the operational status signal monitors the offline status of the electronic device 100 when an image-based inspection is not actively being performed. Specifically, in some embodiments, a method is implemented to track the status of an electronic system configured for image processing or other operations, regardless of whether the electronic system is active or offline. The electronic system acquires one or more of a plurality of system settings 902, a plurality of input / output (I / O) signals 904 (e.g., from an I / O interface), a plurality of status signals 906 (e.g., generated based on the I / O signals), a plurality of metric signals 908, and a plurality of statistical signals 910. In some embodiments, the electronic system identifies data or abnormal events based on the settings 902 and / or the signals 904-910. Dynamically and in real time, the electronic system visualizes at least a subset of the settings 902 and the signals 904-910 simultaneously, at least with respect to a time axis, in the user interface. The metrics are used to analyze the performance of the electronic system related to image processing, digital input / output, or other operations. For example, the electronic system may be applied to analyze and discover timing anomalies between a standalone communication process and a programmable logic controller (PLC). In another example, the electronic system may be applied to find performance issues for a particular image processing library under load. In some embodiments, multiple software applications 230 run simultaneously to receive streams of timing data from multiple sources. Example applications include, but are not limited to, setup software, standalone data visualization and / or analysis applications.

[0113] FIG. 14A is a portion of an exemplary user interface 1400 visualizing cycle status signals 906 and I / O signals 904 for three image processing cycles 404, according to some embodiments, and FIG. 14B is a portion of another exemplary user interface 1450 displaying a histogram of capture times for multiple image processing cycles 404, according to some embodiments. In some embodiments, a region of the user interface 1300 is selected and displayed in a zoom mode. For example, the user interface 1400 corresponds to the selected region 1316, where the cycle status of the three image processing cycles 404 may be reviewed in detail. In another example, the user interface 1450 corresponds to the selected region 1318. A histogram 1304 of statistical signals of image capture times 1302 is visualized in the user interface 1450, representing the distribution of image capture times 1302 for 177 image processing cycles 404. The median 1306, mean 1308, 2x standard deviation value 1310, and 3x standard deviation value 1312 are marked on the histogram 1304. One or more outliers 1314 are detected above three times the normal value 1312 and correlated with external and internal events. In some embodiments, the event data 912 associated with the outliers 1314 is analyzed after the anomaly occurs.

[0114] 15 is a flow diagram of a method 1500 for tracking image processing cycles in real time, according to some embodiments. Method 1500 is implemented in an electronic system for tracking image processing cycles 404 in real time. The electronic system has a memory and one or more processors and includes electronic device 100 and computing device 302 (FIG. 3). In some embodiments, electronic device 100 includes one or more cameras 112 configured to capture images. In some embodiments, electronic device 100 is coupled to a camera or a separate electronic device having a camera and configured to acquire image data including a sequence of images.

[0115] The electronic system implements (1502) a plurality of image processing cycles 404 associated with a temporal sequence of triggers, with each image processing cycle 404 being created (1504) in response to one or more respective trigger events 602 (FIG. 6). The electronic system acquires (1506) a plurality of input / output (I / O) signals 904 for the plurality of image processing cycles 404 and generates (1508) a plurality of cycle status signals 906 based on the I / O signals 904. Each cycle status signal 906 is associated (1510) with a sequence of timestamps 914 indicating the progress of the respective image processing cycle 404. The sequence of timestamps 914 has a temporal resolution higher than a predetermined resolution (1512). While the image processing cycles 404 are being executed, the electronic system dynamically and in real time simultaneously visualizes (1514) the plurality of cycle status signals 906 with respect to at least the time axis 1110 of a user interface (e.g., the user interface shown in FIGS. 11A-14B).

[0116] In some embodiments, the electronic system obtains one or more metric signals 908 indicative of overall system performance associated with the plurality of image processing cycles 404 and visualizes the one or more metric signals 908 simultaneously with the plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to a time axis 1110. Furthermore, in some embodiments, the electronic system detects an outlier for at least one metric signal based on an outlier criterion. In response to detecting the outlier, the electronic system highlights a portion of the at least one metric signal 908 associated with the outlier in a user interface. Furthermore, in some embodiments, the at least one metric signal includes two or more metric signals 908. The electronic system monitors a correlation between the two or more metric signals 908, and the outlier criterion is associated with a variation in the correlation. In some embodiments, the at least one metric signal includes one or more of a CPU usage level, a CPU operating time, a core temperature, a memory usage, and a network bandwidth. In response to detecting the outlier, the electronic system initiates an action on CPU usage by a subset of the plurality of image processing cycles 404.

[0117] In some embodiments, each of the one or more metric signals 908 is one of image acquisition time, CPU core load percentage, CPU temperature, memory usage, number of threads, and network traffic. In some embodiments, the one or more metric signals 908 include one of a period, a frequency, a duty cycle, a pulse width, a high time, and a low time of one of the plurality of I / O signals 904.

[0118] In some embodiments, the electronic system visualizes a subset of the plurality of I / O signals 904 simultaneously with the plurality of cycle status signals 906, each of the subsets of the plurality of I / O signals 904 being displayed with respect to the time axis 1110.

[0119] In some embodiments, the electronic system generates one or more statistical signals 910 based on a subset of the plurality of system settings 902, the plurality of cycle status signals 906, the plurality of I / O signals 904, and the one or more metric signals 908. The electronic system further visualizes the one or more statistical signals 910 simultaneously with the plurality of cycle status signals 906 on a non-time axis 1110.

[0120] In some embodiments, visualizing the plurality of cycle status signals 906 further includes displaying each of the plurality of cycle status signals 906 in a stacked manner. The plurality of cycle status signals 906 are displayed synchronously with respect to the same time axis 1110.

[0121] In some embodiments, the time axis 1110 includes a single time axis 1110, and visualizing the plurality of cycle status signals 906 further includes identifying, for each cycle status signal 906, an active period associated with an active portion of the respective status signal, and displaying the active portions of each cycle status signal 906 in a row relative to the single time axis 1110 according to the associated active periods. Furthermore, in some embodiments, in accordance with determining that the active portions of the first cycle status signal 906-1 of the first image processing cycle and the active portions of the second cycle status signal 906-2 of the second image processing cycle overlap each other on the single time axis 1110, the electronic system determines that at least one of the first image processing cycle and the second image processing cycle encounters an abnormal condition. Furthermore, in some embodiments, the electronic system automatically widens the inter-image time gap in accordance with determining that the active portions of the first cycle status signal 906-1 of the first image processing cycle and the active portions of the second cycle status signal 906-2 of the second image processing cycle overlap each other on the single time axis 1110.

[0122] In some embodiments, in response to detecting an overlap of two cycle status signals 906, the electronic system highlights the overlapping portions of each of the two cycle status signals 906 in a user interface.

[0123] In some embodiments, the plurality of I / O signals 904 includes one or more images captured in response to each trigger, a trigger signal for each trigger, and barcode information extracted from the one or more images.

[0124] In some embodiments, multiple I / O signals 904 are obtained from two or more sources and associated with correlated timestamps according to a precision time protocol.

[0125] In some embodiments, the predetermined resolution is 20 nanoseconds.

[0126] In some embodiments, each image processing cycle 404 includes one or more imaging stages of triggering, acquiring, configuring, decoding, and image processing, and corresponds to a respective cycle status signal 906-2 that is color-coded based on the one or more imaging stages.

[0127] In some embodiments, the first image processing cycle includes an image processing stage in which the first image is processed by multiple parallel processing threads.

[0128] In some embodiments, an electronic system running a digital oscilloscope application enables a user interface to be displayed on the screen of a computing device.

[0129] In some embodiments, the electronic system allows a user interface to be displayed on the screen of a computing device via a browser application.

[0130] In some embodiments, each system setting 902 is one of a plurality of camera settings 902A, a plurality of image processing settings 902B, and a plurality of processor settings 902C. The plurality of camera settings 902A include one or more of exposure time, focal length, gain, image filtering options, lighting parameters, window of interest parameters, color conversion parameters, linear and non-linear calibration settings, contrast correction parameters, and histogram normalization parameters. The plurality of image processing settings 902B include one or more of symbology settings, effort level, modifier settings, region of interest, and number of retries. The plurality of processor settings 902C include one or more of thread pool size, thread priority, thread affinity (CPU core assignment), trigger delay, stack size, timeout, throttling parameters, and heartbeat frequency.

[0131] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 206 stores a subset of the above-identified modules and data structures. Additionally, memory 206 can store additional modules or data structures not described above.

[0132] The terminology used in the description of the invention herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will further be understood that the terms "comprises" and "comprising," when used herein, specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0133] As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" refers to both "based only on" and "based at least on."

[0134] As used herein, the term "exemplary" means "serving as an example, instance, or illustration" and does not necessarily imply a preference or superiority of the example over other configurations or implementations.

[0135] As used herein, the term "and / or" includes any combination of the listed elements. For example, "A, B, and / or C" includes the following sets of elements: A only, B only, C only, A and B without C, A and C without B, B and C without A, and combinations of all three elements: A, B, and C.

[0136] The above description has been set forth with reference to specific implementations for purposes of explanation. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, so that others skilled in the art can utilize the invention to its fullest extent and in various implementations with various modifications suited to the particular uses contemplated.

Claims

1. 1. A method for tracking image processing cycles in real time, implemented in an electronic system having a memory and one or more processors, the method comprising: performing a plurality of image processing cycles associated with a temporal sequence of triggers, each image processing cycle being produced in response to one or more respective trigger events; acquiring a plurality of input / output (I / O) signals of the plurality of image processing cycles; generating a plurality of cycle status signals based on the I / O signals, each cycle status signal being associated with a sequence of timestamps indicating a progress of a respective image processing cycle, the sequence of timestamps having a time resolution greater than a predetermined resolution; and dynamically and in real time simultaneously visualizing said plurality of cycle status signals with respect to at least a time axis of a user interface while said image processing cycle is being performed. method.

2. obtaining one or more metric signals indicative of overall system performance associated with the plurality of image processing cycles; visualizing one or more metric signals simultaneously with the plurality of cycle status signals; Equipped with each of the one or more metric signals is displayed with respect to the time axis; The method of claim 1.

3. Detecting an outlier in at least one metric signal based on an outlier criterion; in response to detecting the outlier, highlighting in the user interface a portion of the at least one metric signal associated with the outlier; and Equipped with The method of claim 2.

4. wherein the at least one metric signal comprises two or more metric signals, the method further comprising: and monitoring a correlation of the two or more metric signals, the outlier criterion being related to a variation in the correlation. The method according to claim 3.

5. the at least one metric signal includes one or more of a CPU usage level, a CPU operating time, a core temperature, an image acquisition time, a memory usage, a number of threads, a network traffic, and a network bandwidth; and initiating an action on CPU usage by the subset of the plurality of image processing cycles in response to detecting the outlier. The method according to claim 3.

6. and visualizing a subset of the plurality of I / O signals simultaneously with the plurality of cycle status signals, each of the subsets of the plurality of I / O signals being displayed with respect to the time axis. The method of claim 1.

7. generating one or more statistical signals based on a subset of a plurality of system settings, the plurality of cycle status signals, the plurality of I / O signals, and the one or more metric signals; visualizing said one or more statistical signals simultaneously with said plurality of cycle status signals on a non-time axis; Equipped with The method of claim 1.

8. visualizing the plurality of cycle status signals, and displaying each of the plurality of cycle status signals in a stacked manner, the plurality of cycle status signals being displayed synchronously with respect to the same time axis. The method of claim 1.

9. wherein the time axis comprises a single time axis, and visualizing the plurality of cycle status signals comprises, for each cycle status signal: identifying an active portion and an associated active duration of each of said status signals; and and displaying the active portions of the respective cycle status signals in a row relative to the single time axis according to the associated active durations. The method of claim 1.

10. determining that at least one of the first image processing cycle and the second image processing cycle encounters an abnormal condition in accordance with a determination that an active portion of a first cycle status signal of a first image processing cycle and an active portion of a second cycle status signal of a second image processing cycle overlap one another on the single time axis.

10. The method of claim 9.

11. automatically widening an inter-image time gap in response to a determination that an active portion of a first cycle state signal of a first image processing cycle and an active portion of a second cycle state signal of a second image processing cycle overlap one another on the single time axis.

10. The method of claim 9.

12. one or more processors; a memory storing one or more programs configured to be executed by the processor; wherein the one or more programs include: instructions for performing a plurality of image processing cycles associated with a temporal sequence of triggers, each image processing cycle being produced in response to one or more respective trigger events; instructions for acquiring a plurality of input / output (I / O) signals of the plurality of image processing cycles; instructions for generating a plurality of cycle status signals based on the I / O signals, each cycle status signal being associated with a sequence of timestamps indicating a progress of a respective image processing cycle, the sequence of timestamps having a time resolution greater than a predetermined resolution; instructions for simultaneously visualizing, in dynamic and real-time, the plurality of cycle status signals with respect to at least a time axis of a user interface while the image processing cycle is being performed. Electronic system.

13. The one or more programs: and instructions for, in response to detecting an overlap of two cycle status signals, highlighting an overlapping portion of each of the two cycle status signals in the user interface.

13. The electronic system of claim 12.

14. the plurality of I / O signals including one or more images captured in response to respective triggers, a trigger signal for each trigger, and barcode information extracted from the one or more images; 13. The electronic system of claim 12.

15. the plurality of I / O signals are obtained from two or more sources and associated with timestamps that are correlated according to a precision time protocol; 13. The electronic system of claim 12.

16. 1. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of an electronic system, the one or more programs comprising: instructions for performing a plurality of image processing cycles associated with a temporal sequence of triggers, each image processing cycle being produced in response to one or more respective trigger events; instructions for acquiring a plurality of input / output (I / O) signals of the plurality of image processing cycles; instructions for generating a plurality of cycle status signals based on the I / O signals, each cycle status signal being associated with a sequence of timestamps indicating a progress of a respective image processing cycle, the sequence of timestamps having a time resolution greater than a predetermined resolution; instructions for simultaneously visualizing, in dynamic and real-time, the plurality of cycle status signals with respect to at least a time axis of a user interface while the image processing cycle is being performed. A non-transitory computer-readable storage medium.

17. 17. The non-transitory computer-readable storage medium of claim 16, wherein each image processing cycle includes one or more imaging stages of triggering, acquiring, configuring, decoding, and image processing, and corresponds to a respective cycle status signal that is color coded based on the one or more imaging stages.

18. 20. The non-transitory computer-readable storage medium of claim 16, wherein the first image processing cycle includes an image processing stage in which the first image is processed by multiple parallel processing threads.

19. The one or more programs are executing a digital oscilloscope application to enable display of said user interface on a screen of a computing device; and and further comprising instructions for one of: enabling display of the user interface on a screen of the computing device via a browser application.

17. The non-transitory computer readable storage medium of claim 16.

20. each system configuration being one of a plurality of camera configurations, a plurality of image processing configurations, and a plurality of processor configurations; the plurality of camera settings include one or more of exposure time, focal length, gain, image filtering options, illumination parameters, window of interest parameters, color transformation parameters, linear and non-linear calibration settings, contrast correction parameters, and histogram normalization parameters; the plurality of image processing settings including one or more of a symbology setting, an effort level, a modifier setting, a region of interest, and a number of retries; the plurality of processor settings including one or more of thread pool size, thread priority, thread affinity (CPU core assignment), trigger delay, stack size, timeout, throttling parameters, and heartbeat frequency; 17. The non-transitory computer readable storage medium of claim 16.

Images