State Management Asynchronous Runtime

Abstract

To provide a method, a system, and a device that enable optimum design, execution, and performance of an electronic device (barcode scanner).SOLUTION: A method includes identifying a plurality of image processing cycles associated with a temporal sequence of triggers. Each image processing cycle is created in response to one or more respective trigger events. The plurality of image processing cycles correspond to a plurality of parallel pipelines. The method also includes, for each parallel pipeline, pulling each cycle data container from a cycle data pool. A first image processing cycle is processed in a first parallel pipeline to generate first report data, independently of processing remaining image processing cycles in respective remaining parallel pipelines. The method further includes reporting the first report data to a client device coupled to an electronic device for further processing (e.g., storage, classification, analysis, and / or visualization).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The disclosed embodiments relate generally to electronic circuits, and more particularly to systems, apparatus, and methods for configuring hardware resources to process multiple image processing cycles in multiple parallel pipelines. [Background technology]

[0002] Computer processors implement parallelism, pipelining, software task overlap, and dedicated hardware acceleration to enhance their processing capabilities. These processing-enhancing techniques are often limited to highly repeatable cycles that perform functionally identical or similar jobs with little or no variation from cycle to cycle. Computer processors are configured using a register model, consistently applying a single configuration to drive runtime execution. For example, processor pipelines often have a fixed number of stages. Multiple stages are staggered, with stages in a current cycle often executing simultaneously with distinct stages in a previous cycle. It is beneficial to develop flexible and highly efficient parallel processing mechanisms while fully utilizing existing computer resources. Summary of the Invention

[0003] Various embodiments of the present application are directed to a core high-performance runtime engine configured to process image data in parallel pipelines for heavy-duty industrial machine vision applications. Machine vision applications are applied to electronic devices (e.g., imaging devices including one or more cameras) and perform functions including, but not limited to, image processing, code reading, product information extraction, and print quality verification. Specifically, a state-managed asynchronous pipelined architecture is applied to the image processing electronics of these machine vision applications. Multiple image processing cycles are executed in parallel in multiple pipelines, leveraging hardware capabilities without bottlenecks. Each image processing cycle includes multiple time-critical stages and operations that are arbitrarily triggered and executed alternately in multiple threads. In some embodiments, a trigger event is received while a previous image processing cycle is still in progress and corresponds to an entirely different image processing cycle assigned to a different pipeline that is executed independently of the previous image processing cycle. In some prior art implementations, a stage of an image processing job uses available hardware capacity and may cause other image processing jobs or other stages of the same image processing job to wait in a queue. Conversely, in this application, a state-managed, asynchronous pipeline architecture leverages available hardware capacity and delivers performance gains by executing multiple image processing cycles in independent parallel pipelines, with the performance gains scaling linearly with the number of processor cores in the imaging device.

[0004] In one aspect, an image processing method is implemented in an electronic device (e.g., an imaging device) having a memory and one or more processors. The method includes identifying a plurality of image processing cycles associated with a temporal sequence of triggers. Each image processing cycle is created in response to one or more respective trigger events. The method further includes assigning the plurality of image processing cycles to a plurality of parallel pipelines (e.g., each image processing cycle is assigned to a respective parallel pipeline) and, for each parallel pipeline, retrieving a respective cycle data container from a cycle data pool. The method further includes processing a first image processing cycle in a first parallel pipeline independently of the processing of the remaining image processing cycles in each remaining parallel pipeline to generate first report data. The method further includes providing the first report data to a client device, for example, for further processing (e.g., storage, classification, analysis, and / or visualization).

[0005] In some embodiments, the cycle data pool includes a predetermined number of cycle data containers before identifying the multiple image processing cycles. Once assigned to a respective image processing cycle, each parallel pipeline directly uses one of the predetermined number of cycle data containers without waiting for the creation of a new cycle data container. In some circumstances, the predetermined number of cycle data containers are fully used by different pipelines, and no cycle data containers can be pulled for a new pipeline. In some embodiments, processing the first image processing cycle further includes inputting image data associated with one or more respective trigger events into the first cycle data container, generating first cycle data including first report data, and storing the first cycle data in the first cycle data container.

[0006] In some embodiments, the imaging device is communicatively coupled to a client device (e.g., a laptop computer, desktop computer, or mobile phone) via a wired or wireless communications link. A user interface for the machine vision application enables the client device to present report data collected from the imaging device in a user-friendly manner. In some embodiments, the client device is located remotely from the imaging device and is coupled to the imaging device via a server. A web-based user interface enables the client device to remotely review report data reported by the imaging device via the server.

[0007] According to some embodiments, an electronic device includes one or more processors, a memory, and one or more programs stored in the memory, the programs configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described herein.

[0008] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by an electronic device having one or more processors and a memory, the one or more programs including instructions for performing any of the methods described herein.

[0009] Thus, a method, system and apparatus are disclosed that enables optimal design, execution and performance of a barcode scanner.

[0010] The various embodiments described above can be combined with other embodiments described herein. The features and advantages described herein are not all-inclusive, and many additional features and advantages will be apparent to those skilled in the art, particularly upon consideration of the drawings, specification, and claims. Furthermore, it should be noted that the language used herein has been selected primarily for ease of reading and explanation, and may not have been selected to define or encompass the entire inventive subject matter. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a perspective view of an electronic device (eg, a scanning device) according to some embodiments. [Figure 2] FIG. 2 is a block diagram of an exemplary electronic device, according to some embodiments. [Figure 3] FIG. 3 is an example of an image processing environment including an electronic device that processes image data using parallel pipelines, according to some embodiments. [Figure 4] FIG. 4 illustrates a process for processing image data in multiple parallel pipelines according to some embodiments. [Figure 5] FIG. 5 illustrates another example of a process for processing image data in multiple parallel pipelines, according to some embodiments. [Figure 6] FIG. 6 illustrates another example of a process for processing image data in multiple parallel pipelines, according to some embodiments. [Figure 7] FIG. 7 is a diagram illustrating multiple image processing cycles performed in multiple parallel pipelines, according to some embodiments. [Figure 8] 8 is a flow diagram of a method for processing image data according to some embodiments. Reference is made to the embodiments illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific details. DETAILED DESCRIPTION OF THE INVENTION

[0012] Various embodiments of the present application are directed to a core high-performance runtime engine configured to process image data in parallel pipelines for heavy-duty industrial machine vision applications. A state-managed, asynchronous pipeline architecture is applied to an electronic device (e.g., an imaging device) for image processing in these machine vision applications. Specifically, the electronic device identifies multiple image processing cycles associated with a temporal sequence of triggers, with each image processing cycle being created in response to one or more respective trigger events (e.g., image acquisition). The electronic device assigns the multiple image processing cycles to multiple parallel pipelines. For each parallel pipeline, an existing cycle data container is drawn from a cycle data pool. A first image processing cycle is processed in the first parallel pipeline to generate first report data independently of the remaining image processing cycles in each remaining parallel pipeline. The first report data is provided to a client device. In some embodiments, each image processing cycle includes multiple time-critical operations and stages that are optionally triggered and executed alternately in multiple threads of a corresponding pipeline. Such a state-managed, asynchronous pipeline architecture leverages available hardware capacity and independently executes multiple image processing cycles in parallel pipelines, resulting in significant performance gains that scale linearly with the number of processor cores (e.g., CPU, FPGA, NPU, and / or GPU) in the electronic device.

[0013] The primary goal of the state-managed, asynchronous pipeline architecture is to fully utilize hardware resources, including but not limited to processors, acquisition sources, input / output, communication channels, processing threads, and processes, to achieve maximum performance for the image processing system. Furthermore, this architecture significantly simplifies user intervention for runtime control, configuration, and monitoring, improving the user experience. In some embodiments, the processing pipeline and associated hardware and software settings may be modified at any time without generating significant overhead. In some embodiments, deep instrumentation enables self-analysis and tuning of the architecture automatically, enabling improvements in image processing performance with little or no user intervention.

[0014] FIG. 1 is a perspective view of an electronic device 100 (e.g., a scanning device) according to some embodiments. In some embodiments, the electronic device 100 may also be referred to as a code reader, barcode scanner, label scanner, optical scanner, or image capturing system. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). The electronic device 100 includes a housing 110 (e.g., a body or exterior case) for protecting components located inside the electronic device 100. In some embodiments, the housing 110 includes integrated fittings or brackets for holding the internal components in place. In some embodiments, the electronic device 100 includes a top cover 102 located on an upper side of the electronic device 100. In some embodiments, the top cover 102 is transparent or partially transparent.

[0015] In some embodiments, the electronic device 100 includes one or more distance sensors 104 located within the electronic device 100 (e.g., internal distance sensors). For example, referring to FIG. 1 , the distance sensors 104 are located inside the electronic device 100 (e.g., adjacent the top cover 102) and face the front end of the electronic device 100. In some embodiments, the distance sensors 104 are included in an aiming module. The distance sensors 104 project a beam of light onto a target when the target is placed on the electronic device 100 to assist in visual alignment of the target. This helps align the camera with the imaging plane or center point of the field of view. In some embodiments, each distance sensor 104 is a time-of-flight (TOF) sensor, an ultrasonic sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, or an infrared (IR) distance sensor. In some embodiments, the electronic device 100 includes two or more distance sensors 104, each of the same type (e.g., each of the two or more distance sensors is a TOF sensor). In some embodiments, electronic device 100 includes two or more distance sensors, at least two of which are of distinguishable types (e.g., electronic device 100 includes a TOF distance sensor and a radar sensor). In some embodiments, electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) the presence or absence of an object within a sensing area. The sensing area is designed for the proximity sensors to operate. In some embodiments, electronic device 100 determines the distance between a target and electronic device 100 using distance measurement techniques, such as an image focus finder, an analog-to-digital conversion (ADC) circuit, and / or a digital-to-analog conversion (DAC) circuit.

[0016] More specifically, in some embodiments, the distance sensor 104 is a TOF sensor. A TOF sensor measures the time elapsed between when a signal (e.g., a wave pulse, an LED pulse, a laser pulse, or infrared light) emitted from the sensor reflects off an object and returns to the sensor. Distance is then calculated using the speed of light in air and the time between the transmission and reception of the signal. In some embodiments, the distance sensor 104 is an ultrasonic sensor. An ultrasonic sensor or sonar sensor detects the distance to an object by emitting high-frequency sound waves. The ultrasonic sensor emits high-frequency sound waves toward the object and a timer is started. The object reflects the sound waves back toward the sensor. A receiver picks up the reflected waves and stops the timer. The time it takes for the sound waves to return is calculated against the speed of sound to determine the distance traveled. In some embodiments, the distance sensor 104 is a radar sensor. A radar sensor (e.g., a radar distance sensor) calculates the distance to an object by transmitting high-frequency radio waves (e.g., microwaves) and measuring the reflection of the waves from the object. In some embodiments, the radar sensor is configured to determine the distance, angle, and radial velocity of an object relative to the position of the electronic device 100. In some embodiments, the distance sensor 104 is a LiDAR sensor, which measures the range of an object through light waves from a laser (e.g., instead of radio or sound waves). In some embodiments, the distance sensor 104 is an infrared (IR) distance sensor. An IR distance sensor operates on the principle of triangulation and measures distance based on the angle of a reflected beam.

[0017] In some embodiments, electronic device 100 further includes a plurality of light sources 106 (e.g., eight light-emitting diodes (LEDs) in FIG. 1 ) mounted on a printed circuit board (PCB) 108. Light source 106 is also referred to as a lighting source, illumination source, or illuminator. In some embodiments, light source 106 is part of an illumination system of electronic device 100, which also includes illuminators (e.g., bright field illuminators and dark field illuminators), a reflector, and a lighting module. Details of the illumination system are described in U.S. Patent Application No. 14 / 298,659, entitled "Combination Dark Field and Bright Field Illuminator," filed June 6, 2014 (now U.S. Patent No. 8,989,569, issued March 24, 2015, which is incorporated herein by reference in its entirety).

[0018] In some embodiments, the light sources 106 have one or more lighting types. Examples of lighting types include, but are not limited to, LED light sources, laser light sources, and liquid crystal display (LCD) lights. Each lighting type has its own lighting characteristics, such as color (e.g., blue, red, or green) and / or intensity. The light sources 106 are mounted (e.g., soldered) to a PCB 108 located within the electronic device 100 (e.g., behind the top cover 102). The PCB 108 includes a front surface that faces the top cover 102 of the electronic device 100. In some embodiments, the light sources mounted to the front surface of the PCB 108 include both long-distance light sources and low-angle light sources.

[0019] In some embodiments, the electronic device 100 includes a camera 112. The lens of the camera 112 is exposed through an opening in the PCB 108 and is physically surrounded by light sources 106. The light sources 106 are grouped into multiple illumination units (e.g., a first illumination unit and a second illumination unit). Each illumination unit is configured to be independently controlled to illuminate a different region of the field of view of the camera 112. In one example, two light sources 106 near the corners of the top cover 102 are grouped together to form an illumination unit. The four illumination units are independently controlled to illuminate respective regions of the field of view of the camera 112, either sequentially or simultaneously.

[0020] In some embodiments, electronic device 100 further includes one or more indicators 114. Each indicator 114 is located on an edge of top cover 102 of electronic device 100 and is configured to be illuminated according to a light pattern in which a single color or different colors are displayed for a sequence of temporal durations determined based on a frequency. In some situations, the light pattern represents a message containing data or status of electronic device 100. For example, indicator 114 may continuously brighten red in response to detecting the presence of a product on or near the top cover, and may turn and remain green for a short period of time in response to successfully scanning a barcode displayed on the product. In some embodiments, each indicator 114 includes one or more LEDs from which light is emitted, the light being displayed on indicator 114 in a substantially uniform and homogeneous manner.

[0021] 2 is a block diagram of an exemplary electronic device 100, according to some embodiments. The electronic device 100 includes one or more distance sensors 104, as described above with respect to FIG. 1. In some embodiments, the one or more distance sensors 104 include one or more of a time-of-flight sensor, an ultrasonic sensor, a radar sensor, or a LiDAR sensor. In some embodiments, the electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) whether an object is present within a sensing area. The sensing area is designed to allow the proximity sensors to operate. In some embodiments, the electronic device 100 determines the distance between an object and the electronic device 100 using distance measurement techniques, such as an image focus finder, analog-to-digital conversion (ADC), and / or digital-to-analog conversion (DAC).

[0022] The electronic device 100 includes a light source 106. In some embodiments, the light source 106 includes a long-distance light source 262, a low-angle light source 264, and / or a dome light source 266, as described in FIG. 3 and in U.S. patent application Ser. No. 14 / 298,659, filed June 6, 2014, and entitled "Combination Dark Field and Bright Field Illuminator" (now U.S. Pat. No. 8,989,569, issued March 24, 2015, which is incorporated by reference herein in its entirety). In some embodiments, the light source 106 provides illumination with visible light. In some embodiments, the light source 106 provides illumination with invisible light (e.g., infrared or ultraviolet light).

[0023] In some embodiments, electronic device 100 includes a decoder 212 for decoding data contained in the barcode and transmitting the data to a computing device. In some embodiments, decoder 212 is part of software application 230. Details of the decoder 212 are described in U.S. patent application Ser. No. 14 / 298,659, filed Jun. 6, 2014, entitled "Combination Dark Field and Bright Field Illuminator" (now U.S. Pat. No. 8,989,569, issued Mar. 24, 2015, the entire contents of which are incorporated herein by reference).

[0024] In some embodiments, electronic device 100 includes one or more input interfaces 210 to facilitate user input. In some embodiments, electronic device 100 is a battery-operated device and includes a rechargeable battery. In this example, input interface 210 may include a charging port for charging the battery.

[0025] In some embodiments, the electronic device 100 includes a camera 112, which includes an image sensor 216 and a lens 218. The lens 218 directs the path of light rays and focuses them onto the image sensor 216 to recreate an image on the image sensor as accurately as possible. The image sensor 216 converts light (e.g., photons) into electrical signals that the electronic device 100 can interpret. In some embodiments, the lens 218 is an optical lens made from glass or other transparent material. In some embodiments, the lens 218 is a liquid lens composed of an optical liquid material, whose shape, focal length, and / or working distance change when a current or voltage is applied to the liquid lens. In some embodiments, the electronic device 100 (e.g., via the one or more processors 202) uses distance information obtained by the distance sensor 104 to determine an optimal current or voltage to apply to the liquid lens 218 so that it has an optimal focal length for decoding barcode data contained in the image. In some embodiments, the camera 112 is configured to capture images in color. In some embodiments, the camera 112 is configured to capture images in black and white.

[0026] The electronic device 100 also includes one or more processors (e.g., CPUs) 202, one or more communication interfaces 204 (e.g., network interfaces), memory 206, and one or more communication buses 208 (sometimes called chipsets) for interconnecting these components.

[0027] In some embodiments, electronic device 100 includes radios 220. Radios 220 enable one or more communication networks, allowing electronic device 100 to communicate with other devices, such as computing devices or servers. In some embodiments, radios 220 are capable of data communication using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, Ultrawide Band (UWB), and / or software defined radio (SDR)), custom or standard wired protocols (e.g., Ethernet or HomePlug), and / or any other suitable communication protocols, including communication protocols not yet developed as of the filing date of this application.

[0028] Memory 206 includes high-speed random-access memory such as DRAM, SRAM, DDRRAM, or other random-access solid-state memory devices. In some embodiments, memory includes non-volatile memory such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. In some embodiments, memory 206 includes one or more storage devices located remotely from one or more processors 202. Memory 206, or alternatively, non-volatile memory within memory 206, includes a non-transitory computer-readable storage medium. In some embodiments, memory 206, or the non-transitory computer-readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof: Operating logic 222, which includes procedures for handling various basic system services and for performing hardware-dependent tasks. A communications module 224 (e.g., a wireless communications module) that connects to and communicates with other network devices (e.g., local networks such as routers providing Internet connectivity, network-attached storage devices, network routing devices, server systems, computing devices, and / or other connected devices) coupled to one or more communications networks via one or more communications interfaces 204 (e.g., wired or wireless). An application 230 that acquires an image including a label (e.g., a barcode), decodes the label, and controls one or more components of electronic device 100 and / or other connected devices according to the determined state. In some embodiments, application 230 includes: An illumination module 232 that selects and deploys a sequence of one or more light sources 106 and / or illumination patterns 234 for the current read cycle (e.g., based on distance measurements, such as direct or indirect measurements from the distance sensor 104). A distance module 236 that determines (eg, selects) a sequence of focal lengths to be employed during the current read cycle based on distance measurements from one or more distance sensors 104 . An exposure and gain module 238 that samples the image 244 captured by the camera 112. An image acquisition and processing module 240 that acquires and processes images, for example, according to the processes shown in any of FIGS. A decoder 212 for decoding the data contained in the barcode and transmitting it to a computing device. Data 242 of the electronic device 100. Data 242 includes, but is not limited to: Image data 244 (e.g., camera data). Symbology data 246 (e.g., type of code, such as barcode). Device settings 248 of the electronic device 100, such as default options, image acquisition settings (e.g., exposure and gain settings) and recommended user settings. 250 user settings, such as the recommended shade of lenses (e.g., photochromic lenses). Sensor data 252 obtained (e.g., measured) from the distance sensor 104 and / or other sensors included in the electronic device 100.

[0029] In some embodiments, the distance sensor 104 is monitored by the illumination module 232. When a user initiates a current read cycle, the distance sensor 104 identifies a distance field (e.g., near field, mid-field, or far field) corresponding to the object's position. The illumination module 232 selects an illumination sequence corresponding to the distance field to execute. If a good read was achieved in the previous read cycle (e.g., a good read from the third illumination pattern in the near-field illumination sequence) and the current read cycle has the same distance field as the previous read cycle, the application 230 starts the current read cycle using the values ​​of the previous good read (e.g., the third illumination pattern in the near-field illumination pattern, the previous focus position, exposure, and / or gain) before starting the illumination sequence from the beginning. Because users typically read many similar parts, the device can achieve a good read more quickly if it starts with known good settings from a previous decode operation. If the previous settings did not result in a good read, the illumination sequence for the current distance field starts from the beginning, repeating capture-after-capture with each sequence.

[0030] In some embodiments, the exposure and gain module 238 rejects images that do not fall within predetermined attribute ranges for "brightness" and / or "sharpness" (e.g., rejected images are not processed by the image acquisition and processing module 240). In some embodiments, the exposure and gain module 238 updates the image acquisition settings (e.g., exposure and gain) for the next image capture to provide optimal "brightness" for image processing.

[0031] In some embodiments, after an image is captured (e.g., using camera 112), electronic device 100 (e.g., via application 230) evaluates the quality of the captured image. For example, electronic device 100 reads (e.g., determines) the image's sharpness value, average light mean value, and / or average dark mean value and determines whether to accept or reject the image. If the result does not meet or exceed predetermined target values, the image is rejected and another image is recaptured. If the result meets or exceeds predetermined target values, the image is processed (e.g., by image acquisition and processing module 240).

[0032] As an example, in some embodiments, a good quality image is an image sample having a light average value between 100 and 170 (out of a range of 0 to 255), a dark average value between 20 and 80 (out of a range of 0 to 255), and a sharpness value above 6000 (out of a range of 0 to approximately 12,000).

[0033] In some embodiments, data collected during image sampling (eg, evaluation) is captured and added (eg, as data 242).

[0034] In some embodiments, after qualifying an image, electronic device 100 (e.g., via application 230) determines whether to adjust exposure or gain settings (e.g., using a light mean correction path or a dark mean correction path) for the next image. If it decides to do so, electronic device 100 collects target light and dark mean values ​​for comparison, develops a proportional-integral (PI) controller transfer function, and calculates the necessary changes in exposure to obtain an ideal exposure for the next image.

[0035] In some embodiments, once an image is successfully decoded, the exposure, gain, and focus values ​​are fed back to the application 230. On the next read cycle, the application 230 checks whether these decode settings are pending. If so, the electronic device 100 attempts to load the camera settings and any previous settings rather than calculating the next setting configuration. If the previous decode settings are used, the application 230 samples the image for data but does not adjust the values ​​of the feedback controllers.

[0036] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned memory devices and correspond to sets of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 206 stores a subset of the modules and data structures identified above. Additionally, memory 206 can store additional modules or data structures not listed above. In some embodiments, a subset of the programs, modules, and / or data stored in memory 206 is stored in and / or executed by a server system and / or an external device (e.g., a computing device).

[0037] FIG. 3 illustrates an exemplary image processing environment 300 including an electronic device 100 that processes image data using a parallel pipeline, according to some embodiments. In some embodiments, the electronic device 100 is an imaging device. In some embodiments, the electronic device 100 is a code reader, a barcode scanner, a label scanner, or an optical scanner. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). In some embodiments, the electronic device 100 is configured to acquire image data including one or more images, process the image data using a parallel pipeline, and provide report data 312 generated from the image data to one or more client devices 302 (e.g., device 302A, device 302B, device 302C, or device 302D). The one or more client devices 302 may be, for example, a desktop computer, a tablet computer, a mobile phone, or an intelligent, multi-sensing, network-connected home device (e.g., a display assistant device). Each client device 302 can collect report data 312 from electronic device 100, receive user input, execute user applications, and present report data 312 or other information in its user interface. In some embodiments, the user application includes an interactive user application. The user interface of the interactive user application is displayed on the client device for receiving user input related to electronic device 100 and for visualizing report data 312 generated by electronic device 100.

[0038] The electronic device 100 is configured to enable multiple parallel pipelines. The electronic device 100 identifies multiple image processing cycles associated with a temporal sequence of triggers, with each image processing cycle being created in response to one or more respective trigger events (e.g., one or more image capture operations). The multiple image processing cycles are assigned to the multiple parallel pipelines. Existing cycle data containers are drawn directly from each parallel pipeline's cycle data pool. The electronic device processes the multiple image processing cycles of the multiple parallel pipelines and generates respective report data independently of each other. In some embodiments, the temporal sequence of triggers corresponds to an ordered sequence of images processed during the image processing cycle. The image processing cycle report data 312 is generated separately from the multiple parallel pipelines, independent of the order of the sequence of images. In some circumstances, the report data of the image processing cycles is provided to the client device 302 organized according to the order of the sequence of the corresponding images.

[0039] In some embodiments, a user application implemented on the client device 302 is driven by a first programming language, and multiple image processing cycles are executed on the electronic device 100 by a second programming language different from the first programming language. The multiple image processing cycles are configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first and second programming languages. For example, the intermediate data representation is implemented in JSON (JavaScript Object Notation). The user application includes a web-based user interface, and the first programming language includes JavaScript. A runtime associated with the image processing cycles uses a second language (e.g., C++). JSON is the native format of the runtime by building JSON support into almost all runtime C++ objects, allowing JavaScript programs to understand workflow or report formats automatically used by the runtime.

[0040] In some embodiments, the report data 312 or user input is processed locally at the client device 302 and / or remotely at the server 304. The one or more servers 304 provide system data (e.g., boot files, operating system images, and user applications) to the client device 302 and, in some embodiments, process the report data 312 and user input received from one or more client devices 302 when the user applications execute on the client device 302. In some embodiments, the data processing environment 300 further includes a storage device 306 for storing data related to the server 304, the client device 302, the electronic device 100, and the user applications executing on the client device 302. For example, the storage device 306 can store video content, static visual content, and a product database.

[0041] The one or more servers 304, the one or more client devices 302, the electronic device 100, and the storage device 306 are communicatively coupled to each other via one or more communications networks 308. The communications network 308 is the medium used to provide a communications link between these devices and computers connected together within the data processing environment 300. The one or more communications networks 308 may include connections, such as wired, wireless communications links, or fiber optic cables. Examples of the one or more communications networks 308 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. The one or more communication networks 308 are optionally implemented using any known network protocol, including various wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. Connection to the one or more communication networks 308 may be established directly (e.g., using a 3G / 4G connection to a wireless carrier), through a network interface 310 (e.g., a router, switch, gateway, hub, or intelligent, dedicated whole-home control node), or through any combination thereof.Thus, one or more communications networks 308 represent the Internet, a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, government, educational, and other electronic systems that route data and messages.

[0042] In some embodiments, electronic device 100 is communicatively coupled to client device 302 (302A) via wired communication link 312. In some embodiments, electronic device 100 is communicatively coupled to client device 302 (302A) via a local communication network 314 enabled by network interface 310. Both electronic device 100 and client device 302 (e.g., 302A) are located at the same location (e.g., a warehouse or factory). In some embodiments, electronic device 100 is remote from client device 302 and is communicatively coupled to client device 302 (e.g., device 302B, device 302C, and / or device 302D) via at least a WAN.

[0043] FIG. 4 illustrates a process 400 for processing image data in multiple parallel pipelines 402 (e.g., 402A, 402B, and 402C) according to some embodiments. The process 400 is implemented by the electronic device 100 having the memory 206 and one or more processors 202. For example, the process 400 is executed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 identifies multiple image processing cycles 404 associated with a temporal sequence of triggers (e.g., image acquisition). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is generated in response to one or more respective trigger events (e.g., capture of a single image or capture of a set of images). The multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from a cycle data pool 408. In some embodiments, the cycle data pool 408 includes a predetermined number of cycle data containers 406 prior to identifying the multiple image processing cycles 404. When assigned to a respective image processing cycle 404, each parallel pipeline 402 directly uses one of the predetermined number of cycle data containers 406 without waiting for the generation of a new cycle data container 406. This expedites the start and execution of each parallel pipeline 402. In some situations, a cycle data container 406 may be completely used by a different pipeline 402 and may not be available for the assigned pipeline 402 in a new image processing cycle 404.

[0044] After the multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402, the multiple image processing cycles 404 are processed independently of one another in the multiple parallel pipelines 402, thereby independently generating report data 312 (e.g., 312A, 312B, and 312C). The report data 312 is provided to the client device 302 for further processing (e.g., storage, classification, analysis, and / or visualization). In some embodiments, the client device 302 is coupled to the electronic device 100 via a local communications network and executes an interactive user application 412. For example, the client device 302 and the electronic device 100 are located in a factory automation network without Internet access. The interactive user application 412 includes a user interface configured to locally configure, adjust, and debug the electronic device 100 and associated workflows and jobs. In some embodiments, the interactive user application 412 is web-based and implemented on the client device 302, which is coupled to a WAN (e.g., the Internet). For example, the web-based user interface is enabled by a browser-based user application 412. The browser-based user application 412 is served directly from the client device 302 and communicates with the electronic device 100 using HTTP and web sockets. A web-based user interface is presented to allow the client device 302 to remotely configure, tune, and debug the electronic device 100 and its associated workflows and jobs.

[0045] In some embodiments, process 400 enables a highly interactive browser-based user experience by exchanging data structures between a web-based user interface of a user application 412 and the runtime of multiple parallel pipelines 402. In some embodiments, user application 412 is implemented based on Hypertext Transfer Protocol (HTTP), and client device 302 executing user application 412 comprises an Internet of Things (IOT) device. In one example, configuration of user application 412 and pipelines 402 is via a data exchange scheme using JavaScript Object Notation (JSON), allowing a JavaScript-driven web-based user interface to natively interact with a core runtime implemented in C++ or other non-dynamic computer language.

[0046] In some embodiments, the first trigger event of the first image processing cycle 404A precedes the second trigger event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Also, in some embodiments, whichever of the first report data 312A or the second report data 312B becomes available first is reported to the client device 302 first, regardless of the temporal order of the first and second trigger events. If the second report data 312B for the second image processing cycle 404B is available before the first report data 312A for the first image processing cycle 404A, the second report data 312B is provided to the client device 302 before the first report data 312A. If the second report data 312B is available after the first report data 312A, the second report data 312B is provided to the client device 302 after the first report data 312A.

[0047] For each parallel pipeline 402, an existing cycle data container 406 is drawn from the cycle data pool 408 and used to store cycle data 414 associated with the corresponding image processing cycle 404. For example, a first image processing cycle 404A is implemented in the first parallel pipeline 402A. The first cycle data 414A includes image data associated with one or more respective trigger events of the first image processing cycle 404A and intermediate data generated by and passed between stages of the first image processing cycle 404A. In some embodiments, the first cycle data 414A includes first report data 312A generated from the intermediate data and reported to the client device 302. In some embodiments, all of the image data, intermediate data, and report data of the first image processing cycle 404A are reported to the client device 302. In some embodiments, only the report data 312A of the first image processing cycle 404A is reported to the client device 302.

[0048] In one example, the electronic device 100 is configured to recognize a barcode in an image. In response to a first trigger event 602 in which an image including a barcode is captured, a first image processing cycle 404A is generated. First cycle data 414 associated with the first image processing cycle 404 includes a serial number corresponding to the barcode. First report data 312A generated from the first image processing cycle 404 includes a product name and a price of the product.

[0049] In some prior art implementations, each pipeline has a fixed number of stages, with the stages executing alternately, such that the first stage of a current cycle executes simultaneously with the second stage of the previous cycle preceding the current cycle. In contrast, in various embodiments herein, parallel pipelines 402 have no such restriction, and each parallel pipeline 402 is switched on for a corresponding image processing cycle 404 without overhead. While the stages of each cycle execute within the same pipeline 402, the pipeline 402 contains a managed number of threads and is associated with a cycle data container 406 drawn from a cycle data pool 408.

[0050] FIG. 5 illustrates another exemplary process 500 for processing image data in multiple parallel pipelines 402, according to some embodiments. The process 500 is performed by the electronic device 100, which has the memory 206 and one or more processors 202 (FIG. 2). For example, the process 500 is performed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 provides an acquisition service 502 that identifies multiple image processing cycles 404 associated with a temporal sequence of triggers (e.g., image acquisition). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more respective trigger events (e.g., capturing a single image, capturing a set of images). Based on the temporal order of the triggers, the multiple image processing cycles are placed in a runnable cycle queue 504. The multiple image processing cycles 404 are assigned to the multiple parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from the cycle data pool 408. Cycle data 414 is associated with or generated for each image processing cycle 404 and stored in the corresponding cycle data container 406 in the course of managing and executing the image processing cycle 404.

[0051] In some embodiments, the first image processing cycle 404A is implemented in a first parallel pipeline 402A and inspects an input image, e.g., to detect stains on a drug bottle label. The first image processing cycle 404A includes multiple stages, e.g., an inspection stage 506, a post-inspection stage 508, a report generation stage 510, and a data return stage 512. The input image is inspected in the inspection stage 506 to generate intermediate data that is stored in the cycle data container 406A along with the cycle data 414. The intermediate data is further processed in the post-inspection stage 508 to provide additional information. The additional information is used to generate first report data 312A in the report generation stage 510. The first report data 312A is placed in a queue of pending data 514 and provided to the client device 302 in the data return stage 512. In some embodiments, the report data 312 is organized in the queue of pending data 514 based on the chronological order of associated trigger events. Also, in some embodiments, report data 312 is organized into queues of pending data 514 based on the chronological order of its own availability, regardless of the chronological order of the associated triggering events.

[0052] Each stage of the image processing cycle 404A is not limited to a fixed hardware allocation but has flexible access to the hardware resources allocated to the corresponding parallel pipeline 402A. In other words, each stage of the image processing cycle 404A can be accelerated using all of the allocated pipeline hardware resources or can be executed in parallel with one or more of the remaining stages of the image processing cycle 404A.

[0053] In some embodiments, the plurality of image processing cycles 404 assigned to the plurality of pipelines 402 include a plurality of first image processing cycles 404-1. The runnable cycle queue 504 further includes a plurality of second image processing cycles 404-2 following the first image processing cycles 404-1. When the plurality of pipelines 402 are fully occupied by the plurality of first image processing cycles 404-1, a status signal 516 (e.g., wait for idle) is sent to the runnable cycle queue 504 to request that the runnable cycle queue 504 hold the plurality of second image processing cycles 404-2 until the plurality of pipelines 402 have completed processing the plurality of first image processing cycles 404-1.

[0054] High-precision instrumentation is built in to provide a deep, intuitive understanding of the effects of adjusting various performance parameters, including thread, process, and core loads, and hardware utilization. This functionality is designed to enable a real-time performance visualizer in a web-based user interface. It also enables on-the-fly deep analysis, with the potential to automatically adjust parameters for maximum performance without user intervention. In some embodiments, while processing the first image processing cycles 404-1 in the parallel pipelines 402, the electronic device 100 dynamically adjusts one or more parameters, including the number of pipelines, thread pool size, thread priority, delay, throttling parameters, core allocation, acquisition parameters, lighting, and camera focus. The second image processing cycles 404-2 are assigned to the parallel pipelines 402 and processed in the parallel pipelines 402 based on the one or more adjusted parameters. Specifically, in one example, the electronic device 100 monitors the utilization of central processing unit (CPU) cores. Based on the CPU core utilization, the electronic device 100 dynamically adjusts the computational resources allocated to each of the parallel pipelines. In this manner, parameters controlling hardware usage limits (e.g., CPU core utilization) are dynamically adjustable and can be changed during runtime. By measuring and automatically adjusting these parameters, it is possible to tailor the runtime for best performance depending on the situation.

[0055] FIG. 6 illustrates another exemplary process 600 for processing image data in multiple parallel pipelines 402, according to some embodiments. The process 600 is implemented by the electronic device 100 having the memory 206 and one or more processors 202 (FIG. 2). For example, the process 600 is executed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 provides an acquisition service 502 that identifies multiple image processing cycles 404 associated with a temporal sequence of triggers 602 (e.g., image acquisition) for an engine queue thread 604. Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more respective trigger events 602 (e.g., capturing a single image, capturing a set of images). In some embodiments, the electronic device 100 further includes a configuration service 606 that provides a job configuration that is applied to define (operation 608) the image processing cycles 404 for processing image data captured by the temporal sequence of triggers 602. Based on the temporal order of the triggers 602, the image processing cycles 404 are placed in the runnable cycle queue 504. The image processing cycles 404 are assigned to the parallel pipelines 402. For each parallel pipeline 402, an existing cycle data container 406 (e.g., 406A) is drawn from the cycle data pool 408. In the course of managing and executing the image processing cycles 404, cycle data 414 is associated with or generated for each image processing cycle 404 and stored in the corresponding cycle data container 406.

[0056] A trigger event 602 includes the acquisition of a single image, which is processed in a corresponding image processing cycle 404. The single image is optionally processed through multiple stages 610 (e.g., 610A and 610B) managed by a single thread 612 or multiple threads 612 of the image processing cycle 404. In other words, in some situations not shown, a first stage 610A is performed at least partially in parallel with a second stage 610B. In some embodiments, multiple trigger events 602 include the acquisition of multiple images. In response to multiple trigger events 602, the image processing cycle 404 executes to process multiple images associated with the multiple trigger events 602 in multiple stages 610. The multiple stages 610 are optionally processed by a single thread 612 or multiple threads 612 of the image processing cycle 404. One of the multiple images is optionally processed through multiple stages managed by a subset of the associated single thread 612 or multiple threads 612 of the image processing cycle 404. In one example, the image processing stage 610B is implemented in a subset of the parallel threads 612. In some embodiments, the electronic device 100 implements each of the successive stages of the image processing cycle 404 sequentially using all or a portion of the first hardware resources allocated to the corresponding parallel pipelines 402.

[0057] Note that multiple images are input to the parallel pipelines 402, with corresponding cycle data containers 406 being provided to the parallel pipelines 402. The cycle data containers 406 optionally store raw image data for multiple images or pre-processed image data for multiple images (e.g., filtered images, corrected images, associated image features). In some embodiments, an image signal processor (ISP) is configured to filter or correct the raw image data captured by the camera. Each image processing cycle 404 is configured to process the raw image data or pre-processed image data.

[0058] In some embodiments, the first image processing cycle 404 includes a first thread 612A and a second thread 612B. The second thread 612B optionally starts before or after the first thread 612. The first thread 612A generates a first subset of first cycle data 414. The second thread 612B starts before the generation of the first subset of first cycle data 414 and is suspended to wait until the first subset of first cycle data 414 has been generated and provided by the first thread 612A.

[0059] In some embodiments, a workflow template is loaded into a first parallel pipeline 402A. The stages 610 of the first image processing cycle 404A are performed uninterrupted by the remaining image processing cycles (e.g., 402B and 402C of FIG. 4). The stages 610 of a different image processing cycle 404 may be performed in a second parallel pipeline 402B using the same or a different workflow template. While these two image processing cycles are performed separately in the two pipelines 402, there is no overhead of switching between workflow templates.

[0060] In some embodiments, the stages 610 of an image processing cycle 404 are forced to follow a temporal ordering with the previous image processing cycle. This forces certain operations (e.g., transmission of result data) to occur sequentially between cycles. For example, both image processing cycles 404A and 404B are processed by the first parallel pipeline 402A, and the second image processing cycle 404B follows the first image processing cycle 404A. The stages 610 of the second image processing cycle 402B are forced to follow a temporal ordering with the first image processing cycle 402A. The first report data 312A generated by the first image processing cycle 402A is optionally used by the second image processing cycle 402B.

[0061] The electronic device 100 includes a state-management asynchronous runtime engine 614 configured to keep the runtime in a known state. Instead of individual commands to change parameters or having to set a mode to perform an operation, the runtime state is always consistent and can only be changed to a new state (or not changed if the new state is not acceptable). This eliminates much of the complexity of managing the user experience (e.g., undo is trivial). In some embodiments, the state used by the image processing cycle 404 is contained in cycle data 414 stored in a cycle data container 406. The cycle data 414 flows through the stages and operations of the image processing cycle 404 implemented in the parallel pipeline 402. The application of cycle data in the cycle data container 406 maximizes parallelism with other services, such as image acquisition. The cycle data container 406 contains all the information necessary to complete the purpose of the corresponding image processing cycle 404 and can be completely replaced for the new purpose of the next image processing cycle 404.

[0062] In some embodiments, cycle data 414 further includes an input queue for data acquired or generated during execution of parallel pipelines 402. For example, image acquisition is optionally overlapped with other processing, and images are pushed to a queue in cycle data 414 when stage 610 needs image data and may be blocked if the data has not yet arrived.

[0063] 7 is a temporal diagram 700 of multiple image processing cycles 404 implemented in multiple parallel pipelines 402, according to some embodiments. Each of the multiple image processing cycles 404 is processed in a respective parallel pipeline 402 independently of the remaining image processing cycles in the respective remaining parallel pipeline.

[0064] 8 is a flow diagram of a method 800 for processing image data, according to some embodiments. The method is implemented in electronic device 100, and in particular in image acquisition and processing module 240 of electronic device 100. In some embodiments, electronic device 100 includes one or more cameras 112 configured to capture images. In some embodiments, electronic device 100 is coupled to a camera or a different electronic device having a camera, and configured to acquire image data including a series of images.

[0065] The electronic device 100 identifies (802) a plurality of image processing cycles 404 associated with the temporal sequence of triggers 602. Each image processing cycle 404 is created in response to one or more respective trigger events. Each trigger event includes the acquisition of a respective image. The electronic device 100 assigns (804) the plurality of image processing cycles 404 to a plurality of parallel pipelines 402. For each parallel pipeline 402, a respective cycle data container 406 is drawn from a cycle data pool 408 (806). The electronic device 100 processes (808) the first image processing cycle 404A of the first parallel pipeline 402A independently of the processing of the remaining image processing cycles 404 (e.g., 404B and 404C of FIG. 4 ) in each remaining parallel pipeline 402, and generates first report data. The first report data is provided (810) to the client device 302 (e.g., 302A of FIG. 3 ), for example, for display on a web-based user interface of the client device 302.

[0066] In some embodiments, a trigger event includes the acquisition of a single image, and the single image is processed in a corresponding image processing cycle 404. Also, in some embodiments, multiple trigger events include the acquisition of multiple images, and in response to the multiple trigger events, the image processing cycle 404 is executed to process the multiple images associated with the multiple trigger events. In a first processing cycle 404A, multiple trigger events are processed by multiple threads. In some situations, a first thread 612A (FIG. 6) generates a first subset of first cycle data 414A that is applied by a second thread 612. The second thread 612 is started and suspended to wait until the first subset of first cycle data 414A is generated and available to the first thread.

[0067] In some embodiments, the cycle data pool 408 includes a predetermined number of cycle data containers 406 before identifying the multiple image processing cycles 404. When assigned to a respective image processing cycle 404, each parallel pipeline 402 directly uses one of the predetermined number of cycle data containers 406 without waiting for the creation of a cycle data container 406. This expedites the start and execution of each parallel pipeline 402. In some situations, the predetermined number of cycle data containers are fully used by different pipelines 402, and a cycle data container 406 cannot be retrieved for the pipeline 402 assigned to the new image processing cycle 404.

[0068] In some embodiments, processing the first image processing cycle 404A further includes inputting (812) image data associated with one or more respective trigger events associated with the first image processing cycle 404A into the first cycle data container 406A, generating (814) first cycle data 414A including the first report data 312A, and storing (816) the first cycle data 414A in the first cycle data container 406A. In some embodiments, after the first image processing cycle 404A is completed, the data stored in the first cycle data container 406A is purged and the first cycle data container 406A is returned to the cycle data pool 408 to await retrieval and use by the next parallel pipeline 402. In some embodiments, the report data 312A reported to the client device 302 for the first image processing cycle 404A includes all of the first cycle data 414A and the image data. The first cycle data includes all intermediate or final data generated by the first parallel pipeline 402A. The client device 302, for example, stores the report data 312 in a database (e.g., storage 306), visualizes the report data 312, and controls the report data 312.

[0069] In some embodiments, the first image processing cycle 404A includes a first thread 612A and a second thread 612B. The first image processing cycle 404A is processed in the first parallel pipeline 402A by, in sequence, generating a first subset of first cycle data by the first thread 612A, starting the second thread 612B before generating the first subset of first cycle data, pausing the second thread 612B, and waiting until the first subset of first cycle data has been generated by the first thread 612A.

[0070] In one example, the first image processing cycle 404A is generated in response to a first trigger event. The first trigger event includes capturing an image including a barcode. The first cycle data includes a serial number corresponding to the barcode. The first report data includes a product name and a price. In another example, the first image processing cycle 404A is generated in response to a first trigger event. The first trigger event includes capturing an image of a label affixed to a medication bottle. In some embodiments, the first report data includes an image of the label. The client device 302 receives the image of the label, determines that the label in the image is damaged, and displays the image of the label with a message indicating that the medication bottle label is damaged. In some embodiments, the electronic device 100 determines that the label in the image is damaged and generates first report data 312A including a message indicating that the medication bottle label is damaged. The client device 302 receives the message and displays the message without the image of the label.

[0071] In some embodiments, the first cycle data 312A includes intermediate data generated by and passed between multiple stages of the first image processing cycle 404A and used to generate the first report data, with multiple data items of the intermediate data forming a queue and stored sequentially in the first cycle data container 406A.

[0072] In some embodiments, the temporal sequence of triggers corresponds to an ordered sequence of images processed during multiple image processing cycles 404 allocated to multiple parallel pipelines 402. A first image is acquired before a second image, and report data 312 corresponding to the first image is provided to the client device before report data 312 corresponding to the second image.

[0073] In some embodiments, the first image processing cycle 404A is generated in response to multiple trigger events, each of which includes the acquisition of a respective image. In some circumstances, the pipeline 402 begins processing a trigger event that precedes the first image processing cycle 404A while waiting for a later trigger event.

[0074] In some embodiments, the first image processing cycle 404A includes a first stage 610A and a second stage 610B (FIG. 6). The first image processing cycle 404A in the first parallel pipeline 402A is processed by performing a portion of the first stage 610A and a portion of the second stage 610B in parallel.

[0075] In some embodiments, the first image processing cycle 404A includes a series of stages (e.g., 506-512 in FIG. 5) that are processed in the first parallel pipeline 402A by sequentially performing each of the series of stages using first hardware resources allocated to the first parallel pipeline 402A.

[0076] In some embodiments, following the first image processing cycle 404A, the electronic device processes a second image processing cycle 404A in a second parallel pipeline 402B, which is different from the plurality of image processing cycles 404. After providing the first report data 312A to the client device, the electronic device 100 provides second report data to the client device 302. In some embodiments, the second report data is visualized on a user interface of the client device 302. In some embodiments, the first trigger event of the first image processing cycle 404A precedes the second trigger event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Also, in some embodiments, regardless of the temporal order of the first and second trigger events, the first report data 312A or the second report data 312B that becomes available first is reported first to the client device 302. If the second report data 312B of the second image processing cycle 404B is available before the first report data 312A of the first image processing cycle 404A, the second report data 312B is provided to the client device 302 before the first report data 312A. If the second report data 312B is available after the first report data 312A, the second report data 312B is provided to the client device 302 after the first report data 312A.

[0077] In some embodiments, the first image processing cycle 404A is performed in the first parallel pipeline 402A by loading a workflow template into the first parallel pipeline 402A and staging multiple stages of the first image processing cycle 404A according to the workflow template without being interrupted by the remaining image processing cycles 404.

[0078] In some embodiments, the electronic device 100 monitors the utilization of a central processing unit (CPU) core. Based on the utilization of the CPU core, the respective computational resources distributed among the multiple parallel pipelines 402 are dynamically adjusted. This is done automatically without user intervention. High-precision instrumentation is built in, allowing deep and intuitive understanding of the effects of adjusting various performance parameters, including thread, process, and core loads, and hardware utilization. In some embodiments, computer performance is visualized in a user interface on the client device 302. Computer performance is analyzed in real time during execution, potentially automatically adjusting parameters to improve performance without user intervention.

[0079] In some embodiments, the plurality of image processing cycles 404 includes a plurality of first image processing cycles 404-1 (FIGS. 5 and 6). While processing the plurality of first image processing cycles 404-1 in the plurality of parallel pipelines 402, the electronic device 100 dynamically adjusts one or more parameters of the number of pipelines, thread pool size, thread priority, delay, throttling parameters, core allocation, acquisition parameters, lighting, and camera focus. A plurality of second image processing cycles 404-2 follow the plurality of first image processing cycles 404-1. The electronic device allocates the plurality of second image processing cycles 404 to the plurality of parallel pipelines 402. Based on the one or more adjusted parameters, the plurality of second image processing cycles 404-2 are processed in the plurality of parallel pipelines 402.

[0080] In some embodiments, the client device 302 executes an interactive user application. The electronic device 200 enables display of a user interface and displays the first report data on the user interface of the client device 302. Furthermore, in some embodiments, the user application is driven by a first programming language, and the plurality of image processing cycles 404 are executed by a second programming language different from the first programming language. The plurality of image processing cycles 404 are configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first and second programming languages. In some embodiments, the interactive user application is implemented locally on the client device 302, which is coupled to the electronic device via a wired communication link or a local communication network. In some embodiments, the interactive user application is web-based and implemented on the client device 302, which is coupled to the electronic device via a WAN (e.g., the Internet). For example, the web-based user interface is enabled in a browser-based application served directly from the client device 302 and communicating with the electronic device 100 using HTTP and web sockets. The user interface is useful for defining, configuring, adjusting, and debugging the electronic device 100 and associated workflows and jobs.

[0081] In some embodiments, the first image processing cycle 404A is implemented in an electronic device 100 that is coupled to a client device via a local communications network.

[0082] Each of the above-identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned memory devices and corresponds to a set of instructions for performing the functions described above. The above-identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; various subsets of these modules may be combined or rearranged in various embodiments. In some embodiments, memory 206 stores a subset of the above-identified modules and data structures. Additionally, memory 206 can store additional modules or data structures not described above.

[0083] The terminology used in the description of the invention herein is for the purpose of describing particular implementations only and is not intended to be limiting of the invention. As used in the description of the invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will further be understood that the terms "comprises" and "comprising," when used herein, specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0084] As used herein, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" refers to both "based only on" and "based at least on."

[0085] As used herein, the term "exemplary" means "serving as an example, instance, or illustration" and does not necessarily imply a preference or superiority of the example over other configurations or implementations.

[0086] As used herein, the term "and / or" includes any combination of the listed elements. For example, "A, B, and / or C" includes the following sets of elements: A only, B only, C only, A and B without C, A and C without B, B and C without A, and combinations of all three elements: A, B, and C.

[0087] The above description has been set forth with reference to specific implementations for purposes of explanation. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, so that others skilled in the art can utilize the invention to its fullest extent and in various implementations with various modifications suited to the particular uses contemplated.

Claims

1. 1. A method of image processing implemented in an electronic device having a memory and one or more processors, comprising: identifying a plurality of image processing cycles associated with a temporal sequence of triggers, each image processing cycle being created in response to one or more respective trigger events; Allocating a plurality of image processing cycles to a plurality of parallel pipelines; for each parallel pipeline, drawing a respective cycle data container from the cycle data pool; processing a first image processing cycle in a first parallel pipeline to generate first report data independently of the processing of remaining image processing cycles in each remaining parallel pipeline; providing the first report data to the client device; Equipped with the plurality of image processing cycles includes a plurality of first image processing cycles; dynamically adjusting one or more parameters of a number of pipelines, a thread pool size, thread priorities, delays, throttling parameters, core allocations, acquisition parameters, lighting, and camera focus while processing the first plurality of image processing cycles in the parallel pipelines; assigning a plurality of second image processing cycles subsequent to the plurality of first image processing cycles to the plurality of parallel pipelines; processing the plurality of second image processing cycles in the plurality of parallel pipelines based on the one or more adjusted parameters.

2. one or more processors; a memory storing one or more programs configured to be executed by the processor; Equipped with the one or more programs: instructions for identifying a plurality of imaging cycles associated with a temporal sequence of triggers, each imaging cycle being created in response to one or more respective trigger events; allocating the image processing cycles to a plurality of parallel pipelines; instructions for extracting a respective cycle data container from the cycle data pool for each parallel pipeline; instructions for processing a first image processing cycle in a first parallel pipeline to generate first report data independently of the processing of remaining image processing cycles in each remaining parallel pipeline; instructions for providing the first report data to a client device; Including, the plurality of image processing cycles includes a plurality of first image processing cycles, and the one or more programs: instructions for dynamically adjusting one or more parameters of a number of pipelines, a thread pool size, thread priorities, delays, throttling parameters, core allocations, acquisition parameters, lighting, and camera focus while processing the first plurality of image processing cycles in the parallel pipelines; instructions for allocating a plurality of second image processing cycles subsequent to the plurality of first image processing cycles to the plurality of parallel pipelines; instructions for processing the plurality of second image processing cycles in the plurality of parallel pipelines based on the one or more adjusted parameters; an electronic system,

3. one or more processors; a memory storing one or more programs configured to be executed by the processor; Equipped with the one or more programs: instructions for identifying a plurality of imaging cycles associated with a temporal sequence of triggers, each imaging cycle being created in response to one or more respective trigger events; allocating the image processing cycles to a plurality of parallel pipelines; instructions for extracting a respective cycle data container from the cycle data pool for each parallel pipeline; instructions for processing a first image processing cycle in a first parallel pipeline to generate first report data independently of the processing of remaining image processing cycles in each remaining parallel pipeline; instructions for providing the first report data to a client device; Including, the one or more programs: An electronic system further comprising instructions for executing a user application on the client device, the user application being driven by a first programming language, the plurality of image processing cycles being executed by a second programming language different from the first programming language, and the plurality of image processing cycles being configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first programming language and the second programming language.

4. 1. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of an electronic system, the one or more programs comprising: instructions for identifying a plurality of imaging cycles associated with a temporal sequence of triggers, each imaging cycle being created in response to one or more respective trigger events; allocating the image processing cycles to a plurality of parallel pipelines; instructions for extracting a respective cycle data container from the cycle data pool for each parallel pipeline; instructions for processing a first image processing cycle in a first parallel pipeline to generate first report data independently of the processing of remaining image processing cycles in each remaining parallel pipeline; instructions for providing the first report data to a client device; Including, the one or more programs:

11. The non-transitory computer-readable storage medium, further comprising instructions for dynamically adjusting one or more parameters of a number of pipelines, a thread pool size, thread priorities, delays, throttling parameters, core allocations, acquisition parameters, lighting, and camera focus while processing the plurality of first image processing cycles in the plurality of parallel pipelines.

5. the plurality of image processing cycles includes a plurality of first image processing cycles; the one or more programs: instructions for allocating a plurality of second image processing cycles subsequent to the plurality of first image processing cycles to the plurality of parallel pipelines; instructions for processing the plurality of second image processing cycles in the plurality of parallel pipelines based on the one or more adjusted parameters; The non-transitory computer-readable storage medium of claim 4 , further comprising:

Images