Real-time visualization at asynchronous runtime via state management

Abstract

The invention relates to real-time visualization at asynchronous runtime via state management. The present application is intended for information visualization for image processing. The electronic system implements a plurality of image processing cycles associated with a time sequence of triggers, and each image processing cycle is created in response to one or more respective trigger events. An electronic system obtains a plurality of input / output (I / O) signals for a plurality of image processing cycles and generates a plurality of cycle state signals. Each period state signal is associated with a sequence of timestamps and indicates a process of a respective image processing period based on the I / O signal. The sequence of timestamps has a temporal resolution higher than the predefined resolution. While implementing the image processing cycle, the electronic system dynamically and simultaneously visualizes a plurality of cycle state signals on the user interface in real time with respect to at least a first time axis.

Description

Technical Field

[0001] The disclosed embodiments relate generally to electronic circuits and, more particularly, to systems, apparatus, and methods for configuring hardware resources to process multiple image processing cycles in multiple parallel pipelines. Background Art

[0002] Computer processors implement parallelism, pipelining, overlapping of software tasks, and dedicated hardware acceleration to enhance their processing capabilities. These processing enhancement techniques are usually limited to highly repeatable cycles that functionally implement the same or similar operations involving little or no change from one cycle to another. Computer processors are configured using a register model, and a single configuration is consistently applied to drive runtime execution. For example, a processor pipeline typically has a fixed number of stages. Multiple stages are interlaced, and the stages of the current cycle typically run simultaneously with different stages of the previous cycle. It would be beneficial to develop a highly flexible and efficient parallel processing mechanism while fully utilizing existing computer resources. Summary of the invention

[0003] Various embodiments of the present application are directed to a core high-performance runtime engine configured to process image data in a parallel pipeline for heavy-duty industrial machine vision applications. Machine vision applications are applied in electronic devices (e.g., imaging devices including one or more cameras) to implement functions including but not limited to image processing, code reading, product information extraction, and print quality verification. Specifically, in these machine vision applications, a state-managed asynchronous pipelined architecture is applied in an electronic device for image processing. Multiple image processing cycles are executed in parallel in multiple pipelines, thereby utilizing hardware capabilities without bottlenecks. Each image processing cycle includes multiple time-critical stages and operations that are optionally triggered and interleaved in multiple threads. In some embodiments, a triggering event is received while a previous image processing cycle is still in progress and corresponds to a completely different image processing cycle assigned to a different pipeline that is implemented independently of the previous image processing cycle. In some prior art implementations, a stage of an image processing job uses available hardware capabilities and can force any other image processing job or other stages of the same image processing job to wait in a queue. In contrast, in the present application, a state-managed asynchronous pipelined architecture exploits available hardware capabilities and provides performance improvements by executing multiple image processing cycles independently of each other in parallel pipelines, and this performance improvement scales linearly with the number of processor cores of the imaging device.

[0004] In one aspect, an image processing method is implemented at an electronic device (e.g., an imaging device) having a memory and one or more processors. The method includes the following steps: identifying multiple image processing cycles associated with a triggered time sequence. Each image processing cycle is created in response to one or more corresponding triggering events. The method also includes the following steps: assigning multiple image processing cycles to multiple parallel pipelines (e.g., each image processing cycle is assigned to a corresponding parallel pipeline), and for each parallel pipeline, pulling a corresponding cycle data container from a cycle data pool. The method also includes the following steps: processing a first image processing cycle in a first parallel pipeline to generate first report data, which is independent of processing the remaining image processing cycles in the corresponding remaining parallel pipelines. The method also includes the following steps: providing the first report data to a client device, for example, for further processing (such as storage, classification, analysis and / or visualization).

[0005] In some embodiments, before identifying multiple image processing cycles, the cycle data pool includes a predefined number of cycle data containers. Once assigned to the corresponding image processing cycle, each parallel pipeline directly uses one of the predefined number of cycle data containers without waiting for a new cycle data container to be created. In some cases, the predefined number of cycle data containers are fully used with different pipelines, and the cycle data container cannot be pulled for a new pipeline. In some embodiments, the step of processing the first image processing cycle also includes: filling the first cycle data container with image data associated with one or more corresponding trigger events; generating first cycle data including first report data; and storing the first cycle data in the first cycle data container.

[0006] In some embodiments, the imaging device is communicatively coupled to a client device (e.g., a laptop computer, desktop computer, or mobile phone) via a wired or wireless communication link. A user interface of a machine vision application is enabled on the client device to present report data collected from the imaging device in a user-friendly manner. In some embodiments, the client device is located remotely from the imaging device and is coupled to the imaging device via a server. A web-based user interface is enabled on the client device to remotely view report data reported by the imaging device through the server.

[0007] Some implementations of the present application relate to performance optimization and analysis of complex multi-process applications, especially those found in fields such as factory automation and quality control. Some implementations focus on systems that require high-precision event recording and interactive tuning, such as machine vision systems, barcode reading, and label verification systems. An advanced system for interactive analysis, data merging, and real-time instrumentation of various event types (including time, digital, analog, and image data) is provided. Some implementations cover real-time system parameter adjustments using intuitive visual interfaces and potential machine learning methods to enhance system performance and reliability.

[0008] More specifically, on the other hand, a method is implemented at an electronic system having a memory and one or more processors for real-time tracking of image processing cycles. The method includes the following steps: implementing multiple image processing cycles associated with a triggered time sequence, and each image processing cycle is created in response to one or more corresponding triggering events. The method also includes the following steps: obtaining multiple input / output (I / O) signals of the multiple image processing cycles; and generating multiple cycle status signals based on the I / O signals. Each cycle status signal is associated with a timestamp sequence and indicates the progress of the corresponding image processing cycle. The timestamp sequence has a time resolution higher than a predefined resolution. The method also includes the following steps: while implementing the image processing cycle, dynamically and in real time, visualizing the multiple cycle status signals simultaneously on a user interface with respect to at least a first time axis.

[0009] According to some embodiments, an electronic system includes one or more processors, a memory, and one or more programs stored in the memory. The programs are configured to be executed by the one or more processors. The one or more programs include instructions for executing any one of the methods described herein.

[0010] According to some embodiments, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by an electronic system having one or more processors and a memory. The one or more programs include instructions for executing any of the methods described herein.

[0011] Thus, methods, systems, and apparatus are disclosed that enable optimal design, implementation, and performance of a bar code scanner.

[0012] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art in view of the drawings, the specification, and the claims. In addition, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes, and may not be selected to describe or limit the entire inventive subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a perspective view of an electronic device (eg, a scanning device) according to some embodiments.

[0014] Figure 2 is a block diagram of an example electronic device in accordance with some implementations.

[0015] Figure 3 is an example image processing environment including an electronic device that uses a parallel pipeline to process image data in accordance with some implementations.

[0016] Figure 4 A process for processing image data in multiple parallel pipelines according to some embodiments is illustrated.

[0017] Figure 5 Another example process for processing image data in multiple parallel pipelines according to some embodiments is illustrated.

[0018] Figure 6 Another example process for processing image data in multiple parallel pipelines according to some embodiments is illustrated.

[0019] Figure 7 is a timing diagram of multiple image processing cycles implemented in multiple parallel pipelines according to some embodiments.

[0020] Figure 8 is a flow chart of a method for processing image data according to some embodiments.

[0021] Fig. 9 is a diagram of data structures of signals and data used in an image processing environment according to some embodiments.

[0022] Fig.10 is a timing diagram of an image processing cycle implemented in one of a plurality of parallel pipelines according to some embodiments.

[0023] FIG. 11A to FIG. 11D are four example timing diagrams of multiple image processing cycles according to some implementations.

[0024] Fig.12is an example collapsed graphical user interface (GUI) for visualizing signals of multiple image processing cycles 404 over an extended duration in accordance with some implementations.

[0025] Fig.13 is another example user interface 1300 for visualizing signals for multiple image processing cycles according to some implementations.

[0026] Fig.14A is a portion of an example user interface that visualizes cycle status signals and I / O signals for three image processing cycles according to some embodiments, and Fig. 14B is a portion of another example user interface displaying a histogram of capture times for multiple image processing cycles in accordance with some implementations.

[0027] Fig.15 is a flow chart of a method for real-time tracking of an image processing cycle according to some implementations.

[0028] Reference will now be made to embodiments, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention may be practiced without these specific details. DETAILED DESCRIPTION

[0029] Various embodiments of the present application are directed to a core high-performance runtime engine that is configured to process image data in parallel pipelines for heavy-duty industrial machine vision applications. In these machine vision applications, an asynchronous pipeline architecture managed by state is applied in an electronic device (e.g., an imaging device) for image processing. Specifically, the electronic device identifies multiple image processing cycles associated with a triggered time sequence, and each image processing cycle is created in response to one or more corresponding triggering events (e.g., the acquisition of an image). The electronic device assigns multiple image processing cycles to multiple parallel pipelines. For each parallel pipeline, a pre-existing cycle data container is pulled from a cycle data pool. A first image processing cycle is processed in a first parallel pipeline to generate first report data, which is independent of the remaining image processing cycles in the corresponding remaining parallel pipelines. The first report data is provided to a client device. In some embodiments, each image processing cycle includes multiple time-critical operations and stages that are optionally triggered and interleaved in multiple threads of the corresponding pipeline. This state-managed asynchronous pipelined architecture takes advantage of available hardware capabilities and significantly improves performance by independently executing multiple image processing cycles in parallel pipelines, and this performance improvement scales linearly with the number of processor cores (e.g., CPU, FPGA, NPU and / or GPU) of the electronic device.

[0030] The main purpose of the state-managed asynchronous pipelined architecture is to achieve maximum performance for an image processing system that fully utilizes hardware resources, including but not limited to processors, acquisition sources, input / output units, communication channels, processing threads, and processes. In addition, the architecture greatly simplifies user intervention in control, configuration, and monitoring runtime, thereby enhancing the user experience. In some embodiments, the processing pipeline and associated hardware and software settings are changed at any time without incurring significant overhead. In some embodiments, self-analysis and adjustment of the architecture is enabled automatically through deep instrumentation to improve image processing performance with no or little user interaction.

[0031] Some implementations of the present application relate to performance optimization and analysis of complex multi-process applications, especially those found in fields such as factory automation and quality control. Some implementations focus on systems that require high-precision event recording and interactive tuning, such as machine vision systems, barcode reading, and label verification systems. An advanced system for interactive analysis, data merging, and real-time instrumentation for various event types (including time, digital, analog, and image data) is provided. Some implementations cover the use of intuitive visual interfaces and potential machine learning methods to perform real-time system parameter adjustments to enhance system performance and reliability.

[0032] Figure 1 is a perspective view of an electronic device 100 (e.g., a scanning device) according to some embodiments. In some embodiments, the electronic device 100 is also referred to as a code reader, a barcode scanner, a label scanner, an optical scanner, or an image capture system. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). The electronic device 100 includes a housing 110 (e.g., a body or housing) for protecting components located inside the electronic device 100. In some embodiments, the housing 110 includes integrated accessories or brackets for holding the internal components in place. In some embodiments, the electronic device 100 includes a top cover 102 located on the top side of the electronic device 100. In some embodiments, the top cover 102 is transparent or partially transparent.

[0033] In some embodiments, the electronic device 100 includes one or more distance sensors 104 (eg, internal distance sensors) located within the electronic device 100. For example, referring to Figure 1, the distance sensor 104 is located inside the electronic device 100 (e.g., adjacent to the top cover 102) and faces the front end of the electronic device 100. In some embodiments, the distance sensor 104 is included in the aiming module. When the target is set on the electronic device 100, the distance sensor 104 projects a light beam at the target to assist the visual alignment of the target. This helps to align the camera with the center point of the imaging plane or field of view. In some embodiments, each distance sensor 104 is a time of flight (TOF) sensor, an ultrasonic sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, or an infrared (IR) distance sensor. In some embodiments, the electronic device 100 includes two or more distance sensors 104, each of which has the same type (e.g., each distance sensor in the two or more distance sensors is a TOF sensor). In some embodiments, the electronic device 100 includes two or more distance sensors, wherein at least two distance sensors are of different types (e.g., the electronic device 100 includes a TOF distance sensor and a radar sensor). In some embodiments, the electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) whether an object is within a sensing area in which the proximity sensor is designed to operate. In some embodiments, the electronic device 100 uses ranging technology (such as an image focus finder, an analog-to-digital conversion (ADC) circuit, and / or a digital-to-analog conversion (DAC) circuit) to determine the distance between the target object and the electronic device 100.

[0034] More specifically, in some embodiments, the distance sensor 104 is a TOF sensor. The TOF sensor measures the time elapsed from the emission of a signal (e.g., a wave pulse, an LED pulse, a laser pulse, or an IR wave) from the sensor to the moment it returns to the sensor after reflecting off an object. The distance is then calculated by using the speed of light in the air and the time between sending / receiving the signal. In some embodiments, the distance sensor 104 is an ultrasonic sensor. An ultrasonic sensor or sonar sensor detects the distance from an object by emitting high-frequency sound waves. The ultrasonic sensor emits high-frequency sound waves toward a target object and starts a timer. The target object reflects the sound waves back to the sensor. The receiver picks up the reflected waves and stops the timer. The time taken for the wave to return is calculated against the speed of sound to determine the distance traveled. In some embodiments, the distance sensor 104 is a radar sensor. A radar sensor (e.g., a radar distance sensor) transmits high-frequency radio waves (e.g., microwaves) and calculates the distance from the object by measuring the reflection of the radio waves from the object. In some embodiments, the radar sensor is configured to determine the distance, angle, and radial velocity of the object relative to the position of the electronic device 100. In some embodiments, the distance sensor 104 is a LiDAR sensor, which measures the distance of a target object by light waves from a laser (e.g., instead of radio waves or sound waves). In some embodiments, the distance sensor 104 is an infrared (IR) distance sensor. IR distance sensors work on the principle of triangulation, where the distance is measured based on the angle of the reflected beam.

[0035] In some embodiments, the electronic device 100 further includes a plurality of light sources 106 (eg, Figure 1 8 light emitting diodes (LEDs) in the light source 106). The light source 106 is also referred to as an illumination source, an illuminating source, or an illuminator. In some embodiments, the light source 106 is part of an illumination system of the electronic device 100, which also includes an illuminator (e.g., a bright field illuminator and a dark field illuminator), a reflector, and an illumination module. More details about the illumination system are described in U.S. Patent Application No. 14 / 298,659, entitled “Combination Dark Field and Bright Field Illuminator” filed on June 6, 2014 (now U.S. Patent No. 8,989,569, issued on March 24, 2015), the entire contents of which are incorporated herein by reference.

[0036] In some embodiments, the light source 106 has one or more types of illumination. Examples of illumination types include, but are not limited to, LED light sources, laser light sources, and liquid crystal display (LCD) lights. Various types of illumination in the illumination types have corresponding illumination characteristics, such as color (e.g., blue, red, or green) and / or intensity. The light source 106 is mounted on a PCB 108 (e.g., soldered on the PCB), which is located in the electronic device 100 (e.g., behind the top cover 102). The PCB 108 includes a front surface of the top cover 102 facing the electronic device 100. In some embodiments, the light source mounted on the front surface of the PCB 108 includes a long-distance light source and a low-angle light source.

[0037] In some embodiments, the electronic device 100 includes a camera 112. The lens of the camera 112 is exposed via an opening of the PCB 108 and is physically surrounded by the light source 106. The light source 106 is grouped into a plurality of lighting units (e.g., a first lighting unit and a second lighting unit). Each lighting unit is configured to be independently controlled to illuminate different areas of the field of view of the camera 112. In an example, every two light sources 106 near the corners of the top cover 102 are grouped to form a lighting unit. The four lighting units are independently controlled to illuminate corresponding areas of the field of view of the camera 112 in a sequential or simultaneous manner.

[0038] In some embodiments, the electronic device 100 also includes one or more indicators 114. Each indicator 114 is located at the edge of the top cover 102 of the electronic device 100 and is configured to be illuminated according to a light pattern in which a single color or different colors are displayed for a series of temporal durations defined based on frequency. In some cases, the light pattern represents a message including data or status of the electronic device 100. For example, in response to detecting the presence of a product on or near the top cover, the indicator 114 is continuously illuminated in red, and in response to a successful scan of a barcode displayed on the product, the indicator is turned green and remains green for a shortened duration. In some embodiments, each indicator 114 includes one or more LEDs that emit light, and the light is displayed on the indicator 114 in a substantially uniform and uniform manner.

[0039] Figure 2 is a block diagram of an example electronic device 100 according to some embodiments. The electronic device 100 includes one or more distance sensors 104, as previously described with reference to Figure 1described. In some embodiments, the one or more distance sensors 104 include one or more of the following: a time-of-flight sensor, an ultrasonic sensor, a radar sensor, or a LiDAR sensor. In some embodiments, the electronic device 100 includes one or more proximity sensors for sensing (e.g., detecting) whether an object is within a sensing area in which the proximity sensor is designed to operate. In some embodiments, the electronic device 100 uses ranging technology (such as an image focus finder, analog-to-digital conversion (ADC) and / or digital-to-analog conversion (DAC)) to determine the distance between the target object and the electronic device 100.

[0040] The electronic device 100 includes a light source 106. In some embodiments, the light source 106 includes a long distance light source 262, a low angle light source 264, and / or a dome light source 266, such as in Figure 3 and described in U.S. Patent Application No. 14 / 298,659, filed on June 6, 2014, entitled “Combination Dark Field and Bright Field Illuminator” (now U.S. Patent No. 8,989,569, issued on March 24, 2015, and incorporated herein by reference in its entirety). In some embodiments, light source 106 provides illumination with visible light. In some embodiments, light source 106 provides illumination with non-visible light (e.g., with infrared light or violet light).

[0041] In some embodiments, the electronic device 100 includes a decoder 212 for decoding data contained in the barcode and sending the data to a computer device. In some embodiments, the decoder 212 is part of the software application 230. Details of the decoder 212 are described in U.S. Patent Application No. 14 / 298,659, entitled “Combination Dark Field and Bright Field Illuminator,” filed on June 6, 2014 (now U.S. Patent No. 8,989,569, issued on March 24, 2015 and incorporated herein by reference in its entirety).

[0042] In some embodiments, the electronic device 100 includes one or more input interfaces 210 to facilitate user input. In some embodiments, the electronic device 100 is a battery-operated device and includes a rechargeable battery. In this case, the input interface 210 may include a charging port for charging the battery.

[0043] In some embodiments, the electronic device 100 includes a camera 112, which includes an image sensor 216 and a lens 218. The lens 218 guides the path of light and focuses them on the image sensor 216 to recreate the image as accurately as possible on the image sensor. The image sensor 216 converts light (e.g., photons) into electrical signals that can be interpreted by the electronic device 100. In some embodiments, the lens 218 is an optical lens and is made of glass or other transparent materials. In some embodiments, the lens 218 is a liquid lens composed of an optical liquid material, and its shape, focal length and / or working distance changes when a current or voltage is applied to the liquid lens. In some embodiments, the electronic device 100 (e.g., via the processor 202) uses the distance information obtained by the distance sensor 104 to determine the optimal current or voltage applied to the liquid lens 218 so as to have an optimal focal length for decoding the barcode data contained in the image. In some embodiments, the camera 112 is configured to capture color images. In some embodiments, the camera 112 is configured to capture black and white images.

[0044] The electronic device 100 also includes one or more processors (eg, CPUs) 202, one or more communication interfaces 204 (eg, network interfaces), memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as a chipset).

[0045] In some embodiments, the electronic device 100 includes a radio 220. The radio 220 enables one or more communication networks and allows the electronic device 100 to communicate with other devices such as computer devices or servers. In some embodiments, the radio 220 can use any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, Ultra Wideband (UWB), and / or Software Defined Radio (SDR)), custom or standard wired protocols (e.g., Ethernet or HomePlug), and / or any other suitable communication protocol (including communication protocols that have not been developed as of the filing date of this patent application) for data communication.

[0046] The memory 206 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, the memory includes a non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. In some embodiments, the memory 206 includes one or more storage devices located away from the one or more processors 202. The memory 206 (or alternatively, the non-volatile memory within the memory 206) includes a non-transitory computer-readable storage medium. In some embodiments, the memory 206 (or the non-transitory computer-readable storage medium of the memory 206) stores the following programs, modules, and data structures or a subset or superset thereof:

[0047] The operation logic unit 222 includes a unit for processing various basic system services and

[0048] Processes used to perform hardware-related tasks;

[0049] A communication module 224 (eg, a radio communication module) that communicates with the communication interface 204

[0050] Connect (e.g., wired or wirelessly) to and communicate with other network devices coupled to one or more communications networks (e.g., a local network such as a router providing Internet connectivity, networked storage devices, network routing devices, server systems, computing devices, and / or other connected devices);

[0051] An application 230 that acquires an image containing a label (e.g., a barcode) and decodes the label, and controls one or more of the electronic device 100 and / or other connected devices according to the determined state.

[0052] In some implementations, application 230 includes:

[0053] o an illumination module 232 that selects and deploys (e.g., based on distance measurements, such as direct measurements from distance sensor 104 or indirect measurements) a sequence of one or more light sources 106 and / or illumination patterns 234 for the current read cycle;

[0054] o a distance module 236 that determines (e.g., selects) which focus distance sequence to employ during a current read cycle based on distance measurements from the distance sensor 104;

[0055] o an exposure and gain module 238 that samples an image 244 captured by the camera 112;

[0056] ○ Image acquisition and processing module 240, the image acquisition and processing module, for example, according to Figures 4 to 6 as well as Figure 8 to acquire and process an image using a process exemplified in one of the embodiments; and

[0057] o a decoder 212 for decoding the data contained in the barcode and sending the data to a computer device;

[0058] Data 242 for the electronic device 100, including but not limited to:

[0059] o Image data 244 (eg, camera data);

[0060] o Symbology data 246 (e.g., code type, such as a barcode);

[0061] o Device settings 248 of the electronic device 100, such as default options, image acquisition settings (eg, exposure and gain settings), and preferred user settings; and

[0062] o User settings 250, such as the preferred shade of the lens (e.g., for photochromic lenses); and

[0063] o Sensor data 252 acquired (eg, measured) from the distance sensor 104 and / or other sensors included in the electronic device 100 .

[0064] In some embodiments, the distance sensor 104 is monitored by the illumination module 232. When the user starts the current reading cycle, the distance sensor 104 identifies the distance field (e.g., near field, mid field, or far field) corresponding to the location of the target object. The illumination module 232 selects an illumination sequence corresponding to the distance field for execution. If a good read was achieved in the previous reading cycle (e.g., a good read from the third illumination pattern in the near field illumination sequence), and the current reading cycle has the same distance field as the previous reading cycle, the application 230 will start the current reading cycle by using the values ​​of the earlier good read (e.g., the third illumination pattern in the near field illumination pattern, the previous focus position, exposure, and / or gain) before starting the illumination sequence from the beginning. The user is typically reading many similar parts, and if the device starts with a known good setting from the last decode operation, the device can achieve a good read faster. If there is no previous setting that enables a good read, the illumination sequence for the current distance field starts from the beginning and iterates through each sequence captured one after another.

[0065] In some embodiments, the exposure and gain module 238 rejects images that do not fall within the predefined attribute ranges of "brightness" and / or "sharpness" (e.g., rejected images are not processed by the image acquisition and processing module 240). In some embodiments, the exposure and gain module 238 updates the image acquisition settings (such as exposure and gain) for the next image capture to provide optimal "brightness" for image processing.

[0066] In some embodiments, after capturing an image (e.g., using camera 112), the electronic device 100 (e.g., via application 230) evaluates the quality of the acquired image. For example, the electronic device 100 reads (e.g., determines) the clarity value, average light mean value, and / or average dark mean value of the image to determine whether the image is qualified or rejected. If the result does not meet or exceed the predefined target value, the image is rejected and another image is captured again. If the result meets or exceeds the predefined target value, the image is processed (e.g., by image acquisition and processing module 240).

[0067] As an example, in some embodiments, a good quality image is an image sample having a light mean score between 100 and 170 (out of a range of 0 to 255), a dark mean score between 20 and 80 (out of a range of 0 to 255), and a sharpness score above 6000 (out of a range of 0 to approximately 12000).

[0068] In some implementations, data collected during image sampling (eg, evaluation) is captured and added (eg, as data 242 ).

[0069] In some embodiments, after qualifying an image, the electronic device 100 (e.g., via the application 230) determines whether to adjust exposure or gain settings for the next image (e.g., using a bright mean correction path or a dark mean correction path). If it decides to do so, the electronic device 100 collects target bright mean and dark mean values ​​for comparison, deploys a proportional and integral (PI) controller transfer function, and calculates the necessary change in exposure to obtain the ideal exposure in the next image.

[0070] In some embodiments, after successfully decoding an image, the exposure, gain, and focus values ​​are fed back to the application 230. In the following read cycle, the application 230 checks whether these decode settings are pending. If these decode settings are pending, the electronic device 100 attempts to load the camera settings and any previous settings, which is compared to calculating the next configuration of settings. In the event that the previous decode settings are used, the application 230 samples the image data, but does not adjust the values ​​of the feedback controller.

[0071] Each of the executable modules, applications or process sets identified above can be stored in one or more memory devices in the previously mentioned memory devices, and corresponds to an instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes or modules, and therefore, in various embodiments, various subsets of these modules can be combined or otherwise rearranged. In some embodiments, memory 206 stores a subset of the modules and data structures identified above. In addition, memory 206 can store additional modules and data structures not described above. In some embodiments, a subset of programs, modules and / or data stored in memory 206 is stored on a server system and / or executed by the server system and / or executed by an external device (e.g., a computing device).

[0072] Figure 3 300 is an example image processing environment 300 of an electronic device 100 including one or more images processed using a parallel pipeline according to some embodiments. In some embodiments, the electronic device 100 is an imaging device. In some embodiments, the electronic device 100 is a code reader, a barcode scanner, a label scanner, or an optical scanner. In some embodiments, the electronic device 100 is part of an optical data reading system (e.g., a label scanning station). In some embodiments, the electronic device 100 is configured to obtain image data including one or more images, process the image data using a parallel pipeline, and provide report data 312 generated from the image data to one or more client devices 302 (e.g., devices 302A, 302B, 302C, or 302D). One or more client devices 302 may be, for example, a desktop computer, a tablet computer, a mobile phone, or a smart, multi-sensing, network-connected home device (e.g., a display auxiliary device). Each client device 302 may collect report data 312 from the electronic device 100, receive user input, execute a user application, and present report data 312 or other information on its user interface. In some implementations, the user application includes an interactive user application. A user interface of the interactive user application is displayed on the client device to receive user input associated with the electronic device 100 and to visualize report data 312 generated by the electronic device 100.

[0073] The electronic device 100 is configured to enable multiple parallel pipelines. The electronic device 100 identifies multiple image processing cycles associated with a triggered time sequence, and each image processing cycle is created in response to one or more corresponding triggering events (e.g., one or more image capture operations). Multiple image processing cycles are assigned to multiple parallel pipelines. For each parallel pipeline, a pre-existing cycle data container is directly pulled from a cycle data pool. The electronic device processes multiple image processing cycles in multiple parallel pipelines to generate corresponding report data, which are independent of each other. In some embodiments, the triggered time sequence corresponds to an ordered sequence of images processed during the image processing cycle. The report data 312 of the image processing cycle is generated separately from the multiple parallel pipelines regardless of the order of the image sequence. In some cases, the report data of the image processing cycle is organized according to the order of the corresponding image sequence and provided to the client device 302.

[0074] In some embodiments, the user application implemented on the client device 302 is driven by a first programming language, and multiple image processing cycles are executed on the electronic device 100 by a second programming language different from the first programming language. Multiple image processing cycles are configured to automatically exchange instructions and data with the user application via intermediate data representations between the first programming language and the second programming language. For example, the intermediate data representation is implemented using JSON (JavaScript Object Notation). The user application includes a web-based user interface, and the first programming language includes JavaScript. The runtime associated with the image processing cycle uses a second language (e.g., C++). JSON is made the native format of the runtime by building JSON support into almost every runtime C++ object, thereby allowing JavaScript programs to automatically understand the workflow format or report format used by the runtime.

[0075] In some embodiments, report data 312 or user input is processed locally at client device 302 and / or remotely by server 304. One or more servers 304 provide system data (e.g., boot files, operating system images, and user applications) to client device 302, and in some embodiments, one or more servers 304 process user input and report data 312 received from client device 302 when a user application is executed on client device 302. In some embodiments, data processing environment 300 also includes storage device 306 for storing data related to server 304, client device 302, electronic device 100, and user applications executed on client device 302. For example, storage device 306 can store video content, static visual content, and product databases.

[0076] One or more servers 304, one or more client devices 302, electronic device 100 and storage device 306 are connected to each other in a communicative manner via one or more communication networks 308, which are media for providing communication links between these devices and computers connected together in data processing environment 300. One or more communication networks 308 can include connections such as wired, wireless communication links or optical cables. Examples of one or more communication networks 308 include local area networks (LANs), wide area networks (WANs) such as the Internet, or combinations thereof. One or more communication networks 308 are optionally implemented using any known network protocols, including various wired or wireless protocols such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi, Voice over Internet Protocol (VoIP), Wi-MAX or any other suitable communication protocol. The connection to the one or more communication networks 308 may be established directly (e.g., using a 3G / 4G connection with a wireless carrier), or through a network interface 310 (e.g., a router, switch, gateway, hub, or intelligent, dedicated whole-home control node), or through any combination thereof. Thus, the one or more communication networks 308 may represent the Internet, a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers that contain thousands of business systems, government systems, educational systems, and other electronic systems that route data and messages.

[0077] In some embodiments, the electronic device 100 is communicatively coupled to the client device 302 (302A) via a wired communication link. In some embodiments, the electronic device 100 is communicatively coupled to the client device 302 (302A) via a local communication network 314 enabled by the network interface 310. The electronic device 100 and the client device 302 (e.g., 302A) are both located in the same place (e.g., a warehouse or factory). In some embodiments, the electronic device 100 is remote from the client device 302 and is communicatively coupled to the client device 302 (e.g., devices 302B, 302C, and / or 302D) via at least a WAN.

[0078] Figure 4A process 400 of processing image data in multiple parallel pipelines (e.g., 402A, 402B, and 402C) according to some embodiments is illustrated. The process 400 is implemented by an electronic device 100 having a memory 206 and one or more processors 202. For example, the process 400 is performed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 identifies multiple image processing cycles 404 associated with a temporal sequence of triggers (e.g., acquisition of images). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more corresponding trigger events (e.g., capturing a single image or capturing a group of images). Multiple image processing cycles 404 are assigned to multiple parallel pipelines 402. For each parallel pipeline 402, a pre-existing cycle data container 406 (e.g., 406A) is pulled from a cycle data pool 408. In some embodiments, before the plurality of image processing cycles 404 are identified, the cycle data pool 408 includes a predefined number of cycle data containers 406. Once assigned to a corresponding image processing cycle 404, each parallel pipeline 402 directly uses one of the predefined number of cycle data containers 406 without waiting for a new cycle data container 406 to be created. This speeds up the startup and execution of each parallel pipeline 402. In some cases, the cycle data container 406 has been fully used with a different pipeline 402, and the cycle data container 406 cannot be pulled for the pipeline 402 to which the new image processing cycle 404 is assigned.

[0079] After the plurality of image processing cycles 404 are assigned to the plurality of parallel pipelines 402, the plurality of image processing cycles 404 are processed independently of each other in the plurality of parallel pipelines 402, thereby independently generating report data 312 (e.g., 312A, 312B, and 312C). The report data 312 is provided to the client device 302 for further processing (e.g., storage, classification, analysis, and / or visualization). In some embodiments, the client device 302 is connected to the electronic device 100 via a local communication network and executes an interactive user application 412. For example, the client device 302 and the electronic device 100 are located in a factory automation network that cannot access the Internet. The interactive user application 412 includes a user interface configured to locally configure, adjust, and debug the electronic device 100 and its associated workflows and jobs. In some embodiments, the interactive user application 412 is web-based and implemented on a client device 302 connected to a WAN (e.g., the Internet). For example, a web-based user interface is enabled to be implemented in a browser-based user application 412. The browser-based user application 412 is served directly from the client device 302 and uses HTTP and web sockets to communicate with the electronic device 100. A web-based user interface is presented to allow the client device 302 to remotely configure, adjust, and debug the electronic device 100 and its associated workflows and jobs.

[0080] In some embodiments, the process 400 enables a highly interactive browser-based user experience by exchanging data structures between a web-based user interface of a user application 412 and runtimes of multiple parallel pipelines 402. In some embodiments, the user application 412 is implemented based on the Hypertext Transfer Protocol (HTTP), and the client device 302 executing the user application 412 includes an Internet of Things (IOT) device. In an example, the configuration of the user application 412 and the pipeline 402 is accomplished via a data exchange scheme using JavaScript Object Notation (JSON), thereby allowing a web-based user interface driven by JavaScript to interact natively with a core runtime implemented in C++ or other non-dynamic computer languages.

[0081] In some embodiments, the first triggering event of the first image processing cycle 404A precedes the second triggering event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Alternatively, in some embodiments, regardless of the temporal order of the first triggering event and the second triggering event, whichever of the first report data 312A and the second report data 312B is available first is reported to the client device 302. If the second report data 312B of the second image processing cycle 404B is available before the first report data 312A of the first image processing cycle 404A, the second report data 312B is provided to the client device 302 before the first report data 312A. If the second report data 312B is available after the first report data 312A, the second report data 312B is provided to the client device 302 after the first report data 312A.

[0082] For each parallel pipeline 402, a pre-existing cycle data container 406 is pulled from the cycle data pool 408 and is used to store cycle data 414 associated with the corresponding image processing cycle 404. For example, a first image processing cycle 404A is implemented in the first parallel pipeline 402A. The first cycle data 414A includes image data associated with one or more corresponding triggering events of the first image processing cycle 404A, and intermediate data generated by and passed through multiple stages of the first image processing cycle 404A. In some embodiments, the first cycle data 414A includes first report data 312A generated from the intermediate data and reported to the client device 302. In some embodiments, all image data, intermediate data, and report data of the first image processing cycle 404A are reported to the client device 302. Alternatively, in some embodiments, only the report data 312A of the first image processing cycle 404A is reported to the client device 302.

[0083] In the example, the electronic device 100 is configured to recognize a barcode in an image. The first image processing cycle 404A is created in response to capturing a first trigger event 602 of an image including a barcode. The first cycle data 414 associated with the first image processing cycle 404 includes a serial number corresponding to the barcode. The first report data 312A generated from the first image processing cycle 404 includes a product name and a product price.

[0084] In some prior art implementations, each pipeline has fixed stages that are staggered so that the first stage of the current cycle runs simultaneously with the second stage of the previous cycle that precedes the current cycle. In contrast, in various implementations of the present application, parallel pipelines 402 do not have this limitation, and each parallel pipeline 402 is turned on for a corresponding image processing cycle 404, so that there is no overhead. The stages of each cycle run within the same pipeline 402, and the pipeline 402 includes a managed number of threads and is associated with a cycle data container 406 extracted from a cycle data pool 408.

[0085] Figure 5 Another example process 500 of processing image data in multiple parallel pipelines 402 according to some embodiments is illustrated. The process 500 is performed by an electronic device 100 ( Figure 2 ). For example, process 500 is performed by processor 202 according to instructions provided by image acquisition and processing module 240 of electronic device 100. Electronic device 100 provides acquisition service 502, which identifies multiple image processing cycles 404 associated with a time sequence of triggers (e.g., acquisition of images). Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more corresponding trigger events (e.g., capturing a single image, capturing a group of images). Based on the time order of the triggers, multiple image processing cycles are arranged in a runnable cycle queue 504. Multiple image processing cycles 404 are assigned to multiple parallel pipelines 402. For each parallel pipeline 402, a pre-existing cycle data container 406 (e.g., 406A) is pulled from a cycle data pool 408. During the process of managing and executing image processing cycles 404 , cycle data 414 is associated with or generated for each image processing cycle 404 and stored in a corresponding cycle data container 406 .

[0086] In some embodiments, a first image processing cycle 404A is implemented in a first parallel pipeline 402A to inspect an input image, for example, to detect smearing of a label on a medicine bottle. The first image processing cycle 404A includes multiple stages, for example, an inspection stage 506, a post-inspection stage 508, a report generation stage 510, and a data return stage 512. In the inspection stage 506, the input image is inspected and intermediate data is generated that is stored in the cycle data container 406A along with the cycle data 414. In the post-inspection stage 508, the intermediate data is further processed to provide additional information that is used to generate first report data 312A in the report generation stage 510. The first report data 312A is placed in a pending data queue 514 to be provided to the client device 302 in the data return stage 512. In some embodiments, the report data 312 is organized in the pending data queue 514 based on the temporal order of the associated triggering events. Alternatively, in some embodiments, report data 312 is organized in pending data queue 514 based on the temporal order of its own availability, regardless of the temporal order of the associated triggering events.

[0087] Each stage of the image processing cycle 404A is not restricted by fixed hardware allocation and can flexibly access the hardware resources allocated to the corresponding parallel pipeline 402A. In other words, each stage of the image processing cycle 404A can use all allocated pipeline hardware resources to accelerate or execute in parallel with one or more remaining stages of the image processing cycle 404A.

[0088] In some embodiments, the plurality of image processing cycles 404 assigned to the plurality of pipelines 402 include a plurality of first image processing cycles 404-1. The runnable cycle queue 504 also includes a plurality of second image processing cycles 404-2 following the first image processing cycles 404-1. When the plurality of pipelines 402 are fully occupied by the plurality of first image processing cycles 404-1, a status signal 516 (e.g., "wait for idle") is sent to the runnable cycle queue 504, requesting the runnable cycle queue 504 to hold the plurality of second image processing cycles 404-2 until the plurality of pipelines 402 completes processing of the plurality of first image processing cycles 404-1.

[0089] High-precision instrumentation is built in, allowing for an in-depth and intuitive understanding of the impact of adjusting various performance parameters, including thread, process, and core loading and hardware utilization. This capability is designed in a way that allows a real-time performance visualizer in a web-based user interface. It also allows for in-depth analysis during execution, thereby having the potential to automatically adjust parameters for maximum performance without user intervention. In some embodiments, when processing multiple first image processing cycles 404-1 in multiple parallel pipelines 402, the electronic device 100 dynamically adjusts one or more parameters in the following items: the number of pipelines, the size of the thread pool, the thread priority, the delay, the throttling parameter, the core assignment, the acquisition parameter, the lighting, and the camera focus. Multiple second image processing cycles 404-2 are assigned to multiple parallel pipelines 402 and are processed in the parallel pipelines 402 based on one or more adjusted parameters. Specifically, in the example, the electronic device 100 monitors the central processing unit (CPU) core utilization. Dynamically based on CPU core utilization, the electronic device 100 adjusts the corresponding computing resources allocated to multiple parallel pipelines. Therefore, the parameters that control the limit on hardware usage (e.g., CPU core utilization) are dynamically adjustable and can be changed at runtime. These parameters can be measured and automatically adjusted to customize the runtime according to the situation to achieve optimal performance.

[0090] Figure 6 Another example process 600 of processing image data in multiple parallel pipelines 402 according to some embodiments is illustrated. The process 600 is performed by an electronic device 100 ( Figure 2). For example, the process 600 is executed by the processor 202 according to instructions provided by the image acquisition and processing module 240 of the electronic device 100. The electronic device 100 provides an acquisition service 502, which identifies a plurality of image processing cycles 404 associated with a time sequence of triggers 602 (e.g., acquisition of images) in an engine queue thread 604. Each image processing cycle 404 (e.g., 404A, 404B, and 404C) is created in response to one or more corresponding trigger events 602 (e.g., capturing a single image, capturing a group of images). In some embodiments, the electronic device 100 also includes a configuration service 606, which provides a job configuration applied to define (operation 608) image processing cycles 404 to process image data captured by the time sequence of triggers 602. Based on the time order of the triggers 602, the plurality of image processing cycles 404 are arranged in the runnable cycle queue 504. The plurality of image processing cycles 404 are assigned to the plurality of parallel pipelines 402. For each parallel pipeline 402, a pre-existing cycle data container 406 (e.g., 406A) is pulled from the cycle data pool 408. During the process of managing and executing image processing cycles 404, cycle data 414 is associated with or generated for each image processing cycle 404 and stored in the corresponding cycle data container 406.

[0091] The trigger event 602 includes the acquisition of a single image, and the single image is processed in the corresponding image processing cycle 404. Optionally, the single image is processed via multiple stages 610 (e.g., 610A and 610B) managed in a single thread 612 or multiple threads 612 of the image processing cycle 404. In other words, in some cases not shown, the first stage 610A is implemented at least partially in parallel with the second stage 610B. In some embodiments, multiple trigger events 602 include the acquisition of multiple images. In response to the multiple trigger events 602, the image processing cycle 404 is executed to process the multiple images associated with the multiple trigger events 602 in multiple stages 610. Optionally, the multiple stages 610 are processed by a single thread 612 or multiple threads 612 of the image processing cycle 404. Optionally, one of the multiple images is processed via multiple stages managed in a subset of the associated single thread 612 or multiple threads 612 of the image processing cycle 404. In the example, the image processing stage 610B is implemented as a subset of the parallel threads 612. In some implementations, the electronic device 100 uses all or part of the first hardware resources allocated to the corresponding parallel pipeline 402 to sequentially implement each stage of the series of stages of the image processing cycle 404.

[0092] It should be noted that the corresponding periodic data container 406 is filled with multiple images provided to the parallel pipeline 402. The periodic data container 406 optionally stores the raw image data of the multiple images or the pre-processed image data of the multiple images (e.g., filtered images, corrected images, associated image features). In some embodiments, the image signal processor (ISP) is configured to filter or correct the raw image data captured by the camera. Each image processing cycle 404 is configured to process the raw image data or the pre-processed image data.

[0093] In some embodiments, the first image processing cycle 404 includes a first thread 612A and a second thread 612B. The second thread 612B optionally starts before or after the first thread 612A. The first thread 612A generates a first subset of the first cycle data 414. The second thread 612B is started before the first subset of the first cycle data 414 is generated, and is paused to wait until the first subset of the first cycle data 414 is generated and provided by the first thread 612A.

[0094] In some embodiments, the workflow template is loaded onto the first parallel pipeline 402A. The plurality of stages 610 of the first image processing cycle 404A are performed in stages that are not used by the remaining image processing cycles (e.g., Figure 4 The same or different workflow templates can be used to implement multiple stages 610 of different image processing cycles 404 in the second parallel pipeline 402B. When the two image processing cycles are implemented in two pipelines 402 respectively, there is no overhead of switching between workflow templates.

[0095] In some embodiments, the stages 610 of the image processing cycle 404 are forced to follow a temporal sequencing with respect to the previous image processing cycle. This causes certain operations (e.g., sending result data) to occur sequentially between cycles. For example, image processing cycles 404A and 404B are both processed by the first parallel pipeline 402A, and the second image processing cycle 404B follows the first image processing cycle 404A. The stages 610 of the second image processing cycle 402B are forced to follow a temporal sequencing with respect to the first image processing cycle 402A. The first report data 312A generated by the first image processing cycle 402A is optionally used by the second image processing cycle 402B.

[0096] The electronic device 100 includes a state-managed asynchronous runtime engine 614 that is configured to keep the runtime in a known state. Instead of individual commands that change parameters or must set the mode of performing operations, the state of the runtime is always coherent and can only be changed to a new state (or cannot be changed if the new state is not accepted). This eliminates many complexities in managing the user experience (e.g., undo is easily resolved). In some embodiments, the state used by the image processing cycle 404 is included in the cycle data 414 stored in the cycle data container 406. The cycle data 414 flows through the stages and operations of the image processing cycle 404 implemented in the parallel pipeline 402. The use of cycle data in the cycle data container 406 maximizes parallelism with other services (such as image acquisition). The cycle data container 406 contains all the information required to complete the goals of the corresponding image processing cycle 404 and can be fully exchanged for new goals on the next image processing cycle 404.

[0097] In some embodiments, cycle data 414 also includes an input queue for data obtained or generated during execution of parallel pipeline 402. For example, the acquisition of an image is optionally overlapped with other processing, and the image is pushed to a queue within cycle data 414 when stage 610 requires image data and is likely to block if the data has not yet arrived.

[0098] Figure 77 is a time diagram 700 of a plurality of image processing cycles 404 implemented in a plurality of parallel pipelines 402 according to some embodiments. Each of the plurality of image processing cycles 404 is processed in a corresponding parallel pipeline 402 independently of the remaining image processing cycles in the corresponding remaining parallel pipelines. Diagram 700 includes ten signal lanes, and each signal lane corresponds to a separate pipeline and displays a corresponding cycle status signal 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, or 722. On each signal lane, a corresponding one of the cycle status signals 702-722 is associated with a timestamp sequence and indicates the progress of the corresponding image processing cycle implemented by the corresponding parallel pipeline 402. The ten corresponding cycle status signals 702-722 are displayed in a synchronized manner on the ten signal lanes of diagram 700. In some embodiments, each of the periodic state signals 702-722 includes a plurality of line segments or bars corresponding to a plurality of imaging stages (e.g., trigger, acquire, set, decode, image processing). The plurality of line segments or bars are optionally coded with different colors. The different periodic state signals 702-722 are sorted, grouped, and filtered to illustrate the relationship between sequential bar code reading operations, and to illustrate how different image processing events overlap and interact with each other. In some embodiments, the cursor moves on the corresponding user interface to hover over one of the different bars of the periodic state signals 702-722 to highlight the relationship between the trajectories of the ten periodic state signals 702-722.

[0099] Figure 8 8 is a flow chart of a method 800 for processing image data according to some embodiments. The method is implemented by the electronic device 100, in particular, by the image acquisition and processing module 240 of the electronic device 100. In some embodiments, the electronic device 100 includes one or more cameras 112 configured to capture images. In some embodiments, the electronic device 100 is connected to a camera or a different electronic device having a camera, and is configured to acquire image data including a sequence of images.

[0100] The electronic device 100 identifies (802) a plurality of image processing cycles 404 associated with a temporal sequence of triggers 602. Each image processing cycle 404 is created in response to one or more corresponding trigger events. Each trigger event includes the acquisition of a corresponding image. The electronic device 100 assigns (804) the plurality of image processing cycles 404 to a plurality of parallel pipelines 402. For each parallel pipeline 402, a corresponding cycle data container 406 is pulled (806) from a cycle data pool 408. The electronic device 100 processes (808) a first image processing cycle 404A in a first parallel pipeline 402A to generate first report data, which is in parallel with processing the remaining image processing cycles 404 (e.g., Figure 4 The first report data is provided (810) to the client device 302 (e.g., Figure 3 302A), for example, for display on a web-based user interface of the client device 302.

[0101] In some embodiments, the trigger event includes the acquisition of a single image, and the single image is processed in the corresponding image processing cycle 404. Alternatively, in some embodiments, multiple trigger events include the acquisition of multiple images, and in response to the multiple trigger events, the image processing cycle 404 is executed to process multiple images associated with the multiple trigger events. In the first processing cycle 404A, the multiple trigger events are processed by multiple threads. In some cases, the first thread 612A ( Figure 6 ) generates a first subset of the first cycle data 414A to be applied by the second thread 612B. The second thread 612B is started and is paused to wait until the first subset of the first cycle data 414A is generated and provided by the first thread.

[0102] In some embodiments, before identifying the plurality of image processing cycles 404, the cycle data pool 408 includes a predefined number of cycle data containers 406. Once assigned to a corresponding image processing cycle 404, each parallel pipeline 402 directly uses one of the predefined number of cycle data containers 406 without waiting for the cycle data container 406 to be created. This speeds up the startup and execution of each parallel pipeline 402. In some cases, the predefined number of cycle data containers are fully used with different pipelines 402, and the cycle data container 406 cannot be pulled for the pipeline 402 to which the new image processing cycle 404 is assigned.

[0103] In some embodiments, processing the first image processing cycle 404A further includes: filling (812) the first cycle data container 406A with image data associated with one or more corresponding triggering events associated with the first image processing cycle 404A, generating (814) first cycle data 414A including first report data 312A, and storing (816) the first cycle data 414A in the first cycle data container 406A. In some embodiments, after completing the first image processing cycle 404A, the data stored in the first cycle data container 406A is cleared and the first cycle data container 406A is returned to the cycle data pool 408 to be pulled and used with the next parallel pipeline 402. In some embodiments, the report data 312A reported to the client device 302 for the first image processing cycle 404A includes all of the first cycle data 414A and the image data. The first cycle data includes all of the intermediate data or final data generated by the first parallel pipeline 402A. The client device 302 stores the reporting data 312 in a database (eg, in the storage device 306 ), visualizes the reporting data 312 , controls the reporting data 312 , and the like.

[0104] In some embodiments, the first image processing cycle 404A includes a first thread 612A and a second thread 612B. The first image processing cycle 404A is processed in the first parallel pipeline 402A by sequentially performing the following operations: generating a first subset of first cycle data by the first thread 612A, starting the second thread 612B before generating the first subset of first cycle data, and pausing the second thread 612B to wait until the first subset of first cycle data is generated by the first thread 612A.

[0105] In an example, a first image processing cycle 404A is created in response to a first trigger event. The first trigger event includes acquiring an image including a barcode. The first cycle data includes a serial number corresponding to the barcode. The first report data includes a product name and a product price. In another example, a first image processing cycle 404A is created in response to a first trigger event. The first trigger event includes acquiring an image of a label attached to a medicine bottle. In some embodiments, the first report data includes an image of the label. The client device 302 receives the image of the label, determines that the label in the image is damaged, and displays the image of the label with a message indicating that the label on the medicine bottle is damaged. Alternatively, in some embodiments, the electronic device 100 determines that the label in the image is damaged, and generates first report data 312A including a message indicating that the label on the medicine bottle is damaged. The client device 302 receives the message and displays the message without the label image.

[0106] In some embodiments, the first cycle data 414A includes intermediate data, which is generated by multiple stages of the first image processing cycle 404A and passed among the multiple stages, and is used to generate first report data, and multiple data items of the intermediate data form a queue and are stored in the first cycle data container 406A in sequence.

[0107] In some implementations, the temporal sequence of triggers corresponds to an ordered sequence of images processed during the plurality of image processing cycles 404 assigned to the plurality of parallel pipelines 402. The first image is acquired before the second image, and the report data 312 corresponding to the first image is provided to the client device before the report data 312 corresponding to the second image.

[0108] In some embodiments, the first image processing cycle 404A is created in response to multiple triggering events, and each of the multiple triggering events includes acquiring a corresponding image. In some cases, when waiting for a later triggering event, the pipeline 402 starts processing an earlier triggering event in the first image processing cycle 404A.

[0109] In some embodiments, the first image processing cycle 404A includes a first phase 610A and a second phase 610B ( Figure 6 ). The first image processing cycle 404A in the first parallel pipeline 402A is processed by implementing a portion of the first stage 610A and a portion of the second stage 610B in parallel.

[0110] In some embodiments, the first image processing cycle 404A includes a series of stages (e.g., Figure 5 The first image processing cycle 404A is processed in the first parallel pipeline 402A by sequentially implementing each stage in the series of stages using the first hardware resources allocated to the first parallel pipeline 402A.

[0111] In some embodiments, after the first image processing cycle 404, the electronic device processes a second image processing cycle 404A different from the plurality of image processing cycles 404 in the second parallel pipeline 402B. After providing the first report data 312A to the client device, the electronic device 100 provides the second report data to the client device 302. In some embodiments, the second report data is visualized on a user interface of the client device 302. In some embodiments, the first trigger event of the first image processing cycle 404A precedes the second trigger event of the second image processing cycle 404B. The second report data 312B of the second image processing cycle 404B is provided to the client device 302 after the first report data 312A of the first image processing cycle 404A, regardless of whether the second report data 312B is available before or after the first report data 312A. Alternatively, in some embodiments, regardless of the temporal order of the first trigger event and the second trigger event, whichever of the first report data 312A and the second report data 312B is available first is reported to the client device 302 first. If second reporting data 312B for second image processing cycle 404B is available before first reporting data 312A for first image processing cycle 404A, second reporting data 312B is provided to client device 302 before first reporting data 312A. If second reporting data 312B is available after first reporting data 312A, second reporting data 312B is provided to client device 302 after first reporting data 312A.

[0112] In some embodiments, the first image processing cycle 404A is implemented in the first parallel pipeline 402A by loading a workflow template on the first parallel pipeline 402A, and implementing multiple stages of the first image processing cycle 404A according to the workflow template without being interrupted by the remaining image processing cycles 404.

[0113] In some embodiments, the electronic device 100 monitors central processing unit (CPU) core utilization. Dynamically based on CPU core utilization, the corresponding computing resources allocated to multiple parallel pipelines 402 are adjusted. This is achieved automatically without user intervention. Built-in high-precision instrumentation allows for a deep and intuitive understanding of the impact of adjusting various performance parameters, including thread, process, and core loading and hardware utilization. In some embodiments, computer performance is visualized in a user interface of the client device 302. Computer performance is analyzed in real time during execution, thereby having the potential to automatically adjust parameters to enhance performance without user intervention.

[0114] In some embodiments, the plurality of image processing cycles 404 include a plurality of first image processing cycles 404-1 ( Figure 5 and Figure 6 ). When processing the plurality of first image processing cycles 404-1 in the plurality of parallel pipelines 402, the electronic device 100 dynamically adjusts one or more parameters of the following items: the number of pipelines, the size of the thread pool, the thread priority, the delay, the throttling parameter, the core assignment, the acquisition parameter, the lighting, and the camera focus. The plurality of second image processing cycles 404-2 are after the plurality of first image processing cycles 404-1. The electronic device assigns the plurality of second image processing cycles 404 to the plurality of parallel pipelines 402. Based on the one or more adjusted parameters, the plurality of second image processing cycles 404-2 are processed in the plurality of parallel pipelines 402.

[0115] In some embodiments, the client device 302 executes an interactive user application. The electronic device 100 enables display of a user interface, and displays the first report data on the user interface of the client device 302. In addition, in some embodiments, the user application is driven by a first programming language, and the plurality of image processing cycles 404 are executed by a second programming language different from the first programming language, wherein the plurality of image processing cycles 404 are configured to automatically exchange instructions and data with the user application via an intermediate data representation between the first programming language and the second programming language. In some embodiments, the interactive user application is implemented locally on the client device 302, and the client device is connected to the electronic device via a wired communication link or a local communication network. In some embodiments, the interactive user application is web-based and is implemented on the client device 302 connected to the electronic device via a WAN (e.g., the Internet). For example, a web-based user interface can be implemented in a browser-based application, which provides services directly from the client device 302 and uses HTTP and web sockets to communicate with the electronic device 100. The user interface can be used to define, set, adjust, and debug the electronic device 100 and its associated workflows and jobs.

[0116] In some implementations, the first image processing cycle 404A is implemented on the electronic device 100, which is coupled to a client device via a local communication network.

[0117] In one aspect of the present application, a multi-process optimization application is implemented in factory automation. Quality control requires monitoring and iterative tuning of multiple system settings (e.g., including camera settings, image processing settings, processor settings). Examples of critical system settings include, but are not limited to, image capture times in multiple image processing cycles, which require capturing image sequences at precise times or at precise intervals. Some embodiments include high-precision instrumentation and interactive analysis of complex multi-process and multi-core applications, and are used to set up embedded devices in real time in factory automation scenarios. In some embodiments, multiple image processing cycles generate a variety of event-type data, including I / O timing, signal data, performance metrics, image data, and associated data changes at one or more levels of time accuracy. The data generated by the image processing cycles is dynamically merged to maintain insights over time. The interactive oscilloscope-type user interface enables real-time visualization, manipulation, and intuitive tuning of system settings, thereby improving the iterative trial and error process. The oscilloscope-type user interface facilitates the software application 230 ( Figure 2 ), the software application 230 acquires an image containing a tag (e.g., a barcode) and decodes the tag, and provides the user with unprecedented insight into the behavior of the electronic device 100 in the image processing environment 300.

[0118] Fig. 9 is a diagram of a data structure 900 of signals and data used in an image processing environment 300 according to some embodiments. The electronic system includes an electronic device 100 having a camera and a computer device 302 for visualizing signals and data. The electronic system implements a plurality of image processing cycles 404 ( Figure 4 ), and in response to one or more corresponding triggering events 602 ( Figure 6 ) creates each image processing cycle 404. The electronic system includes a system setting 902 for setting a plurality of image processing cycles 404. During the image processing cycle 404, the electronic system obtains a plurality of input / output (I / O) signals 904, which are optionally obtained at an input or output interface of the electronic system. Based on the plurality of I / O signals 904, the electronic system generates a plurality of cycle status signals 906, and each cycle status signal 906 is associated with a sequence of timestamps 914 and indicates the progress of the corresponding image processing cycle 404. In some embodiments, the cycle status signal 906 is generated with high precision, and the sequence of timestamps 914 has a time resolution higher than a predefined resolution (e.g., every two consecutive timestamps have a time difference of less than 20 nanoseconds (ns)). For example, the predefined resolution is 20ns, and the time resolution is optionally 2ns, 5ns, 10ns, and 13ns.

[0119] In some embodiments, the system settings 902 include a set of operating configurations, and the electronic system includes a configuration service 606 ( Figure 6 ), wherein the set of job configurations is applied to define an image processing cycle 404 to process image data captured in response to the time sequence of triggers 602. In some embodiments, system settings 902 (e.g., job configurations) of the electronic system include, but are not limited to, one or more of the following: camera settings 902A, image processing settings 902B, and processor settings 902C. Examples of camera settings 902A include, but are not limited to, camera exposure time, focal length, and gain. Examples of image processing settings 902B include, but are not limited to, image filtering options, lighting parameters, window of interest (WOI), color conversion, linear and nonlinear calibration settings, contrast correction, and histogram normalization. Examples of processor settings 902C include, but are not limited to, thread pool size, thread priority, thread affinity (core assignment), trigger delay, stack size, timeout, throttling, and heartbeat frequency. In some embodiments, the system settings 902 also include the software application 230 ( Figure 2 ) application settings. Application settings include user configurable settings (e.g., symbol type, threshold, region of interest (ROI) size) and internal system settings (e.g., thread priority, delay, timeout, network usage). Application settings optionally overlap with image processing settings 902B or processor settings 902C.

[0120] In some embodiments, multiple I / O signals 904 are obtained from two or more sources and are associated with timestamps 914 related to a precise time protocol. Each source includes an internal source or an external source. An example of a precise time protocol is IEEE 1588. In some embodiments, multiple I / O signals 904 include one or more of the following: digital signals 904A, event signals 904B, and blip signals 904C. In an example, multiple I / O signals 904 include one or more images captured in response to each trigger 603, a trigger signal for each trigger 602, and barcode information extracted from one or more images. In some embodiments, digital signal 904A has a low to high transition edge and a high to low transition edge, and one or both of these edges correspond to a virtual or physical I / O event. In some embodiments, event signal 904B is obtained for one or more of the following: error or exception, mode change, external command received, process or trigger limit exceeded, data mismatch, unexpected sequence number, stop event, start event, external device connected and / or external device disconnected. In some embodiments, flag signal 904C indicates that the electronic system, parallel pipeline 402 ( Figure 4 ) and thread 612( Figure 6) duration associated with the image. Examples of marker signals 904 include, but are not limited to, start and end times for image acquisition, image processing, pre-processing, post-processing, formatting, report generation, network transmission, and parsing. In addition, in some embodiments, the marker signal 904 is stored with identifiers of the image thumbnail and the full-resolution image.

[0121] In addition, in some embodiments, the electronic system also includes one or more metric signals 908 indicating overall system performance associated with the plurality of image processing cycles 404. The one or more metric signals 908 are optionally measured directly by the electronic system or generated based on the I / O signal 904. In some embodiments, the one or more metric signals 908 include a hardware performance signal 908A and a signal quality signal 908B. Examples of hardware performance signals 908A include, but are not limited to, image acquisition time, CPU load, core temperature, memory usage level, number of threads, network traffic, and parameter changes. In some embodiments, a subset of the metric signals 908 includes secondary signals generated based on real-time analysis of the I / O signal 904. For example, the duration is measured between transitions of the digital signal 904 to determine the duty cycle, time period (e.g., pulse width), high level time, low time level, and frequency of the cycle, which corresponds to the secondary signal of the digital signal 904 and is presented as an analog signal for further processing and rendering.

[0122] Additionally, in some implementations, the electronic system generates one or more statistical signals 910 based on one or more metric signals 908, I / O signals 904, cycle status signals 906, and a subset of system settings 902. For example, the one or more statistical signals 910 include one or more of a mean, a median, a minimum, a maximum, a variance, a histogram, and an outlier of the image capture times set in response to the trigger event 602. In some cases, the statistical signal 910 is visualized as a histogram, a trend graph, a scatter plot, etc.

[0123] In some embodiments, events are identified based on system settings 902 and signals 904-910, and subsets of system settings 902 and signals 904-910 are organized according to events as event data 912 for further data storage and / or visualization. In some embodiments, events correspond to event signals 904B directly measured in association with the events. Alternatively, in some embodiments, events are detected in real time based on system settings 902 and signals 904-910. Examples of events include, but are not limited to, physical input / output events, topic publishing events, and topic subscription events.

[0124] Some implementations of the present application are intended for easy and efficient event recording. In some embodiments, the image processing environment 300 is configured to record different types of events with adaptive timestamp accuracy and adaptive data formats. Event data 912 includes digital signals, metric signals, errors or exceptions (including data such as stack traces), execution timing, context information, image acquisition time and thumbnails, counts or more. By these means, data recording is easy to use and implemented with controlled overhead, thereby allowing electronic systems to receive data from multiple sources, the range of the multiple sources from shared memory to the network, which conforms to the user datagram protocol (UDP), the transmission control protocol (TCP) or WebSocket.

[0125] Fig.10 1000 is a timing diagram of an image processing cycle 404 implemented in one of a plurality of parallel pipelines 402 according to some embodiments. The electronic system implements a plurality of image processing cycles 404 associated with a temporal sequence of triggers 602 ( Figure 4 ), and in response to one or more corresponding triggering events 602 ( Figure 6 ) creates each image processing cycle 404. During the image processing cycle 404, the electronic system obtains a plurality of input / output (I / O) signals 904, which are optionally obtained at an input or output interface of the electronic system. Based on the plurality of I / O signals 904, the electronic system generates one or more cycle status signals 906 for each image processing cycle 404, and each cycle status signal 906 is associated with a sequence of timestamps 914 and indicates the progress of the corresponding image processing cycle 404. In some embodiments, the cycle status signal 906 is generated with high precision, and the sequence of timestamps 914 has a time resolution higher than a predefined resolution (e.g., every two consecutive timestamps have a time difference of less than 20 nanoseconds (ns)).

[0126] In some embodiments, the timing diagram 1000 corresponds to one of a plurality of signal channels. On each signal channel, one or more cycle status signals 906 indicate the progress of a corresponding image processing cycle 404 implemented by a corresponding parallel pipeline 402. For example, the cycle status signals 906 include an image acquisition status signal 906A, a pipeline thread status signal 906B, and an image processing thread status signal 906C. The image acquisition status signal 906A includes a trigger event 602 ( Figure 6) and an image acquisition status bar 1004. Pipeline thread status signal 906B includes an extended line with multiple segments 1006, and each segment represents a corresponding stage of the pipeline thread. Image processing thread status signal 906C includes an extended line with multiple segments 1008 and a circular indicator 1010, which indicates the detection of a barcode on an image captured in response to trigger event 602. Cycle status signal 906 includes multiple segments 1006 and 1008 corresponding to multiple imaging stages (e.g., trigger, acquisition, setup, decoding, image processing). The multiple segments are optionally coded with different colors.

[0127] FIG. 11A to FIG. 11D 1 are four example time diagrams 1100, 1120, 1140, and 1160 of a plurality of image processing cycles 404 according to some embodiments. Each time diagram 1100, 1120, 1140, or 1160 visualizes the cycle state signals 906 of a plurality of image processing cycles 404 in a plurality of parallel signal channels. The plurality of parallel signal channels are arranged in a stacked manner, and the cycle state signals 906 of different signal channels are displayed in a synchronized manner with respect to the same time axis 1110. In other words, each cycle state signal 906 is associated with a sequence of timestamps 914, and the cycle state signals 906 of the plurality of image processing cycles 404 are synchronized with each other. On each signal channel, one or more cycle state signals 906 indicate the progress of the corresponding image processing cycle 404 implemented by the corresponding image processing pipeline 402. In some embodiments, each signal channel includes a cycle time 1104 and a cycle identifier 1102 indicating how long ago the corresponding image was captured.

[0128] refer to Fig.11A and Fig. 11B In some embodiments, the cycle status signal 906 of each image processing cycle 404 includes at least an image acquisition status signal 906A and a pipeline thread status signal 906B. The image acquisition status signal 906A includes a trigger event 602 ( Figure 6 ) trigger event indicator 1002 and image acquisition status bar 1004. Pipeline thread status signal 906B includes an extended bar with multiple segments 1006, and each segment 1006 represents a corresponding stage of the pipeline thread (e.g., pipeline setup, decoding). In addition, in some cases ( Fig.11A), multiple image processing cycles 404 are implemented by a single processor core (e.g., by a single image processing pipeline 402). After the previous image is completely processed in the previous image processing cycle 404P within the single image processing pipeline 402, the trigger event 602 for the next image processing cycle 404N occurs, and the next image is captured and then processed in the next image processing cycle 404N within the next image processing pipeline 402N. Alternatively, in some cases ( Fig. 11B ), multiple image processing cycles 404 are implemented in multiple parallel pipelines 402. When a previous image is processed in the previous image processing cycle 404P in the previous image processing pipeline 402P, a trigger event 602 of a next image processing cycle 404N occurs, and the next image is captured, and then the next image is processed in the next image processing cycle 404N in the next image processing pipeline 402N. The overlap of the next image processing pipeline 402N with the previous image processing pipeline increases the image processing rate.

[0129] refer to Fig. 11C and Fig.11D In some embodiments, the cycle status signal 906 of each image processing cycle 404 includes an image acquisition status signal 906A, a pipeline thread status signal 906B, and a plurality of image processing thread status signals 906C. The pipeline thread status signal 906B and the image processing thread status signal 906C are arranged in parallel with each other and are at least partially aligned with the same portion of the time axis 1110. The image processing thread status signal 906C includes a plurality of thread indicators 1152, which indicate the plurality of processing threads 612 ( Figure 6 ) process. In some cases ( Fig. 11C ), the trigger event 602 of the next image processing cycle 404N occurs only after the previous image is completely processed in the previous image processing cycle 404P, and the next image is captured and then processed by the multiple parallel threads 1152 of the single image processing pipeline 402 in the next image processing cycle 404N. Alternatively, in some cases ( Fig.11D ), while a previous image is being processed in a previous image processing cycle 404P of a previous image processing pipeline 402P, a trigger event 602 for a next image processing cycle 404N occurs, and a next image is captured and then processed within the next image processing pipeline 402N in the next image processing cycle 404N by the plurality of parallel threads 1152. Furthermore, in some embodiments, the combination of the plurality of image processing pipelines 402 and the plurality of processing threads 612 increases the processing rate to be equal to or faster than the image frame rate at which the sequence of images is captured.

[0130] Fig.12is an example collapsed graphical user interface (GUI) 1200 for visualizing signals of multiple image processing cycles 404 over an extended duration according to some embodiments. In some embodiments, a computer device (e.g., Figure 3 The client device 302 in the embodiment executes a dedicated digital oscilloscope application (e.g., Figure 2 230 in the computer device) and enables the user interface 1200 to be displayed on the screen of the computer device. Alternatively, in some embodiments, the computer device (e.g., Figure 3 The client device 302 in the embodiment of the present invention enables the user interface 200 to be displayed on its screen via a general browser application. The progress of the plurality of image processing cycles 404 is tracked by the plurality of cycle status signals 906 visualized on the user interface 1200. When the image processing cycle 404 is implemented by the electronic system, the user interface 1200 dynamically and in real time simultaneously visualizes the plurality of cycle status signals 906 with respect to at least the timeline 1100. In some embodiments, a subset of the plurality of I / O signals 904 is simultaneously visualized with the plurality of cycle status signals 906, and each of the subsets of the plurality of I / O signals 904-1 and 904-2 is displayed with respect to the timeline. In some embodiments, one or more metric signals 908 indicate the overall system performance associated with the plurality of image processing cycles 404, and are simultaneously visualized with the plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to the timeline 1100.

[0131] In some cases, given the length of the extended duration, each of the plurality of I / O signals 904-1 and 904-2 and the plurality of periodic state signals 906 are folded along the time axis 1100, and the details of the signals 904-1, 904-2, and 906 are shown in FIG. Fig.12 In addition, in some cases, Fig.12 The zoom operation is implemented on the time axis 1100 to extend the time axis 1100 to Fig.13 In some embodiments, although the details of signals 904-1, 904-2, and 906 are missing, the application Fig.12 The signal density shown on the graph is used to detect an abnormal event 1202 of the periodic state signal 906. In addition, in some embodiments, the extended duration helps to detect another abnormal event 1204 (also referred to as an outlier 1204) of the metric signal 908 on the user interface 1200. Data associated with the abnormal events 1202 and 1204 are extracted and stored for further review and analysis. Such abnormal events 1202 and 1204 are optionally reported to the user of the user interface 1200 in real time by displaying an alert on the user interface 1200 or by including the detection of the abnormal event in an output report.

[0132] In other words, in some embodiments, an outlier 1204 of at least one metric signal 908 is detected based on an outlier criterion. In response to detecting the outlier 1204, a portion of at least one metric signal 908 associated with the outlier 1204 is highlighted on the user interface 1200. In some cases, in response to detecting the outlier 1204, additional coincident data is recorded to facilitate further review and analysis of the outlier 1204. In addition, in some embodiments, the at least one metric signal 908 includes two or more metric signals 908. The electronic system monitors the correlation of the two or more metric signals 908, and the outlier criterion is associated with the change in the correlation. In addition, in some embodiments, the at least one metric signal 908 includes one or more of the following items: CPU usage level, CPU operation duration, core temperature, memory usage, and network bandwidth. In response to detecting the outlier 1204, an action is initiated to control CPU usage of a subset of the plurality of image processing cycles 404. In these ways, abnormal events are monitored and applied to adjust the allocation of computing, storage, and / or network resources between image processing cycles 404, thereby allowing for more efficient use of resources.

[0133] The electronic system is configured for dynamic data aggregation and merging with outlier detection. In some embodiments, the electronic system dynamically merges recorded data (e.g., I / O signals 904) based on the age of the data. Recent data is stored with higher precision, while older data is progressively aggregated, thereby maintaining key statistical information or reducing image resolution while ensuring the continued availability of historical information. In addition, in some embodiments, an outlier condition is specified. Based on determining that a subset of settings 902 and signals 904-912 meets an outlier condition (i.e., in response to detecting an outlier condition), the corresponding event data 912 includes the subset of settings 902 and signals 904-912 and is stored for further review and analysis.

[0134] Fig.13 404 is another example user interface 1300 for visualizing signals from a plurality of image processing cycles 404 according to some embodiments. Figure 3 The client device 302 in the embodiment executes a dedicated digital oscilloscope application (e.g., Figure 2 230) or a general browser application, and enables the user interface 1300 to be displayed on the screen of the computer device. Fig.12 Compared to user interface 1200 in FIG. 1 , user interface 1300 corresponds to time window 1206 having a shorter duration ( Fig.12), thereby providing more signal details. The progress of multiple image processing cycles 404 is tracked by multiple cycle status signals 906 visualized on user interface 1300. When image processing cycles 404 are implemented by the electronic system, user interface 1300 dynamically and in real time simultaneously visualizes multiple cycle status signals 906 with respect to at least time axis 1100.

[0135] In some embodiments, the timeline 1100 includes a single timeline. For each cycle status signal 906, an active portion and an associated activity duration of the corresponding status signal 906 are identified. Based on the associated activity duration, the user interface 1300 displays the active portion of the corresponding cycle status signal 906 on a row 1320 with respect to the single timeline. For example, each cycle status signal 906 is cropped to an image processing thread status signal 906C including a plurality of thread indicators 1152. The thread indicators 1152 of the plurality of cycle status signals 906 are shifted to the same row 1320. In addition, in some embodiments, based on determining that an active portion of a first cycle status signal 906-1 of a first image processing cycle and an active portion of a second cycle status signal 906-2 of a second image processing cycle overlap each other on the single timeline, it is determined that at least one of the first image processing cycle and the second image processing cycle has encountered an abnormal condition. In some embodiments, based on determining that an active portion of a first cycle state signal 906-1 for a first image processing cycle and an active portion of a second cycle state signal 906-2 for a second image processing cycle overlap with each other on a single time axis 1110, the electronic system automatically extends an inter-image time gap between two triggers 602 configured to enable capture of two consecutive images.

[0136] In some embodiments, in response to detecting the overlap of the two periodic state signals 906-1 and 906-2, the user interface 1300 is displayed so that the overlapping portion of each of the two periodic state signals is highlighted. In addition, in some embodiments, in response to detecting the overlap of the two periodic state signals 906-1 and 906-2, the signals 902, 904, and 908 are monitored to identify an outlier 1322 corresponding to an abnormal event, and the outlier 1322 is optionally highlighted on the user interface 1300.

[0137] In some embodiments, a subset of the plurality of I / O signals 904 (e.g., flag signal 904C) is visualized simultaneously with the plurality of cycle status signals 906, and each of the subset of the plurality of I / O signals 904 is displayed with respect to the timeline 1100. In some embodiments not shown, one or more metric signals 908 indicate overall system performance associated with the plurality of image processing cycles 404, and are visualized simultaneously with the plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to the timeline 1100.

[0138] In some implementations, one or more statistical signals 910 are generated based on a subset of one or more metric signals 908, multiple I / O signals 904, multiple cycle status signals 906, and multiple system settings 902. The one or more statistical signals 910 are visualized simultaneously with the multiple cycle status signals 906 and about an axis different from the time axis 1110. Fig.13 In some embodiments, a statistical signal 910 of the image capture time 1302 is determined over an extended period of time (e.g., for 177 image processing cycles 404). A histogram 1304 of the statistical signal of the image capture time 1302 is visualized on the user interface 1300, thereby representing the distribution of the image capture time 1302 for the 177 image processing cycles 404. A median 1306, a mean 1308, two standard deviation values ​​1310, and a triple standard value 1312 are marked on the histogram 1304. One or more outliers 1314 are detected that exceed the triple standard value 1312 and are associated with external and internal events. In some embodiments, event data 912 associated with the outlier 1314 is analyzed after the anomaly occurs.

[0139] In some embodiments, the electronic system enables display of a user interface 1300 on a screen of the computer device 302 via a browser application. The user interface 1300 implements an advanced browser-based environment and is similar to the user interface of a digital oscilloscope. The user interface 1300 navigates across a wide range of time scales and renders appropriate signals, events, or data, including I / O signals 904, cycle status signals 906, metric signals 908, timing durations, statistical signals 910, image thumbnails, and parameter information. The user interface 1300 provides the software application 230 ( Figure 2 ) intuitive visual representation of behavior over time. In some embodiments, user interface 1300 implements one or more features of sorting, filtering, grouping, and logging.

[0140] In some embodiments, the electronic system implements interactive data analysis. The user interface 1300 allows the user to identify signal interactions, such as correlations between high CPU usage and duration of operation, for example, by providing features that track and display such relationships. These signal interactions trigger outlier detection and actions. For example, if an outlier condition occurs that requires further review, the event data 912 including associated time records and statistics is permanently stored in the database 242 ( Figure 2 ) for further review and analysis.

[0141] In some embodiments, the electronic system implements image-based data rendering. The electronic system renders the image-based data so that overlaps between images become apparent. In an example, images of long boxes moving on a conveyor are captured from a fixed camera position. The images have overlaps for comprehensive surface inspection. The electronic system allows a user to visualize these overlaps on a timeline on the user interface 1300. In some embodiments, the user interface 1300 allows adjustment of inter-image timing based on one or more factors. In addition, in some cases, inter-image timing is automatically inferred based on a subset of signals 904-912.

[0142] In some embodiments, the image processing environment 300 is adjusted or optimized in real time. The user interface 1300 allows for analysis of timing data for real-time system optimization. Based on the user interface 1300, the user can interactively and in real time adjust the system settings 902 while observing the impact of the system settings 902 on the system performance, which is reflected by the signals 904-912. In some embodiments, machine learning or other automated methods are used to analyze the signals 904-912. Based on the analysis results, the electronic system automatically, dynamically, and in real time adjusts the system settings 902.

[0143] refer to Fig.13In some embodiments, the user interface 1300 also displays an operational status signal different from the periodic status signal 906. For example, when an imaging-based examination is not actively running, the operational status signal monitors the offline status of the electronic device 100. Specifically, in some embodiments, a method is implemented to track the status of an electronic system configured for image processing or other operations, regardless of whether the electronic system is active or offline. The electronic system obtains one or more of the following items: multiple system settings 902, multiple input / output (I / O) signals 904 (e.g., from an I / O interface), multiple status signals 906 (e.g., generated based on the I / O signals), multiple metric signals 908, and multiple statistical signals 910. In some embodiments, the electronic system identifies data or abnormal events based on the settings 902 and / or signals 904-910. The electronic system dynamically and in real time simultaneously visualizes at least a subset of the settings 902 and signals 904-910 on the user interface with respect to at least a time axis. The method is used to analyze the performance of an electronic system related to image processing, digital input / output, or other operations. For example, the electronic system is applied to analyze and discover timing anomalies between independent communication processes and programmable logic controllers (PLCs). In another example, the electronic system is applied to discover performance issues of certain image processing libraries under load. In some embodiments, multiple software applications 230 are executed simultaneously to receive timing data streams from multiple sources. Examples of applications include, but are not limited to, setup software and independent data visualization and / or analysis applications.

[0144] Fig.14A is a portion of an example user interface 1400 that visualizes cycle status signals 906 and I / O signals 904 for three image processing cycles 404 according to some embodiments, and Fig. 14B 14 is a portion of another example user interface 1450 showing a histogram of capture times of a plurality of image processing cycles 404 according to some embodiments. In some embodiments, an area of ​​the user interface 1300 is selected and displayed in a zoom mode. For example, the user interface 1400 corresponds to the selected area 1316, in which the cycle status of three image processing cycles 404 can be viewed in detail. In another example, the user interface 1450 corresponds to the selected area 1318. A histogram 1304 of a statistical signal of image capture time 1302 is visualized on the user interface 1450, representing the distribution of image capture time 1302 of 177 image processing cycles 404. The median 1306, the mean 1308, the two standard deviation values ​​1310, and the triple standard value 1312 are marked on the histogram 1304. One or more outliers 1314 are detected to exceed the triple standard value 1312 and are associated with external events and internal events. In some embodiments, the event data 912 associated with the outlier 1314 is analyzed after the anomaly occurs.

[0145] Fig.15 1 is a flow chart of a method 1500 for real-time tracking of an image processing cycle according to some embodiments. The method 1500 is implemented at an electronic system for real-time tracking of an image processing cycle 404. The electronic system has a memory and one or more processors and includes an electronic device 100 and a computer device 302 ( Figure 3 ). In some embodiments, the electronic device 100 includes one or more cameras 112 configured to capture images. In some embodiments, the electronic device 100 is coupled to a camera or a different electronic device having a camera, and is configured to acquire image data including a sequence of images.

[0146] The electronic system implements (1502) a plurality of image processing cycles 404 associated with a time sequence of triggers, and in response to one or more corresponding trigger events 602 ( Figure 6 ) creates (1504) each image processing cycle 404. The electronic system obtains (1506) a plurality of input / output (I / O) signals 904 for the plurality of image processing cycles 404, and generates (1508) a plurality of cycle status signals 906 based on the I / O signals 904. Each cycle status signal 906 is associated (1510) with a sequence of time stamps 914 and indicates the progress of the corresponding image processing cycle 404. The sequence of time stamps 914 has (1512) a temporal resolution that is higher than a predefined resolution. When implementing the image processing cycle 404, the electronic system dynamically and in real time displays the image processing cycle 404 in a user interface (e.g., FIG. 11A to FIG. 14B Multiple periodic status signals 906 are simultaneously visualized (1514) on a user interface (shown).

[0147] In some embodiments, the electronic system obtains one or more metric signals 908 indicating overall system performance associated with a plurality of image processing cycles 404, and visualizes the one or more metric signals 908 simultaneously with a plurality of cycle status signals 906. Each of the one or more metric signals 908 is displayed with respect to a timeline 1110. In addition, in some embodiments, the electronic system detects an outlier of at least one metric signal based on an outlier criterion. In response to detecting an outlier, a portion of at least one metric signal 908 associated with the outlier is highlighted on a user interface. In addition, in some embodiments, at least one metric signal includes two or more metric signals 908. The electronic system monitors the correlation of the two or more metric signals 908, and the outlier criterion is associated with a change in the correlation. In some embodiments, at least one metric signal includes one or more of the following items: CPU usage level, CPU operation duration, core temperature, memory usage, and network bandwidth. In response to detecting an outlier, the electronic system initiates an action on CPU usage by a subset of the plurality of image processing cycles 404.

[0148] In some embodiments, each of the one or more metric signals 908 is one of the following: image acquisition time, CPU core load percentage, CPU temperature, memory usage, number of threads, and network traffic. In some embodiments, the one or more metric signals 908 include one of the following: period, frequency, duty cycle, pulse width, high time, and low time of one of the plurality of I / O signals 904.

[0149] In some implementations, the electronic system visualizes a subset of the plurality of I / O signals 904 simultaneously with the plurality of periodic status signals 906 , and displays each of the subset of the plurality of I / O signals 904 with respect to the timeline 1110 .

[0150] In some implementations, the electronic system generates one or more statistical signals 910 based on the one or more metric signals 908, the plurality of I / O signals 904, the plurality of cycle status signals 906, and a subset of the plurality of system settings 902. The electronic system also visualizes the one or more statistical signals 910 simultaneously with the plurality of cycle status signals 906 and about a non-time axis 1110.

[0151] In some implementations, visualizing multiple periodic state signals 906 further includes displaying each of multiple periodic state signals 906 in a stacked manner. Multiple periodic state signals 906 are displayed with respect to the same timeline 1110 in a synchronized manner.

[0152] In some embodiments, the timeline 1110 includes a single timeline 1110, and visualizing the plurality of periodic status signals 906 further includes, for each periodic status signal 906, identifying an active portion of the corresponding status signal and an associated active duration; and displaying the active portion of the corresponding periodic status signal 906 on a row with respect to the single timeline 1110 according to the associated active duration. Further, in some embodiments, based on determining that an active portion of a first periodic status signal 906-1 of a first image processing cycle and an active portion of a second periodic status signal 906-2 of a second image processing cycle overlap each other on the single timeline 1110, the electronic system determines that at least one of the first image processing cycle and the second image processing cycle has encountered an abnormal condition. Further, in some embodiments, based on determining that an active portion of a first periodic status signal 906-1 of a first image processing cycle and an active portion of a second periodic status signal 906-2 of a second image processing cycle overlap each other on the single timeline 1110, the electronic system automatically extends the inter-image time gap.

[0153] In some implementations, in response to detecting an overlap of two periodic status signals 906 , the electronic system highlights the overlapping portion of each of the two periodic status signals 906 on a user interface.

[0154] In some implementations, the plurality of I / O signals 904 include one or more images captured in response to each trigger, a trigger signal for each trigger, and barcode information extracted from the one or more images.

[0155] In some implementations, the plurality of I / O signals 904 are obtained from two or more sources and are associated with time stamps that are related according to a precision time protocol.

[0156] In some embodiments, the predefined resolution is 20 nanoseconds.

[0157] In some embodiments, each image processing cycle 404 includes one or more imaging phases of: trigger, acquire, setup, decode, and image processing, and corresponds to a respective cycle status signal 906 - 2 that is color-coded based on the one or more imaging phases.

[0158] In some implementations, the first image processing cycle includes an image processing phase in which the first image is processed by a plurality of parallel processing threads.

[0159] In some implementations, an electronic system executes a digital oscilloscope application including a user interface that enables display on a screen of a computer device.

[0160] In some implementations, the electronic system enables display of a user interface on a screen of a computer device via a browser application.

[0161] In some embodiments, each system setting 902 is one of a plurality of camera settings 902A, a plurality of image processing settings 902B, and a plurality of processor settings 902C. The plurality of camera settings 902A includes one or more of the following: exposure time, focal length, gain, image filtering options, lighting parameters, window of interest parameters, color conversion parameters, linear and nonlinear calibration settings, contrast correction parameters, and histogram normalization parameters. The plurality of image processing settings 902B includes one or more of the following: symbol settings, effort level, qualifier settings, region of interest, and number of retries. The plurality of processor settings 902C includes one or more of the following: thread pool size, thread priority, thread affinity (CPU core assignment), trigger delay, stack size, timeout, throttling parameters, and heartbeat frequency.

[0162] Each of the executable modules, applications or process sets identified above can be stored in one or more memory devices in the previously mentioned memory devices, and corresponds to an instruction set for performing the above functions. The modules or programs (i.e., instruction sets) identified above do not need to be implemented as separate software programs, processes or modules, and therefore, in various embodiments, various subsets of these modules can be combined or otherwise rearranged. In some embodiments, memory 206 stores a subset of the modules and data structures identified above. In addition, memory 206 can store additional modules and data structures not described above.

[0163] The terms used in the description of the present invention are only used to describe specific embodiments and are not intended to limit the present invention. As used in the description of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are also intended to include plural forms. It will also be understood that the terms "and / or" used herein refer to and cover any one of one or more related enumerated items and all possible combinations thereof. It will also be understood that the terms "comprise" and "comprising" when used in this specification specify the presence of the features, steps, operations, elements and / or components, rather than excluding the presence or addition of one or more other features, steps, operations, elements, components and / or combinations thereof.

[0164] As used herein, unless explicitly specified otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0165] As used herein, the word “exemplary” means “serving as an example, instance, or illustration,” and does not necessarily indicate any preference or superiority of an example over any other configurations or implementations.

[0166] As used herein, the term "and / or" encompasses any combination of the listed elements. For example, "A, B, and / or C" includes the following element sets: only A, only B, only C, A and B without C, A and C without B, B and C without A, or a combination of all three elements A, B, and C.

[0167] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exclusive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and various embodiments with various modifications suitable for the intended specific use.

[0168] Related Applications

[0169] This application is a continuation-in-part of and claims priority to U.S. patent application No. 18 / 217,417, filed on June 30, 2023, entitled “State Managed Asynchronous Runtime,” which is incorporated herein by reference in its entirety.

Claims

1. A method for real-time tracking of an image processing cycle, the method being implemented at an electronic system having a memory and one or more processors, the method comprising the following steps: implementing a plurality of image processing cycles associated with the time sequence of triggers, each image processing cycle being created in response to one or more respective triggering events; Obtaining a plurality of input / output I / O signals of the plurality of image processing cycles; generating a plurality of periodic status signals based on the I / O signal, wherein each periodic status signal is associated with a time stamp sequence and indicates a progress of a corresponding image processing cycle, the time stamp sequence having a time resolution higher than a predefined resolution; as well as When the image processing cycle is implemented, the plurality of cycle status signals are simultaneously visualized on a user interface dynamically and in real time with respect to at least a time axis.

2. The method according to claim 1, further comprising the steps of: obtaining one or more metric signals indicative of overall system performance associated with the plurality of image processing cycles; as well as The one or more metric signals are visualized simultaneously with the plurality of periodic status signals, each of the one or more metric signals being displayed with respect to the time axis.

3. The method according to claim 2, further comprising the steps of: detecting outliers of at least one metric signal based on an outlier criterion; as well as In response to detecting the outlier, a portion of the at least one metric signal associated with the outlier is highlighted on the user interface.

4. The method according to claim 3, wherein: The at least one metric signal comprises two or more metric signals, and the method further comprises the following steps: A correlation of the two or more metric signals is monitored, wherein the outlier criterion is associated with a change in the correlation.

5. The method according to claim 3, wherein: The at least one metric signal comprises one or more of the following: CPU usage level, CPU operation duration, core temperature, image acquisition time, memory usage, number of threads, network traffic, and network bandwidth, the method further comprising the steps of: In response to detecting the outlier, an action on CPU usage of a subset of the plurality of image processing cycles is initiated.

6. The method according to claim 1, further comprising the steps of: A subset of the plurality of I / O signals is visualized simultaneously with the plurality of periodic state signals, each of the subset of the plurality of I / O signals being displayed with respect to the time axis.

7. The method according to claim 1, further comprising the steps of: generating one or more statistical signals based on a subset of a plurality of system settings, the plurality of cycle state signals, the plurality of I / O signals, and one or more metric signals; as well as The one or more statistical signals are visualized simultaneously with the plurality of periodic state signals and with respect to a non-time axis.

8. The method according to claim 1, wherein: The step of visualizing the plurality of periodic status signals further comprises: Each of the plurality of periodic state signals is displayed in a stacked manner, wherein the plurality of periodic state signals are displayed in a synchronized manner with respect to a same time axis.

9. The method according to claim 1, wherein: The time axis includes a single time axis, and the step of visualizing the plurality of periodic state signals further includes: for each periodic state signal: identifying an active portion of a corresponding status signal and an associated active duration; and The active portion of the corresponding periodic status signal is displayed on a row with respect to the single time axis according to the associated active duration.

10. The method according to claim 9, further comprising the steps of: Based on determining that the active portion of the first cycle status signal of the first image processing cycle and the active portion of the second cycle status signal of the second image processing cycle overlap with each other on the single time axis, it is determined that at least one of the first image processing cycle and the second image processing cycle has encountered an abnormal condition.

11. The method according to claim 9, further comprising the steps of: The inter-image time gap is automatically extended based on determining that an active portion of a first periodic state signal of a first image processing period and an active portion of a second periodic state signal of a second image processing period overlap each other on the single time axis.

12. An electronic system, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the processor, the one or more programs further comprising instructions for: implementing a plurality of image processing cycles associated with the time sequence of triggers, each image processing cycle being created in response to one or more respective triggering events; Obtaining a plurality of input / output I / O signals of the plurality of image processing cycles; generating a plurality of periodic status signals based on the I / O signal, wherein each periodic status signal is associated with a time stamp sequence and indicates a progress of a corresponding image processing cycle, the time stamp sequence having a time resolution higher than a predefined resolution; as well as When the image processing cycle is implemented, the plurality of cycle status signals are simultaneously visualized on a user interface dynamically and in real time with respect to at least a time axis.

13. The electronic system of claim 12, wherein the one or more programs further comprise instructions for: In response to detecting an overlap of two periodic status signals, an overlapping portion of each of the two periodic status signals is highlighted on the user interface.

14. The electronic system according to claim 12, wherein: The plurality of I / O signals include one or more images captured in response to each trigger, a trigger signal for each trigger, and barcode information extracted from the one or more images.

15. The electronic system according to claim 12, wherein: The plurality of I / O signals are obtained from two or more sources and are associated with time stamps that are related according to a precise time protocol.

16. A non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of an electronic system, the one or more programs comprising instructions for: implementing a plurality of image processing cycles associated with the time sequence of triggers, each image processing cycle being created in response to one or more respective triggering events; Obtaining a plurality of input / output I / O signals of the plurality of image processing cycles; A plurality of periodic status signals are generated based on the I / O signal, wherein: Each cycle status signal is associated with a time stamp sequence having a temporal resolution higher than a predefined resolution and indicates the progress of a corresponding image processing cycle; as well as When the image processing cycle is implemented, the plurality of cycle status signals are simultaneously visualized on a user interface dynamically and in real time with respect to at least a time axis.

17. The non-transitory computer-readable storage medium of claim 16, wherein: Each image processing cycle includes one or more imaging stages of: triggering, acquiring, setting, decoding, and image processing, and each image processing cycle corresponds to a respective cycle status signal coded by color based on the one or more imaging stages.

18. The non-transitory computer-readable storage medium of claim 16, wherein: The first image processing cycle includes an image processing phase in which a first image is processed by a plurality of parallel processing threads.

19. The non-transitory computer-readable storage medium of claim 16, the one or more programs further comprising instructions for one of: executing a digital oscilloscope application including enabling display of the user interface on a screen of a computer device; and Display of the user interface on the screen of the computer device is enabled via a browser application.

20. The non-transitory computer-readable storage medium of claim 16, wherein: Each system setting is one of a plurality of camera settings, a plurality of image processing settings, and a plurality of processor settings; The plurality of camera settings include one or more of: exposure time, focal length, gain, image filtering options, illumination parameters, window of interest parameters, color conversion parameters, linear and non-linear calibration settings, contrast correction parameters, and histogram normalization parameters; The plurality of image processing settings include one or more of: a symbol setting, an effort level, a qualifier setting, an area of ​​interest, and a number of retries; and The plurality of processor settings include one or more of: thread pool size, thread priority, thread affinity CPU core assignment, trigger delay, stack size, timeout, throttling parameters, and heartbeat frequency.

Images