Industrial visual data processing method based on netty adaptive space-time window

Abstract

The application discloses an industrial visual data processing method based on a Netty adaptive space-time window, relates to the technical field of industrial visual data processing, and comprises the following steps: establishing a data acquisition layer, performing space-time header analysis on a data packet through a protocol decoding module, and setting a space-time constraint processor; and performing dynamic window verification on image data of each station. The application effectively solves the frame disorder problem of data transmission in the same station, significantly reduces the cross-station quality inspection false detection and missed detection caused by network delay jitter, realizes strong adaptive capability for a dynamic network environment, and improves the real-time performance, stability and quality inspection reliability of the industrial visual detection system.

Description

Technical Field

[0001] This invention relates to the field of industrial vision data processing technology, and in particular to an industrial vision data processing method based on Netty's adaptive spatiotemporal window. Background Technology

[0002] Currently, the existing technology has limitations. Traditional detection hardware uses high-performance industrial cameras, smart sensors, and edge computing devices, while software combines deep learning models and real-time operating systems (RTOS) to achieve efficient processing.

[0003] Despite the fact that existing technologies have achieved efficient detection, the following pain points still exist: strong network dependence, edge devices rely on stable networks, and network latency or jitter can lead to out-of-order detection; rigid fixed thresholds, traditional time windows cannot adapt to dynamic network environments, there is redundant latency during stable periods, and a high false negative rate during jitter periods; weak multi-station collaboration, global unified buffering strategies ignore network differences between stations, resulting in low resource utilization. Summary of the Invention

[0004] In view of the problems existing in the existing industrial vision data processing methods based on Netty adaptive spatiotemporal windows, this invention is proposed.

[0005] Therefore, the problem to be solved by this invention is: how to overcome the problems of missed detection, false detection and low resource utilization caused by data disorder due to network latency jitter, rigid fixed time windows and weak multi-station collaboration in existing industrial vision inspection systems, and to realize a spatiotemporal window data processing method that can adapt to dynamic network environment and be independently adjusted according to station.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide an industrial vision data processing method based on Netty's adaptive spatiotemporal window, which includes establishing a data acquisition layer and performing timestamp synchronization and structured encapsulation of image data from each workstation in the data acquisition layer.

[0008] The protocol decoding module performs spatiotemporal header parsing on the data packets and sets a spatiotemporal constraint processor, which includes an event mode and a timed mode.

[0009] Dynamic window verification is performed on the image data of each workstation to complete the industrial vision data processing with adaptive spatiotemporal windowing.

[0010] As a preferred embodiment of the industrial vision data processing method based on Netty adaptive spatiotemporal window described in this invention, the timestamp synchronization includes microsecond-level clock synchronization using the IEEE 1588 protocol, and the structured encapsulation includes embedding an 8-byte timestamp, a 2-byte workstation ID, and a 4-byte frame sequence number in the data packet.

[0011] As a preferred embodiment of the industrial vision data processing method based on Netty's adaptive spatiotemporal window described in this invention, the spatiotemporal header parsing of data packets through the protocol decoding module includes verifying the validity based on the 2-byte workstation ID, diverting data from different workstations to independent receiving queues, and sorting and storing the data in each receiving queue according to the frame sequence number to achieve temporal consistency of data within the same workstation ID.

[0012] As a preferred embodiment of the industrial visual data processing method based on Netty's adaptive spatiotemporal window described in this invention, the spatiotemporal constraint processor includes an event mode and a timing mode.

[0013] The event mode includes the following technical steps:

[0014] When data arrives, the network monitor is notified to update network metrics.

[0015] Data is stored in the corresponding queue according to the frame number, based on the different workstations and their respective queues.

[0016] Retrieve the data at the head of the queue and obtain the dynamic window from the dynamic window calculator;

[0017] The process of obtaining a dynamic window from the dynamic window calculator includes the following technical steps:

[0018] The time difference is calculated by subtracting the absolute value of the current time from the data's timestamp.

[0019] When the time difference is less than or equal to the time window (i.e., the time window verification is successful), the data is pushed to TensorRT inference, and the verified data is passed to the time window to optimize the time window calculation.

[0020] Conversely, if time window verification fails, the dynamic window calculator is used to handle the time window verification failure and optimize the time window calculation.

[0021] As a preferred embodiment of the industrial vision data processing method based on Netty's adaptive spatiotemporal window described in this invention, the timing mode includes re-checking the queue status of all different workstations at fixed time intervals.

[0022] If the processing time difference between the current time and the previous time is greater than 5 seconds and the queue size is 0, it is considered to be in an idle state, and the time window value is reset to the base window value.

[0023] When the backlog of queues at different workstations exceeds the threshold If the dynamic time window value is recalculated, the recalculation of the dynamic time window value is part of the dynamic time window adaptive mechanism, which is used to adjust the time tolerance threshold according to the queue load status of the workstation.

[0024] As a preferred embodiment of the industrial visual data processing method based on Netty adaptive spatiotemporal window described in this invention, the dynamic window calculator includes generating a duration value of a dynamic time window for judging whether the data is valid based on the average jitter, out-of-order rate and current queue backlog of each different workstation provided by the network monitor.

[0025] The duration of the dynamic time window refers to the window size of the dynamic time window, which is used in the spatiotemporal constraint processor to determine whether the received visual data has arrived within a reasonable time range, thereby determining whether the data can be sent into the subsequent inference and detection process.

[0026] The network monitor is used to update network metrics, and the updating of network metrics includes the following steps:

[0027] Calculate the total transmission delay experienced by each data packet from the acquisition end to the receiving end;

[0028] Calculate the average degree of arrival delay variation among consecutively arriving data packets at the same workstation, i.e., the average value of arrival interval jitter;

[0029] The adaptive adjustment for dynamic time windows includes two parts: adopting the index-weighted moving average algorithm recommended by the IETF and calculating the out-of-order rate.

[0030] When the out-of-order rate is less than 1%, the basic safety margin is used; when the out-of-order rate is greater than 1%, it is dynamically increased proportionally.

[0031] As a preferred embodiment of the industrial visual data processing method based on Netty's adaptive spatiotemporal window described in this invention, the basic window value is used to reflect the minimum buffer value under network conditions, so that the network has a buffer, and is set to 20ms.

[0032] The average jitter includes configuring a jitter amplification coefficient and obtaining the average arrival interval jitter value of the workstation.

[0033] The out-of-order rate includes a safety margin based on the out-of-order rate, used to deal with jitter or measurement errors;

[0034] The current queue backlog includes an analysis of the current queue backlog based on the current queue backlog status. When the queue backlog increases, the dynamic time window is increased to allow data to be processed. When the queue backlog decreases, the dynamic time window decreases accordingly.

[0035] The dynamic window calculator also includes a triggering mechanism for calculating dynamic time windows, which includes timed triggering and event triggering;

[0036] The timed triggering refers to recalculating the dynamic time window size of all workstations every fixed time interval, which is 200ms by default, to ensure that the dynamic time window can adapt to the gradual changes in network status.

[0037] The event triggering includes the following steps:

[0038] When the dynamic time window verification fails (i.e., times out), the latest dynamic time window is obtained and enlarged to 1.2 times.

[0039] If the minimum value of the maximum dynamic time window is taken as the new dynamic time window, then the latest dynamic time window value for the current workstation is recalculated.

[0040] Secondly, embodiments of the present invention provide an industrial vision data processing system based on Netty's adaptive spatiotemporal window, which includes: a data acquisition module, which establishes a data acquisition layer and performs timestamp synchronization and structured encapsulation on the image data of each workstation in the data acquisition layer;

[0041] The protocol parsing and processing module parses the spatiotemporal header of the data packet through the protocol decoding module and sets the spatiotemporal constraint processor, which includes event mode and timer mode.

[0042] The dynamic window verification module performs dynamic window verification on the image data of each workstation, completing the industrial vision data processing with adaptive spatiotemporal windows.

[0043] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the above-described industrial visual data processing method based on Netty adaptive spatiotemporal window.

[0044] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the above-described industrial visual data processing method based on Netty adaptive spatiotemporal window.

[0045] The beneficial effects of this invention are as follows: By introducing a mechanism based on workstation-based queueing and frame sequence number sorting, this invention effectively solves the problem of intra-frame out-of-order data transmission within the same workstation. Combined with dynamic adaptive time window verification, it significantly reduces false detections and missed detections in cross-workstation quality inspection caused by network latency jitter. By constructing a dynamic window calculation model that includes a basic window, jitter compensation, safety margin, and queue backlog factor, and integrating a timing and event-based dual-trigger mechanism, the system can intelligently adjust the time window under different network conditions: when the network is stable, it maintains the minimum window (20ms) to improve processing speed and resource utilization; when there is a sudden high jitter, it quickly expands the window within 200ms to protect critical data; when there is queue backlog, it linearly increases the window to prevent processing avalanche; and when idle, it automatically resets the basic window to save memory and CPU overhead. Thus, it achieves strong adaptability to dynamic network environments and improves the real-time performance, stability, and quality inspection reliability of the industrial vision inspection system. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0047] Figure 1 A flowchart of an industrial vision data processing method based on Netty adaptive spatiotemporal window provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the structure of a medium for an industrial visual data processing method based on Netty's adaptive spatiotemporal window.

[0049] Figure 3 This is a schematic diagram of a computing device for an industrial vision data processing method based on Netty's adaptive spatiotemporal window. Detailed Implementation

[0050] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0054] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0055] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0056] Example

[0057] Reference Figures 1-3 This is the first embodiment of the present invention, which provides an industrial vision data processing method based on Netty's adaptive spatiotemporal window, including:

[0058] S1: Establish a data acquisition layer, and perform timestamp synchronization and structured encapsulation of image data from each workstation in the data acquisition layer.

[0059] The timestamp synchronization includes using the IEEE 1588 protocol to achieve microsecond-level clock synchronization, and the structured encapsulation includes embedding an 8-byte timestamp, a 2-byte workstation ID, and a 4-byte frame sequence number in the data packet.

[0060] Furthermore, the timestamps of each camera are synchronized to the microsecond level using the IEEE 1588 clock.

[0061] Data packet structure: [8-byte timestamp|2-byte workstation ID|4-byte frame sequence number|N-byte image data] Example: [0x6554F3E04D3E0F3B|0002|00000067|424D360000000000...].

[0062] IEEE 1588 is a precision time protocol, an international standard protocol used to achieve high-precision clock synchronization in distributed networks.

[0063] S2: The protocol decoding module performs spatiotemporal header parsing on the data packets and sets the spatiotemporal constraint processor, which includes event mode and timer mode.

[0064] The protocol decoding module performs time-space header parsing on data packets, including verifying validity based on the 2-byte workstation ID, diverting data from different workstations to independent receiving queues, and storing the data in each receiving queue according to the frame sequence number to achieve time sequence consistency of data within the same workstation ID.

[0065] S2.1: The spatiotemporal constraint processor includes event mode and timer mode;

[0066] The event pattern includes the following technical steps:

[0067] When data arrives, the network monitor is notified to update network metrics.

[0068] Data is stored in the corresponding queue according to the frame number, based on the different workstations and their respective queues.

[0069] Retrieve the data at the head of the queue and obtain the dynamic window from the dynamic window calculator;

[0070] Obtaining a dynamic window from the dynamic window calculator involves the following technical steps:

[0071] The time difference is calculated by subtracting the absolute value of the current time from the data's timestamp.

[0072] When the time difference is less than or equal to the time window (i.e., the time window verification is successful), the data is pushed to TensorRT inference, and the verified data is passed to the time window to optimize the time window calculation.

[0073] Conversely, if time window verification fails, the dynamic window calculator is used to handle the time window verification failure and optimize the time window calculation.

[0074] TensorRT inference refers to the process of using the high-performance deep learning inference optimization engine TensorRT (TensorRuntime) to perform efficient and low-latency image recognition and defect detection calculations on a pre-trained neural network model.

[0075] The timed mode includes rechecking the queue status of all different workstations at fixed time intervals;

[0076] If the processing time difference between the current time and the previous time is greater than 5 seconds and the queue size is 0, it is considered to be in an idle state, and the time window value is reset to the base window value.

[0077] When the backlog of queues at different workstations exceeds the threshold If this happens, the dynamic time window value is recalculated. Recalculating the dynamic time window value is part of the dynamic time window adaptive mechanism, which is used to adjust the time tolerance threshold according to the queue load status of the workstation.

[0078] When the queue backlog exceeds the threshold of 50, the dynamic time window value is recalculated.

[0079] The dynamic window calculator generates a dynamic time window duration value to determine whether the data is valid, based on the average jitter, out-of-order rate, and current queue backlog of each workstation provided by the network monitor.

[0080] The duration of the dynamic time window refers to the window size of the dynamic time window, which is used in the spatiotemporal constraint processor to determine whether the received visual data has arrived within a reasonable time range, thereby determining whether the data can be sent into the subsequent inference and detection process.

[0081] Network monitors are used to update network metrics. Updating network metrics involves the following steps:

[0082] Calculate the total transmission delay experienced by each data packet from the acquisition end to the receiving end;

[0083] Calculate the average degree of arrival delay variation among consecutively arriving data packets at the same workstation, i.e., the average value of arrival interval jitter;

[0084] The adaptive adjustment for dynamic time windows includes two parts: adopting the index-weighted moving average algorithm recommended by the IETF and calculating the out-of-order rate.

[0085] When the out-of-order rate is less than 1%, the basic safety margin is used; when the out-of-order rate is greater than 1%, it is dynamically increased proportionally.

[0086] Furthermore, the index-weighted moving average algorithm recommended by the IETF is adopted, where the best practice value for the smoothing factor α is 0.125.

[0087] Calculate the total transmission delay experienced by each data packet from the acquisition end to the receiving end. Arrival delay = arrival timestamp of the data packet - timestamp carried by the data packet, reflecting the total delay of the data packet from acquisition to reception.

[0088] S3: Perform dynamic window verification on the image data of each workstation to complete the industrial vision data processing with adaptive spatiotemporal window.

[0089] The base window value is used to reflect the minimum buffer value under network conditions, so that the network has a buffer, and is set to 20ms.

[0090] Average jitter includes configuring the jitter amplification factor and obtaining the average arrival interval jitter value of the workstation;

[0091] The out-of-order rate includes a safety margin based on the out-of-order rate to account for jitter or measurement errors;

[0092] The current queue backlog includes items based on the current queue backlog status for current queue backlog analysis. When the queue backlog increases, the dynamic time window is increased to allow data to be processed. When the queue backlog decreases, the dynamic time window decreases accordingly.

[0093] The dynamic window calculator also includes a triggering mechanism for calculating dynamic time windows, which includes timed triggering and event triggering.

[0094] Timed triggering means that the dynamic time window size of all workstations is recalculated every fixed time interval, which is 200ms by default, to ensure that the dynamic time window can adapt to the gradual changes in network status.

[0095] Event triggering includes the following steps:

[0096] When the dynamic time window verification fails (i.e., times out), the latest dynamic time window is obtained and enlarged to 1.2 times.

[0097] If the minimum value of the maximum dynamic time window is taken as the new dynamic time window, then the latest dynamic time window value for the current workstation is recalculated.

[0098] The base window is a small value that reflects the minimum necessary buffer under ideal network conditions. Even under good network conditions, a basic buffer exists. This paper sets it to 20ms for the following reasons:

[0099] The theoretical transmission time of the smallest transmission unit of an industrial vision system on Gigabit Ethernet is: (1280*1024*1) / 125,000,000 ≈ 10.5ms. An additional 9.5ms of redundancy is added to cover the following overheads: switch forwarding delay, protocol encapsulation overhead, and clock synchronization error. This design is only used as an initial baseline value. The actual window = BASE_WINDOW_MS + dynamic compensation value.

[0100] The jitter amplification factor is a configurable constant, such as (2 or 3), used to amplify the effects of jitter and ensure that the window can cover most fluctuations. The K_FACTOR value can be set based on experience or business requirements for robustness.

[0101] The reason for setting K_FACTOR=2.5 in this article is as follows:

[0102] When the value is too large (e.g., >3.0): slight jitter will cause a large window to expand, resulting in excessive delay;

[0103] Values ​​that are too small (e.g., <1.5): Insufficient high-jitter window, leading to out-of-order missed detections;

[0104] By using the RFC 5481 (Enterprise Network Jitter Control Guideline) standard: buffer time ≥ 2 x historical maximum jitter value, taking its conservative value K_FACTOR×avgJitter≈2.5×avgJitter, it covers 95% of jitter scenarios (normal distribution).

[0105] Average jitter is the average arrival interval jitter value from the recent period for this workstation, reflecting the degree of network instability; safety margin is an additional dynamic margin value.

[0106] The calculation is based on the out-of-order rate (OutOfOrderRate) to cope with sudden extreme jitter or measurement errors. The safety margin is set to max(5.0, outOfOrderRate*10), and the scaling factor of 10 allows the safety margin to increase linearly with the out-of-order rate. Each 1% out-of-order rate corresponds to a margin of 10ms. For out-of-order rates below 0.5%, a base value of 5ms is used, and for out-of-order rates above 0.5%, the margin increases linearly.

[0107] The queue factor is a term that reflects the current queue backlog situation. QueueFactor = Weight * QueueSize. When the queue backlog is severe, the window is appropriately increased to allow older data to be processed and prevent avalanche. When the queue is idle, it can be decreased. Weight is a weight coefficient. Here, it is set to weight=0.1 because TensorRT processes one image data in about 10ms. Its physical meaning comes from the fact that the standard number of frames that the disk system can process in 1 second is 100 frames / second.

[0108] The window size was increased by 1.2 here based on experience and experimental settings, for the following reasons:

[0109] An adjustment factor that is too small (such as 1.05) will result in too many adjustments to reach the required window size, causing the system to respond slowly.

[0110] An excessively large adjustment factor (such as 1.5) may cause the window to grow too fast, resulting in an excessively large window, which in turn introduces too much latency (because it takes longer to determine a timeout), increases memory usage, and may mask the real network problem.

[0111] At this point, 1.2 is a compromise, which makes the window grow exponentially in the case of consecutive timeouts, but the growth is relatively slow. For example, the first timeout window becomes 1.2 times the original, the second timeout becomes 1.44 (1.2^2) times the original, the third time becomes 1.728 (1.2^3) times the original, and so on.

[0112] At the same time, to prevent the window from expanding indefinitely, an upper limit (MAX_WINDOW_MS) is set to prevent it from expanding indefinitely in an avalanche-like manner.

[0113] In a preferred embodiment, an industrial vision data processing system based on Netty's adaptive spatiotemporal window is provided. This system includes a data acquisition module that establishes a data acquisition layer, where image data from each workstation is timestamped and structurally encapsulated; a protocol parsing and processing module that performs spatiotemporal header parsing on data packets through a protocol decoding module and sets a spatiotemporal constraint processor, which includes event mode and timed mode; and a dynamic window verification module that performs dynamic window verification on image data from each workstation, thus completing the adaptive spatiotemporal window industrial vision data processing.

[0114] The above-mentioned unit modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0115] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen of the computer device may be an LCD screen or an e-ink display screen. The input device of the computer device may be a touch layer covering the display screen, or buttons, a trackball, or a touchpad located on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0116] In summary, this invention effectively solves the problem of intra-frame out-of-order data transmission within the same workstation by introducing a mechanism based on workstation-based queueing and frame sequence number sorting. Combined with dynamic adaptive time window verification, it significantly reduces false positives and false negatives in cross-workstation quality inspection caused by network latency jitter. By constructing a dynamic window calculation model that includes a basic window, jitter compensation, safety margin, and queue backlog factor, and integrating a timing and event-triggered mechanism, the system can intelligently adjust the time window under different network conditions: maintaining the minimum window (20ms) when the network is stable to improve processing speed and resource utilization; rapidly expanding the window within 200ms to protect critical data during sudden high jitter; linearly increasing the window to prevent processing avalanche during queue backlog; and automatically resetting the basic window during idle periods to save memory and CPU overhead. This achieves strong adaptability to dynamic network environments and improves the real-time performance, stability, and quality inspection reliability of the industrial vision inspection system.

[0117] After introducing the method and system of exemplary embodiments of the present invention, the following references are made. Figure 2 A computer-readable storage medium according to exemplary embodiments of the present invention will be described, please refer to... Figure 2 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it implements the steps described in the above method implementation, such as: establishing a data acquisition layer; performing timestamp synchronization and structured encapsulation on the image data of each workstation in the data acquisition layer; parsing the spatiotemporal header of data packets through a protocol decoding module; setting a spatiotemporal constraint processor, which includes an event mode and a timed mode; and performing dynamic window verification on the image data of each workstation to complete the industrial vision data processing of adaptive spatiotemporal windows. The specific implementation methods of each step will not be repeated here.

[0118] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0119] After introducing the methods and media of exemplary embodiments of the present invention, the following references are made. Figure 3 A computational device for adaptive recovery of low-voltage power grid self-healing control according to an exemplary embodiment of the present invention.

[0120] Figure 3 A block diagram is shown of an exemplary computing device 40 suitable for implementing embodiments of the present invention. The computing device 40 may be a computer system or a server. Figure 3 The computing device 40 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0121] like Figure 3 As shown, the components of computing device 40 may include, but are not limited to: one or more processors or processing units 401, system memory 402, and bus 403 connecting different system components (including system memory 402 and processing unit 401).

[0122] The computing device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computing device 40, including volatile and non-volatile media, and removable and non-removable media.

[0123] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 4021 and / or cache memory 4022. Computing device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, ROM 4023 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 3 (Not shown in the image, usually referred to as "hard drive"). Although not shown in... Figure 3The diagram illustrates that disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) can be provided. In these cases, each drive can be connected to bus 403 via one or more data media interfaces. System memory 402 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0124] A program / utility 4025 having a set (at least one) of program modules 4024 may be stored, for example, in system memory 402, and such program modules 4024 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment. Program modules 4024 typically perform the functions and / or methods described in the embodiments of the present invention.

[0125] The computing device 40 can also communicate with one or more external devices 404 (such as a keyboard, pointing device, display, etc.). This communication can be performed via the input / output (I / O) interface 405. Furthermore, the computing device 40 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 406. Figure 3 As shown, network adapter 406 communicates with other modules of computing device 40 (such as processing unit 401) via bus 403. It should be understood that, although... Figure 3 As not shown, it can be used in conjunction with computing device 40 with other hardware and / or software modules.

[0126] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402. For example, it establishes a data acquisition layer, performs timestamp synchronization and structured encapsulation of image data from each workstation in the data acquisition layer; it performs spatiotemporal header parsing of data packets through a protocol decoding module, sets a spatiotemporal constraint processor, which includes event mode and timer mode; and it performs dynamic window verification on the image data from each workstation to complete industrial vision data processing with adaptive spatiotemporal windows.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0131] If the functionality is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0133] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An industrial vision data processing method based on Netty's adaptive spatiotemporal window, characterized in that: include, Establish a data acquisition layer, and perform timestamp synchronization and structured encapsulation of image data from each workstation in the data acquisition layer; The protocol decoding module performs spatiotemporal header parsing on the data packets and sets up a spatiotemporal constraint processor. The spatiotemporal constraint processor includes an event mode and a timed mode. Specifically, it includes: verifying the validity based on the 2-byte workstation ID, diverting data from different workstations to independent receiving queues, and storing the data in each receiving queue according to the frame sequence number to ensure the temporal consistency of data within the same workstation ID; and generating a dynamic time window duration value for judging the validity of data based on the average jitter, out-of-order rate, and current queue backlog of each different workstation provided by the network monitor. The duration value of the dynamic time window refers to the window size of the dynamic time window, which is used in the spatiotemporal constraint processor to determine whether the received visual data has arrived within a reasonable time range, thereby determining whether the data can be sent to the subsequent inference and detection process. Dynamic window verification is performed on the image data of each workstation to complete the industrial vision data processing with adaptive spatiotemporal windowing.

2. The industrial vision data processing method based on Netty adaptive spatiotemporal window as described in claim 1, characterized in that: The timestamp synchronization includes microsecond-level clock synchronization using the IEEE 1588 protocol, and the structured encapsulation includes embedding an 8-byte timestamp, a 2-byte workstation ID, and a 4-byte frame sequence number in the data packet.

3. The industrial vision data processing method based on Netty's adaptive spatiotemporal window as described in claim 2, characterized in that: The event mode includes the following technical steps: When data arrives, the network monitor is notified to update network metrics. Data is stored in the corresponding queue according to the frame number, based on the different workstations and their respective queues. Retrieve the data at the head of the queue and obtain the dynamic window from the dynamic window calculator; The process of obtaining a dynamic window from the dynamic window calculator includes the following technical steps: The time difference is calculated by subtracting the absolute value of the current time from the data timestamp. When the time difference is less than or equal to the time window, i.e. the time window verification is successful, the data is pushed to TensorRT inference, the verified data is passed to the time window, and the time window calculation is optimized. Conversely, if time window verification fails, the dynamic window calculator is used to handle the time window verification failure and optimize the time window calculation.

4. The industrial vision data processing method based on Netty's adaptive spatiotemporal window as described in claim 3, characterized in that: The timing mode includes rechecking the queue status of all different workstations at fixed time intervals; If the processing time difference between the current time and the previous time is greater than 5 seconds and the queue size is 0, it is considered to be in an idle state, and the time window value is reset to the base window value. When the backlog of queues at different workstations exceeds the threshold If the dynamic time window value is recalculated, the recalculation of the dynamic time window value is part of the dynamic time window adaptive mechanism, which is used to adjust the time tolerance threshold according to the queue load status of the workstation.

5. The industrial vision data processing method based on Netty's adaptive spatiotemporal window as described in claim 4, characterized in that: The network monitor is used to update network metrics, and the updating of network metrics includes the following steps: Calculate the total transmission delay experienced by each data packet from the acquisition end to the receiving end; Calculate the average degree of arrival delay variation among consecutively arriving data packets at the same workstation, i.e., the average value of arrival interval jitter; The adaptive adjustment for dynamic time windows includes two parts: adopting the index-weighted moving average algorithm recommended by the IETF and calculating the out-of-order rate. When the out-of-order rate is less than 1%, the basic safety margin is used; when the out-of-order rate is greater than 1%, it is dynamically increased proportionally.

6. The industrial vision data processing method based on Netty's adaptive spatiotemporal window as described in claim 5, characterized in that: The base window value is used to reflect the minimum buffer value under network conditions, so that the network has a buffer, and is set to 20ms. The average jitter includes configuring a jitter amplification coefficient and obtaining the average arrival interval jitter value of the workstation. The out-of-order rate includes a safety margin based on the out-of-order rate, used to deal with jitter or measurement errors; The current queue backlog includes an analysis of the current queue backlog based on the current queue backlog status. When the queue backlog increases, the dynamic time window is increased to allow data to be processed. When the queue backlog decreases, the dynamic time window decreases accordingly. The dynamic window calculator also includes a triggering mechanism for calculating dynamic time windows, which includes timed triggering and event triggering; The timed triggering refers to recalculating the dynamic time window size of all workstations every fixed time interval, which is 200ms by default, to ensure that the dynamic time window can adapt to the gradual changes in network status. The event triggering includes the following steps: When the dynamic time window verification fails (i.e., times out), the latest dynamic time window is obtained and enlarged to 1.2 times. If the minimum value of the maximum dynamic time window is taken as the new dynamic time window, then the latest dynamic time window value for the current workstation is recalculated.

7. An industrial vision data processing system based on Netty's adaptive spatiotemporal window, based on the industrial vision data processing method based on Netty's adaptive spatiotemporal window as described in any one of claims 1 to 6, characterized in that: include, The data acquisition module establishes a data acquisition layer, which performs timestamp synchronization and structured encapsulation of image data from each workstation. The protocol parsing and processing module parses the spatiotemporal header of data packets through the protocol decoding module and sets up a spatiotemporal constraint processor. The spatiotemporal constraint processor includes event mode and timed mode, specifically including: verifying validity based on a 2-byte workstation ID; diverting data from different workstations to independent receiving queues; storing data within each receiving queue in sequence according to frame number to ensure temporal consistency of data within the same workstation ID; and generating a dynamic time window duration value for determining data validity based on the average jitter, out-of-order rate, and current queue backlog provided by the network monitor for each different workstation. The dynamic time window duration value refers to the window size, used in the spatiotemporal constraint processor to determine whether received visual data arrives within a reasonable time range, thereby deciding whether the data can be sent to the subsequent inference and detection process. The dynamic window verification module performs dynamic window verification on the image data of each workstation, completing the industrial vision data processing with adaptive spatiotemporal windows.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the industrial visual data processing method based on Netty adaptive spatiotemporal window as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the industrial visual data processing method based on Netty adaptive spatiotemporal window as described in any one of claims 1 to 6.

Images