Multi-modal inference pipeline parallel acceleration method oriented to edge-cloud collaboration

Abstract

This application provides a parallel acceleration method for multimodal inference pipelines oriented towards edge-cloud collaboration, applied in the field of industrial vision inspection technology. The method acquires industrial sensor data, extracts frozen background prefix features in response to physical roll-change signals, calculates pixel-level differences to generate residual feature tensors, and concatenates them with process parameter vectors to form a multimodal fusion vector. After being segmented by micro-batch, the pipeline input is cascaded into convolutional and fully connected layers. The output of each layer is converted into an intermediate confidence interval. The final control decision interval is obtained through the intersection operation of adjacent cascaded stage intervals and sent to the actuator. Simultaneously, in response to the next roll-change signal, background feature recalculation is triggered and the storage area is atomically updated. This solution effectively improves inference throughput, eliminates roll-change false alarms, enhances decision robustness, and provides rapid system recovery capabilities.

Description

Technical Field

[0001] This application relates to the field of industrial vision inspection technology, and in particular to a parallel acceleration method for multimodal inference pipelines oriented towards edge-cloud collaboration. Background Technology

[0002] With the increase in industrial camera resolution and production line operating speed, the amount of data per frame image has increased dramatically. Traditional deep learning inference architectures based on single nodes and single batches, limited by serial computing mode, can no longer meet the real-time requirements of millisecond-level latency for surface defect detection of rolled materials, resulting in delayed detection results and failure to drive actuators for closed-loop control in a timely manner.

[0003] In edge-cloud collaborative deployment scenarios, edge devices have limited computing power, while cloud servers, despite their powerful computing capabilities, suffer from network transmission latency. Existing technologies lack reasonable partitioning and dynamic load balancing strategies for inference tasks, making it difficult to achieve an optimal solution between computing resources and communication overhead. This results in low hardware utilization and an inability to fully leverage the potential of heterogeneous computing platforms.

[0004] When changing rolls, the old and new rolls have inherent differences in substrate texture. Using the old background features will misjudge normal differences as defects, causing a large number of false alarms and unplanned downtime. At the same time, traditional point estimation inference cannot quantify model uncertainty. Errors accumulate layer by layer in the cascaded network, which seriously reduces the system's detection reliability and decision credibility under complex dynamic conditions. Summary of the Invention

[0005] The embodiments of this application provide a pipelined parallel acceleration method for multimodal inference in edge-cloud collaboration. This method improves real-time throughput through pipeline parallelism, eliminates false alarms caused by physical volume switching triggering background feature recalculation, enhances decision reliability through interval estimation and concatenated intersection operations, and possesses rapid recovery capabilities. It achieves high real-time performance and high accuracy in edge-cloud collaborative multimodal inference control. To achieve the above objectives, this application adopts the following technical solution: A method for accelerating multimodal inference pipeline parallelism in edge-cloud collaboration, the method comprising: Acquire industrial sensor data at the current moment, including high-resolution images and process parameters; In response to the received physical roll change signal, the stored frozen background prefix features are extracted; Calculate the pixel-level difference between the high-resolution image and the frozen background prefix features, and generate a residual feature tensor; The parameter vector obtained by combining the residual feature tensor with the process parameters is concatenated by channel to generate a multimodal fusion vector. The multimodal fusion vector is divided into pipeline micro-batches and sequentially input into cascaded convolutional and fully connected layers. The element-wise activation function is taken for the output of each layer to obtain the intermediate confidence interval. Perform an interval intersection operation on the intermediate confidence intervals output from adjacent cascaded stages to obtain the updated upper and lower bounds of the intervals, and output them as the final control decision intervals. The final control decision interval is sent as a control command to the production line execution mechanism; In response to the next physical roll change signal received from the production line actuator, the frozen background prefix feature is recalculated, and the recalculated frozen background prefix feature is used for pixel-level difference calculation at the next moment.

[0006] In some possible implementations, the extraction of the stored frozen background prefix features includes: Analyze the production line number carried in the physical roll change signal; The corresponding frozen background prefix feature is read from the non-volatile storage area according to the production line number; The obtained frozen background prefix features are output as the frozen background prefix features of the current inference cycle.

[0007] In some possible implementations, calculating the pixel-level difference between the high-resolution image and the frozen background prefix features to generate a residual feature tensor includes: Perform the same normalization transformation as the frozen background prefix feature on the high-resolution image to obtain the normalized image tensor; The normalized image tensor and the frozen background prefix features are subtracted element-wise at corresponding pixel positions to obtain the original residual image; Perform element-wise absolute value operation on the original residual map to generate a residual magnitude map; The residual magnitude map is input into a preset threshold filtering layer to generate a residual feature tensor.

[0008] In some possible implementations, the step of concatenating the residual feature tensor with the parameter vector obtained from the process parameters by channel to generate a multimodal fusion vector includes: The process parameters were analyzed to extract the values ​​of temperature, tension, and speed. The temperature, tension, and velocity values ​​are mapped to embedding vectors that match the channel dimensions of the residual feature tensor, respectively. Perform layer normalization on the embedded vector to generate a parameter vector; The residual feature tensor and parameter vector are concatenated along the channel dimension to generate a multimodal fusion vector.

[0009] In some possible implementations, the step of dividing the multimodal fusion vector into pipelined micro-batches and sequentially inputting them into cascaded convolutional and fully connected layers includes: Obtain the current memory capacity of the inference pipeline and the feature size of a single image; Based on the video memory capacity and the feature size of a single image, calculate the maximum number of image frames that a single batch can accommodate, and use this as the micro-batch size; According to the micro-batch size, the multimodal fusion vector is sliced ​​along the batch dimension to generate multiple micro-batch sub-vectors; Record the original batch index of each micro-batch sub-vector in the multimodal fusion vector; Following the order of the original batch index, each micro-batch sub-vector is sequentially input into the cascaded convolutional layer and fully connected layer.

[0010] In some possible implementations, the step of taking an element-wise activation function on the output of each layer to obtain the intermediate confidence interval includes: The original tensors output by the convolutional and fully connected layers are divided into lower bound channel groups and upper bound channel groups according to the channel dimension. Perform an element-wise exponential normalization transformation on the lower bound channel group to generate a normalized lower bound subtensor; Perform an element-wise exponential normalization transformation on the upper limit channel group to generate a normalized upper limit subtensor; An element-wise comparison is performed between the normalized lower bound subtensor and the normalized upper bound subtensor to ensure that the lower bound is not greater than the upper bound. The normalized lower bound subtensor and the normalized upper bound subtensor are concatenated sequentially along the channel dimension to form the intermediate confidence interval.

[0011] In some possible implementations, the step of performing an interval intersection operation on the intermediate confidence intervals output by adjacent concatenated stages to obtain updated interval upper and lower bounds includes: Extract the first lower bound tensor and the first upper bound tensor from the output of the first cascade stage; Extract the second lower bound tensor and the second upper bound tensor from the output of the second cascade stage; For each tensor position, take the larger of the two lower bound tensors as the lower bound of the interval after the intersection operation, and take the smaller of the two upper bound tensors as the upper bound of the interval after the intersection operation. The lower bound tensor and the upper bound tensor obtained from the intersection operation are combined to form the updated upper and lower bounds of the interval.

[0012] In some possible implementations, sending the final control decision interval as a control command to the production line actuator includes: Analyze the lower and upper bound tensors of the final control decision interval; Calculate the center values ​​of the lower limit tensor and the upper limit tensor along the decision dimension to generate control quantity values. Use the control quantity values ​​as the driving signal of the actuator. Reserve the lower limit tensor and the upper limit tensor for confidence verification in the next inference cycle. Based on the control quantity value, a control instruction including the execution action code and duration is generated; The control commands are written into the communication port connected to the production line actuator.

[0013] In some possible implementations, the response to the next physical roll change signal received from the production line actuator includes: Monitor the feedback channel connected to the production line actuator; Analyze the device status messages transmitted in the feedback channel; When the device status message indicates that the production line has entered the roll change state, it is determined that the next physical roll change signal has been received. Extract the new roll texture image carried in the next physical roll change signal; Based on the new roll texture image, trigger the recalculation process of freezing background prefix features; The recalculated frozen background prefix features are associated with the inference period corresponding to the next physical roll-over signal.

[0014] In some possible implementations, the step of using the recalculated frozen background prefix features for pixel-level difference calculation at the next time step includes: The recalculated frozen background prefix features are written to the non-volatile storage area, replacing the features in the original storage area; When acquiring the current high-resolution image in the next moment, the replaced frozen background prefix feature is synchronously invoked; Perform the same normalization transformation on the replaced frozen background prefix features as on the high-resolution image; The normalized replacement features are used as the reference tensor for pixel-level difference calculation.

[0015] As can be seen from the above technical solution, this application has the following beneficial effects: 1. This application breaks down the inference process into multiple pipeline stages and executes them in parallel at a micro-batch granularity, resulting in a high degree of temporal overlap between the calculation stages. This effectively improves end-to-end inference throughput and fully meets the real-time requirements of high-speed production line operation. Simultaneously, by introducing frozen background prefix features and combining them with a background recalculation mechanism triggered by physical roll change signals, it ensures that the detection benchmark always remains consistent with the actual surface characteristics of the current roll material. This fundamentally eliminates systematic false alarms caused by changes in the background substrate, significantly reduces the frequency of unplanned downtime, and substantially lowers production costs and operational disruptions caused by false detections on the production line.

[0016] 2. This application extends the output of each layer of the neural network from traditional point estimation to interval estimation, and combines cascaded interval intersection operations to achieve step-by-step consistency filtering, effectively suppressing the accumulation and propagation of errors in deep networks, and significantly improving the reliability and accuracy of the model's decision-making under complex dynamic conditions. Furthermore, by establishing a cross-cycle decision interval consistency verification mechanism, erroneous actions caused by single-frame image anomalies are effectively prevented. Simultaneously, an atomic writing method is used to store the updated background features in a non-volatile storage area, enabling the system to quickly resume detection without re-acquisition and calibration after a restart, greatly enhancing the continuous availability and fault tolerance of the production line. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 The main flowchart of the overall reasoning and control closed loop provided in the embodiments of this application; Figure 2 A flowchart of the extraction of frozen background prefix features and roll-changing response provided for embodiments of this application; Figure 3 A flowchart of multimodal fusion vector generation provided in the embodiments of this application; Figure 4 A flowchart of parallel inference and interval estimation in a micro-batch pipeline provided for embodiments of this application; Figure 5 The flowchart for cascaded interval intersection operation and decision consistency verification provided in the embodiments of this application is shown. Detailed Implementation

[0019] The terms "first," "second," and "third," etc., in this application specification, claims, and drawings are used to distinguish different objects, not to define a specific order.

[0020] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as exemplary or for example in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0021] Research has revealed that existing technologies employ a single-node serial inference architecture, which struggles to meet the real-time requirements of high-speed production lines with millisecond-level latency. Furthermore, the lack of reasonable task partitioning and load balancing strategies in edge-cloud collaborative deployments leads to low utilization of computing resources. Existing technologies rely on fixed background features for defect detection, which can easily misjudge normal baseline differences as defects during roll changes, resulting in numerous false alarms. Additionally, the use of point estimation outputs fails to quantify model uncertainty, and errors accumulate layer by layer in cascaded networks, severely reducing the system's detection reliability under complex dynamic conditions.

[0022] To address the aforementioned issues, this application provides a parallel acceleration method for multimodal inference pipelines oriented towards edge-cloud collaboration: Example To solve the above problems, such as Figures 1-5 As shown, in industrial production lines, roll-type products (such as metal foil, film, paper, and textiles) are highly susceptible to surface defects such as scratches, foreign objects, holes, and wrinkles during unwinding, transport, and rewinding due to fluctuations in production processes, environmental interference, and equipment vibration. To ensure product quality, continuous, non-contact inspection of the roll surface is required while the production line is running. Based on the inspection results, control commands are sent to the actuators to achieve closed-loop control actions such as defect marking, deceleration, or shutdown.

[0023] With the increasing resolution of industrial cameras and the speed of production line operation, the amount of data per frame has increased dramatically. Traditional inference architectures based on single nodes and single batches can no longer meet the real-time requirements of millisecond-level latency. At the same time, in the edge-cloud collaborative deployment mode, the computing power of edge devices (such as industrial cameras and edge gateways) is limited, while cloud devices (such as GPU servers), although powerful, suffer from network transmission latency. How to reasonably divide inference tasks and balance computing load and transmission overhead has become an urgent technical problem to be solved.

[0024] Crucially, the roll changeover moment is the most vulnerable window for the inspection system. Before and after roll changeover, the new and old roll materials have inherent differences in physical properties such as substrate texture, color, gloss, and surface microstructure. If the inspection system continues to use the background features of the old roll material as the inspection benchmark for the new roll material, these normal substrate differences will be misjudged as defects, leading to a large number of false alarms, which in turn will cause unplanned production line downtime and result in huge economic losses. In addition, traditional deep learning inference often uses point estimation output, which cannot quantify model uncertainty. In cascaded networks, errors are prone to accumulate layer by layer, further reducing the reliability of the system under complex operating conditions.

[0025] To address the aforementioned technical challenges, this application proposes a parallel acceleration method for multimodal inference pipelines oriented towards edge-cloud collaboration. This method improves throughput through micro-batch pipeline segmentation, eliminates false alarms during roll change by triggering background feature recalculation through physical roll change signals, and suppresses error propagation through interval estimation and cascaded intersection operations. Thus, it achieves highly reliable defect detection and control under high real-time constraints.

[0026] II. Detailed description of the specific implementation method.

[0027] This embodiment details the complete implementation process of the method of the present invention. This method can be implemented on general-purpose industrial computing devices (such as edge servers, industrial PCs, and embedded AI boxes). These devices have computing units such as CPUs, GPUs, or NPUs, as well as storage units such as RAM and flash memory, and run real-time operating systems or general-purpose operating systems.

[0028] The method consists of ten core steps executed sequentially, with clear data dependencies and logical coupling between each step, forming a closed-loop reasoning and control pipeline.

[0029] (a) Step 1: Acquire the industrial sensor data at the current moment.

[0030] Input data: Raw image data streams output from high-resolution industrial cameras, and analog or digital signals output from various process parameter sensors.

[0031] Processing Logic: At the inspection station of the industrial production line, one or more high-resolution industrial cameras are installed. Global shutter cameras are preferred to avoid the rolling shutter effect that occurs when photographing high-speed moving objects. The camera resolution is determined based on the required inspection accuracy, typically requiring that the physical size of a single pixel is smaller than the minimum size of the defect to be inspected. To ensure strict synchronization between image acquisition and the movement of the roll material, the camera's acquisition trigger signal is connected to the production line encoder pulse signal. The encoder is mounted on the output shaft of the drive motor or the driven roller shaft, generating periodic pulses as the roller shaft rotates. The system counts the encoder pulses, triggering the camera to acquire one frame of image every fixed physical distance the roll material advances. This physical position-based triggering method effectively eliminates image stretching or compression caused by production line speed fluctuations, ensuring a one-to-one correspondence between the image sequence and the roll material surface.

[0032] Upstream and downstream of the inspection station, a group of process parameter sensors is deployed. This sensor group includes at least: a non-contact temperature sensor for monitoring the surface temperature of the roll material; a tension sensor for monitoring the tension of the roll material during operation; and a speed sensor for monitoring the real-time running speed of the roll material. These sensors are connected to the inference system via an industrial bus or analog acquisition card, and output detection values ​​with the same sampling period as the camera.

[0033] All data from cameras and sensors is packaged into a unified format data frame in the edge preprocessing unit. The header of the data frame includes metadata such as timestamp, production line number, and equipment status word, while the data portion includes raw image data and instantaneous values ​​of various process parameters. This data frame is transmitted via Industrial Ethernet to the edge buffer of the inference system, awaiting further processing.

[0034] Output data: A structured data frame, including a high-resolution two-dimensional image (spatial dimension , where is the image height in pixels, is the image width in pixels, and 3 represents the RGB color channel) and several scalar process parameter values.

[0035] (ii) Step 2: In response to the received physical roll change signal, extract the stored frozen background prefix features.

[0036] Input data: Physical roll change signals sent by the production line actuators, which are transmitted to the inference system in the form of hardware interrupts or high-priority messages.

[0037] Processing Logic: The physical roll change signal is the key event triggering the background update in this invention. This signal is generated by a roll diameter detection sensor or a roll presence sensor in the production line actuator. These sensors continuously monitor the roll status on the unwinding or take-up rolls. When the remaining roll diameter is detected to have decreased to a preset roll change threshold, or when a new and old roll joint is detected to have passed, the actuator determines that a roll change event is about to occur or has already occurred, and immediately sends a physical roll change signal to the inference system.

[0038] To ensure the real-time performance and reliability of the signal, the physical roll change signal is preferably transmitted via hardware interrupt, bypassing the conventional task scheduling of the operating system and ensuring that the inference system can respond to the event within microseconds. The data frame of the physical roll change signal includes a production line number field, which is uniformly assigned by the production line's upper control system during system initialization. Each production line has a globally unique number used to distinguish between multiple production lines in parallel production environments.

[0039] After receiving the physical roll change signal, the inference system first parses out the production line number. Then, using this number as an index, the system performs a retrieval in the non-volatile memory area. Within the memory area, a hash index table or B+ tree index structure is built according to the production line number. Each number corresponds to an independent storage area used to store the currently used frozen background prefix features and their associated metadata for that production line.

[0040] The frozen background prefix feature is one of the core data structures of this invention. It is defined as follows: a pure background image captured by a high-resolution industrial camera under no-load conditions on the production line, after preprocessing such as denoising, distortion correction, and normalization, is input into a lightweight encoder network to extract high-dimensional tensor data. This feature tensor has the same spatial dimension as the high-resolution image ( ), but the number of channels C is determined by the encoder network structure. Its numerical type is single-precision or half-precision floating-point to support subsequent element-wise arithmetic operations. This feature serves as the benchmark for all subsequent residual calculations and remains unchanged throughout the inference cycle.

[0041] Output data: Frozen background prefix features corresponding to the current production line number, with spatial dimensions of , which have been loaded into high-speed memory.

[0042] (III) Step 3: Calculate the pixel-level difference between the high-resolution image and the frozen background prefix features to generate the residual feature tensor.

[0043] Input data: The high-resolution image tensor and frozen background prefix features at the current time step, both of which have the same spatial dimension and number of channels.

[0044] Processing Logic: The core idea of ​​this step is background subtraction. The system first performs a normalization transformation on the high-resolution image acquired at the current moment that is identical to that of the frozen background prefix features, including color space conversion, data type conversion, and numerical range scaling. All the above transformation parameters are recorded along with the frozen background prefix features during storage. The system directly calls these parameters when performing normalization, ensuring that the two undergo mathematically equivalent transformations.

[0045] After normalization, the system performs element-wise subtraction on the normalized image tensor and the frozen background prefix features at each pixel location to obtain the original residual map. Then, it performs an absolute value operation on each element of the original residual map, unifying all differences into magnitudes. Finally, the residual magnitude map is input into a preset threshold filtering layer. The filtering layer compares the residual magnitude at each pixel location with a preset noise tolerance threshold: if the magnitude is less than the threshold, it is set to zero and considered background noise; if the magnitude is greater than or equal to the threshold, it is retained as a valid residual.

[0046] The noise tolerance threshold is calibrated as follows: Under no-load conditions on the production line, several frames of images are continuously acquired. The residual amplitude of each frame relative to the frozen background prefix features is calculated, and the mean and standard deviation of all residual amplitudes are statistically analyzed. The noise tolerance threshold is calculated using the following formula: The preset confidence coefficient is used to balance the false alarm rate and the false negative rate. The threshold determined by this statistical method can adapt to the noise level, lighting conditions and camera characteristics of different production lines, avoiding the problem of poor generalization caused by manually setting the threshold.

[0047] Output data: residual feature tensor, with spatial dimension n and values ​​ranging from 0 to 1. In most background regions, the residual values ​​are set to zero by the thresholding filter layer; only in defect regions does the residual feature tensor have a significant non-zero amplitude, thus achieving initial enhancement of the defect signal and effective suppression of background noise.

[0048] (iv) Step 4: Concatenate the residual feature tensor and the parameter vector obtained from the process parameters by channel to generate a multimodal fusion vector.

[0049] Input data: residual feature tensor (spatial dimension 1) and three scalar process parameters: temperature, tension, and velocity.

[0050] Processing Logic: The system first parses the temperature, tension, and velocity values ​​from the data packet. These three values ​​have different physical dimensions and numerical ranges; directly concatenating them would lead to a numerical scale imbalance. Therefore, the system configures an independent embedding mapping module for each parameter. Each scalar value is mapped to an embedding vector with a length matching the number of channels in the residual feature tensor through a linear transformation.

[0051] After calculating the three embedding vectors, the system performs element-wise addition at the element level, merging them into a single parameter vector that integrates all process information. The physical meaning of element-wise addition is that each of the three process parameters contributes to the feature representation of the channel dimension, which is then merged to form a unified feature representation.

[0052] The merged parameter vector is then subjected to layer normalization. Layer normalization transforms the vector into a distribution with zero mean and unit variance by calculating the mean and standard deviation within the vector. This effectively eliminates numerical instability caused by differences in dimensions and numerical ranges between different parameters, ensuring the convergence stability of the subsequent neural network training and inference processes.

[0053] Finally, the residual feature tensor and the normalized parameter vector are concatenated along the channel dimension. Since the parameter vector is one-dimensional while the residual feature tensor is three-dimensional, the system first expands the one-dimensional parameter vector spatially, copying the same parameter vector at each spatial location to form a three-dimensional tensor. Then, this three-dimensional tensor is concatenated with the residual feature tensor along the channel dimension to obtain a multimodal fusion vector of dimension . This concatenation method ensures that the features at each pixel location include not only local residual information but also global process context, thus significantly improving the model's adaptability to complex and dynamic operating conditions.

[0054] Output data: Multimodal fusion vector, spatial dimension , numerical type is floating point, ready to be input into subsequent convolutional neural networks for deep feature extraction.

[0055] (v) Step 5: Divide the multimodal fusion vector into pipeline micro-batch segments and input them sequentially into the cascaded convolutional and fully connected layers.

[0056] It should be noted that the image features in the single image feature size do not refer to the original high-resolution image, but rather to the multimodal fusion feature tensor output in step four. The dimension of this tensor is [dimension number missing], and its memory usage is the amount of single-frame data used for subsequent micro-batch segmentation. Those skilled in the art should understand that using the fusion feature, rather than the original image, as the basis for GPU memory computation is because the fusion feature is the actual data format input to the convolutional neural network.

[0057] Input data: multimodal fusion vector (batch dimension) and available GPU memory capacity of the current inference device.

[0058] Processing Logic: The inference system first obtains the currently available video memory capacity through the operating system API or hardware driver interface. Simultaneously, the system calculates the storage space occupied by a single image feature. Considering that floating-point data typically occupies bytes, and the feature map contains elements, it can be calculated using the following formula: The calculation of the micro-batch size is a key engineering feature of this invention. The system divides the available video memory capacity by the feature size of a single image and rounds down to obtain the maximum number of image frames that a single batch can accommodate. Here, represents the floor function. The operation in the formula is to ensure that the system can still run in single-frame mode even under extreme memory constraints, avoiding crashes. The purpose of this calculation method is to maximize the number of frames processed in parallel each time, thereby improving hardware utilization, within the constraints of memory capacity.

[0059] After determining the micro-batch size, the system slices the multimodal fusion vector along the batch dimension. Assuming the original batch contains frames, slicing yields 10 micro-batch sub-vectors. The first 10 sub-vectors each contain frames, and the last contains the remaining frames. During slicing, the system establishes an index mapping table to record which micro-batch each frame is assigned to and its position offset within the sub-vector, ensuring that the final output order matches the input order.

[0060] Subsequently, the system submits each micro-batch sub-vector to the cascaded convolutional and fully connected layers sequentially, according to the original batch index order. Here, "sequentially" does not mean serial execution, but rather pipelined overlapping execution: while the first micro-batch sub-vector is passing through the convolutional layer, the second micro-batch sub-vector can begin loading or preprocessing; while the first micro-batch sub-vector enters the fully connected layer, the second micro-batch sub-vector can enter the convolutional layer. This pipelined parallel mechanism keeps each computational unit constantly busy, significantly improving end-to-end throughput.

[0061] Cascaded convolutional layers and fully connected layers constitute the backbone inference network of this invention. The convolutional layer section consists of multiple stacked convolutional layers, each including several convolutional kernels with a stride typically of 1. A padding method that maintains the spatial dimension is employed, with the number of channels increasing progressively to extract multi-scale features from low-level edges to high-level semantics. The fully connected layer section follows the convolutional layers. It first performs global average pooling on the three-dimensional tensor output by the convolutional layers, compressing the spatial information of each channel into a single value to obtain a fixed-length feature vector. Then, through one or more fully connected layers, the feature vector is mapped to the final decision dimension D.

[0062] For the output tensor of each layer, the system does not directly output a point estimate, but instead takes an element-wise activation function to obtain an intermediate confidence interval. This design extends traditional neural networks from point estimation to interval estimation, laying the mathematical foundation for subsequent uncertainty quantification and consistency filtering.

[0063] Output data: The intermediate confidence interval tensor of each stage of the cascaded neural network.

[0064] (vi) Step 6: Take the element-wise activation function for each layer output to obtain the intermediate confidence interval.

[0065] Input data: The original tensor output by a certain layer of the cascaded neural network, with a number of channels.

[0066] Processing logic: The system first divides the output tensor into two equal groups along the channel dimension. The first half of the channels is designated as the lower bound channel group, and the second half as the upper bound channel group. When the number of channels is odd, zero-padding or rounding can be used to ensure that the number of channels in the two groups is as equal as possible.

[0067] For the value of the i-th channel at a spatial location in the lower limit channel group, the system performs an exponential normalization transformation: Perform the exact same transformation on the upper channel group: Wherein is the number of channels in the lower or upper limit channel group. The mathematical properties of this transformation guarantee that each normalized value is in the interval between 0 and 1, and the sum of all lower limit channel values ​​at the same spatial location is 1, and the sum of all upper limit channel values ​​is 1. This allows the lower and upper limit values ​​to have a probabilistic interpretation, which can be understood as the lower and upper confidence bounds of the probability of an event occurring.

[0068] After normalization, the system performs element-wise comparisons between the lower bound subtensor and the upper bound subtensor. For each position, the system compares whether the lower bound value is not greater than the upper bound value. If there is a position where the lower bound is greater than the upper bound, the lower bound and upper bound values ​​at that position are swapped to ensure that the lower bound is always no greater than the upper bound. This is a fundamental requirement for interval representation; otherwise, the interval would lose its mathematical meaning.

[0069] As another preferred implementation, the system can also divide the output tensor into a central channel group and a radius channel group along the channel dimension. After performing exponential normalization transformation on the central channel group and the radius channel group respectively, the normalized central value is denoted as and the normalized radius value is denoted as . The lower limit and upper limit are calculated according to the following formula: Since the radius value is always non-negative, this calculation method naturally guarantees that the lower limit value is no greater than the upper limit value, eliminating the need for post-calculation operations. The physical meaning of this implementation is: the model directly predicts the center and radius of the confidence interval for each location. The interval width reflects the prediction uncertainty; the larger the width, the more uncertain the model's prediction for that location; the smaller the width, the more confident the model is.

[0070] The aforementioned implementation method, which divides the output tensor into a lower-bound channel group and an upper-bound channel group and performs a post-transformation, is equivalent to the preferred implementation method in terms of final output effect. Both ultimately output a confidence interval tensor whose lower bound does not exceed the upper bound; the difference lies only in the intermediate transformation process. Those skilled in the art can choose an appropriate implementation method based on actual hardware conditions and model training requirements.

[0071] Finally, the system concatenates the adjusted lower bound subtensor and upper bound subtensor along the channel dimension in the order of "lower bound first, upper bound last" to form an intermediate confidence interval tensor. This tensor explicitly encodes the model's prediction uncertainty at each location: the larger the interval width, the more uncertain the model's prediction at that location; the smaller the interval width, the more confident the model.

[0072] Output data: intermediate confidence interval tensor.

[0073] (vii) Step 7: Perform an interval intersection operation on the intermediate confidence intervals output by adjacent cascaded stages to obtain the updated upper and lower bounds of the intervals.

[0074] Input data: The intermediate confidence interval between the outputs of two adjacent stages.

[0075] Processing logic: For each position in the tensor, perform an interval intersection operation. The calculation method is as follows: Where represents the lower bound tensor and upper bound tensor output in the first stage; and represents the lower bound tensor and upper bound tensor output in the second stage. The geometric meaning of this operation is to find the intersection of the two intervals on the real number axis. If , the intersection is non-empty, indicating that the two levels of judgment at that position overlap and the decision is reliable; if , the intersection is empty, indicating that the two levels of judgment at that position do not overlap and the decision is unreliable. The information at that position will be ignored or downgraded in subsequent steps.

[0076] If the cascaded network consists of three or more stages, the system uses the intersection result of the first and second stages as the new first interval, and then performs the same intersection operation with the output of the third stage, and so on. This cascaded intersection mechanism ensures that errors are filtered out step by step during propagation, and the width of the final decision interval will not exceed the width of any single-layer interval, thereby significantly improving the reliability of the system.

[0077] Output data: Updated interval upper and lower bound tensors.

[0078] (viii) Step eight: Send the final control decision range as a control command to the production line execution mechanism.

[0079] Input data: Final control decision interval (final lower bound tensor and final upper bound tensor).

[0080] Processing logic: If the final control decision interval is a high-dimensional tensor (i.e., spatial dimension greater than 1), the system first performs global average pooling along the spatial dimensions on the lower and upper bound tensors, compressing the spatial information of each channel into a single value, resulting in the lower and upper bound decision vectors. If the final output is already a scalar or a one-dimensional vector, then this output is directly used as the decision vector, without needing to perform pooling operations. The decision dimension refers to the dimension of the pooled decision vector, with each dimension corresponding to a specific control decision indicator.

[0081] In terms of decision-making, the center value between the lower and upper bounds after pooling is calculated: The central value is used because, under symmetric or approximately symmetric conditions, it is an unbiased estimate of the true decision value. Using the lower bound would lead to an overly conservative system, potentially missing serious defects; using the upper bound would lead to an overly aggressive system, potentially causing unwarranted production line shutdowns. The central value strikes a balance between these two approaches.

[0082] It should be noted that the control values ​​are used to generate the action commands that drive the actuators, while the final lower and upper limit tensors themselves do not participate in the current execution action, but are reserved for confidence verification in the next inference cycle. This separation design ensures real-time execution while providing redundant information for system safety monitoring, and is an important means by which this invention improves system reliability.

[0083] The system compares the control variable value with a preset action threshold to determine the specific action code. The specific value of the action threshold can be obtained through optimization using historical data. The action duration can also be calculated based on the control variable value, for example, set as a basic duration multiplied by an additional duration coefficient, to ensure that the duration increases monotonically with the severity of the defect.

[0084] After generating control commands that include action codes and durations, the system encapsulates them into data frames according to the industrial Ethernet protocol and sends them to the production line actuators through the communication port.

[0085] The lower and upper bound tensors of the final control decision interval are stored in memory for confidence verification in the next inference cycle. The verification method can be to calculate the intersection-union ratio (IUR) of the intervals stored in this round and the intervals calculated in the next round along the decision dimension. If the IUR is lower than a preset consistency threshold, the system will trigger a self-check and suspend the execution of new instructions to prevent erroneous actions caused by abnormal single-frame images.

[0086] Output data: Control instruction data frames sent to the production line actuators, and upper and lower limit tensors stored in memory for the next round of confidence verification.

[0087] (ix) Step nine: In response to the next physical roll change signal received from the production line actuator.

[0088] Input data: Equipment status messages sent by the production line actuators.

[0089] Processing logic: The inference system continuously runs a high-priority listening thread in the background, receiving device status messages transmitted through the feedback channel in a non-blocking manner. Each message includes the operating status of the actuator, sensor readings, and device operation logs.

[0090] The system parses each message and extracts relevant fields reflecting the production line's operating status. When a device status message indicates that the production line has entered a roll change state, the system determines that it has received the next physical roll change signal. For example, whether the production line has entered a roll change state can be determined by detecting whether the roll identification code carried in the device status message has changed. New rolls typically have a new globally unique identification code, and the change of this code is a deterministic indicator of a roll change event. Using status determination instead of simply detecting signal edges effectively avoids multiple false triggers caused by sensor jitter during the roll change process.

[0091] After a roll change signal is detected, the system further extracts the new roll texture image carried in the equipment status message. Based on this texture image, the system triggers a recalculation process for the frozen background prefix feature. During the window period before the roll joint arrives at the inspection station, the system controls the main camera at the inspection station to continuously acquire multiple frames of empty background images with the same parameters. After performing normalization transformation, the pixel-level average value is taken as the new frozen background prefix feature. The new feature is associated with the current inference cycle for the next round of difference calculation.

[0092] Output data: Recalculated frozen background prefix features.

[0093] The roll change signal triggers the preparation process for background feature recalculation. In actual industrial scenarios, the texture of the new roll material may not yet be in the camera's field of view when the roll change signal occurs. Therefore, after receiving the roll change signal, the system continuously monitors the production line's operating status. Only after the roll material joint passes the inspection station and the new roll material stably enters the camera's field of view does it execute the actual empty background image acquisition and feature recalculation. If the texture of the new roll material is already in the camera's field of view when the roll change signal occurs, acquisition and recalculation are performed directly. The above timing control ensures that the acquired texture image is a defect-free empty background image, avoiding the misuse of the texture of the roll material joint or transition section as the background reference.

[0094] (X) Step 10: Use the recalculated frozen background prefix features for the pixel-level difference calculation at the next time step.

[0095] Input data: The recalculated frozen background prefix features.

[0096] Processing logic: After feature recalculation, the system writes the new feature data to the storage area corresponding to the current production line number in the non-volatile storage area, overwriting the old features. The write operation is executed atomically: first, the new feature data is written to a temporary backup area; after writing is completed and data integrity is verified, the pointer is switched to the new data area through a single atomic operation. This mechanism avoids the storage area being in a partially updated and inconsistent state due to power failure during the write process, ensuring data reliability after system restart.

[0097] When the system acquires the current high-resolution image in the next moment, it reads the replaced frozen background prefix features from the non-volatile storage area. The read features must undergo the exact same normalization transformation as the high-resolution image, with the transformation parameters stored together with the feature data to ensure consistency between the two transformations. Then, the normalized replacement features are used as the reference tensor for pixel-level difference calculation, and the residual feature tensor is generated by subtracting the reference tensor from the current image's normalized tensor. Subsequent processes are consistent with steps three through eight, thus forming a complete and adaptive detection loop.

[0098] Output data: The updated state of the non-volatile storage area and the baseline tensor for the next inference cycle.

[0099] In industrial roll material production scenarios, the detection of surface defects in roll materials faces a fundamental contradiction between real-time requirements and massive computational demands. This application addresses this issue by breaking down the inference process into multiple pipeline stages and executing them in parallel at a micro-batch granularity, allowing the stages to overlap in time and effectively improving end-to-end throughput.

[0100] By introducing a mechanism that freezes the background prefix feature and triggers recalculation with the physical roll change signal, the background feature always remains consistent with the actual surface characteristics of the current roll material, fundamentally eliminating systematic false detections caused by background mismatch.

[0101] By extending the output of each layer of the neural network from point estimation to interval estimation, and combining it with cascaded interval intersection operation to achieve consistency filtering, the cascaded error is suppressed into stepwise filtering, thereby improving the accuracy of the final decision.

[0102] By establishing a cross-cycle decision consistency check mechanism, erroneous system shutdowns caused by single-frame image anomalies are effectively prevented. By writing new feature atoms into non-volatile storage, the system can immediately load the latest features for detection after a restart.

Claims

1. A method for parallel acceleration of multimodal inference pipelines for edge-cloud collaboration, characterized in that, The method includes: Acquire industrial sensor data at the current moment, including high-resolution images and process parameters; In response to the received physical roll change signal, the stored frozen background prefix features are extracted; Calculate the pixel-level difference between the high-resolution image and the frozen background prefix features, and generate a residual feature tensor; The parameter vector obtained by combining the residual feature tensor with the process parameters is concatenated by channel to generate a multimodal fusion vector. The multimodal fusion vector is divided into pipeline micro-batches and sequentially input into cascaded convolutional and fully connected layers. The element-wise activation function is taken for the output of each layer to obtain the intermediate confidence interval. Perform an interval intersection operation on the intermediate confidence intervals output from adjacent cascaded stages to obtain the updated upper and lower bounds of the intervals, and output them as the final control decision intervals. The final control decision interval is sent as a control command to the production line execution mechanism; In response to the next physical roll change signal received from the production line actuator, the frozen background prefix feature is recalculated, and the recalculated frozen background prefix feature is used for pixel-level difference calculation at the next moment.

2. The method according to claim 1, characterized in that, The extraction of stored frozen background prefix features includes: Analyze the production line number carried in the physical roll change signal; The corresponding frozen background prefix feature is read from the non-volatile storage area according to the production line number; The obtained frozen background prefix features are output as the frozen background prefix features of the current inference cycle.

3. The method according to claim 2, characterized in that, The step of calculating the pixel-level difference between the high-resolution image and the frozen background prefix features to generate a residual feature tensor includes: Perform the same normalization transformation as the frozen background prefix feature on the high-resolution image to obtain the normalized image tensor; The normalized image tensor and the frozen background prefix features are subtracted element-wise at corresponding pixel positions to obtain the original residual image; Perform element-wise absolute value operation on the original residual map to generate a residual magnitude map; The residual magnitude map is input into a preset threshold filtering layer to generate a residual feature tensor.

4. The method according to claim 1, characterized in that, The step of concatenating the parameter vector obtained by combining the residual feature tensor with the process parameters by channel to generate a multimodal fusion vector includes: The process parameters were analyzed to extract the values ​​of temperature, tension, and speed. The temperature, tension, and velocity values ​​are mapped to embedding vectors that match the channel dimensions of the residual feature tensor, respectively. Perform layer normalization on the embedded vector to generate a parameter vector; The residual feature tensor and parameter vector are concatenated along the channel dimension to generate a multimodal fusion vector.

5. The method according to claim 1, characterized in that, The step of dividing the multimodal fusion vector into pipeline micro-batches and sequentially inputting it into cascaded convolutional and fully connected layers includes: Obtain the current memory capacity of the inference pipeline and the feature size of a single image; Based on the video memory capacity and the feature size of a single image, calculate the maximum number of image frames that a single batch can accommodate, and use this as the micro-batch size; According to the micro-batch size, the multimodal fusion vector is sliced ​​along the batch dimension to generate multiple micro-batch sub-vectors; Record the original batch index of each micro-batch sub-vector in the multimodal fusion vector; Following the order of the original batch index, each micro-batch sub-vector is sequentially input into the cascaded convolutional layer and fully connected layer.

6. The method according to claim 1, characterized in that, The stepwise activation function applied to the output of each layer is used to obtain the intermediate confidence interval, including: The original tensors output by the convolutional and fully connected layers are divided into lower bound channel groups and upper bound channel groups according to the channel dimension. Perform an element-wise exponential normalization transformation on the lower bound channel group to generate a normalized lower bound subtensor; Perform an element-wise exponential normalization transformation on the upper limit channel group to generate a normalized upper limit subtensor; An element-wise comparison is performed between the normalized lower bound subtensor and the normalized upper bound subtensor to ensure that the lower bound is not greater than the upper bound. The normalized lower bound subtensor and the normalized upper bound subtensor are concatenated sequentially along the channel dimension to form the intermediate confidence interval.

7. The method according to claim 6, characterized in that, The step of performing an interval intersection operation on the intermediate confidence intervals output from adjacent cascaded stages to obtain updated interval upper and lower bounds includes: Extract the first lower bound tensor and the first upper bound tensor from the output of the first cascade stage; Extract the second lower bound tensor and the second upper bound tensor from the output of the second cascade stage; For each tensor position, take the larger of the two lower bound tensors as the lower bound of the interval after the intersection operation, and take the smaller of the two upper bound tensors as the upper bound of the interval after the intersection operation. The lower bound tensor and the upper bound tensor obtained from the intersection operation are combined to form the updated upper and lower bounds of the interval.

8. The method according to claim 1, characterized in that, Sending the final control decision interval as a control command to the production line execution mechanism includes: Analyze the lower and upper bound tensors of the final control decision interval; Calculate the center values ​​of the lower limit tensor and the upper limit tensor along the decision dimension to generate control quantity values. Use the control quantity values ​​as the driving signal of the actuator. Reserve the lower limit tensor and the upper limit tensor for confidence verification in the next inference cycle. Based on the control quantity value, a control instruction including the execution action code and duration is generated; The control commands are written into the communication port connected to the production line actuator.

9. The method according to claim 1, characterized in that, The response to the next physical roll change signal received from the production line actuator includes: Monitor the feedback channel connected to the production line actuator; Analyze the device status messages transmitted in the feedback channel; When the device status message indicates that the production line has entered the roll change state, it is determined that the next physical roll change signal has been received. Extract the new roll texture image carried in the next physical roll change signal; Based on the new roll texture image, trigger the recalculation process of freezing background prefix features; The recalculated frozen background prefix features are associated with the inference period corresponding to the next physical roll-over signal.

10. The method according to claim 1, characterized in that, The step of using the recalculated frozen background prefix features for pixel-level difference calculation at the next time step includes: The recalculated frozen background prefix features are written to the non-volatile storage area, replacing the features in the original storage area; When acquiring the current high-resolution image in the next moment, the replaced frozen background prefix feature is synchronously invoked; Perform the same normalization transformation on the replaced frozen background prefix features as on the high-resolution image; The normalized replacement features are used as the reference tensor for pixel-level difference calculation.

Images