A man-machine collaborative quality detection and early warning system for a smart factory

Abstract

A man-machine collaborative quality detection and early warning system for intelligent factories, comprising an edge visual acquisition terminal, a cognitive anchor generation module and a hybrid intelligent inference engine, the hybrid intelligent inference engine extracts a high-dimensional feature vector of a workpiece image using a residual network and calculates a Mahalanobis distance. When the distance is in a fuzzy decision interval, a man-machine collaboration request is triggered, and a feature heat map is displayed on the terminal. The cognitive anchor generation module receives defect key pixel points labeled by an operator, maps them to a high-dimensional feature space to construct a feature repulsion sphere, and adds the feature repulsion sphere as a regularization term to the loss function of the residual network for online fine-tuning. The scheme corrects the feature boundary in a targeted manner without changing the weight distribution of the model main body, overcomes long-tail defect missed defects, and improves the feature boundary stability of the detection model under continuous production conditions.

Description

Technical Field

[0001] This invention belongs to the field of hybrid intelligence computing technology and discloses a human-machine collaborative quality detection and early warning system for smart factories. Background Technology

[0002] Current smart factory quality inspection typically employs deep residual networks to extract high-dimensional feature vectors from workpiece surface images and classify them into known defect categories. In actual deployments, the model relies on historically collected sample datasets for offline training. When rare defects deviating from the learned distribution are encountered during production, the model outputs low-confidence scores. The conventional approach is to set a static confidence threshold; when the score falls below this threshold, the system pushes the image to a manual re-inspection terminal for offline judgment. Quality inspectors view the original images on a 2D monitor and annotate the defects. Subsequently, engineers compile the newly annotated images and merge them into the original training set at fixed intervals for global retraining of the neural network model.

[0003] In the aforementioned conventional approach, manual judgment and machine model updates are separated into two independent offline processes. Faced with long-tail defects and extremely low training sample sizes, the model lacks sufficient feature representation. Periodically retraining the model globally by mixing newly labeled images with large amounts of historical data disrupts the established feature classification boundaries, leading to a degradation in the model's generalization ability on normal samples and causing false positives. Because real-time feedback from manual judgment cannot be used for localized corrections in the specific feature space, the model consistently misses detections when facing long-tail defects and lacks the ability to dynamically adapt to new defect features without disrupting the existing feature distribution. Summary of the Invention

[0004] The purpose of this invention is to provide a solution to the problems described in the background section.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A human-machine collaborative quality inspection and early warning system for smart factories includes an edge vision acquisition terminal, a cognitive anchor point generation module, and a hybrid intelligent inference engine. The edge vision acquisition terminal is communicatively connected to the hybrid intelligent inference engine, and the hybrid intelligent inference engine is communicatively connected to the cognitive anchor point generation module. The edge vision acquisition terminal is used to acquire images of the workpiece surface and input them into the hybrid intelligent inference engine; The hybrid intelligent inference engine uses a residual network to extract high-dimensional feature vectors from the workpiece surface image and calculates the Mahalanobis distance between the high-dimensional feature vectors and the known defect category centers. When the Mahalanobis distance is within a preset fuzzy decision range, the hybrid intelligent inference engine triggers a human-machine collaboration request and displays a feature heatmap on the operator's augmented reality terminal. The cognitive anchor generation module receives the defect key pixels marked by the operator based on the feature heatmap, maps the defect key pixels to the high-dimensional feature space, constructs a feature repulsion sphere with the mapped coordinates of the defect key pixels as the center and a preset attenuation coefficient as the radius, and adds the feature repulsion sphere as a regularization term to the loss function of the residual network for online fine-tuning.

[0006] Preferably, when calculating the Mahalanobis distance, the hybrid intelligent inference engine maintains a shared covariance matrix corresponding to the known defect category. The shared covariance matrix is ​​iteratively updated by collecting the high-dimensional feature vectors of historical batches through a sliding window mechanism. After extracting the high-dimensional feature vector, the hybrid intelligent inference engine performs global average pooling on the high-dimensional feature vector, inputs the pooled feature vector into the metric space, and performs matrix multiplication on the difference between the inverse of the shared covariance matrix and the known defect category center to obtain the Mahalanobis distance. The metric space increases the Euclidean distance between different defect category centers by introducing a contrastive learning loss function.

[0007] Preferably, the preset fuzzy decision interval is dynamically defined by the historical Mahalanobis distance distribution of the known defect categories; The hybrid intelligent inference engine statistically analyzes the Mahalanobis distance sets corresponding to each of the known defect categories within a preset time window, and calculates the kernel density estimation curve of the Mahalanobis distance sets. For each known defect category, the first Mahalanobis distance corresponding to the trough position of the kernel density estimation curve is used as the lower boundary of the fuzzy decision interval, and the second Mahalanobis distance corresponding to the intersection of the kernel density estimation curves of adjacent defect categories is used as the upper boundary of the fuzzy decision interval. The hybrid intelligent inference engine matches the corresponding lower and upper boundaries in real time according to the defect category to which the current high-dimensional feature vector belongs.

[0008] Preferably, when generating the feature heatmap, the hybrid intelligent inference engine obtains the feature map output by the last convolutional layer of the residual network and calculates the gradient information of the feature map relative to the high-dimensional feature vector. The hybrid intelligent inference engine performs global average pooling on the gradient information along the channel dimension to obtain the weight matrix of each pixel. The weight matrix is ​​multiplied element-wise with the feature map output by the last convolutional layer to obtain the initial heatmap; The hybrid intelligent inference engine performs bilinear interpolation upsampling on the initial heatmap to make the size of the initial heatmap consistent with the size of the workpiece surface image, and then superimposes the upsampled heatmap onto the workpiece surface image displayed on the augmented reality terminal.

[0009] Preferably, when the cognitive anchor point generation module maps the defect key pixel to the high-dimensional feature space, it extracts the local feature map patch corresponding to the last convolutional layer of the residual network for the defect key pixel, inputs the local feature map patch into a pre-trained back-projection multilayer perceptron, and outputs the mapped coordinates. When constructing the feature rejection sphere, the cognitive anchor generation module calculates the initial Euclidean distance between the mapped coordinates and the high-dimensional feature vector that triggers the human-machine collaboration request. It then scales the initial Euclidean distance using a preset baseline attenuation factor to obtain the preset attenuation coefficient. Finally, it binds and stores the mapped coordinates, the preset attenuation coefficient, and the known defect category label to which the high-dimensional feature vector originally belonged.

[0010] Preferably, when the feature rejection ball is added as a regularization term to the loss function of the residual network, the loss function includes a classification cross-entropy loss term and a rejection ball regularization loss term; The repulsion sphere regularization loss term is configured to apply an exponential penalty gradient based on the distance from the high-dimensional feature vector of the sample to the mapped coordinates when the high-dimensional feature vector of the subsequently input sample falls inside the feature repulsion sphere. When performing the online fine-tuning, the hybrid intelligent inference engine freezes the weight parameters of the remaining network layers in the residual network except for the last two fully connected layers, and only uses the loss function containing the repulsion ball regularization loss term to perform backpropagation updates on the last two fully connected layers.

[0011] Preferably, the residual network includes a shallow feature extraction branch, a middle feature extraction branch, and a deep feature extraction branch connected in sequence; The shallow feature extraction branch is used to extract the edge texture features of the workpiece surface image, the middle feature extraction branch is used to extract local morphological features, and the deep feature extraction branch is used to extract global semantic features. After obtaining the outputs of the shallow feature extraction branch, the middle feature extraction branch, and the deep feature extraction branch, the hybrid intelligent inference engine unifies feature maps of different scales into feature tensors of the same size through a spatial pyramid pooling layer. The feature tensors are then concatenated along the channel dimension, and the concatenated feature tensors are used as input to the high-dimensional feature vector processed by the global average pooling.

[0012] Preferably, the hybrid intelligent inference engine performs an edge-preserving filter operation on the upsampled heatmap before overlaying it onto the workpiece surface image; The hybrid intelligent inference engine extracts connected components in the upsampled heatmap whose weights are greater than a preset weight threshold, and calculates the bounding rectangle of the connected components. Inside the outer rectangle, a bilateral filter is used to smooth the upsampled heatmap; outside the outer rectangle, the pixel values ​​of the upsampled heatmap are masked to zero. When the heatmap after the edge-preserving filtering operation is superimposed on the augmented reality terminal, only the area inside the outer rectangle is retained, and it is mapped into a pseudo-color spectrum from red to blue according to the heat value from high to low.

[0013] Preferably, the cognitive anchor generation module is equipped with a preset baseline decay factor that dynamically adjusts over time; The cognitive anchor generation module records the creation timestamp of the feature repulsion ball, and obtains the time difference between the current system time and the creation timestamp when calculating the preset attenuation coefficient; The cognitive anchor generation module uses an exponential decay function to decay the preset baseline decay factor to obtain the actual decay factor at the current moment. When the time difference exceeds the preset lifecycle threshold, the cognitive anchor generation module sets the actual decay factor to zero, deletes the mapped coordinates and the corresponding known defect category labels from the storage space, and releases the memory resources occupied by the feature rejection ball.

[0014] Preferably, when the hybrid intelligent inference engine performs backpropagation updates on the last two fully connected layers using a loss function that includes the repulsion ball regularization loss term, it simultaneously initiates a sample replay mechanism. The hybrid intelligent inference engine maintains a local circular buffer, which stores images of normal workpiece surfaces that have been historically determined to be non-fuzzy decision intervals and their corresponding defect category labels in the circular buffer according to a first-in-first-out strategy. Before each backpropagation calculation for online fine-tuning, a batch of replay samples is randomly drawn from the circular buffer. The replay samples are then input into the residual network after the weight parameters are frozen. The classification cross-entropy loss of the replay samples is calculated, and the classification cross-entropy loss of the replay samples is weighted and summed with the repulsion ball regularization loss term to obtain the final total loss function.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention directly transforms human judgment of defects into local geometric constraints in a high-dimensional feature space through a cognitive anchor point generation module. The system maps the operator-annotated key defect pixels to the feature space to construct a feature repulsion sphere, and adds it as a regularization term to the loss function for online fine-tuning. During fine-tuning, the hybrid intelligent inference engine freezes the weights of the residual network layers except for the last two fully connected layers. This mechanism, without changing the original feature distribution structure of the model, locally adjusts the classification boundary of the fuzzy decision interval to match human cognition, solving the technical problems of periodic global retraining destroying feature boundaries and long-tail defect under-detection.

[0016] 2. This invention utilizes a sliding window mechanism to update the shared covariance matrix and dynamically defines the upper and lower boundaries of the fuzzy decision interval using a kernel density estimation curve, enabling the metric space's decision criteria to adapt to data drift in continuous production. By extracting the gradient information from the last convolutional layer to generate a pseudo-color feature heatmap with edge-preserving filtering, operators are guided to focus their annotations on questionable areas, improving the accuracy of the mapping coordinates output by the back-projected multilayer perceptron. Combining the feature repulsion sphere's lifecycle decay mechanism with a sample replay mechanism including a circular buffer, the model maintains the stability of its memory of historical normal samples when introducing new cognitive anchors and releases memory resources when constraints fail, ensuring the system stability of the hybrid intelligent inference engine under long-term operating conditions. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall operation of a human-machine collaborative quality inspection and early warning system for smart factories. Figure 2 This is a flowchart of Mahalanobis distance calculation based on shared covariance matrix in a hybrid intelligent inference engine. Figure 3 A flowchart for dynamically defining fuzzy decision intervals based on kernel density estimation curves; Figure 4 Flowchart for feature heatmap generation and rendering incorporating edge-preserving filtering; Figure 5 A flowchart for constructing feature rejection spheres and online fine-tuning based on regularization terms; Figure 6 The flowchart shows the total loss calculation and backpropagation process combined with the circular buffer sample playback mechanism. Detailed Implementation

[0018] In one embodiment, a human-machine collaborative quality inspection and early warning system for smart factories includes an edge vision acquisition terminal, a cognitive anchor point generation module, and a hybrid intelligent inference engine. The edge vision acquisition terminal is communicatively connected to the hybrid intelligent inference engine, which in turn is communicatively connected to the cognitive anchor point generation module. The edge vision acquisition terminal is configured to acquire workpiece surface images and input them into the hybrid intelligent inference engine. Specifically, the edge vision acquisition terminal is deployed above the workstations on the production line of the smart factory and includes an industrial-grade high-resolution camera and a ring light source. The industrial-grade high-resolution camera continuously acquires images of the workpiece surfaces transported by the conveyor belt on the production line. During the acquisition process, the ring light source provides strobe illumination at a specific frequency to eliminate metallic reflections on the workpiece surface. The acquired workpiece surface images are converted into digital signals by an image acquisition card and transmitted to the hybrid intelligent inference engine via a gigabit Ethernet interface. After acquiring the workpiece surface images, the edge vision acquisition terminal performs preprocessing operations on the images. The preprocessing operations include Gaussian filtering for noise reduction and histogram equalization. Gaussian filtering for noise reduction uses a two-dimensional Gaussian kernel function with a size of 5×5 and a standard deviation of [missing information]. Set as For any pixel in the workpiece surface image Its Gaussian filtered grayscale value The calculation formula is: Among them, I Indicates the original image in coordinates The grayscale value at the specified location. Histogram equalization stretches the image contrast by calculating the cumulative grayscale distribution function, enhancing the contrast between workpiece surface defects and the normal background. The preprocessed workpiece surface image is adjusted to a fixed resolution size, such as 224×224, with a standard deviation of [missing value]. Set as For any pixel in the workpiece surface image Its Gaussian filtered grayscale value The pixels are then fed into the hybrid intelligent inference engine.

[0019] The hybrid intelligent inference engine is deployed on an edge computing server. The pixel matrix of the received workpiece surface image is input into a pre-constructed residual network. The residual network performs multi-layer convolution operations on the workpiece surface image to extract a high-dimensional feature vector. The extracted high-dimensional feature vector is a multi-dimensional floating-point array representing the texture, morphology, and semantic information of the workpiece surface. After obtaining the high-dimensional feature vector, the hybrid intelligent inference engine calculates the Mahalanobis distance between this high-dimensional feature vector and the known defect category center. The known defect category center is the reference coordinate point obtained by clustering the feature vectors of various known defects during the system's historical operation. When the calculated Mahalanobis distance is within a preset fuzzy decision interval, the hybrid intelligent inference engine determines that the current workpiece surface image is in the fuzzy zone of the model's cognition, triggering a human-machine collaboration request. The triggered human-machine collaboration request is transmitted via the network to the augmented reality terminal worn by the operator. Upon receiving the request, the augmented reality terminal displays the current workpiece surface image on the display interface and overlays a feature heatmap on the image. The feature heatmap uses different color mappings to visually demonstrate which regions in the image the residual network prioritizes during feature extraction. Operators observe the feature heatmap displayed on the augmented reality terminal and, based on their professional experience, annotate key defect pixels on the terminal's touchscreen. The cognitive anchor generation module receives the coordinates of these key defect pixels annotated by the operator based on the feature heatmap. Upon receiving the key defect pixels, the cognitive anchor generation module maps them to the high-dimensional feature space output by the residual network, obtaining the mapped coordinates. After obtaining the mapped coordinates, the cognitive anchor generation module constructs a feature repulsion sphere in the high-dimensional feature space, centered on these coordinates and with a preset attenuation coefficient as the radius. The feature repulsion sphere defines a geometric region in the high-dimensional space. After constructing the feature repulsion sphere, the cognitive anchor generation module transforms it into a regularization term and adds it to the residual network's loss function. The hybrid intelligent inference engine then fine-tunes the residual network online based on the loss function after adding the regularization term. The online fine-tuning process is completed in real time on the edge computing server, without the need to upload data to the cloud or wait for engineers to perform offline global retraining.

[0020] Specifically, the residual network comprises sequentially connected shallow feature extraction branches, mid-level feature extraction branches, and deep feature extraction branches. The weights of all convolutional layers in the residual network are initialized using the Kaiming normal distribution method. For features such as... A convolutional layer with n input nodes, whose weights W range from a mean of 0 and a variance of 1. The sampling is performed within a normal distribution. The shallow feature extraction branch is configured to extract edge texture features from the workpiece surface image. The shallow feature extraction branch contains two cascaded convolutional layers, each with a kernel size of 3×3, a stride of 1, and padding of 1. The first convolutional layer has 3 input channels, corresponding to the red, green, and blue channels, and 64 output channels. The second convolutional layer has 64 input and 64 output channels. After each convolutional layer, a batch normalization layer and a modified linear unit activation function are configured. The batch normalization layer normalizes the two-dimensional feature map of each channel using the following formula: in, The feature map output by the convolutional layer is represented in the channel. ,high ,width The value at that location, and These represent the channels in the current batch. For all spatial locations, the mean and variance, Set as To prevent the denominator from being zero, This represents the standardized output. The output feature map size of the shallow feature extraction branch is consistent with the size of the input workpiece surface image.

[0021] The mid-level feature extraction branch is configured to extract local morphological features. This branch contains three residual blocks, each containing two 3×3 convolutional layers, with element-wise addition performed between the main path and the residual path. For the first convolutional layer in the residual block, its output... The calculation formula is: in, This represents the convolution operation. and This represents the convolution kernel weights and biases. The final output of the residual block. for: in, The input feature map is the residual block. After the first residual block in the middle-layer feature extraction branch, a 3×3 max-pooling layer with a stride of 2 is configured to downsample the shallow feature map. After downsampling, the feature map output by the middle-layer feature extraction branch is halved in both height and width dimensions, and the number of channels is expanded to 128. The middle-layer feature extraction branch captures local morphological structures such as scratches and dents on the workpiece surface by expanding the receptive field. The deep feature extraction branch is configured to extract global semantic features. The deep feature extraction branch contains four residual blocks, and a 3×3 max-pooling layer is configured after the first residual block. The output channel number of the deep feature extraction branch is set to 256. The deep feature extraction branch abstracts local morphological structures into discriminative global semantic information through deep convolution operations.

[0022] After obtaining the outputs from the shallow, mid-level, and deep feature extraction branches, the hybrid intelligent inference engine uses a spatial pyramid pooling layer to unify feature maps of different scales into feature tensors of the same size. Specifically, the spatial pyramid pooling layer performs different levels of pooling operations on the feature maps output from the shallow feature extraction branch, the shallow feature maps after downsampling with a stride of 2, the feature maps output from the mid-level feature extraction branch, and the feature maps output from the deep feature extraction branch. The pyramid level set is set to 1, 2, and 4. For an input size of... The feature map is used to calculate the grid pooling results for the spatial pyramid pooling layers at levels 11, 22, and 44. The feature map is uniformly divided into height and width dimensions. The first grid region. For the first... Each grid region covers a feature map row index range of [number]. The column index range is Global average pooling is performed within the grid region to convert feature maps of different scales into fixed-length feature vectors. The feature vectors output from the shallow, mid-level, and deep feature extraction branches, after processing by the spatial pyramid pooling layer, are concatenated along the channel dimension. Mathematically, this concatenation operation involves cascading the feature vectors output from each branch along the channel axis, and the resulting feature tensor is used as the input to the high-dimensional feature vector for global average pooling.

[0023] The specific computational process of global average pooling is as follows: for the concatenated feature tensor... ,in Indicates the number of channels. and These represent the height and width dimensions of the feature map, respectively. In terms of spatial dimensions... For each channel The summation of the elements of a two-dimensional matrix and the division by the total number of spatial pixels is calculated using the following formula: in, Indicates the first The scalar output values ​​of each channel after global average pooling. Represents the feature tensor in the channel ,high ,width The characteristic response value at that point. Arrange the scalar output values ​​of all channels to form a high-dimensional feature vector. This high-dimensional feature vector integrates multi-scale edge texture, local morphology, and global semantic information, possessing extremely strong defect representation capabilities.

[0024] In one embodiment, reference Figure 2 The hybrid intelligent inference engine maintains a shared covariance matrix corresponding to known defect categories when calculating Mahalanobis distance. This shared covariance matrix is ​​iteratively updated by collecting high-dimensional feature vectors from historical batches using a sliding window mechanism. Specifically, the hybrid intelligent inference engine maintains a length of [missing information - likely a specific value] in local memory. A first-in, first-out (FIFO) queue is used as the sliding window. Whenever the residual network outputs a new high-dimensional feature vector that is not classified as falling within the fuzzy decision region, this high-dimensional feature vector is added to the tail of the sliding window. When the number of vectors in the sliding window exceeds... At time 10, the oldest vector at the head of the sliding window is removed. To reduce the computational complexity of recalculating the covariance matrix for each batch, the hybrid intelligent inference engine uses an online recursive algorithm to update the mean vector and the shared covariance matrix. Assume that at time 10... The sliding window contains There are n samples, and the mean vector is... The covariance matrix is At that moment The system receives new high-dimensional feature vectors. If the sliding window is not full, the sample count is updated. .time mean vector The recursive calculation formula is: time covariance matrix The recursive calculation formula is: If the sliding window is full, new samples Replace the oldest sample The sample size remains unchanged at this point. Mean vector The recursive calculation formula is updated as follows: covariance matrix The recursive calculation formula is updated as follows: in, This represents the regularization coefficient to prevent matrix singularity, and its value is... , Let represent the identity matrix. Using the above recursive formula, the system only requires constant-level matrix operations to iteratively update the shared covariance matrix each time a new sample is received, avoiding the huge computational overhead of traversing the entire sliding window.

[0025] After extracting high-dimensional feature vectors, the hybrid intelligent inference engine performs global average pooling on these vectors and inputs the pooled feature vectors into the metric space. The metric space amplifies the Euclidean distance between different defect category centers by introducing a contrastive learning loss function. This contrastive learning loss function is configured as a triplet-based metric learning loss, applied to the feature vectors of samples in each batch. Randomly select positive sample feature vectors from the same defect category Feature vectors of hard-to-bear samples are randomly selected from different defect categories. The strategy for mining hard-to-bear samples is as follows: In the current batch, calculate the feature vectors of all different classes and... The Euclidean distance is used to select samples with the smallest Euclidean distance but different classes as hard negative samples. A contrastive learning loss function is then applied. The mathematical expression is: in, This indicates the calculation of Euclidean distance. Norm, This represents the preset interval hyperparameter, with a value of 0.5. This indicates that the value is zero when the internal value is less than zero. This loss function generates a gradient during backpropagation, forcibly increasing the Euclidean distance between different defect category centers, causing different defects to form compact and far apart clusters in the metric space. In the metric space, the Mahalanobis distance is obtained by performing matrix multiplication on the difference between the inverse of the shared covariance matrix and the known defect category centers. The known defect category centers are represented as... , where j represents the defect category index. The current input high-dimensional feature vector is Mahalanobis distance The calculation formula is: in, This represents the inverse of the shared covariance matrix. Since a regularization term is added to the shared covariance matrix... Therefore, it must be a positive definite matrix and its inverse matrix. It necessarily exists and can be efficiently solved using Cholisky decomposition. Mahalanobis distance, by considering the variance and covariance of the feature vector in different dimensions, eliminates the interference of correlation between feature dimensions and accurately reflects the true deviation of the current feature vector from the known defect category center in the probability distribution.

[0026] The preset fuzzy decision interval is dynamically defined by the historical Mahalanobis distance distribution of known defect categories. The hybrid intelligent inference engine statistically analyzes the Mahalanobis distance sets corresponding to each known defect category within a preset time window. The preset time window is configured as the past T production hours. For each known defect category, the hybrid intelligent inference engine collects all historical Mahalanobis distance values ​​under that category, forming a Mahalanobis distance set. Calculate the kernel density estimation curve for the Mahalanobis distance set. The kernel density estimation uses a Gaussian kernel function. The bandwidth parameter of the Gaussian kernel function is... Determined using Silverman's rule, the calculation formula is: in, Represents the set of Mahalanobis distances The standard deviation of is calculated using the following formula: , Let represent the mean of the set of Mahalanobis distances. For any distance value... Its kernel density estimate The calculation formula is: refer to Figure 3 For each known defect category, the first Mahalanobis distance corresponding to the trough position of the kernel density estimation curve is used as the lower boundary of the fuzzy decision interval. The trough position is found using the Newton-Raphson iteration method. First, the starting point of the search is determined. Set as the mean Mahalanobis distance corresponding to the known defect category centers. Shift of standard deviation outward The position, that is The first derivative of the kernel density estimation curve and second derivative The parsing expressions are as follows: In the l-th iteration of Newton's method, according to the formula Update the distance value. When the iteration step size... Less than the preset convergence accuracy Stop iteration when the time is reached. Verify the final convergence point. Does it meet the requirements? If satisfied, then To determine the true trough location, we take it as the first Mahalanobis distance. That is, the lower boundary of the fuzzy decision interval.

[0027] The second Mahalanobis distance corresponding to the intersection of the kernel density estimation curves of adjacent defect categories is used as the upper boundary of the fuzzy decision interval. Specifically, for the current category... and its nearest neighboring class in the feature space Solve the equation In the current category center Center of adjacent categories The equation usually has real solutions along the line connecting the two points. Since an analytical solution cannot be obtained, a bisection method is used for numerical solution. The initial left endpoint of the bisection method is set. Initial right endpoint For category With category The midpoint of the Mahalanobis distance is the center point. In each iteration, the midpoint is calculated. Calculate the difference .like This indicates that the intersection point is on the left, so the right endpoint is updated to... ;like This indicates that the intersection point is on the right, so the left endpoint is updated to... .when When the iteration stops, take the final result. Second Mahalanobis distance This refers to the upper boundary of the fuzzy decision interval. The hybrid intelligent inference engine matches the corresponding lower and upper boundaries in real time based on the defect category to which the current high-dimensional feature vector belongs. After calculating the Mahalanobis distance of the current high-dimensional feature vector with respect to each category, it finds the category with the smallest Mahalanobis distance. Determine the minimum Mahalanobis distance. Does it meet the requirements? If the conditions are met, the Mahalanobis distance is determined to be within a preset fuzzy decision range, triggering a human-machine collaboration request. This dynamic definition mechanism enables the determination criteria of the measurement space to adapt in real time to data drift caused by equipment wear and light changes during continuous production.

[0028] When generating the feature heatmap, the hybrid intelligent inference engine obtains the feature map output from the last convolutional layer of the residual network and calculates the gradient information of the feature map relative to the high-dimensional feature vector. Specifically, the feature map output from the last convolutional layer of the residual network is represented as... ,in This indicates the number of output channels of the last convolutional layer. and This represents the height and width of the feature map. The high-dimensional feature vector is passed through a global average pooling layer and then input into the final fully connected classification layer to obtain the corresponding category. logical value The hybrid intelligent inference engine utilizes backpropagation to calculate logical values. Output feature map for the last convolutional layer The gradient information is obtained by taking the partial derivatives of the derivatives. The formula for calculating gradient information is: The hybrid intelligent inference engine performs global average pooling on the gradient information along the channel dimension to obtain the weight matrix for each pixel. The formula for calculating the weight matrix is: in, Indicates at altitude ,width Pixel weights at that location Indicates channel In position The gradient value at that point. The weight matrix is ​​then multiplied element-wise with the feature map output from the last convolutional layer to obtain the initial heatmap. The calculation formula is: in, This represents a modified linear unit activation function, used to filter out negatively correlated feature responses.

[0029] The hybrid intelligent inference engine performs bilinear interpolation upsampling on the initial heatmap to ensure that the size of the initial heatmap matches the size of the workpiece surface image. Assume the size of the initial heatmap is... The size of the target workpiece surface image is For any pixel coordinate in the target image Calculate its virtual coordinates on the initial heatmap. The calculation formula is: Find the four integer coordinate points surrounding the virtual coordinates. , , , ,in , , , Calculate the interpolation weights in the horizontal and vertical directions. Horizontal weights. Vertical weights The thermal values ​​of the target pixels after upsampling. The calculation formula is: Through the above pixel-by-pixel bilinear interpolation calculation, the initial heat map is smoothly enlarged to match the size of the workpiece surface image.

[0030] Before overlaying the upsampled heatmap onto the workpiece surface image, the hybrid intelligent inference engine performs edge-preserving filtering on the upsampled heatmap. The engine extracts connected components in the upsampled heatmap whose weights are greater than a preset weight threshold. The preset weight threshold is configured as 0.3 times the maximum weight value in the upsampled heatmap. A depth-first search algorithm traverses all pixels in the heatmap, marking adjacent pixels with weights greater than the preset weight threshold as the same connected component. For each connected component, its bounding rectangle is calculated. The coordinates of the bounding rectangle are determined by taking the minimum and maximum values ​​of the x and y coordinates of all pixels within the connected component. After obtaining the bounding rectangle, a bilateral filter is used to smooth the upsampled heatmap inside the bounding rectangle, referencing... Figure 4 The kernel function of a bilateral filter combines spatial distance and pixel value differences. The Gaussian kernel function in the spatial domain... The calculation formula is: in, and Representing pixels and Two-dimensional coordinates on a heatmap, The spatial domain standard deviation is configured to 2.0 pixel units. The Gaussian kernel function for the range. The calculation formula is: in, and Representing pixels and The heat value, This represents the standard deviation of the value range, configured to 0.1. Normalization factor. The calculation formula is: in, Represented by pixels Centered on, with radius The square neighborhood. The bilateral filter is calculated only when the pixel... thermal value With the center pixel thermal value The weight of the Gaussian kernel function in the range is relatively large only when the values ​​are close; if the difference between the two exceeds a certain threshold, the weight of the kernel function in the range will be larger. If the difference between the internal thermal values ​​and the external zero-mask region is large, the weight of the range tends to zero, thus maintaining the sharpness of the edge.

[0031] Outside the circumscribed rectangle, the pixel values ​​of the upsampled heatmap are zeroed out using a masking process, setting the heatmap values ​​of all pixels outside the circumscribed rectangle to 0. When the heatmap, after edge-preserving filtering, is overlaid onto the augmented reality terminal, only the area inside the circumscribed rectangle is retained. For the heatmap values ​​within the retained area, they are mapped to a pseudo-color spectrum from red to blue, from high to low. The specific mapping rule is configured as follows: heatmap values ​​are normalized to the [0,1] interval; a value of 1 is mapped to red, a value of 0.5 to green, and a value of 0 to blue. Intermediate values ​​are calculated using piecewise linear interpolation to obtain the corresponding red, green, and blue channel color values. For the red channel... The calculation formula is: For green channel The calculation formula is: For the Blue Channel The calculation formula is: in, This represents the normalized thermal value. The calculated pseudo-color spectrum is overlaid semi-transparently onto the workpiece surface image displayed on the augmented reality terminal, guiding the operator to focus annotations on areas of concern.

[0032] The cognitive anchor point generation module extracts the local feature patches corresponding to the key defect pixels in the last convolutional layer of the residual network when mapping the key defect pixels to a high-dimensional feature space. Because the residual network involves multiple downsampling operations, each pixel on the workpiece surface image corresponds to a receptive field region of a certain size in the last convolutional layer. The cognitive anchor point generation module then uses the coordinates of the key defect pixels in the workpiece surface image as labeled by the operator. Combined with the total downsampling factor of the residual network Calculate the mapping coordinates of the pixel on the feature map of the last convolutional layer. The calculation formula is: Centered on the mapped coordinates, extract a local feature patch of size P×P. The local feature map patch is input into a pre-trained back-projection multilayer perceptron, which outputs mapped coordinates. The back-projection multilayer perceptron consists of three fully connected layers. The weight matrix of the first fully connected layer is... The bias vector is The weight matrix of the second fully connected layer is: The bias vector is The weight matrix of the third fully connected layer is: The bias vector is Local feature patches Flattened into a one-dimensional vector The forward propagation calculation process of the back-projection multilayer perceptron is as follows: in, These are the output mapped coordinates. Through a multi-layer nonlinear mapping network, the system backprojects the features of local two-dimensional image patches into a high-dimensional semantic space, establishing a correspondence between pixel-level annotations and feature-level representations.

[0033] When constructing the feature repulsion sphere, the cognitive anchor generation module calculates the initial Euclidean distance between the mapped coordinates and the high-dimensional feature vector that triggered the human-machine collaboration request. The high-dimensional feature vector that triggered the human-machine collaboration request is represented as... Initial Euclidean distance The calculation formula is: The initial Euclidean distance is scaled using a preset baseline attenuation factor to obtain a preset attenuation coefficient. (Preset attenuation coefficient) The calculation formula is: in, This represents the preset baseline attenuation factor, with a value of 0.8. Map the coordinates... Preset attenuation coefficient And the known defect category labels to which the high-dimensional feature vectors originally belonged. The data is bound and stored to form a cognitive anchor record, which is then stored in the memory of the cognitive anchor generation module.

[0034] In one embodiment, reference Figure 5 The cognitive anchor generation module includes a preset baseline decay factor that dynamically adjusts over time. The cognitive anchor generation module also records the creation timestamp of the feature repulsion sphere. When calculating the preset attenuation coefficient, the current system time is obtained. Calculate the time difference between the current system time and the creation timestamp. The cognitive anchor generation module uses an exponential decay function to attenuate a preset baseline decay factor, obtaining the actual decay factor at the current moment. The formula for calculating the exponential decay function is: in, This represents the decay rate hyperparameter. At this point, the preset decay coefficient for the current time is updated to... When the time difference Exceeding the preset lifecycle threshold At that time, the cognitive anchor generation module sets the actual attenuation factor to zero, that is... This makes the preset attenuation coefficient at the current moment... Simultaneously, the cognitive anchor generation module removes the mapping coordinates corresponding to the feature rejection sphere from the storage space. and corresponding known defect category labels This releases the memory resources occupied by the feature rejection sphere. The lifecycle decay mechanism reflects the timeliness of manual annotation. As production batches progress, early annotations for specific working conditions may no longer be applicable. Through automatic decay and deletion mechanisms, the accumulation of invalid feature space constraints is avoided, preventing memory overflow.

[0035] When feature repulsion balls are added as regularization terms to the loss function of the residual network, the loss function includes a classification cross-entropy loss term and a repulsion ball regularization loss term. Classification cross-entropy loss term The calculation formula is: in, This represents the total number of known defect categories. The one-hot encoded vector representing the true category. This indicates the category to which the residual network output belongs. The softmax probability value. The repulsion sphere regularization loss term is configured to apply an exponential penalty gradient based on the distance from the high-dimensional feature vector of the sample to the mapped coordinates when the high-dimensional feature vector of the subsequent input sample falls inside the feature repulsion sphere. Repulsion sphere regularization loss term The calculation formula is: in, This represents the total number of valid feature rejection balls currently stored in memory. Indicates the first The mapped coordinates of the repulsion spheres, Indicates the first The current preset attenuation coefficient of the feature repulsion sphere. Indicates the first Known defect category labels bound to a feature exclusion sphere. Indicates the first The penalty weight coefficient for each feature rejection sphere, This indicates the indicator function. It represents the true class label of the current input sample. Labels bound to feature repulsion spheres When the values ​​are the same, the indicator function takes a value of 1; otherwise, it takes a value of 0. When the sample's high-dimensional feature vector... Distance Mapped Coordinates The Euclidean distance is less than the preset attenuation coefficient. At that time, it falls into the interior of the characteristic repulsion sphere, and the exponential term The value is in The interval will result in a penalty loss.

[0036] During backpropagation, it is necessary to calculate the repulsion sphere regularization loss term relative to the high-dimensional feature vector. The partial derivatives are used to generate an exponentially penalized gradient. For a feature repulsion sphere that satisfies the label conditions... Its loss component is Using the chain rule, calculate right gradient: Due to the European distance The gradient is Substituting it into the above formula, we get: The gradient formula indicates that the direction of the gradient is from the mapped coordinates. Pointing to the current high-dimensional feature vector The gradient magnitude is modulated by the exponential term. A fixed-strength pushing force is generated when the high-dimensional feature vector Z of a subsequent input sample just falls inside the feature repulsion sphere. As Z continues to move towards the mapped coordinates... As the distance decreases, the value of the exponential term increases sharply and approaches 1, resulting in a large divergence gradient. Meanwhile, the denominator... It serves to adjust the magnitude of the thrust, when the preset attenuation coefficient is used. As the lifetime decays and the magnitude decreases, the magnitude of the push-away gradient is further amplified, ensuring that even with a shrinking repulsion sphere radius, sufficient penalty force can still be generated to push away erroneous features.

[0037] During online fine-tuning, the hybrid intelligent inference engine freezes the weight parameters of all layers in the residual network except for the last two fully connected layers. Specifically, when constructing the deep learning computation graph, for all convolutional kernel weight tensors in the shallow feature extraction branch, the mid-layer feature extraction branch, the deep feature extraction branch, and the affine transformation parameters of the spatial pyramid pooling layer, the computation graph separation operation is invoked to block the gradient backpropagation path. When the gradient signal of the total loss function backpropagates to the global average pooling layer, the gradient signal is truncated and no longer propagates forward to the preceding convolutional layers. Therefore, when the parameter optimizer performs update operations, these frozen weight parameters remain unchanged. The hybrid intelligent inference engine uses the Adam optimizer to update the weight parameters of the last two fully connected layers. The Adam optimizer maintains a first-order moment estimate for each weight parameter. and second-order moment estimation At that moment Any weight parameter in the last two fully connected layers The gradient is denoted as The update formulas for the first-order moment estimate and the second-order moment estimate are as follows: in, Set to 0.9, Set to 0.999. To eliminate initialization bias, bias corrections are applied to the first-order and second-order moment estimates: Weight parameters The final update formula is: in, This represents the learning rate of the Adam optimizer, configured as follows: , Indicates the numerical stability term, configured as follows Through the Adam optimizer, the hybrid intelligent inference engine can adaptively adjust the learning step size of the last two fully connected layers, accelerating the convergence of the online fine-tuning process. This mechanism, without altering the original feature distribution structure of the model, locally and directionally adjusts the classification boundaries of the fuzzy decision interval to align with human cognition, solving the technical problems of periodic global retraining destroying feature boundaries and missing long-tail defects.

[0038] In one embodiment, reference Figure 6The hybrid intelligent inference engine simultaneously initiates a sample replay mechanism when backpropagating the last two fully connected layers using a loss function that includes a repulsion ball regularization loss term. The hybrid intelligent inference engine maintains a local circular buffer. The underlying layer of the circular buffer uses a one-dimensional array. Implemented, the size of the array is The configuration is set to 10000. Each element in the array stores a structure containing two fields: the image pixel matrix and the defect category label. A circular buffer and a head pointer are used for maintenance. Sum of tail pointers Initially, all pointers point to array index 0. When a normal workpiece surface image and its corresponding defect category label, which have been historically determined to be within a non-fuzzy decision range, are received, an enqueue operation is performed. The image pixel matrix and defect category label are then stored in... Then move the tail pointer one position to the right, using the following formula: .when After moving, it equals When the buffer is full, the overwrite strategy is executed, moving the head pointer one position to the right. The calculation formula is: Discard the oldest historical samples.

[0039] Before each backpropagation computation for online fine-tuning, the system needs to randomly draw replay samples of batch size B from the circular buffer. The batch size is configured to be 32. The drawing process generates... One in A non-repeating random integer sequence within the range, where This represents the number of valid samples in the current buffer. For each random integer... Calculate its actual index in the circular buffer array. By index from The corresponding replay samples are read from the database. The extracted replay samples are input into the residual network after freezing the weight parameters. Through forward propagation, the high-dimensional feature vectors and classification prediction results of the replay samples are obtained. The classification cross-entropy loss of the replay samples is calculated. The final total loss function is obtained by weighted summation of the classification cross-entropy loss and the repulsion sphere regularization loss term from the replay samples. The calculation formula is: in, This represents the balanced weight hyperparameter of the repulsion sphere regularization loss term. During backpropagation, this total loss function is used to calculate the gradient and update the weight parameters of the last two fully connected layers. The sample replay mechanism continuously inputs historical normal samples into the network each time a new artificial cognitive anchor is introduced for local fine-tuning. This forces the network to maintain the stability of its memory of the learned normal sample feature distribution while adjusting the classification boundary of specific long-tail defects, thus avoiding catastrophic forgetting of the model during long-term operation.

[0040] In one embodiment, a human-machine collaborative quality inspection and early warning system for smart factories directly transforms operator judgments of long-tail defects into local geometric constraints in a high-dimensional feature space through a cognitive anchor point generation module. The system maps operator-annotated key defect pixels to the feature space to construct a feature repulsion sphere, which is then added as a regularization term to the loss function for online fine-tuning. During fine-tuning, the hybrid intelligent inference engine freezes the weights of the residual network layers except for the last two fully connected layers. This mechanism, without altering the original feature distribution structure of the model, locally adjusts the classification boundaries of the fuzzy decision interval to align with human cognition, solving the technical problems of periodic global retraining destroying feature boundaries and missed detection of long-tail defects. The system utilizes a sliding window mechanism to collect historical batch high-dimensional feature vectors for iterative updates to the shared covariance matrix, and dynamically defines the upper and lower boundaries of the fuzzy decision interval using a kernel density estimation curve, adapting the judgment criteria of the metric space to changes in equipment status and data distribution shifts caused by illumination drift during continuous production. By extracting the gradient information from the last convolutional layer to generate a pseudo-color feature heatmap with edge-preserving filtering, operators are guided to focus their annotations on suspicious areas, eliminating background noise interference and improving the accuracy of the mapping coordinates output by the back-projected multilayer perceptron. Combining the lifecycle decay mechanism of the feature repulsion sphere with a sample replay mechanism including a circular buffer, the model maintains the stability of its memory of historical normal samples when introducing new cognitive anchors, and automatically releases memory resources when constraints fail, ensuring the system resource stability and detection accuracy of the hybrid intelligent inference engine under long-term continuous operation.

Claims

1. A human-machine collaborative quality inspection and early warning system for smart factories, characterized in that, It includes an edge vision acquisition terminal, a cognitive anchor point generation module, and a hybrid intelligent inference engine. The edge vision acquisition terminal is communicatively connected to the hybrid intelligent inference engine, and the hybrid intelligent inference engine is communicatively connected to the cognitive anchor point generation module. The edge vision acquisition terminal is used to acquire images of the workpiece surface and input them into the hybrid intelligent inference engine; The hybrid intelligent inference engine uses a residual network to extract high-dimensional feature vectors from the workpiece surface image and calculates the Mahalanobis distance between the high-dimensional feature vectors and the known defect category centers. When the Mahalanobis distance is within a preset fuzzy decision range, the hybrid intelligent inference engine triggers a human-machine collaboration request and displays a feature heatmap on the operator's augmented reality terminal. The cognitive anchor generation module receives the defect key pixels marked by the operator based on the feature heatmap, maps the defect key pixels to the high-dimensional feature space, constructs a feature repulsion sphere with the mapped coordinates of the defect key pixels as the center and a preset attenuation coefficient as the radius, and adds the feature repulsion sphere as a regularization term to the loss function of the residual network for online fine-tuning.

2. The human-machine collaborative quality detection and early warning system for smart factories according to claim 1, characterized in that, When calculating the Mahalanobis distance, the hybrid intelligent inference engine maintains a shared covariance matrix corresponding to the known defect category. The shared covariance matrix is ​​iteratively updated by collecting the high-dimensional feature vectors of historical batches through a sliding window mechanism. After extracting the high-dimensional feature vector, the hybrid intelligent inference engine performs global average pooling on the high-dimensional feature vector, inputs the pooled feature vector into the metric space, and performs matrix multiplication on the difference between the inverse of the shared covariance matrix and the known defect category center to obtain the Mahalanobis distance. The metric space increases the Euclidean distance between different defect category centers by introducing a contrastive learning loss function.

3. The human-machine collaborative quality detection and early warning system for smart factories according to claim 1, characterized in that, The preset fuzzy decision interval is dynamically defined by the historical Mahalanobis distance distribution of the known defect categories; The hybrid intelligent inference engine statistically analyzes the Mahalanobis distance sets corresponding to each of the known defect categories within a preset time window, and calculates the kernel density estimation curve of the Mahalanobis distance sets. For each known defect category, the first Mahalanobis distance corresponding to the trough position of the kernel density estimation curve is used as the lower boundary of the fuzzy decision interval, and the second Mahalanobis distance corresponding to the intersection of the kernel density estimation curves of adjacent defect categories is used as the upper boundary of the fuzzy decision interval. The hybrid intelligent inference engine matches the corresponding lower and upper boundaries in real time according to the defect category to which the current high-dimensional feature vector belongs.

4. The human-machine collaborative quality detection and early warning system for smart factories according to claim 1, characterized in that, When generating the feature heatmap, the hybrid intelligent inference engine obtains the feature map output by the last convolutional layer of the residual network and calculates the gradient information of the feature map relative to the high-dimensional feature vector. The hybrid intelligent inference engine performs global average pooling on the gradient information along the channel dimension to obtain the weight matrix of each pixel. The weight matrix is ​​multiplied element-wise with the feature map output by the last convolutional layer to obtain the initial heatmap; The hybrid intelligent inference engine performs bilinear interpolation upsampling on the initial heatmap to make the size of the initial heatmap consistent with the size of the workpiece surface image, and then superimposes the upsampled heatmap onto the workpiece surface image displayed on the augmented reality terminal.

5. The human-machine collaborative quality detection and early warning system for smart factories according to claim 1, characterized in that, When mapping the defect key pixel to the high-dimensional feature space, the cognitive anchor point generation module extracts the local feature map patch corresponding to the last convolutional layer of the residual network for the defect key pixel, inputs the local feature map patch into a pre-trained back-projection multilayer perceptron, and outputs the mapped coordinates. When constructing the feature rejection sphere, the cognitive anchor generation module calculates the initial Euclidean distance between the mapped coordinates and the high-dimensional feature vector that triggers the human-machine collaboration request. It then scales the initial Euclidean distance using a preset baseline attenuation factor to obtain the preset attenuation coefficient. Finally, it binds and stores the mapped coordinates, the preset attenuation coefficient, and the known defect category label to which the high-dimensional feature vector originally belonged.

6. The human-machine collaborative quality detection and early warning system for smart factories according to claim 1, characterized in that, When the feature rejection ball is added as a regularization term to the loss function of the residual network, the loss function includes a classification cross-entropy loss term and a rejection ball regularization loss term. The repulsion sphere regularization loss term is configured to apply an exponential penalty gradient based on the distance from the high-dimensional feature vector of the sample to the mapped coordinates when the high-dimensional feature vector of the subsequently input sample falls inside the feature repulsion sphere. When performing the online fine-tuning, the hybrid intelligent inference engine freezes the weight parameters of the remaining network layers in the residual network except for the last two fully connected layers, and only uses the loss function containing the repulsion ball regularization loss term to perform backpropagation updates on the last two fully connected layers.

7. The human-machine collaborative quality detection and early warning system for smart factories according to claim 2, characterized in that, The residual network includes a shallow feature extraction branch, a middle feature extraction branch, and a deep feature extraction branch connected in sequence; The shallow feature extraction branch is used to extract the edge texture features of the workpiece surface image, the middle feature extraction branch is used to extract local morphological features, and the deep feature extraction branch is used to extract global semantic features. After obtaining the outputs of the shallow feature extraction branch, the middle feature extraction branch, and the deep feature extraction branch, the hybrid intelligent inference engine unifies feature maps of different scales into feature tensors of the same size through a spatial pyramid pooling layer. The feature tensors are then concatenated along the channel dimension, and the concatenated feature tensors are used as input to the high-dimensional feature vector processed by the global average pooling.

8. The human-machine collaborative quality detection and early warning system for smart factories according to claim 4, characterized in that, Before overlaying the upsampled heatmap onto the workpiece surface image, the hybrid intelligent inference engine performs an edge-preserving filter operation on the upsampled heatmap. The hybrid intelligent inference engine extracts connected components in the upsampled heatmap whose weights are greater than a preset weight threshold, and calculates the bounding rectangle of the connected components. Inside the outer rectangle, a bilateral filter is used to smooth the upsampled heatmap; outside the outer rectangle, the pixel values ​​of the upsampled heatmap are masked to zero. When the heatmap after the edge-preserving filtering operation is superimposed on the augmented reality terminal, only the area inside the outer rectangle is retained, and it is mapped into a pseudo-color spectrum from red to blue according to the heat value from high to low.

9. The human-machine collaborative quality detection and early warning system for smart factories according to claim 5, characterized in that, The cognitive anchor generation module is equipped with a preset baseline decay factor that dynamically adjusts over time. The cognitive anchor generation module records the creation timestamp of the feature repulsion ball, and obtains the time difference between the current system time and the creation timestamp when calculating the preset attenuation coefficient; The cognitive anchor generation module uses an exponential decay function to decay the preset baseline decay factor to obtain the actual decay factor at the current moment. When the time difference exceeds the preset lifecycle threshold, the cognitive anchor generation module sets the actual decay factor to zero, deletes the mapped coordinates and the corresponding known defect category labels from the storage space, and releases the memory resources occupied by the feature rejection ball.

10. The human-machine collaborative quality detection and early warning system for smart factories according to claim 6, characterized in that, When the hybrid intelligent inference engine performs backpropagation updates on the last two fully connected layers using a loss function that includes the repulsion ball regularization loss term, it simultaneously initiates a sample replay mechanism. The hybrid intelligent inference engine maintains a local circular buffer, which stores images of normal workpiece surfaces that have been historically determined to be non-fuzzy decision intervals and their corresponding defect category labels in the circular buffer according to a first-in-first-out strategy. Before each backpropagation calculation for online fine-tuning, a batch of replay samples is randomly drawn from the circular buffer. The replay samples are then input into the residual network after the weight parameters are frozen. The classification cross-entropy loss of the replay samples is calculated, and the classification cross-entropy loss of the replay samples is weighted and summed with the repulsion ball regularization loss term to obtain the final total loss function.

Images