An automated analysis system for detection data of radio frequency devices

Abstract

This invention relates to the field of radio frequency identification (RFID) data analysis, specifically an automated analysis system for detection data from RFID devices. It includes a data preprocessing module: receiving raw tag reading events and segmenting them into time windows; a state construction module: obtaining latent factors from the time windows based on a temporal convolutional network and a decoupled representation model, combining them into a multidimensional state vector; a deep Q-network module: receiving the multidimensional state vector and selecting actions to receive the multidimensional true vector through a Q-function neural network; an intrinsic curiosity module: obtaining intrinsic curiosity rewards through inverse dynamics and forward dynamics models; and obtaining the total reward through a reward shaping framework; and a learning module: assembling the multidimensional state vector, actions, total reward, and multidimensional true vector into experience tuples, storing them in a learning experience replay pool for further learning. This invention provides deeper diagnostic depth through interpretable state representation based on weak supervision, improving operational efficiency and response speed.

Description

Technical Field

[0001] This invention relates to the field of radio frequency identification (RFID) data analysis, specifically to an automated analysis system for detection data of radio frequency (RF) devices. Background Technology

[0002] In modern industrial automation, particularly in warehousing, logistics, and production line management, the core of achieving intelligence lies in building a cognitive engine capable of understanding and predicting dynamic processes. Multi-sensor networks, represented by technologies such as RFID, can continuously capture high-dimensional, unstructured temporal feature vectors reflecting the state of industrial processes. However, the evolution of industrial processes, such as process optimization or layout changes, manifests as a continuous "conceptual drift" in data distribution. This fundamental characteristic requires the system to transcend traditional programming paradigms and shift towards an adaptive modeling method centered on weakly supervised learning and representation learning, thereby learning the inherent manifold or potential generative laws of normal industrial behavior.

[0003] Current systems, in understanding and adapting to dynamic industrial environments, rely on non-neural, algorithmic data processing logic rather than learning-based connectionist models. These systems cannot learn autonomously; they lack a deep neural network structure to automatically extract abstract features from high-dimensional, highly correlated inputs. All processing logic requires manual design and cannot handle complex patterns. Moreover, their models are static, lacking any form of online model weight updates or continuous learning mechanisms, making them unable to adapt to changes in data distribution caused by "concept drift," leading to rapid model obsolescence.

[0004] In summary, there is currently a lack of an adaptive analysis system that can continuously learn from the dynamic baseline of “normal” data patterns in an environment and autonomously identify unknown anomalous patterns that differ significantly from the baseline in statistical characteristics but were not previously explicitly labeled in the training data.

[0005] To address this, an automated analysis system for detection data of radio frequency devices is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide an automated analysis system for detection data of radio frequency devices to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an automated analysis system for detection data of radio frequency devices, comprising: Data preprocessing module: Receives raw tag reading events from the radio frequency device and divides them into time windows of fixed duration; State construction module: Captures deep temporal features for each time window through a temporal convolutional network; decomposes the deep temporal features through a decoupled representation model to obtain latent factors, which are then combined into a multidimensional state vector; Deep Q-network module: Receives a multi-dimensional state vector, analyzes the estimated values ​​of actions labeled as normal and abnormal through a Q-function neural network, generates random numbers and compares them with the ε parameter, executes a strategy based on the comparison result and the estimated value, selects an action, executes the action, and receives a multi-dimensional true vector; Intrinsic Curiosity Module: Receives multidimensional state vectors and multidimensional reality vectors, obtains intrinsic curiosity rewards through inverse dynamics models and forward dynamics models utilizing attention mechanisms, and obtains total rewards by shaping the framework through rewards; Course learning module: It combines multidimensional state vectors, actions, total rewards, and multidimensional real vectors into experience tuples and stores them in the course experience replay pool; it learns according to the preset course learning outline and updates the deep Q network through the gradient descent algorithm.

[0008] Preferably, the specific process of the data preprocessing module includes: The system receives raw tag reading events from the radio frequency device, which include tag ID, reader ID, timestamp, and received signal strength indication. A periodic timer is used to aggregate all raw tag reading events into a single time window at preset fixed time intervals.

[0009] Preferably, the specific process of the state construction module includes: A temporal convolutional network traverses all original label reading events within a time window. Through multiple layers of causal and dilated convolutions, it automatically learns deep temporal features from the original label reading events that can capture long-distance temporal dependencies. These deep temporal features are then fed into a decoupled representation learning model. This model uses the operational stage labels of warehouse management as weakly supervised signals. By applying regularization constraints and combining them with an auxiliary stage classification task, the deep temporal features are decoupled into multiple latent factors. The numerical vector composed of these latent factors is combined into a multidimensional state vector. These latent factors include a counting factor, an unknown factor, a signal strength factor, and a frequency distribution factor.

[0010] Preferably, the specific generation process of the deep Q-network module includes: The deep Q-network agent inputs a multi-dimensional state vector into its internal Q-function neural network. The Q-function neural network outputs two values, representing the estimated long-term expected rewards of performing actions labeled as normal and abnormal in the current state. A random number is generated and compared with a preset parameter ε. If the random number is less than ε, the deep Q-network agent executes an exploration strategy; otherwise, if the random number is not less than ε, the deep Q-network agent executes a greedy strategy, selecting the action with the higher estimated value. The selected action is executed, and the agent receives the multi-dimensional true vector for the next time step.

[0011] Preferably, the specific process of rewarding intrinsic curiosity includes: The system receives a multidimensional state vector and a multidimensional true vector, which are converted into feature representations by a feature encoder. The inverse dynamics model infers the action to be performed by the deep Q-network agent based on the feature representation, and the error between the inverse and the selected action is used to train the feature encoder. The forward dynamics model uses an attention mechanism to dynamically assign weights to each dimension of the feature representation of the multidimensional state vector based on the feature representation of the multidimensional state vector and the selected action, and generates the feature representation of the predicted future state vector. The intrinsic curiosity reward is obtained by calculating the overall prediction error between the feature representation of the generated future state vector and the feature representation of the multidimensional true vector.

[0012] Preferably, the specific generation process of the reward shaping framework includes: In the reward shaping framework, a diverse meta-training task containing historical normal and abnormal data is used to learn and generate a potential function-based reward shaping function. The reward shaping function receives intrinsic curiosity rewards and sparse external rewards received from external human intervention, and combines a small amount of feedback to dynamically calculate a total reward value to guide the deep Q-network agent.

[0013] Preferably, the specific processing procedure for empirical tuples includes: The system receives a multidimensional state vector, a selected action, a total reward, and a multidimensional truth vector. These four core data items are combined into a structured experience tuple. A deep Q-network is used to calculate the value estimate for the multidimensional state vector and action. A target value based on the total reward and the multidimensional truth vector is then calculated. The temporal difference error between the value estimate and the target value is calculated as an unexpectedness indicator. The learning difficulty of the experience tuple is evaluated using intrinsic curiosity reward and the multidimensional state vector. The experience tuple, along with the unexpectedness indicator and the learning difficulty, is stored in the course experience replay pool.

[0014] Preferably, the specific process of learning according to the pre-set course syllabus includes: When the number of experience tuples accumulated in the replay pool reaches the preset training threshold, the model update is initiated. According to the preset course learning outline, in the early stage of training, the sampler prioritizes the learning of experience tuples with lower learning difficulty. As the performance of the deep Q network agent improves, the course difficulty gradually increases, and the sampler gradually increases the sampling weight of experience tuples with higher learning difficulty. Using this small batch of data sampled by course, the network weights of the deep Q network, the inverse dynamics model, and the forward dynamics model are updated through the gradient descent algorithm.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By using interpretable state representation based on weak supervision, the ability to detect anomalies is improved, enabling it not only to identify events that deviate significantly from the learned normal pattern baseline, but also to provide diagnostic information. Deep temporal features are extracted from raw RFID data using temporal convolutional networks (TCNs), and then decoupled representation learning techniques are introduced to decompose the mixed features extracted by TCNs into multiple independent latent factors with interpretable representational meaning. This provides warehouse operations and maintenance personnel with a deeper diagnostic depth, improving operational efficiency and response speed.

[0016] 2. By employing an attention-weighted, associative curiosity exploration mechanism, efficient and intelligent learning and adaptation are achieved. The Intrinsic Curiosity Module (ICM) utilizes attention to focus on exploring environmental changes highly correlated with the behavior of the deep Q-network agent, learning to distinguish between core state changes and random or irrelevant background disturbances. This allows the deep Q-network agent's learning process to be focused, enabling it to adapt to core changes in the warehousing process more quickly. It avoids ineffective exploration in irrelevant environmental dynamics, thus improving the model's sample efficiency and environmental adaptability when dealing with concept drift problems.

[0017] 3. By designing a curriculum for the learning path of the deep Q-network agent, the learning difficulty of each experience tuple is first assessed, and in the early stages of training, the deep Q-network agent is prioritized to learn simple experiences. As the deep Q-network agent establishes a solid understanding of these basic patterns, the difficulty of the curriculum gradually increases, introducing more challenging experiences step by step. This learning approach, progressing from easy to difficult, effectively avoids interference from localized and complex noisy data in the early stages of model training, thereby significantly improving training stability and final performance, ensuring that the final trained model is comprehensive and robust. Attached Figure Description

[0018] Figure 1 This is a structural diagram of an automated analysis system for detection data of radio frequency devices proposed in an embodiment of this invention application; Figure 2This is a schematic diagram of the decision-making and training process of the deep Q-network agent proposed in an embodiment of this invention. Figure 3 This is a schematic diagram of an application scenario proposed in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1-3 The present invention provides an automated analysis system for detection data of radio frequency devices, the specific modules of which are as follows: Data preprocessing module: Receives raw tag reading events from the radio frequency device and divides them into time windows of fixed duration; State construction module: Captures deep temporal features for each time window through a temporal convolutional network; decomposes the deep temporal features through a decoupled representation model to obtain latent factors, which are then combined into a multidimensional state vector; Deep Q-network module: Receives a multi-dimensional state vector, analyzes the estimated values ​​of actions labeled as normal and abnormal through a Q-function neural network, generates random numbers and compares them with the ε parameter, executes a strategy based on the comparison result and the estimated value, selects an action, executes the action, and receives a multi-dimensional true vector; Intrinsic Curiosity Module: Receives multidimensional state vectors and multidimensional reality vectors, obtains intrinsic curiosity rewards through inverse dynamics models and forward dynamics models utilizing attention mechanisms, and obtains total rewards by shaping the framework through rewards; Course learning module: It combines multidimensional state vectors, actions, total rewards, and multidimensional real vectors into experience tuples and stores them in the course experience replay pool; it learns according to the preset course learning outline and updates the deep Q network through the gradient descent algorithm.

[0021] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0022] Example 1: This application discloses an automated analysis system for detection data of radio frequency devices, see below. Figure 1The specific modules proposed in this invention include: a data preprocessing module: receiving raw tag reading events from radio frequency devices and dividing them into time windows of fixed duration; a state construction module: capturing deep temporal features for each time window through a temporal convolutional network; decomposing them through a decoupled representation model to obtain latent factors, which are then combined into a multidimensional state vector; a deep Q-network module: receiving the multidimensional state vector, analyzing the estimated values ​​of actions labeled as normal and abnormal through a Q-function neural network, generating random numbers and comparing them with the ε parameter, executing a strategy based on the comparison results and estimated values, selecting an action; executing the action, and receiving the multidimensional real vector; an intrinsic curiosity module: receiving the multidimensional state vector and the multidimensional real vector, obtaining an intrinsic curiosity reward through an inverse dynamics model and a forward dynamics model utilizing an attention mechanism; obtaining the total reward through a reward shaping framework; and a course learning module: forming experience tuples from the multidimensional state vector, actions, total reward, and multidimensional real vector, storing them in a course experience replay pool; learning according to a preset course learning outline, and updating the deep Q-network through a gradient descent algorithm.

[0023] Furthermore, the raw tag reading events from the radio frequency device are received and divided into time windows of fixed duration; the specific implementation process includes: In industrial environments, the system described in this invention primarily completes computationally intensive tasks offline or near-online on cloud servers or local data centers with high-performance computing resources. Tasks with high real-time requirements are deployed on edge computing servers located closer to the data source.

[0024] The system interfaces with the Warehouse Management System (WMS) host system via standard interfaces (such as API, message queues, or database polling) to receive data streams uploaded by all RFID readers in real time. Each raw event record is a tuple containing {TagID, ReaderID, Timestamp, RSSI}.

[0025] Due to environmental interference and the physical characteristics of tags, raw RFID data streams often contain a large number of redundant and false read events. This module uses time smoothing window technology to normalize and synchronize the data stream. To determine the optimal time window duration, it needs to be systematically set rather than simply relying on experience. Through video analysis or stopwatch timing, N independent business events (e.g., N pallets passing through) in a target scenario (e.g., a warehouse entrance) are sampled, and the duration of each event is recorded. Then, the arithmetic mean (i.e., average duration) and standard deviation of these durations are calculated. The minimum interval between two consecutive independent business events is recorded. The time window duration can be determined using an empirical formula, which is the product of the average duration, the confidence coefficient k, and the standard deviation. Here, k is the confidence coefficient (e.g., k=3 to cover 99.7% event integrity), and the final calculated time window duration must be less than the recorded minimum business event interval. For example, by sampling 100 forklift pallet passages through the doorway, the average passage time was measured to be 450ms, and the standard deviation was 50ms. To ensure 99.7% of passing events are captured completely, k=3 is chosen. The calculated time window length is 450ms plus the product of 3 and 50ms, which is 600ms. Field investigation shows that the minimum interval between two consecutive forklifts passing is 1500ms. 600ms is much shorter than this interval, therefore setting the time window length to 600ms is reasonable. To ensure event continuity, a 50% overlap rate can be set between adjacent time windows.

[0026] The data aggregation operation is triggered by a periodic timer. This periodic timer is a software module implemented based on the kernel timer of the server operating system. It uses setInterval or a similar programming language function to trigger the data aggregation operation with a period of the determined time window duration (600ms in this example). To cope with timer drift that may be caused by system load, the module also includes time calibration logic that synchronizes with an NTP (Network Time Protocol) server to ensure the long-term stability of the timing period.

[0027] All raw tag read events within the window are grouped according to their {TagID, ReaderID} combination. Events within each group are aggregated. Aggregated events retain unique TagID and ReaderID. Their RSSI value is set to the arithmetic mean of the RSSI values ​​of all events within the group. Their timestamp is set to the weighted average of the timestamps of all events within the group (weighted by RSSI) or a direct average, rather than the window start time, to retain more accurate timing information. If no raw tag read events occur within a time window, the system generates one or more records representing 'no signal'. For example, for each reader known to be active, a record containing {TagID: null, ReaderID: specific ID, Timestamp: window center time, RSSI: -100dBm} is generated.

[0028] After filtering and aggregation, the system sorts all related events according to timestamps to construct a time series describing the complete movement trajectory of the item within the warehouse.

[0029] By setting periodic timers and aggregating raw tag reading events into fixed time windows, data stream normalization and synchronization are achieved. Batch processing of data avoids performing complex calculations and analyses on each raw reading event, significantly improving computational efficiency and reducing the system's processing load.

[0030] Furthermore, deep temporal features are captured through a temporal convolutional network for each time window; the latent factors are decomposed by decoupling the representation model and combined into a multidimensional state vector; corresponding state construction modules are also included; see [link / reference]. Figure 2 The specific process includes: The input time window dataset is preprocessed to transform it into a normalized tensor. Key parameters in this process include: discretizing the 200-millisecond time window into L=40 equal-length time slices; calculating C=2-dimensional features within each time slice, namely the read count and the average RSSI. If there are no read events in a time slice, the read count is recorded as 0, and the average RSSI is assigned a default value of -100dBm to represent no signal. After this process, the data from one time window is transformed into a two-dimensional tensor of shape (2, 40).

[0031] The Temporal Convolutional Network (TCN) traverses all original label reading events containing two-dimensional tensors within a time window. Through multiple layers of causal and dilated convolutions, it automatically learns deep temporal features from these events that capture long-range temporal dependencies. The network's specific structure and parameter configuration are as follows: the network consists of four stacked residual blocks, where residual connections prevent the vanishing gradient problem in deep networks; the kernel size is set to 3; the dilation factor sequence for the dilated convolutions is set to 1, 2, 4, and 8; all convolutional layers have 64 channels; the activation function is the Modified Linear Unit (ReLU); and Dropout is applied after each residual block with a dropout rate of 0.2. This network strictly adheres to causality, ensuring that the output at time t depends only on the input at time t and earlier. The final output of the TCN is a 128-dimensional vector, representing the deep temporal features.

[0032] The deep temporal features are fed into a decoupled representation learning model, which uses operational stage labels of warehouse management as weak supervision signals and is jointly optimized through a well-defined loss function. This loss function mainly consists of a weighted sum of three parts: reconstruction loss, auxiliary classification task loss, and a total relevance regularization term designed to decouple the factors, thus decoupling the deep temporal features into multiple low-relevance latent factors. To ensure the balanced contribution of each loss term in gradient updates and to reflect their respective importance, each weight coefficient is set according to its impact on the final model performance and the numerical scale of each loss term. In this embodiment, the weights are configured as follows: the weight coefficient of the reconstruction loss is set to a baseline value of 1.0, ensuring that the decoded latent factors can restore the input features as closely as possible; the weight coefficient of the auxiliary classification task loss is set to 0.5, aiming to guide the model to learn physically meaningful representations, with a weight slightly lower than the reconstruction loss to avoid the model focusing excessively on the classification task and sacrificing the detailed characterization of the data itself; and the weight coefficient of the total relevance regularization term is set to 10.0. Since the total correlation loss is usually much smaller than the reconstruction loss, setting a higher weight value is to numerically amplify its contribution to the total loss.

[0033] The operational stage labels for warehouse management are discrete events extracted from the WMS system logs, such as inbound scanning, outbound picking, and inventory transfer. The system uses timestamp matching to associate each time window with a unique operational stage label. If the timestamp range of a time window overlaps with multiple operational stage labels, the label with the longest duration within that window is prioritized for matching; if the durations are the same, the label with the earlier start time is selected. If no valid WMS operational stage label is matched within a time window and the time range of the window before and after it, it is assigned the 'idle' label. The 'auxiliary stage classification task' involves adding a fully connected layer and a Softmax activation function to a branch of the decoupled representation learning model to predict the operational stage corresponding to the current time window. Its loss is calculated using the standard cross-entropy loss function.

[0034] The total relevance regularization term aims to penalize the statistical dependencies between latent factors, prompting the model to learn latent representations with low relevance. In this embodiment, the value of the total relevance can be approximately estimated and optimized by introducing a discriminator network for adversarial training. This discriminator distinguishes between real joint distribution samples and samples synthesized from marginal distributions, and is trained using a standard binary cross-entropy loss function. Its goal is to accurately identify real joint distribution samples as "true" (output probability of 1) and synthetic samples as "false" (output probability of 0). The total relevance regularization term of the decoupled representation learning model, on the other hand, has the opposite training objective of the discriminator. Its goal is to adjust the distribution of latent factors to "deceive" the discriminator, striving to output a probability close to 1 for synthetic samples even if the discriminator cannot distinguish between synthetic and real samples. Through this mini-maximum game between the generator (decoupled representation model) and the discriminator, the total relevance between latent factors can be effectively reduced. The discriminator network is a multilayer perceptron. Its input layer receives the concatenated latent factors, followed by two hidden layers, each containing 128 neurons and using the ReLU activation function. The output layer consists of a single neuron using the sigmoid activation function to output a probability value between 0 and 1. The four decoder heads have identical structures. Each decoder head is a fully connected network containing a hidden layer with 64 neurons and a ReLU activation function, and a linear output layer that outputs a 16-dimensional latent factor.

[0035] The latent factors include counting factors, unknown factors, signal strength factors, and frequency distribution factors. The model comprises four parallel decoding heads, each an independent fully connected network, responsible for decoding a specific 16-dimensional latent factor from shared deep temporal features and predicting the corresponding factor. To decouple the factors, the model incorporates a regularization constraint based on minimizing total correlation during training, while utilizing supervision signals from the auxiliary classification task to ensure the decomposed factors have practical representational meaning. The reconstruction loss is the sum of errors between the predicted values ​​of these four prediction tasks and the baseline value (e.g., calculated using mean squared error, MSE).

[0036] The baseline values ​​for the four latent factors are calculated as follows: (1) Counting factor: Its significance lies in quantifying the scale of RFID events within a time window. Specifically, it consists of two statistics: a) the number of unique and independent tag IDs (TagIDs) detected within the time window; b) the total number of times all tags are read within the time window.

[0037] (2) Signal strength factor: Its significance lies in describing the overall quality and stability of the signal. It consists of three statistics: a) the arithmetic mean of the Received Signal Strength Indication (RSSI) of all events within the window; b) the standard deviation of all RSSI values, used to measure signal volatility; c) the difference between the maximum and minimum values ​​of all RSSI values, used to measure the dynamic range of the signal.

[0038] (3) Frequency distribution factor: Its significance lies in describing the distribution pattern of read events over time. The specific calculation method is as follows: First, a fixed time window (e.g., 200 milliseconds) is divided into 10 equal-length sub-intervals (each 20 milliseconds). Then, the number of read events falling within each sub-interval is counted to form a 10-dimensional distribution histogram. This factor is the probability distribution after the histogram is normalized.

[0039] (4) Unknown Factor: This quantifies the predictability of the current event pattern, i.e., whether the pattern is common. Specifically, it is calculated using a lightweight multilayer perceptron (MLP) with 32 neurons and a ReLU activation function in the hidden layer. This model predicts the sequence for the next 10 time slices based on the read counts and average RSSI sequence of the first 30 time slices within a window consisting of 40 time slices. This factor is composed of the mean squared error (MSE) between the predicted and true sequences; the larger the error, the more "unknown" or "abnormal" the pattern.

[0040] The numerical vectors composed of the latent factors are combined into a multidimensional state vector. Specifically, the four 16-dimensional latent factor numerical vectors decoded by the model are concatenated and combined in a predetermined order to finally generate a 64-dimensional numerical vector, which is the final quantitative description of the current time window state—the multidimensional state vector.

[0041] By employing a Temporal Convolutional Network (TCN), deep temporal features can be automatically learned and extracted from event sequences read from the original labels, generating a feature representation that better reflects the true physical dynamics of the warehouse entity. By introducing a decoupled representation learning model and using the warehouse operation stage as a weakly supervised signal, the final generated multidimensional state vector is not only rich in information dimensions but also has a clear structure and strong interpretability.

[0042] Furthermore, a multidimensional state vector is received, and the value of actions labeled as normal and abnormal is analyzed using a Q-function neural network. Random numbers are generated and compared with the ε parameter. Based on the comparison result, a strategy is executed to select an action; the action is then executed, and a multidimensional true vector is received; this corresponds to a deep Q-network module; see [link / reference]. Figure 2 The specific implementation process includes: The deep Q-network agent inputs a multi-dimensional state vector S_t into an internal Q-function neural network. The Q-function neural network outputs two values, representing the estimated long-term expected rewards for performing the actions "marked as normal" and "marked as abnormal" in the current state. The specific structural parameters of this Q-function neural network are configured as follows: the input layer contains 64 neurons, matching the dimension of S_t; it is followed by two fully connected hidden layers, each containing 128 neurons, using a modified linear unit (ReLU) as the activation function to introduce non-linear expressive power; the output layer contains 2 neurons, using linear activation. The use of a linear activation function ensures that the output Q-value can take any real value, not limited to a specific range. When the state vector S_t is input into the network, after forward propagation, the two neurons in the output layer produce Q-value estimates corresponding to the discrete actions "marked as normal" and "marked as abnormal," denoted as Q(S_t, A_normal) and Q(S_t, A_abnormal), respectively.

[0043] The output layer of a Q-function neural network can be expanded to four neurons, corresponding to the actions at the following four levels: Action level 0: Ignore: This is considered a deterministic normal event and no action is taken.

[0044] Action Level 1: Record to be investigated: "Attention" events that are judged to be slightly deviating from the baseline will not trigger real-time alarms, but will be stored in the log to be analyzed.

[0045] Action Level 2: Request for Verification: If the event is determined to be a moderately uncertain suspicious event, cross-validation will be performed automatically.

[0046] Action Level 3: Immediate Alarm: If the event is determined to be a serious anomaly with high confidence, the highest level alarm will be triggered immediately, and relevant personnel will be notified.

[0047] Through learning, deep Q-networks output the Q-values ​​of the four actions for the current state S_t. The deep Q-network agent still uses an ε-greedy strategy to select the action with the highest Q-value for execution. This hierarchical response mechanism enables the system to take the most cost-effective action based on the actual risk level of the event, avoiding a simple "one-size-fits-all" alarm mode and improving the efficiency and intelligence of the entire operation and maintenance system.

[0048] The system generates a random number and compares it with a preset parameter ε. If the random number is less than ε, the deep Q-network agent will execute an exploration strategy; otherwise, if the random number is not less than ε, the deep Q-network agent will execute a greedy strategy, selecting the action with the higher estimate given by the Q-function network. The action selection here adopts an ε-greedy strategy. The ε parameter dynamically decays from high to low values ​​during model training, but in the practical application phase described in this embodiment, this parameter is set to a fixed value of 0.05. The specific decision logic is as follows: the system generates a random number in the interval [0, 1). If the number is less than 0.05, an exploration strategy is executed, randomly selecting an action with equal probability; if it is not less than 0.05, a greedy strategy is executed, that is, explicitly comparing the magnitudes of Q(S_t, A_normal) and Q(S_t, A_abnormal), and selecting the action corresponding to the higher value as the final decision.

[0049] The system executes the action selected in the previous step, such as archiving the event or triggering an alarm. The execution of this action marks the end of the current decision-making cycle. In the subsequent time window, the warehouse environment continues to evolve, and the deployed RF equipment collects a new batch of raw tag reading events. After these events undergo the complete processing flow of the aforementioned state construction module (including TCN and decoupled representation model), a new 64-dimensional vector is generated. This vector is the multi-dimensional true vector S_{t+1} for the next time step and serves as the input for the deep Q-network agent in the next decision-making cycle, forming a closed-loop operation.

[0050] By employing a Deep Q-Network (DQN) agent for decision-making, a leap has been achieved from traditional anomaly detection based on static rules or thresholds to adaptive intelligent decision-making based on long-term expected returns. This means that the Deep Q-Network agent can understand the subtle differences between different states and their long-term impact. Furthermore, since the Q-function is approximated by a deep neural network, the model can learn complex nonlinear relationships from high-dimensional state vectors, giving it strong generalization capabilities and enabling it to adapt to changing warehousing environments and operational processes without requiring frequent manual rule adjustments.

[0051] Furthermore, receiving multidimensional state vectors and multidimensional reality vectors, the intrinsic curiosity reward is obtained through an inverse dynamics model and a forward dynamics model utilizing an attention mechanism; the total reward is obtained through a reward-shaping framework; this corresponds to the intrinsic curiosity module; see [link / reference]. Figure 2 The specific implementation process includes: The system receives a multidimensional state vector and a multidimensional true vector, which are converted into feature representations by a feature encoder. The inverse dynamics model infers the action to be performed by the deep Q-network agent based on the feature representations, and the error between the inverse dynamics model and the selected action is used to train the feature encoder. The feature encoder here is a multilayer perceptron (MLP) with a single hidden layer. Its input layer has 64 neurons, the hidden layer has 32 neurons (ReLU activation), and the output layer has 32 neurons (linear activation), converting S_t and S_{t+1} into feature representations φ(S_t) and φ(S_{t+1}), respectively. The inverse dynamics model is also an MLP, with a 64-dimensional input (composed of concatenated φ(S_t) and φ(S_{t+1})) and an output layer that predicts the action using a softmax function. The cross-entropy loss between the predicted action and the actual action A_t is used to synchronously update the weights of the inverse dynamics model and the feature encoder.

[0052] The forward dynamics model finds the corresponding 32-dimensional vector representation for each possible discrete action through an action embedding layer. Then, using an attention mechanism, it dynamically assigns weights to each dimension of the multi-dimensional state vector's feature representation based on the selected action, generating a predicted future state vector's feature representation. An intrinsic curiosity reward is generated by calculating the overall prediction error between the generated future state vector's feature representation and the multi-dimensional true vector's feature representation. The 32-dimensional action vector is concatenated with the 32-dimensional state feature representation φ(S_t) to form a 64-dimensional fusion vector. This fusion vector generates a query vector through a linear transformation layer, while the state feature vector φ(S_t) itself generates a key vector and a value vector through two other independent linear transformation layers. The attention mechanism here is a 4-head multi-head attention mechanism, with each head having a dimension of 8. The calculated context vector is residually concatenated with the original state feature vector φ(S_t), and then normalized by layers. Finally, a multilayer perceptron (MLP) outputs the predicted future state feature representation φ'(S_{t+1}). The intrinsic curiosity reward r_i is determined by calculating the mean square error (MSE) between φ'(S_{t+1}) and φ(S_{t+1}), the magnitude of which reflects the uncertainty of the deep Q network agent's prediction of the dynamic environment.

[0053] Before generating intrinsic curiosity rewards, a correlational perturbation injection mechanism can be introduced. A multivariate Gaussian distribution N(μ, Σ) is defined as a random noise model. A sequence of state vectors labeled 'normal' within a continuous time window is selected from a historical database. The difference vector ΔS = S_{t+1} - S_t between the state vectors is calculated, and this difference vector is used as a noise sample. The mean μ and covariance matrix Σ of the random noise model are obtained by calculating the sample mean and sample covariance of all noise samples. When injecting a perturbation, a noise vector ε is sampled from the random noise model, multiplied by a fixed perturbation coefficient, and added to the current state vector S_t to generate a simulated state vector S'_t. The forward dynamics model predicts not only the future state of S_t but also the future state of S'_t. The difference between these two predictions is calculated and compared. If a small, common perturbation leads to a large change in the predicted future state, it proves that the current state is at a "critical" or "unstable" decision boundary. The perturbation coefficient is set to 0.05 in this embodiment, ensuring that the change in the signal strength-related dimension of the normalized state vector S_t after the perturbation is superimposed, when mapped back to the physical world, will not significantly exceed a reasonable range of ±3dBm. The system quantifies this prediction "sensitivity" and multiplies it by the original prediction error as an enhancement factor, reflecting its tendency to explore "system vulnerabilities." This mechanism incentivizes the deep Q-network agent to actively test and learn how to make correct decisions under boundary conditions where minor perturbations are most likely to lead to significant errors, thus improving the model's robustness.

[0054] In the reward shaping framework, a meta-training task containing historical normal and abnormal data is used to learn and generate a reward shaping function based on a potential function. The learning process employs a model-independent meta-learning algorithm to learn a general model initialization parameter, enabling it to quickly adapt to new tasks with only a small amount of new data. The construction process of the meta-training task is as follows: First, from the historical data of various typical fault categories, including label missed reading, label misreading, and device offline, a specific abnormal event type (such as label missed reading) is randomly selected. Then, N (e.g., N=10) state-reward data pairs (s, r) are sampled from the logs related to this type as the support set for the task. Next, M (e.g., M=20) data pairs are sampled from the data related to the task but different as the query set. An inner loop update is performed, which replicates the current potential function model and uses the data on the support set to update the parameters of the replicated model through a gradient descent step, making it initially adaptable to the current task. An outer loop update is then performed, using the model updated by the inner loop to calculate the loss on the query set, and the original potential function model parameters are updated based on this loss. The potential function Φ(S) represented by the multilayer perceptron includes an input layer that matches the 64-dimensional state vector dimension, two hidden layers each containing 64 neurons and a ReLU activation function, and a linear output layer that outputs a single scalar value.

[0055] The reward shaping function receives intrinsic curiosity rewards and sparse external rewards received manually, and dynamically calculates the total reward value used to guide the deep Q-network agent, incorporating a small amount of feedback. The framework trains a potential function Φ(S) represented by an MLP through meta-learning. The value of the reward shaping function F is equal to the discount factor γ (set to 0.99) multiplied by the potential function value Φ(S_{t+1}) of the future state, and then subtracted from the potential function value Φ(S_t) of the current state.

[0056] The external rewards are generated in conjunction with the verification module of the Warehouse Management System (WMS). For example, when the system marks an event as an immediate alarm, and subsequent inventory checks or manual verification confirm that the alarm is a genuine anomaly (such as a damaged label or incorrect goods), the system records a reward of +10 on that experience tuple. Conversely, if it is confirmed as a false alarm, a penalty of -10 is recorded. If the system marks an event as 'normal,' but the event is subsequently proven to be a missed event that caused an inventory anomaly, a penalty of -20 is imposed. These reward signals are asynchronously fed back to the corresponding high-value feedback buffer in the course experience replay pool. The system sets an update threshold; for example, whenever 10 new experience tuples accumulate in the high-value feedback buffer, a fine-tuning of the potential function network is triggered. During the fine-tuning process, the system randomly samples a small batch of data (e.g., 8) from the buffer and performs a gradient descent update on the weights of the potential function network using a small learning rate (e.g., 1e-5).

[0057] The final total reward value R_t is calculated using a summation formula, which is the sum of the sparse external reward r_e, the product of the hyperparameter β (set to 0.1) and the intrinsic curiosity reward r_i, and the aforementioned reward shaping term F. This total reward will serve as the final training signal.

[0058] By generating intrinsic curiosity rewards through a self-supervised dynamic prediction model, a continuous and dense internal incentive signal can be provided to the deep Q-network agent, effectively avoiding the problem of the deep Q-network agent stopping learning due to getting trapped in local optima, and enhancing its autonomous learning ability in complex environments. Through a meta-learning-based reward shaping framework, the system can learn to generate a universal, potential-based reward shaping function from diverse historical data. By combining these two approaches, an information-rich and robust total reward value is ultimately generated, accelerating the convergence speed of the deep Q-network agent and improving sample utilization efficiency.

[0059] Furthermore, the multidimensional state vector, action, total reward, and multidimensional true vector are combined to form an experience tuple, which is stored in the course experience replay pool; learning is performed according to the preset course learning outline, and the deep Q-network is updated using the gradient descent algorithm; this corresponds to the course learning module; the specific implementation process includes: The system receives a multidimensional state vector, a selected action, a total reward, and a multidimensional true vector. These four core data items are combined into a structured empirical tuple. A deep Q-network is used to calculate the value estimate for the multidimensional state vector and action, and then a target value is calculated based on the total reward and the multidimensional true vector. The temporal difference error between the value estimate and the target value is calculated as a surprise metric. First, the input data is combined into an empirical tuple (S_t, A_t, R_t, S_{t+1}). Then, the surprise metric is determined by calculating the temporal difference error (TD-Error). This calculation involves the value estimate Q(S_t, A_t) obtained from the deep Q-network (main network) and a target value Y_t. The target value Y_t is calculated using the Bellman equation, and its value is the sum of R_t multiplied by a discount factor γ (set to 0.99) multiplied by the maximum value of the next state output by the target network. The surprise metric is the absolute value of the difference between the value estimate and the target value; its magnitude directly reflects the degree of "surprise" of the empirical value to the current model.

[0060] The system evaluates the learning difficulty of the experience tuples using intrinsic curiosity rewards and multidimensional state vectors. The learning difficulty D_t is calculated using a weighted formula, where the value is the product of the weight coefficient α (set to 0.6) and the intrinsic curiosity reward r_i, plus (1-α) and the state novelty function N(S_t). The state novelty N(S_t) is inversely proportional to the frequency of the multidimensional state vector S_t in the replay pool. When a new state vector S_t enters, the K-nearest neighbor algorithm (K=5) finds the five multidimensional state vectors with the closest Euclidean distance in the set of all multidimensional state vectors stored in the current course's experience replay pool. The reciprocal of the average frequency of these five vectors is used as the initial value of the state novelty, quantifying the rarity of a state.

[0061] The experience tuples, along with their unexpectedness metrics and learning difficulty, are stored in a course experience replay pool. When the number of experience tuples accumulated in the replay pool reaches a preset training threshold, a model update is initiated. The experience tuples, their associated unexpectedness metrics, and learning difficulty are stored together in a course experience replay pool with a capacity of 50,000. Before starting the course experience replay pool, hundreds (e.g., 500) of validated, clearly labeled typical events are extracted from the historical database as initial experiences, including confirmed equipment failures and standard operating procedures. High-quality experience tuples are constructed from these samples, where action A_t is the true label, and the total reward R_t is a clear, high-intensity external reward value (e.g., -20 represents confirmed failure). These experience tuples are preferentially populated into the experience replay pool to provide an initial training set for the deep Q-network agent and feature encoder.

[0062] For a fixed-capacity experience replay pool, a value-stratified long and short-term memory replay pool can be introduced. Each experience tuple stored in the pool is assigned a long-term value score, which is a weighted sum of the experience tuple's unexpectedness index, state novelty, and the absolute value of its total reward. The experience pool is logically divided into two regions: the short-term memory region occupies the majority of the pool's capacity and uses a traditional first-in, first-out (FIFO) elimination strategy to store and learn routine experiences; the long-term memory region occupies the remaining capacity and is specifically used to archive experiences with the highest long-term value scores, employing a last-place elimination system. A new experience tuple will only replace the lowest-scoring experience tuple when its long-term value score is higher than the lowest-scoring experience tuple in that region. The unexpectedness index directly reflects the cognitive impact of the current event on the model and is crucial for correcting cognitive biases and learning unknown patterns; therefore, it is given the highest weight (0.5). State novelty measures the rarity of an event, helping the model explore and cover a more comprehensive state space and cope with low-probability events; its importance is secondary, therefore, it is given a moderate weight (0.3). While the total reward represents experience with clear supervisory signals, it is typically sparse in industrial scenarios. A weight of (0.2) is assigned. In this embodiment, the short-term memory region occupies 90% of the total pool capacity, and the long-term memory region occupies the remaining 10%. Those skilled in the art can adjust the proportion of the long-term memory region within the range of [5%, 20%] based on the sparsity of high-risk events in actual application scenarios. Periodically sampling a small amount of experience from the long-term memory region for model consolidation training ensures that "golden experiences" related to major failures and critical process changes are prioritized for long-term retention and repeated learning, improving the model's robustness against low-probability, high-risk events. Once the amount of data in the pool reaches the training threshold of 2000, the model update process is initiated, ensuring that sufficiently diverse initial data is available at the start of training.

[0063] According to the preset course syllabus, in the early stages of training, the sampler prioritizes learning from experience tuples with lower learning difficulty. As the performance of the deep Q-network agent improves, the course difficulty gradually increases, and the sampler gradually increases the sampling weight of experience tuples with higher learning difficulty. The sampling mechanism is based on the priority experience replay algorithm. The sampling priority p_i of each experience tuple i is calculated according to the index defined in the current training stage (initially the reciprocal of the learning difficulty D_i, and later the absolute value of the temporal difference error TD-Error_i). To correct the bias caused by priority sampling, an importance sampling weight w_i needs to be calculated for each sampled experience tuple when updating the model parameters. The formula for calculating this weight is: w_i = (N*P(i))^(-β), where N is the total capacity of the experience pool, P(i) is the sampling probability, and β is the bias correction exponent, whose value gradually anneals from an initial value (e.g., 0.4) to 1.0 with the number of training steps. When calculating the gradient of the loss function, the loss value of each sample needs to be multiplied by its corresponding weight w_i to ensure that the model update is unbiased.

[0064] The training process follows a three-stage curriculum outline: In the initial stage (steps 1-10000), experiences with low learning difficulty are prioritized; in the middle stage (steps 10001-50000), uniform sampling is performed; and in the later stage (after step 50001), experiences with high unexpectedity metrics are prioritized. Upon entering the middle stage (i.e., the 10001st training step), the sampler resets the sampling priority p_i of all experience tuples in the curriculum experience replay pool to the same constant value, e.g., 1.0. During this stage, the priority of any newly added experience tuple is also directly set to 1.0. Since all experiences have equal priority, the priority-based sampling probability P(i) will also be the same for all experiences, thus achieving de facto uniform sampling. Simultaneously, to maintain consistency with the sampling strategy, the importance sampling weight w_i used to correct bias is also fixed to 1.0 for all sampled experience tuples during this stage. At each training step, the sampler draws a mini-batch of 64 data points according to the current policy and updates the weights of the deep Q-network agent, inverse dynamics model, and forward dynamics model using gradient descent with the Adam optimizer (learning rate set to 1e-4). The weights of the target network are updated by the weights of the main network every 100 training steps.

[0065] By calculating two key indicators—unexpectedness and learning difficulty—for each experience tuple, the originally homogeneous experience data was quantified and stratified based on value and difficulty. This provides crucial decision-making support for subsequent curriculum-based training, eliminating the randomness and blindness of traditional experience replay. Based on a pre-set three-stage learning syllabus from easy to difficult, this not only significantly improves learning efficiency and reduces the training time required for the model to reach optimal performance, but also makes the final trained model more robust and reliable.

[0066] This invention elevates radio frequency data analysis from traditional passive rule matching to proactive intelligent analysis through a complete closed-loop learning framework. The system analyzes the value of actions marked as normal and abnormal using a "deep Q-network module," making decisions with long-term reward expectations. This allows for dynamic learning of the environmental baseline, effectively addressing the "concept drift" problem. Furthermore, an "intrinsic curiosity module" generates a "total reward," endowing the system with self-driven learning capabilities. In industrial scenarios with sparse external rewards, it can proactively explore and quickly adapt to new processes, achieving efficient adaptive learning. Finally, a "course learning module" trains the model according to a "preset course syllabus." Through a structured learning path from easy to difficult, it effectively avoids interference from complex noise samples in the early stages of training, improving the instability of deep reinforcement learning training and ensuring the stability and robustness of the final model in industrial deployment.

[0067] Example 2: This second embodiment demonstrates a specific planning process for applying the present invention to warehouse management organization A; see reference Figure 3 A represents the tray labeled TAG-ABC, and B represents the reader RDR-01; the specific implementation method is as follows: When a tray labeled TAG-ABC quickly passes the RDR-01 reader at the doorway, the edge computing server continuously receives a series of structured raw tag reading events generated by the reader. These event streams are logically continuous; for example, the server might receive records such as {Label ID: TAG-ABC, Reader ID: RDR-01, Timestamp: 1762377601123, RSSI: -65}, {Label ID: TAG-ABC, Reader ID: RDR-01, Timestamp: 1762377601155, RSSI: -55}, and {Label ID: TAG-ABC, Reader ID: RDR-01, Timestamp: 1762377601189, RSSI: -68}, clearly reflecting the process of the tray moving from far to near and then back to far. The data processing module uses a periodic timer to establish a fixed time window, such as an interval extending from time 1762377601000 to 1762377601200. The system determines that the timestamps of all the events mentioned above fall within this interval, and therefore stores them all in the data buffer of the current time window. When the time window ends, all relevant read events in the buffer are encapsulated into a set and output as a single, complete time window dataset.

[0068] The Temporal Convolutional Network (TCN) receives a dataset within a time window, containing a series of raw label reading events generated when a tray labeled TAG-ABC rapidly passes through a reader RDR-01. The signal strength (RSSI) of these events exhibits a parabolic pattern of first increasing and then decreasing, with reading events concentrated in the middle of the time window. The TCN analyzes this pattern through its multi-layer convolutional structure, ultimately outputting a high-dimensional numerical vector, which is a deep temporal feature that condenses and abstracts the entire "rapid passage" event. This deep temporal feature is then fed into a decoupled representation learning model, which decomposes it into several latent factors with clear representational meanings: the count factor reflects that only one unique label (TAG-ABC) was identified and the number of readings was small; the unknown factor has a low value due to the clear and predictable pattern; the signal strength factor quantifies the "increase-then-decrease" trend of the signal; and the frequency distribution factor indicates the temporal concentration of reading events. These decoded latent factor vectors are concatenated in a predetermined order to form the final multidimensional state vector.

[0069] A multidimensional state vector representing the "pallet passes quickly" event is input into the Q-function neural network within the deep Q-network agent. Because the inherent pattern of this state vector (e.g., signal strength initially increases then decreases, and the reading events are concentrated in a short period) matches a large number of "normal passage" cases in historical data, the deep Q-network agent has learned that this is a routine operation. Therefore, its Q-function neural network outputs two estimates representing the expected long-term reward, where the estimate for the "mark as normal" action is significantly higher than that for the "mark as abnormal" action, for example, an output of [18.5, -12.0]. Subsequently, the deep Q-network agent selects the action with the higher estimate according to its predetermined strategy. In this scenario, it selects the "mark as normal" action. After the action is executed, the system records this "pallet passes quickly" event as a compliant storage operation, without triggering any alarms. After this decision cycle ends, the environment enters the next time step. Since the pallet has left, the reader no longer receives its signal, and the subsequent state construction module generates a new multidimensional true vector representing the "idle" state. The deep Q-network agent then prepares to make the next decision based on this new state.

[0070] The system receives the "pallet passes quickly" state vector before the decision, the "mark as normal" action ultimately selected by the deep Q-network agent, and the "idle" state vector after the decision, representing a return to calm. The internal forward dynamics model predicts what should happen next based on the "pallet passes quickly" state and the "mark as normal" action. Since this "passes through and is then idle" transition is extremely common in warehousing operations, the model's prediction is very accurate; it anticipates that the next state will be characterized by no signal and no reads. Therefore, when it compares this accurate prediction with the actual "idle" state vector, the calculated prediction error is very small. This tiny error value is used as an intrinsic curiosity reward; its low value means that this event is "expected" for the deep Q-network agent and lacks novelty. Meanwhile, because this is a routine operation, the system does not receive any external artificial rewards or penalties. Finally, the reward shaping framework combines this low intrinsic curiosity reward with a zero external reward to calculate the total reward value used to guide the deep Q-network agent. This near-zero or tiny positive total reward serves as a feedback signal, reinforcing the deep Q-network agent's correct judgment on this routine operation.

[0071] First, the system receives a state vector representing "pallet passing quickly," the "marked as normal" action selected by the deep Q-network agent, a small positive total reward value, and a subsequent "idle" state vector. These four data points are combined into a structured experience tuple. The system then evaluates this experience tuple for two key metrics: because this is a very common event accurately predicted by the deep Q-network agent, the difference between its value estimate and the target value is minimal, resulting in a low unexpectedness metric; simultaneously, combined with its low intrinsic curiosity reward and the universality of the state, its learning difficulty metric is also rated as low. This experience tuple, along with its low unexpectedness and low difficulty metrics, is stored in the course experience replay pool. In the early stages of model training, the course sampler prioritizes extracting experiences with lower learning difficulty from the replay pool according to a pre-defined learning syllabus. Therefore, the "pallet fast pass" experience tuple, which represents the basic and routine operation, will be frequently selected. Using small batches of data including this tuple, the system updates the weights of the deep Q network and related models through the gradient descent algorithm. This process continuously strengthens the model's recognition of this routine operation.

[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automated analysis system for detection data of radio frequency equipment, characterized in that, include: Data preprocessing module: Receives raw tag reading events from the radio frequency device and divides them into time windows of fixed duration; State building module: Captures deep temporal features for each time window using a temporal convolutional network; By decoupling the representation model, the deep temporal features are decomposed to obtain latent factors, which are then combined into a multidimensional state vector. The specific process includes: a temporal convolutional network traverses all original label reading events within a time window, and automatically learns deep temporal features from the original label reading events through multi-layer causal and dilated convolutions to capture long-distance temporal dependencies; these deep temporal features are fed into a decoupled representation learning model, which uses the operational stage labels of warehouse management as weak supervision signals, and decouples the deep temporal features into multiple latent factors by applying regularization constraints and combining them with an auxiliary stage classification task; the numerical vector composed of these latent factors is combined into a multi-dimensional state vector; these latent factors include a counting factor, an unknown factor, a signal strength factor, and a frequency distribution factor. Deep Q-network module: Receives a multi-dimensional state vector, analyzes the estimated values ​​of actions labeled as normal and abnormal through a Q-function neural network, generates random numbers and compares them with the ε parameter, executes a strategy based on the comparison result and the estimated value, selects an action, executes the action, and receives a multi-dimensional true vector; The intrinsic curiosity module receives a multidimensional state vector and a multidimensional true vector. It obtains an intrinsic curiosity reward through an inverse dynamics model and a forward dynamics model utilizing an attention mechanism. The total reward is then derived by shaping the framework using this reward. The specific process includes: receiving the multidimensional state vector and the multidimensional true vector; converting them into feature representations using a feature encoder; the inverse dynamics model inferring the action to be performed by the deep Q-network agent based on the feature representation, and using the error between the inverse and selected actions to train the feature encoder; the forward dynamics model dynamically assigns weights to each dimension of the feature representation of the multidimensional state vector based on the feature representation and the selected action, and generates a predicted feature representation of the future state vector; the intrinsic curiosity reward is obtained by calculating the overall prediction error between the generated feature representation of the future state vector and the feature representation of the multidimensional true vector. The course learning module consists of experience tuples composed of multidimensional state vectors, actions, total rewards, and multidimensional true vectors. The temporal difference error between the value estimate and the target value is used as an unexpectedness indicator, and the learning difficulty is evaluated by intrinsic curiosity rewards and state novelty. The tuples are stored in the course experience replay pool. The learning is carried out according to the preset course learning outline, and the deep Q network is updated by the gradient descent algorithm.

2. The automated analysis system for detection data of radio frequency equipment according to claim 1, characterized in that, The specific process of the data preprocessing module includes: The system receives raw tag reading events from the radio frequency device, which include tag ID, reader ID, timestamp, and received signal strength indication. A periodic timer is used to aggregate all raw tag reading events into a single time window at preset fixed time intervals.

3. The automated analysis system for detection data of radio frequency equipment according to claim 1, characterized in that, The specific generation process of the deep Q network module includes: The deep Q-network agent inputs a multi-dimensional state vector into its internal Q-function neural network. The Q-function neural network outputs two values, representing the estimated long-term expected rewards of performing actions labeled as normal and abnormal in the current state. A random number is generated and compared with a preset parameter ε. If the random number is less than ε, the deep Q-network agent executes an exploration strategy; otherwise, if the random number is not less than ε, the deep Q-network agent executes a greedy strategy, selecting the action with the higher estimated value. The selected action is executed, and the agent receives the multi-dimensional true vector for the next time step.

4. The automated analysis system for detection data of radio frequency equipment according to claim 1, characterized in that, The specific generation process of the reward shaping framework includes: In the reward shaping framework, a diverse meta-training task containing historical normal and abnormal data is used to learn and generate a potential function-based reward shaping function. The reward shaping function receives intrinsic curiosity rewards and sparse external rewards received from external human intervention, and combines a small amount of feedback to dynamically calculate a total reward value to guide the deep Q-network agent.

5. An automated analysis system for detection data of radio frequency equipment according to claim 1, characterized in that, The specific processing steps for empirical tuples include: The system receives a multidimensional state vector, a selected action, a total reward, and a multidimensional truth vector. These four core data items are combined into a structured experience tuple. A deep Q-network is used to calculate the value estimate for the multidimensional state vector and action. A target value based on the total reward and the multidimensional truth vector is then calculated. The temporal difference error between the value estimate and the target value is calculated as an unexpectedness indicator. The learning difficulty of the experience tuple is evaluated using intrinsic curiosity reward and the multidimensional state vector. The experience tuple, along with the unexpectedness indicator and the learning difficulty, is stored in the course experience replay pool.

6. The automated analysis system for detection data of radio frequency equipment according to claim 1, characterized in that, The specific process of learning according to the pre-set course syllabus includes: When the number of experience tuples accumulated in the replay pool reaches the preset training threshold, the model update is initiated. According to the preset course learning outline, in the early stage of training, the sampler prioritizes the learning of experience tuples with lower learning difficulty. As the performance of the deep Q network agent improves, the course difficulty gradually increases, and the sampler gradually increases the sampling weight of experience tuples with higher learning difficulty. Using this small batch of data sampled by course, the network weights of the deep Q network, the inverse dynamics model, and the forward dynamics model are updated through the gradient descent algorithm.

Images