A training and examination evaluation method based on RFID and visual multi-modal fusion

Abstract

The present application relates to the technical field of intelligent education and sports informationization evaluation, and particularly relates to a training and examination evaluation method based on RFID and visual multi-modal fusion, comprising: demodulating a raw radio frequency signal sequence of a pre-deployed reference tag to obtain a reference radio frequency spatial coordinate, and calculating a multi-modal spatial perspective transformation matrix thereof to an image pixel coordinate; cascading and binding examinee identity information and a mobile tag unique identifier to synchronously lock a target examinee in a visual and radio frequency path; synchronously collecting a raw radio frequency signal and a motion video frame with a homologous hardware timestamp, and after time sequence alignment, mapping the demodulated real-time radio frequency spatial coordinate to a pixel coordinate system by using the transformation matrix to lock the target examinee; running a visual AI inference and an RFID rule checking path in parallel, respectively outputting an action specification degree and an action count preliminary judgment result, and according to the confidence degrees of the two, real-time evaluating an environmental working condition to adaptively switch a score output mode.

Description

Technical Field

[0001] This invention relates to the field of intelligent education and sports information-based evaluation technology, and in particular to a training and assessment evaluation method based on RFID and visual multimodal fusion. Background Technology

[0002] In existing training assessment and physical fitness testing scenarios, pure vision AI training and assessment systems have been widely used. These systems typically use cameras deployed in front of the assessment site to capture motion video frames of trainees, then use edge or cloud-based pose estimation models to extract the coordinates of key points on the trainee's skeleton, and rely on image frame difference methods or target tracking algorithms to identify the trainee's movements and perform standardized movement counting.

[0003] Existing pure vision AI training and evaluation systems are easily affected by changes in ambient light, interference from training equipment, or physical limitations caused by severe self-occlusion of human limbs in practical applications. When the image quality deteriorates due to backlighting, low light overexposure, or large-area occlusion of the target, the confidence of the skeletal key point extraction of the pose estimation model will drop significantly, leading to frequent missed and false judgments in action counting. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a training and evaluation method based on the fusion of RFID and visual multimodal imaging, aiming to improve the problem of frequent missed judgments and misjudgments caused by the deterioration of image quality in backlight or limb occlusion scenarios in existing pure visual evaluation schemes.

[0005] This invention provides the following technical solution: a training and evaluation method based on RFID and visual multimodal fusion, comprising: S1. Spatial demodulation is performed on the original radio frequency signal sequences of multiple pre-deployed reference tags to obtain reference radio frequency spatial coordinates, and the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates is calculated. S2. Bind the candidate's identity information with the unique identifier of the mobile tag, and simultaneously lock the target candidate in the visual and radio frequency channels; S3. Synchronously acquire the original radio frequency signal sequence with the same hardware timestamp and the motion video frame. After time-series alignment by a time sliding window, spatial demodulate the aligned original radio frequency signal sequence to obtain real-time radio frequency spatial coordinates. Then, use a multimodal spatial perspective transformation matrix to map the real-time radio frequency spatial coordinates to the pixel coordinate system of the motion video frame to lock the target examinee. S4. Run the visual AI reasoning path and the RFID rule verification path in parallel, and output the preliminary judgment results of the action standardization and the action count for the target examinee, respectively. S5. Based on the confidence levels of the initial judgment results of motion standardization and motion count, the environmental conditions are evaluated in real time. Under normal conditions, the motion performance is output by fusing the initial judgment results of motion standardization and motion count. Under visual failure conditions, the motion performance is output primarily based on the initial judgment results of motion count. Under radio frequency abnormal conditions, the original radio frequency signal sequence is shielded and the motion performance is output entirely based on the visual AI inference path.

[0006] Preferably, in S1, spatial demodulation of the raw radio frequency signal sequences of multiple pre-deployed reference tags to obtain reference radio frequency spatial coordinates includes: The system periodically polls the original radio frequency signal sequences transmitted by the multiple reference tags through multiple antennas and extracts the Received Signal Strength Indication (RSSI) timing sequence from the original radio frequency signal sequences. The RSSI time series is subjected to moving average filtering to obtain a standard RSSI sequence; Based on a preset field strength propagation attenuation model, the standard RSSI sequence is converted into the relative distances between multiple reference tags and each antenna. Using a trilateration algorithm, geometric intersection is calculated based on the relative distance, and the three-dimensional homogeneous geometric coordinates of multiple reference tags in the physical field are output as reference radio frequency spatial coordinates.

[0007] Preferably, in S1, calculating the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates includes: Control the industrial camera to acquire calibration image frames containing multiple reference tags; The image pixel coordinates of multiple reference labels in the calibration image frame are identified using an object detection algorithm. By combining the reference radio frequency spatial coordinates of the multiple reference tags with the corresponding image pixel coordinates, a set of spatial linear mapping equations in a homogeneous coordinate system is constructed. The reprojection error of the spatial linear mapping equations is iteratively optimized using the least squares method to solve for the multimodal spatial perspective transformation matrix.

[0008] Preferably, in S2, binding the examinee's identity information with the unique identifier of the mobile tag includes: The system reads the candidate's electronic certificate information through a data interface and simultaneously collects the candidate's real-time biometric data. The real-time biometric data is compared with the registration data in the electronic certificate information to verify the candidate's physical identity; When the comparison result meets the preset consistency judgment conditions, the RFID reader is controlled to read the unique identifier of the mobile tag worn by the examinee in the near-field training area. The electronic certificate information, the unique identifier of the mobile tag, and the current assessment item are associated and stored in a temporary database to complete the cascading binding of the candidate's identity information and the unique identifier of the mobile tag.

[0009] Preferably, in S3, the synchronous acquisition of the original radio frequency signal sequence and motion video frames with the same hardware timestamp includes: A globally unique microsecond-level hardware clock source is generated by a unified clock synchronization module; In the physical driving layer of the industrial camera's exposure output motion video frames, the hardware clock source is injected as a visual frame hardware timestamp into each motion video frame. In the baseband data layer of the RFID reader demodulates the original radio frequency signal sequence, the hardware clock source is synchronously injected as a radio frequency hardware timestamp into each radio frequency data packet of the original radio frequency signal sequence.

[0010] Preferably, in S3, the timing alignment of the time sliding window includes: A dynamic sliding time window is established with a preset time width; Extract the visual frame hardware timestamp and radio frequency hardware timestamp within the dynamic sliding time window; The original radio frequency signal sequence with the smallest hardware timestamp deviation between the two channels is bound and registered one-to-one with the corresponding motion video frame. If radio frequency packet loss occurs within a specific dynamic sliding time window, an interpolation algorithm is invoked to perform timing feature fitting and completion based on the original radio frequency signal sequences of adjacent windows.

[0011] Preferably, in S3, the step of mapping the real-time radio frequency spatial coordinates to the pixel coordinate system of the moving video frame using a multimodal spatial perspective transformation matrix to lock onto the target examinee includes: The multimodal spatial perspective transformation matrix is ​​used to convert the real-time radio frequency spatial coordinates into mapped pixel coordinates in the motion video frame; A multi-target tracking algorithm was used to extract the motion trajectory of the centroid of the human skeleton of each examinee in the motion video frame; Calculate the Pearson correlation coefficient between the temporal trajectory of the mapped pixel coordinates and the motion trajectory of each human skeletal centroid; When the Pearson correlation coefficient is greater than the set consistency threshold, the corresponding human skeleton model is locked as the target candidate.

[0012] Preferably, in S4, the visual AI inference pathway outputs preliminary judgments on the standardization of the target examinee's actions, including: The motion video frames are input into a preset edge pose estimation model to extract the coordinates of multiple human core topological skeleton key points and the corresponding key point confidence scores of the target examinee. Based on the changes in the coordinates of multiple key points of the human core topology skeleton over time, the dynamic joint angles, displacement amplitudes of key parts, and movement trends of the target examinee during the movement process are continuously calculated. The calculated joint dynamic angle, displacement amplitude of key parts and motion trend are compared with the preset standard motion feature templates in the local rule base in real time. Based on the comparison results, determine the compliant items and violation types of the target candidate's actions within the current action cycle, and generate a preliminary judgment result on the standardization of actions.

[0013] Preferably, in S4, the RFID rule verification path outputs the preliminary judgment results of the action count for the target examinee, including: The original radio frequency signal sequence of the target candidate was digitally filtered to extract the periodic characteristic curve of the Received Signal Strength Indicator (RSSI). The peak detection algorithm is used to capture the time-series peaks and troughs in the RSSI periodic characteristic curve to determine the initial number of action cycles for the target examinee. Synchronously acquire hard timing activation status data of low-frequency point triggers deployed at specific physical boundaries of the assessment site; By combining the initial number of action cycles with the hard-time activation state data, a boundary closed-loop verification is performed to determine whether the target examinee accurately passes through a specific physical boundary within a single action cycle, and the initial action count result is output.

[0014] Preferably, in S5, the real-time assessment of environmental conditions based on the confidence levels of the initial judgment results of action standardization and action count includes: Extract the average confidence score of the coordinates of each human core topological skeleton key point in the preliminary judgment result of the action standardization, and extract the radio frequency stability index of the original radio frequency signal sequence in the current time period. When the average confidence score is greater than the first judgment threshold and the radio frequency stability index is greater than the second judgment threshold, the current environmental condition is determined to be a normal operating condition. When the average confidence score is lower than or equal to the first determination threshold for a consecutive preset number of frames, the current environmental condition is determined to be a visual failure condition. When the radio frequency stability index is lower than or equal to the second determination threshold, the current environmental condition is determined to be an abnormal radio frequency condition.

[0015] The present invention has the following beneficial effects: 1. In this invention, by using the periodic fluctuation pattern of the received signal strength indication in the original radio frequency signal sequence as the physical constraint benchmark for action counting, and combining it with the compliance verification of the visual AI inference path, an adaptive dual closed-loop discrimination for action cycle counting is realized, which solves the problem of frequent missed judgments and misjudgments caused by the decline in image quality in backlight or limb occlusion scenarios in traditional pure visual evaluation schemes.

[0016] 2. In this invention, by leveraging the technical advantage that the radio frequency sensing link is completely unaffected by physical interference from lighting conditions, and by seamlessly downgrading from a state machine to a performance output mode based on the radio frequency path when the imaging quality of the visual path deteriorates, continuous assessment and evaluation are achieved in the full lighting range and under dense occlusion conditions. This solves the problem of target tracking loss, which leads to a precipitous drop in recognition accuracy or assessment interruption when the traditional pure vision system is overexposed in strong light, backlight, or low light at night with a lot of noise, or when the human body is largely obscured by the equipment.

[0017] 3. In this invention, the radio frequency spatial coordinates demodulated in real time are dynamically mapped to the pixel coordinate system of the video frame through a multimodal spatial perspective transformation matrix, and the spatiotemporal consistency of the Pearson correlation coefficient is checked with the human skeleton centroid motion trajectory extracted by visual tracking. This realizes the provision of a continuous physical identity reference point for the visual tracking algorithm that does not depend on appearance features, and solves the problem that traditional multi-person testing systems are prone to target identification switching errors and score mismatches when candidates wear similar clothes or are too close together due to limb overlap. Attached Figure Description

[0018] Figure 1 This is a flowchart of a training and evaluation method based on RFID and visual multimodal fusion proposed in this invention; Figure 2 This is a flowchart illustrating the calibration and multimodal matrix construction process of a training and evaluation method based on RFID and visual multimodal fusion proposed in this invention. Figure 3 This is a flowchart of the temporal alignment and spatial mapping locking process for a training and evaluation method based on RFID and visual multimodal fusion proposed in this invention. Figure 4 This is a flowchart of a state machine multi-condition real-time evaluation and switching process for a training and evaluation method based on RFID and visual multimodal fusion proposed in this invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In embodiments of the present invention, the present invention provides a training and evaluation method based on RFID and visual multimodal fusion, such as... Figures 1-4 As shown, it includes: S1. Spatial demodulation is performed on the original radio frequency signal sequences of multiple pre-deployed reference tags to obtain reference radio frequency spatial coordinates, and the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates is calculated. Furthermore, in S1, spatial demodulation of the raw radio frequency signal sequences of multiple pre-deployed reference tags to obtain reference radio frequency spatial coordinates includes: The system periodically polls multiple antennas to receive raw radio frequency signal sequences transmitted by multiple reference tags and extracts the Received Signal Strength Indicator (RSSI) timing sequence from the raw radio frequency signal sequences. The RSSI time series is processed by moving average filtering to obtain the standard RSSI sequence; Based on a pre-defined field strength propagation attenuation model, the standard RSSI sequence is converted into the relative distances between multiple reference tags and each antenna. The trilateration algorithm is used to perform geometric intersection calculations based on relative distances, and outputs the three-dimensional homogeneous geometric coordinates of multiple reference tags in the physical field as reference radio frequency spatial coordinates.

[0021] Furthermore, in S1, calculating the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates includes: Control the industrial camera to acquire calibration image frames containing multiple reference tags; The image pixel coordinates of multiple reference labels in the calibration image frame are identified using an object detection algorithm. By combining the reference radio frequency spatial coordinates of multiple reference tags with the corresponding image pixel coordinates, a set of spatial linear mapping equations in a homogeneous coordinate system is constructed. The reprojection error of the spatial linear mapping equations is iteratively optimized using the least squares method to solve for the multimodal spatial perspective transformation matrix.

[0022] Specifically, in the initialization and calibration process of the multimodal fusion training and evaluation method, the spatial recalibration of the physical site and the construction of the multimodal perspective transformation matrix are performed first. The method periodically receives the original radio frequency signal sequence transmitted by multiple reference tags pre-deployed in the assessment site, and uses multiple antennas to perform periodic polling reception, thereby extracting the received signal strength indication timing sequence from the original radio frequency signal sequence.

[0023] Due to multipath interference in the physical environment of the examination room, the extracted received signal strength indication time series exhibits discrete and fluctuating characteristics. Therefore, the method employs a moving average filtering algorithm to smooth and denoise the received signal strength indication time series. This algorithm eliminates random noise fluctuations by truncating time series data within a fixed window length and calculating the arithmetic mean of all received signal strength indication samples within that window, thus outputting a continuous standard received signal strength indication sequence.

[0024] After standard signal cleaning, the method, based on a pre-defined field strength propagation attenuation model, converts the standard received signal strength indication corresponding to each reference tag into the physical relative distance between that reference tag and each antenna. The field strength propagation attenuation model is expressed by the following formula: ; in, The standard received signal strength indication; d represents the required relative distance between the reference tag and the specific antenna; Indicates the reference distance; This represents the standard received signal strength reference value at the reference reference distance; This represents the environmental path attenuation factor, which is adaptively configured based on the physical building structure type of the examination room. Let represent a Gaussian random variable with zero mean and constant standard deviation. The method involves substituting the extracted standard received signal strength indication into the above formula to solve for the unknown. This yields the set of relative distances between each reference tag and each antenna.

[0025] After obtaining the set of relative distances, the method runs a trilateration algorithm to perform geometric intersection calculations. This algorithm constructs multiple intersecting spherical equations within the physical space, using the known three-dimensional spatial coordinates of each receiving antenna as the center and the calculated physical relative distance as the radius. By simultaneously solving multiple sets of spherical equations to form a linear system of equations, and using a matrix pseudo-inverse operator to solve for the optimal intersection point of this system, the algorithm finally calculates and outputs the three-dimensional homogeneous geometric coordinates of each reference tag in the physical field, serving as the reference radio frequency spatial coordinates.

[0026] While performing spatial localization calculations, the method simultaneously controls the camera to acquire calibration image frames containing image features of multiple reference labels. The acquired calibration image frames are then input into a pre-defined target detection algorithm. This algorithm extracts texture and edge geometric features from the calibration image frames through shallow convolution operations, accurately locates the bounding box of each reference label in the image via bounding box regression and classification mechanisms, and calculates the geometric center point of the bounding box as the image pixel coordinates of that reference label in the image coordinate system.

[0027] After simultaneously acquiring the reference radio frequency spatial coordinates and image pixel coordinates, the method establishes a spatial mapping link between the two. By combining the 3D homogeneous geometric coordinates of the same reference label with the corresponding 2D image pixel coordinates, a system of spatial linear mapping equations in the homogeneous coordinate system is constructed based on the principle of homogeneous coordinate transformation. This system of equations projects 3D spatial points onto the 2D pixel plane through a 3x4 unknown perspective transformation matrix.

[0028] To solve for the unknown matrix elements in the perspective transformation matrix, the method employs the least squares approach to fit the entire system of linear mapping equations in space. This fitting process constructs a global reprojection error objective function by calculating the Euclidean distance between the pixel coordinates predicted by projection and the actual pixel coordinates identified by object detection.

[0029] Subsequently, the method employs a Gauss-Newton iterative optimization algorithm to solve the objective function until the error change between two adjacent iterations is less than a preset termination threshold. At this point, the iteration converges and outputs the optimal solution, thus obtaining the complete multimodal spatial perspective transformation matrix. This matrix is ​​directly used as a core global configuration parameter in the subsequent spatial projection registration of real-time radio frequency positioning points to video frame pixel coordinates during the movement of the trainee.

[0030] S2. Bind the candidate's identity information with the unique identifier of the mobile tag, and simultaneously lock the target candidate in the visual and radio frequency channels; Furthermore, in S2, binding the candidate's identity information with the unique identifier of the mobile tag includes: The system reads the candidate's electronic certificate information through a data interface and simultaneously collects the candidate's real-time biometric data. The real-time biometric data is compared with the registration data in the electronic certificate information to verify the candidate's physical identity; When the comparison result meets the preset consistency judgment conditions, the RFID reader is controlled to read the unique identifier of the mobile tag worn by the examinee in the near-field training area. The electronic certificate information, the unique identifier of the mobile tag, and the current assessment item are associated and stored in a temporary database to complete the cascading binding of the candidate's identity information and the unique identifier of the mobile tag.

[0031] Specifically, when a candidate enters the assessment area, a dynamic binding process between the candidate's identity information and the unique mobile tag identifier is initiated. This process reads the candidate's electronic identification information in real time through a specific data interface, extracting the pre-stored text identity data within the identification document. Simultaneously, the method controls a biometric data acquisition device to obtain the candidate's real-time biometric data, which is dynamically acquired as real-time facial image data.

[0032] After acquiring the dual-source data, the method performs local comparison and verification, comparing the real-time biometric data collected in real time with the registration feature data extracted from the electronic certificate information in a two-level feature similarity comparison. The specific comparison process utilizes the feature vector similarity formula for quantitative calculation: ; in, This represents a high-dimensional topological feature vector extracted from real-time facial image data; This represents the standard facial feature vector extracted from the registration feature data of electronic certificate information; This represents the cosine similarity score between the two. The method uses this score to verify the authenticity of the candidate's physical identity.

[0033] When the cosine similarity score output by feature comparison meets the preset consistency judgment condition, that is, the score is greater than or equal to the preset similarity threshold, the method determines that the candidate's identity document matches, and then triggers near-field pairing control. The method sends a read activation command to the radio frequency reading device in the near-field training area, controls the radio frequency reading device to perform non-contact scanning within the limited physical boundary of the training area, and reads the unique identifier of the mobile tag worn by the trainee in real time.

[0034] After successfully reading the unique identifier of the mobile tag, the method establishes a deep cascading link in the data chain at the system's underlying layer. The method strongly associates and combines the previously read electronic certificate information, the newly parsed unique identifier of the mobile tag, and the name of the current assessment item, encapsulating them into a standard data dictionary record.

[0035] Subsequently, the method writes the standard data dictionary record into and associates it with a local temporary database, completing the cascading binding of the examinee's identity information with the unique identifier of the mobile tag. This cascading mapping pair will serve as a globally unique identity retrieval primary key, residing in the system's cache, and will be used to ensure data flow between the visual target tracking trajectory and the radio frequency positioning trajectory throughout the entire training process, guaranteeing that the visual and radio frequency pathways can synchronously and uniquely lock onto the target examinee.

[0036] S3. Synchronously acquire the original radio frequency signal sequence with the same hardware timestamp and the motion video frame. After time-series alignment by a time sliding window, spatial demodulate the aligned original radio frequency signal sequence to obtain real-time radio frequency spatial coordinates. Then, use a multimodal spatial perspective transformation matrix to map the real-time radio frequency spatial coordinates to the pixel coordinate system of the motion video frame to lock the target examinee. Furthermore, in S3, the synchronous acquisition of raw radio frequency signal sequences and motion video frames with the same hardware timestamps includes: A globally unique microsecond-level hardware clock source is generated by a unified clock synchronization module; In the physical driving layer of the industrial camera's exposure output motion video frames, a hardware clock source is injected as a visual frame hardware timestamp into each motion video frame. In the baseband data layer of the RFID reader, the original radio frequency signal sequence is demodulated, and a hardware clock source is synchronously injected as a radio frequency hardware timestamp into each radio frequency data packet of the original radio frequency signal sequence.

[0037] Furthermore, in S3, the timing alignment of the time sliding window includes: A dynamic sliding time window is established with a preset time width; Extract the hardware timestamp and radio frequency hardware timestamp of the visual frame within the dynamic sliding time window; The original radio frequency signal sequence with the smallest hardware timestamp deviation between the two channels is bound and registered one-to-one with the corresponding motion video frame. If radio frequency packet loss occurs within a specific dynamic sliding time window, an interpolation algorithm is invoked to fit and complete the timing features based on the original radio frequency signal sequences of adjacent windows.

[0038] Furthermore, in S3, the real-time radio frequency spatial coordinates are mapped to the pixel coordinate system of the moving video frame using a multimodal spatial perspective transformation matrix to lock onto the target examinee, including: By using a multimodal spatial perspective transformation matrix, real-time radio frequency spatial coordinates are transformed into mapped pixel coordinates in moving video frames; A multi-target tracking algorithm was used to extract the motion trajectory of the centroid of the human skeleton of each examinee in the motion video frame; Calculate the Pearson correlation coefficient between the temporal trajectory of the mapped pixel coordinates and the motion trajectory of each human skeletal centroid; When the Pearson correlation coefficient is greater than the set consistency threshold, the corresponding human skeletal model is locked as the target candidate.

[0039] Specifically, in the data acquisition and registration phase of the multimodal fusion training and evaluation method, the method first overcomes the time asynchrony problem of heterogeneous data sources at the underlying driver and baseband protocol levels. The method relies on the system's built-in unified clock synchronization module to generate a globally unique microsecond-level hardware clock source.

[0040] During the data acquisition process, when the industrial camera exposes and outputs motion video frames, the method directly uses the current value of the microsecond-level hardware clock source as the hardware timestamp of the visual frame in the camera's physical driver layer, and writes and injects it into the image metadata header of each motion video frame.

[0041] Synchronously, when a signal is received from the mobile tag, the baseband data layer of the original radio frequency signal sequence is demodulated by the reader. The method also uses the real-time value of the microsecond-level hardware clock source as the radio frequency hardware timestamp and seamlessly injects it into each radio frequency data packet of the original radio frequency signal sequence, thus eliminating the time jitter caused by the asynchronous transmission link between the optical path and the radio frequency path from the hardware source.

[0042] After acquiring dual-channel data streams with identical hardware timestamps, the method performs precise alignment of heterogeneous data at the time domain. The method establishes a dynamic sliding time window with a preset time width, which continuously slides forward over time. The method extracts the hardware timestamps and radio frequency hardware timestamps of all visual frames covered within the current dynamic sliding time window in real time and calculates the absolute difference between them.

[0043] The method utilizes a time-series registration algorithm to perform one-to-one binding and registration between the original RF signal sequences with the smallest absolute value of the hardware timestamp deviation between two channels and their corresponding moving video frames. If RF data loss occurs within a specific dynamic sliding time window due to complex electromagnetic interference or obstructions in the examination room, the method calls a first-order linear interpolation algorithm for adaptive data completion. The interpolation calculation is expressed by the following formula: ; in, The hardware timestamp indicating the specific moment when radio frequency packet loss occurred; and These represent the radio frequency hardware timestamps that were successfully received within adjacent windows before and after a specific moment at the target. and They represent in and The known original radio frequency signal feature vector corresponding to the given time; This represents the virtual radio frequency timing characteristics of the target at a specific moment after fitting and completion, thereby ensuring the continuity and interlocking of the dual-modal data stream in the time domain.

[0044] After completing temporal alignment, the method performs geometric spatial demodulation on the original RF signal sequence to be aligned, outputting dynamic real-time RF spatial coordinates, and performs multimodal fusion registration at the spatial layer. The method calls the multimodal spatial perspective transformation matrix obtained in the aforementioned initialization and calibration stage, and transforms the real-time RF spatial coordinates into mapped pixel coordinates in the corresponding motion video frame pixel coordinate system through matrix multiplication and projection transformation.

[0045] Meanwhile, the method runs a multi-target tracking algorithm in the visual pathway to perform image segmentation and topological skeleton extraction on each candidate appearing in the current motion video frame, and calculates and generates multiple human skeleton centroid motion trajectories for each candidate in real time.

[0046] To accurately identify examinees and prevent cheating in complex examination environments with dense crowds and potential limb occlusion, this method introduces a spatiotemporal trajectory correlation check. Within a defined temporal translation window, the method continuously calculates the Pearson correlation coefficient between the temporal trajectory of mapped pixel coordinates and the visually extracted centroid motion trajectories of the human skeleton. The Pearson correlation coefficient is expressed by the following formula: ; in, This indicates the number of consecutive sampling frames within the current evaluation window; Indicates the first The geometric position of the radio frequency mapped pixel coordinates on the pixel plane at frame time; Indicates the first The geometric position of the centroid of a specific human skeleton on the pixel plane at a given frame time; and These represent the arithmetic mean of the corresponding coordinates within the window; This represents the spatiotemporal correlation coefficient between the radio frequency mapping trajectory and the visual centroid trajectory.

[0047] The method compares the calculated Pearson correlation coefficients with a system-preset consistency threshold. When the Pearson correlation coefficient between a human skeleton centroid motion trajectory and the mapped pixel coordinate temporal trajectory is greater than the consistency threshold, the method determines that the physical identity of the visual human model strongly matches the motion tag worn by the examinee. Subsequently, the system internally hard-locks the corresponding human skeleton model as the target examinee, completely avoiding the examinee identity confusion error that is easily caused by simple visual tracking when target intersection occurs.

[0048] S4. Run the visual AI reasoning path and the RFID rule verification path in parallel, and output the preliminary judgment results of the action standardization and the action count for the target examinee, respectively. Furthermore, in S4, the visual AI inference pathway outputs preliminary judgments on the standardization of the target examinee's actions, including: The motion video frames are input into a preset edge pose estimation model to extract the coordinates of multiple human core topological skeleton key points of the target examinee and the corresponding key point confidence. Based on the changes in the coordinates of multiple key points of the human core topology skeleton over time, the dynamic joint angles, displacement amplitudes of key parts, and movement trends of the target examinee during the movement process are continuously calculated. The calculated joint dynamic angles, displacement amplitudes of key parts, and motion trends are compared with the preset standard motion feature templates in the local rule base in real time. Based on the comparison results, determine the compliant items and violation types of the target candidate's actions within the current action cycle, and generate a preliminary judgment result on the standardization of actions.

[0049] Furthermore, in S4, the RFID rule verification path outputs the preliminary judgment results of the action count for the target examinee, including: The original radio frequency signal sequence of the target examinee is digitally filtered to extract the periodic characteristic curve of the Received Signal Strength Indicator (RSSI). The peak detection algorithm is used to capture the time-series peaks and troughs in the RSSI periodic characteristic curve to determine the initial number of action cycles for the target examinee. Synchronously acquire hard timing activation status data of low-frequency point triggers deployed at specific physical boundaries of the assessment site; By combining the initial number of action cycles with the hard-sequence activation state data, a boundary closed-loop verification is performed to determine whether the target examinee accurately passes through a specific physical boundary within a single action cycle, and the initial action count result is output.

[0050] Specifically, in the parallel initial judgment stage of the multimodal fusion training and evaluation method, the method enables a dual-path parallel processing pipeline, running the visual AI inference path and the radio frequency rule verification path in parallel in completely independent underlying threads to ensure low-latency response in high-concurrency training and evaluation environments.

[0051] In the visual AI inference pathway, the method uses temporally aligned motion video frames as image input streams, which are fed into a pre-defined lightweight edge pose estimation model in real time. This model extracts multi-level deep semantic features of the image through a deep convolutional neural network, outputs the coordinates of multiple core topological skeletal key points of the target examinee, and outputs the corresponding key point confidence scores for each point.

[0052] The method extracts the dynamic trajectory of multiple key points in the core topological skeleton of the human body along a time series, and continuously calculates the dynamic joint angles, displacement amplitude of key parts, and movement trend of the target examinee during the movement process. The dynamic joint angles are quantified using a vector angle formula. ; in, and They represent at specific moments. Two topological limb direction vectors are constructed by connecting the coordinates of adjacent skeletal key points; This represents the dynamic angle of the joint at a specific moment. The method constructs a temporal motion feature matrix by real-time calculation of the angle change rate of key joints, the displacement amplitude of key parts on the vertical axis, and the motion trend of the overall centroid.

[0053] Subsequently, the method performs real-time feature verification between the temporal motion feature matrix and the preset standard action feature templates in the local rule base. By calculating the temporal dynamic time warping distance between the current feature matrix and the standard template, the degree of deviation of the action is quantified. Based on the comparison results, the method accurately identifies and determines the compliant items and specific violation types of the target examinee's actions within the current action cycle, generating an initial judgment result of action standardization including fine-grained standardization scores.

[0054] Synchronously, in the radio frequency rule verification path, the method performs digital signal processing on the acquired raw radio frequency signal sequence of the target examinee. By running a low-pass digital filtering algorithm, high-frequency Gaussian white noise caused by the electromagnetic environment and multipath reflections is filtered out, and a smooth received signal strength indication periodic characteristic curve is accurately extracted from it.

[0055] The method employs an adaptive peak detection algorithm to capture time-series peaks and troughs in the periodic characteristic curve of the received signal strength indication. This algorithm dynamically sets a sliding time window and an amplitude change rate threshold to capture inflection points on the characteristic curve that satisfy the waveform's geometric characteristics. The waveform inflection point determination satisfies the following time-series differential equation constraint: ; When the above differential constraints are satisfied, the method determines the current time step. There is a temporal peak. The method determines the initial number of action cycles for the target candidate by counting the number of alternations between the temporal peak and trough.

[0056] To avoid false counts caused by body movement or radio frequency multipath propagation, the method establishes a physical checkpoint verification mechanism. The method synchronously acquires the hard timing activation status data of low-frequency point triggers deployed at specific physical boundaries of the assessment site.

[0057] The method combines the initial action cycle count with hard-time activation state data for boundary closed-loop verification. The method matches the time-domain overlap between the timestamp of the radio frequency peak and the hardware timestamp of the low-frequency trigger being activated by the pulse to determine whether the target examinee accurately and completely passes the preset specific physical boundary within a single action cycle, eliminating false counts of incomplete or partial acceleration, and finally outputting an accurate preliminary judgment result of the action count.

[0058] S5. Based on the confidence levels of the initial judgment results of motion standardization and motion count, the environmental conditions are evaluated in real time. Under normal conditions, the motion performance is output by fusing the initial judgment results of motion standardization and motion count. Under visual failure conditions, the motion performance is output primarily based on the initial judgment results of motion count. Under radio frequency abnormal conditions, the original radio frequency signal sequence is shielded and the motion performance is output entirely based on the visual AI inference path.

[0059] Furthermore, in S5, the environmental conditions are assessed in real time based on the confidence levels of the initial judgment results of motion standardization and motion count, including: The average confidence score of the coordinates of key points of the human core topology skeleton in the preliminary judgment of the motion standardization is extracted, and the radio frequency stability index of the original radio frequency signal sequence in the current time period is extracted. When the average confidence score is greater than the first judgment threshold and the radio frequency stability index is greater than the second judgment threshold, the current environmental condition is determined to be a normal operating condition. When the average confidence score is lower than or equal to the first judgment threshold for a consecutive preset number of frames, the current environmental condition is determined to be a visual failure condition. When the radio frequency stability index is lower than or equal to the second judgment threshold, the current environmental condition is determined to be an abnormal radio frequency condition.

[0060] Specifically, in the decision-making and arbitration phase of the multimodal fusion training and assessment method, the method activates an adaptive decision state machine to quantitatively evaluate and dynamically switch the real-time environmental conditions of the current examination room based on the confidence characteristics of the dual-path output results.

[0061] The method first calls the feature extraction module to extract the average confidence score of the coordinates of each key point in the core topology of the human body from the initial judgment of motion standardization in real time. This score is calculated by averaging the probability estimates given synchronously by the edge pose estimation model when outputting the key point coordinates in the time domain. Simultaneously, the method extracts the radio frequency stability index of the original radio frequency signal sequence within the current sliding time period. The radio frequency stability index is comprehensively quantified by calculating the variance of the received signal strength indication and the real-time packet loss rate of the data packets within the current time period.

[0062] After obtaining the core confidence indices from both the visual and radio frequency channels, the method runs a multi-condition state machine switching algorithm and performs a dual-threshold conditional branch decision. The decision logic of the multi-condition state machine satisfies the following piecewise function constraints: ; in, The average confidence score represents the coordinates of key points in the human core topology skeleton of the current frame. This indicates the radio frequency stability index during the current sliding period; This indicates the system's preset first judgment threshold; This indicates the system's preset second judgment threshold; Indicates the first Real-time skeletal keypoint confidence of a frame; Indicates the preset number of consecutive sliding frames window length; This indicates the current deadline timestamp.

[0063] The method determines the real-time environmental conditions by executing the aforementioned segmented logic. When the average confidence score is greater than the first judgment threshold and the radio frequency stability index is greater than the second judgment threshold, the method determines that the current environmental conditions meet the triggering conditions for normal operation. Under these conditions, the examination room is well-lit with no large-area limb obstruction, and the electromagnetic environment is also normal. The method then calls a multimodal data fusion algorithm to dynamically weight and fuse the initial judgment results of the action standardization output from the visual path with the initial judgment results of the action count output from the radio frequency path, and outputs the candidate's final motion score.

[0064] When the real-time confidence score of the skeletal keypoints is lower than or equal to the first judgment threshold for a consecutive preset number of frames, the method determines that the current environmental condition has transitioned to a visual failure condition. This state typically corresponds to the examinee's body being completely obscured by the referee, equipment, or other examinees, or severe light attenuation at the edge of the examination room. At this time, the method automatically adjusts the decision weights and seamlessly switches to a judgment mode based primarily on the initial judgment result of the action count output by the radio frequency path, combined with the hard-time physical checkpoint status of the low-frequency point trigger to output the motion score, thus shielding against misjudgment interference caused by the loss of visual targets or ID confusion.

[0065] When the radio frequency stability index is lower than or equal to the second judgment threshold, the method determines that the current environmental conditions meet the activation conditions for abnormal radio frequency conditions. This state typically corresponds to a sudden high-frequency electromagnetic interference in the examination room, or the mobile tag worn by the examinee being accidentally shielded by a metal object or a large area of ​​the human body, causing the RSSI waveform to experience a step-like abnormal fluctuation or severe packet loss. At this time, the state machine actively executes circuit breaker protection, directly shielding the data stream input of the entire original radio frequency signal sequence at the underlying level, and completely switching the system decision-making mechanism to rely on the action standardization and timing posture periodicity terms output by the visual AI inference path, independently outputting the examinee's motion score, thereby ensuring the high robustness and high availability of the training and evaluation system in complex and variable physical environments.

[0066] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training and evaluation method based on RFID and visual multimodal fusion, characterized in that, include: S1. Spatial demodulation is performed on the original radio frequency signal sequences of multiple pre-deployed reference tags to obtain reference radio frequency spatial coordinates, and the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates is calculated. S2. Bind the candidate's identity information with the unique identifier of the mobile tag, and simultaneously lock the target candidate in the visual and radio frequency channels; S3. Synchronously acquire the original radio frequency signal sequence with the same hardware timestamp and the motion video frame. After time-series alignment by a time sliding window, spatial demodulate the aligned original radio frequency signal sequence to obtain real-time radio frequency spatial coordinates. Then, use a multimodal spatial perspective transformation matrix to map the real-time radio frequency spatial coordinates to the pixel coordinate system of the motion video frame to lock the target examinee. S4. Run the visual AI reasoning path and the RFID rule verification path in parallel, and output the preliminary judgment results of the action standardization and the action count for the target examinee, respectively. S5. Based on the confidence level of the initial judgment results of motion standardization and motion count, the environmental conditions are evaluated in real time. Under normal conditions, the motion standardization and motion count results are combined to output the sports performance. In the case of visual failure, switch to outputting motion results based primarily on the initial judgment of motion counts. Under abnormal radio frequency conditions, the original radio frequency signal sequence is shielded and the motion results are output entirely based on the visual AI inference path.

2. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S1, spatial demodulation of the raw radio frequency signal sequences of multiple pre-deployed reference tags yields the reference radio frequency spatial coordinates, including: The system periodically polls the original radio frequency signal sequences transmitted by the multiple reference tags through multiple antennas and extracts the Received Signal Strength Indication (RSSI) timing sequence from the original radio frequency signal sequences. The RSSI time series is subjected to moving average filtering to obtain a standard RSSI sequence; Based on a preset field strength propagation attenuation model, the standard RSSI sequence is converted into the relative distances between multiple reference tags and each antenna. Using a trilateration algorithm, geometric intersection is calculated based on the relative distance, and the three-dimensional homogeneous geometric coordinates of multiple reference tags in the physical field are output as reference radio frequency spatial coordinates.

3. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S1, calculating the multimodal spatial perspective transformation matrix from the reference radio frequency spatial coordinates to the image pixel coordinates includes: Control the industrial camera to acquire calibration image frames containing multiple reference tags; The image pixel coordinates of multiple reference labels in the calibration image frame are identified using an object detection algorithm. By combining the reference radio frequency spatial coordinates of the multiple reference tags with the corresponding image pixel coordinates, a set of spatial linear mapping equations in a homogeneous coordinate system is constructed. The reprojection error of the spatial linear mapping equations is iteratively optimized using the least squares method to solve for the multimodal spatial perspective transformation matrix.

4. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S2, binding the candidate's identity information with the unique identifier of the mobile tag includes: The system reads the candidate's electronic certificate information through a data interface and simultaneously collects the candidate's real-time biometric data. The real-time biometric data is compared with the registration data in the electronic certificate information to verify the candidate's physical identity; When the comparison result meets the preset consistency judgment conditions, the RFID reader is controlled to read the unique identifier of the mobile tag worn by the examinee in the near-field training area. The electronic certificate information, the unique identifier of the mobile tag, and the current assessment item are associated and stored in a temporary database to complete the cascading binding of the candidate's identity information and the unique identifier of the mobile tag.

5. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S3, the synchronous acquisition of the original radio frequency signal sequence and motion video frames with the same hardware timestamp includes: A globally unique microsecond-level hardware clock source is generated by a unified clock synchronization module; In the physical driving layer of the industrial camera's exposure output motion video frames, the hardware clock source is injected as a visual frame hardware timestamp into each motion video frame. In the baseband data layer of the RFID reader demodulates the original radio frequency signal sequence, the hardware clock source is synchronously injected as a radio frequency hardware timestamp into each radio frequency data packet of the original radio frequency signal sequence.

6. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S3, the time sliding window, after timing alignment, includes: A dynamic sliding time window is established with a preset time width; Extract the visual frame hardware timestamp and radio frequency hardware timestamp within the dynamic sliding time window; The original radio frequency signal sequence with the smallest hardware timestamp deviation between the two channels is bound and registered one-to-one with the corresponding motion video frame. If radio frequency packet loss occurs within a specific dynamic sliding time window, an interpolation algorithm is invoked to perform timing feature fitting and completion based on the original radio frequency signal sequences of adjacent windows.

7. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S3, the step of using a multimodal spatial perspective transformation matrix to map the real-time radio frequency spatial coordinates to the pixel coordinate system of the moving video frame to lock onto the target examinee includes: The multimodal spatial perspective transformation matrix is ​​used to convert the real-time radio frequency spatial coordinates into mapped pixel coordinates in the motion video frame; A multi-target tracking algorithm was used to extract the motion trajectory of the centroid of the human skeleton of each examinee in the motion video frame; Calculate the Pearson correlation coefficient between the temporal trajectory of the mapped pixel coordinates and the motion trajectory of each human skeletal centroid; When the Pearson correlation coefficient is greater than the set consistency threshold, the corresponding human skeleton model is locked as the target candidate.

8. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S4, the visual AI inference pathway outputs preliminary judgments on the behavioral norms of the target examinee, including: The motion video frames are input into a preset edge pose estimation model to extract the coordinates of multiple human core topological skeleton key points and the corresponding key point confidence scores of the target examinee. Based on the changes in the coordinates of multiple key points of the human core topology skeleton over time, the dynamic joint angles, displacement amplitudes of key parts, and movement trends of the target examinee during the movement process are continuously calculated. The calculated joint dynamic angle, displacement amplitude of key parts and motion trend are compared with the preset standard motion feature templates in the local rule base in real time. Based on the comparison results, determine the compliant items and violation types of the target candidate's actions within the current action cycle, and generate a preliminary judgment result on the standardization of actions.

9. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S4, the RFID rule verification path outputs the preliminary judgment results of the action count for the target examinee, including: The original radio frequency signal sequence of the target candidate was digitally filtered to extract the periodic characteristic curve of the Received Signal Strength Indicator (RSSI). The peak detection algorithm is used to capture the time-series peaks and troughs in the RSSI periodic characteristic curve to determine the initial number of action cycles for the target examinee. Synchronously acquire hard timing activation status data of low-frequency point triggers deployed at specific physical boundaries of the assessment site; By combining the initial number of action cycles with the hard-time activation state data, a boundary closed-loop verification is performed to determine whether the target examinee accurately passes through a specific physical boundary within a single action cycle, and the initial action count result is output.

10. The training and evaluation method based on RFID and visual multimodal fusion according to claim 1, characterized in that, In S5, the real-time assessment of environmental conditions based on the confidence levels of the initial judgment results of action standardization and action count includes: Extract the average confidence score of the coordinates of each human core topological skeleton key point in the preliminary judgment result of the action standardization, and extract the radio frequency stability index of the original radio frequency signal sequence in the current time period. When the average confidence score is greater than the first judgment threshold and the radio frequency stability index is greater than the second judgment threshold, the current environmental condition is determined to be a normal operating condition. When the average confidence score is lower than or equal to the first determination threshold for a consecutive preset number of frames, the current environmental condition is determined to be a visual failure condition. When the radio frequency stability index is lower than or equal to the second determination threshold, the current environmental condition is determined to be an abnormal radio frequency condition.

Images