Multi-mode human body activity data acquisition based on RFID and RGB

Abstract

The invention provides an automatic human body activity data acquisition and processing system based on RFID (Radio Frequency Identification) and RGB (Red, Green and Blue) multi-modal data. The system hardware comprises an Impinj Speedway R420 RFID (Radio Frequency Identification Device) reader, a Lird S9028 PCR (Polymerase Chain Reaction) antenna, 16 E41C Impinj tags and a high-definition camera. According to the software part, PyQt5 is adopted to create a user interface, OpenCV is utilized to process video data, and synchronous acquisition and processing of RFID and RGB data are realized through a multi-thread architecture. According to the system, firstly, space-time information of human body activities is collected from RFID and RGB data sources, a video timestamp is identified by using a YOLOv8 model, and high-precision time synchronization is realized through noise reduction of an RANSAC algorithm. The processed data comprises dynamic response of skeleton nodes and time alignment information of active video frames, and a data synchronization error is controlled at a millisecond level. And a system interface provides real-time skeleton node display and data acquisition quantity feedback, so that the visualization and integrity of data acquisition are ensured. The system has good robustness and real-time performance in a complex environment, and is suitable for human body activity identification application in the fields of intelligent monitoring, behavior analysis and the like.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, wireless communication and human-computer interaction. Its applications include medical monitoring, smart home, security monitoring, workplace safety, etc. By combining RGB visual data and RFID signals, the accuracy and robustness of human activity recognition can be improved. This fusion technology can cope with occlusion problems in complex environments and enhance detection capabilities under low-light conditions, and has a wide range of practical application value. Background Art

[0002] With the widespread application of smart devices, human activity recognition has gradually become one of the core technologies in the fields of human-computer interaction, smart security, medical rehabilitation, etc. However, traditional human activity recognition systems often rely on a single-modal data source (such as RGB video or inertial sensor), which is easily affected in complex or harsh environments, resulting in reduced recognition accuracy. In order to improve the robustness and recognition accuracy of the system, multimodal data fusion technology has emerged. RFID (radio frequency identification) technology is a technology that obtains object location information through radio waves. Its biggest advantage is that it is not affected by environmental factors such as light and occlusion. Therefore, RFID technology can make up for the shortcomings of the visual system under conditions such as weak light and complex occlusion, and provide more dimensional motion information for human activity recognition. Compared with the visual system, RFID signals can also pass through objects, so that they can still obtain position information when they are blocked. RGB cameras are widely used in the field of human activity recognition. They can directly obtain the visual features of human external movements and capture rich spatial and posture information. However, the recognition effect of the RGB system is easily reduced when there is insufficient light or occlusion. The combination of RGB and RFID can improve the recognition ability in complex scenes through the complementary advantages of multimodal data. Although the combination of RFID and RGB has significant advantages, the data characteristics of the two are quite different, including different acquisition frequencies, spatial resolutions, and data formats. Therefore, how to synchronize and fuse the data of these two modalities in time and space has become a research focus. In summary, the multimodal human activity data acquisition technology based on RFID and RGB overcomes the limitations of a single data source by combining information from the two modalities. In the fields of intelligent monitoring, medical rehabilitation, etc., this fusion technology shows broad application prospects. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In order to solve the problems raised in the above background technology, the present invention adopts the following technical solutions.

[0005] like Figure 1 As shown in the figure, the hardware components of the multimodal human activity data collection system based on RFID and RGB include an Impinj Speedway R420 reader, a Laird S9028PCR antenna, 16 E41C Impinj tags, and a camera. In the software part, we use PyQt5 to create a user graphical interface, use OpenCV to process videos, and call Java classes through the jpype library. The system uses multi-threading to process RFID and RGB data collection, realizing an efficient and automated multimodal data collection and processing system.

[0006] like Figure 2 As shown in the figure, the flowchart shows a real-time RFID and RGB data acquisition and processing system for human motion recognition. The process starts with the data acquisition stage, where human motion data is synchronously collected through RFID tags and cameras. The RFID tag and antenna transmit the signal to the card reader, and the camera records the video in real time and inserts the timestamp in the video frame. The collected raw RFID and video data then enter the processing stage. In the data processing stage, the RFID timestamp is first extracted, the timestamp in the video is extracted using YOLOv8, and the video timestamp is denoised using the RANSAC algorithm. The system then synchronizes the timestamps of the RFID and video to ensure accurate alignment of the two modal data. The processed RFID, video data, tags and timestamps are then saved. In addition, the system can display the skeleton nodes of the tag signal response in real time and show the amount of collected data, thereby improving the visualization effect of data monitoring and collection integrity feedback.

[0007] Compared with the prior art, the present invention has the following beneficial effects:

[0008] (1) The present invention supports a multi-threaded data collection architecture, and RFID and RGB data collection can be performed simultaneously, making data processing close to real-time. During the collection process, users can repeat the collection operation multiple times, which is suitable for scenarios with high real-time monitoring requirements, thereby greatly improving the flexibility and efficiency of data.

[0009] (2) The present invention achieves high-precision spatiotemporal alignment by integrating automatic acquisition software and utilizing the timestamp synchronization mechanism of RFID tag data and RGB video, ensuring that the two data modalities are integrated within the millisecond synchronization error range. Compared with the problem that traditional multimodal systems are difficult to effectively synchronize data, this synchronization mechanism greatly improves data consistency.

[0010] (3) Simplify the operation process and improve the collection efficiency. Users only need to click the "RFID" and "Open Camera" buttons to complete multimodal data collection. This design significantly reduces the difficulty of operation and the probability of error, making the collection process more convenient and efficient, and meeting the needs of tasks that require a large amount of real-time data.

[0011] (4) The system supports real-time display of RFID tag signal response data, dynamically presents the status of nodes, and can display the amount of collected data, helping users to monitor the progress and amount of data collection in real time. This design improves the user's visualization of system operations and the convenience of data management, making the data collection process more intuitive and controllable. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is composed of the acquisition system of the present invention.

[0013] Figure 2 This is a flow chart of automated data collection according to the present invention. DETAILED DESCRIPTION

[0014] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0015] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0016] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments. The present invention provides the following embodiments.

[0017] like Figure 1 As shown in the figure, the hardware components of the multimodal human activity data acquisition system based on RFID and RGB include:

[0018] Impinj Speedway R420 reader: This is a high-performance RFID reading device that reads the data signal of the RFID tag on the human body and provides positioning data for the node position of the human skeleton.

[0019] Laird S9028PCR antenna: This antenna is connected to the RFID reader to receive the reflected signal from the RFID tag in the space. The direction and position of the antenna are optimized to cover the target activity area, ensuring stable reception of the tag signal throughout the activity space.

[0020] 16 E41C Impinj tags: These tags are attached to key locations on the human body (such as limbs, torso, head, etc.). Each tag has a unique ID and provides real-time location information of each skeleton node through the received signal strength indication (RSSI) and timestamp.

[0021] HD camera: used to synchronously record RGB video data of the human body. The camera resolution is set to 1920x1080 and the frame rate is 30 frames per second to ensure clear images. The fixed angle of the camera is designed to cover the activity area and avoid visual blind spots, so as to capture complete activity data.

[0022] In the software part, the system uses PyQt5 to create a user graphical interface, which allows users to easily monitor and operate the data acquisition process. The graphical interface contains operation buttons, status indicators, and real-time data display windows to enhance the user experience and intuitive operation. The OpenCV library is used to process the RGB video collected by the camera, including video stream processing, frame extraction, image preprocessing, and timestamp embedding. In addition, the system calls Java classes through the jpype library to interact with external Java modules in order to perform complex data processing tasks and improve system compatibility. The system uses a multi-threaded architecture to achieve synchronous collection and processing of RFID and RGB data, so that each step of data collection can run efficiently and independently, ensuring an efficient and automated data collection process.

[0023] like Figure 2 As shown in the figure, the data collection and processing flow of the system includes the following steps:

[0024] 1. Data collection stage

[0025] The system first starts data collection from RFID and RGB cameras. The RFID tag transmits location information to the Laird S9028PCR antenna through reflected signals, and the signal received by the antenna is transmitted to the system through the Impinj Speedway R420 reader. Each RFID tag carries information such as timestamp and RSSI (signal strength), allowing the system to track the position of the human skeleton nodes in real time based on these data. At the same time, the high-definition camera synchronously records the video of human movements. The camera inserts timestamp information in each frame of video, so that the RGB video data has a time tag, which is convenient for time alignment with the RFID signal.

[0026] 2. Data processing stage

[0027] After data collection is completed, the system enters the data processing stage. The processing steps are as follows:

[0028] RFID timestamp extraction: Extract timestamps from RFID data streams, record the signal response time of each tag, and provide a time reference for data synchronization.

[0029] Video timestamp extraction and noise reduction: The system uses the YOLOv8 model to identify and extract the timestamp of each frame from the RGB video. The timestamp in the video may contain noise. To ensure the stability of the timestamp, the system uses the RANSAC (random sampling consensus) algorithm to reduce the noise of the timestamp. This algorithm can effectively filter out the timestamp deviation caused by video frame damage or blur, ensuring the accuracy of time information.

[0030] Data synchronization: After completing timestamp extraction and noise reduction, the system synchronizes RFID and RGB data based on timestamps. Through precise alignment of timestamps, the RFID tag location data is accurately matched with the activity data in the video frame, achieving time synchronization of the two modal data.

[0031] 3. Data storage and display

[0032] After completing data processing, the system stores the processed RFID data, RGB video frames, tag IDs and timestamp data to form a complete, time-space aligned multimodal data record. The system supports real-time data visualization:

[0033] Skeleton node display: The system displays the signal response position of each RFID tag in real time, dynamically presenting the skeleton node structure of the human body. This enables users to intuitively view the movement status of each node, monitor the response of each tag, and ensure the integrity of the data.

[0034] Data collection volume display: The system interface will display the current amount of data collected, including the number of records for each modality, providing users with feedback on the progress of data collection, making it easier to monitor the completion of overall data collection.

[0035] In summary, the system provides an automated and reliable human activity data collection and processing platform through efficient synchronous collection of RFID and RGB multimodal data, real-time skeleton node display and data volume feedback.

[0036] The above content is a further detailed description of the present invention in combination with specific implementation methods. It cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the scope of protection determined by the claims submitted for the present invention.

Claims

1. A multimodal human activity data collection and processing system based on RFID and RGB. The present invention belongs to the fields of computer vision, wireless communication and human-computer interaction, and is suitable for scenes such as human activity recognition, intelligent monitoring, behavior analysis, etc., especially in complex environments such as medical monitoring, smart home and security monitoring. Existing human activity recognition technology is often difficult to achieve high-precision recognition in complex environments. Single data sources such as RGB video are easily affected by occlusion and insufficient light, while RFID technology can make up for these defects. The present invention provides an automated human activity data collection and processing system based on RFID and RGB multimodal data. The hardware part includes an Impinj Speedway R420 RFID reader, Laird S9028PCR antenna, 16 E41C Impinj tags and a high-definition camera; the software part uses PyQt5 to create a user interface, and combines OpenCV and jpype libraries to realize the synchronous collection and processing of multimodal data. The system uses the YOLOv8 model to identify the timestamp in the video, reduces noise through the RANSAC algorithm to ensure high-precision time synchronization, and displays the skeleton node data and collection progress in real time. The present invention provides an automated human activity data collection and processing system, which, through the spatiotemporal alignment and visual display of multimodal data, is suitable for intelligent monitoring, behavior analysis and other data collection scenarios that require high precision and real-time performance.

2. The hardware part of the multimodal human activity data acquisition and processing system based on RFID and RGB includes an RFID reader, an RFID antenna, multiple RFID tags and a camera, and is characterized by: a) RFID tags are attached to human joints and can collect human skeleton data in real time; b) The camera synchronously collects RGB video data of human activities and embeds timestamp information in the video frames; c) The system can extract the timestamp information of the video frame through the YOLOv8 model and use the RANSAC algorithm to perform timestamp denoising to achieve high-precision data synchronization.

3. The multimodal human activity data acquisition and processing device according to claim 1, characterized in that: The system uses a multi-threaded architecture to synchronously process RFID data and RGB video data to achieve near real-time multimodal data acquisition.

4. The multimodal human activity data acquisition and processing device according to claim 2, characterized in that: The system user interface is built based on PyQt5, which provides operation buttons and status indicators, displays the amount of data collected in real time, and can dynamically present the skeleton nodes of RFID tags.

5. The multimodal human activity data acquisition and processing device according to claim 3, characterized in that: The system can store the processed RFID and RGB data and synchronized timestamp information to form a multimodal data record to support subsequent behavior analysis and human activity recognition.

6. The multimodal human activity data acquisition and processing device according to claim 4, characterized in that: The data processing module can achieve spatiotemporal synchronization of RFID and RGB data within millisecond error range, improving data consistency and fusion effect.

Images