Self-adaptive book checking system and method based on RFID and vision bimodal fusion

Abstract

The invention provides a self-adaptive book checking system and method based on RFID and vision bimodal fusion, and aims to solve the problems that in an existing checking system, a single technology is poor in robustness and low in recognition rate in a complex library environment, and accurate wrong shelf / missing scanning judgment cannot be achieved. According to the method, real-time detection and active probing of book missing scanning are achieved through bimodal sequence cross validation, metal interference is dynamically diagnosed through mobile scanning RFID signal profile analysis, the contribution degree of RFID and visual recognition is adaptively adjusted through a real-time quality evaluation and dynamic weight fusion mechanism, and an autonomous learning mechanism is introduced to continuously optimize system performance. According to the system, the technologies are integrated on a mobile checking robot platform, a four-wheel omnidirectional mobile chassis and a controllable lifting device are utilized, path planning is performed in combination with laser navigation, high-precision, high-robustness, self-adaptive and evolvable automatic book checking in a complex indoor environment is finally realized, and the checking efficiency and the data integrity are remarkably improved.

Description

Technical Field

[0001] This application relates to the fields of library automation management, robotics and Internet of Things applications, and in particular to an adaptive book inventory system and method based on the fusion of RFID and visual dual-modal approaches. Background Technology

[0002] With the development of smart libraries, automated book inventory technology is gradually replacing time-consuming and labor-intensive manual inventory. Currently, the mainstream solutions are mainly based on RFID technology or machine vision technology, or a simple combination of both.

[0003] RFID-based inventory robots (such as CN105512593B) can read tags in batches without contact, which is highly efficient. However, near metal bookshelves commonly found in libraries, electromagnetic waves are severely reflected or absorbed, leading to tag reading failures (missed reads) or signal distortion (misreads), a problem known as "metal interference." Existing solutions, such as deploying multi-antenna arrays for positioning (such as CN114997193B), are costly and complex to modify; while pre-set interference maps cannot adapt to the dynamic changes caused by the movement of books.

[0004] Vision-based inventory checks use OCR to identify book spine information, which is unaffected by metal, but is highly sensitive to lighting, occlusion, book color, and spine wear, resulting in insufficient recognition stability. Simple combinations of RFID and vision (such as the matching mentioned in CN120765916A) typically employ fixed rules or simple thresholds at the decision-making level, failing to cope with complex and changing field environments. For example, in areas with metallic interference, there may still be over-reliance on invalid RFID signals; and when vision is obstructed, valid RFID information may be missed.

[0005] In addition, existing systems rely heavily on post-inventory comparison with the database to address the "missed scan" problem during the inventory process, lacking real-time detection and immediate recovery capabilities during the inventory process, which leads to the accumulation of errors or the need for rework.

[0006] Therefore, there is an urgent need for a robust book inventory system that can dynamically sense environmental changes, intelligently assess the reliability of data from various modalities, adaptively adjust identification strategies, and self-verify and repair itself during the process. Summary of the Invention

[0007] The purpose of this application is to solve the technical problems of lag and passivity in missed scan detection, dynamic interference from metal bookshelves, rigidity of multimodal fusion, and lack of adaptive capability in the existing technology.

[0008] To achieve the above objectives, this application provides the following technical solution.

[0009] An adaptive book inventory method based on RFID and visual dual-modal fusion, characterized by:

[0010] S1: Dynamic environment perception and synchronous data acquisition: The robot moves along the planned inventory path, uses laser navigation sensors and UWB / SLAM fusion for precise positioning, and simultaneously acquires continuous signal streams from book RFID tags and bookshelf image streams.

[0011] S2: Real-time detection and recovery of missed scans based on dual-modal sequence cross-validation: The RFID tag ID sequence obtained in the same bookshelf channel and the book title sequence recognized by vision are time-series aligned and associated through a dynamic time warping algorithm; anomalies that exist only in a single modality are identified and marked to generate a list of suspected missed scans; the robot's active exploration mechanism is triggered to perform a secondary scan of the suspicious area by adjusting the RFID antenna power, the height of the lifting device, or the robot's micro-motion pose to confirm or recover the missed scan information;

[0012] S3: Metal Interference Diagnosis Based on Mobile Scanning RFID Signal Profile Analysis: For each continuously read book tag, a spatiotemporal profile curve of its RSSI and phase signals as a function of robot displacement is constructed; feature values ​​characterizing metal interference are extracted from the profile curve, including RSSI curve concavity, phase curve disorder, and profile noise level; by comparing the feature values ​​of adjacent book tags, a statistical outlier detection algorithm is used to dynamically diagnose whether the current book is in a metal interference area and output the interference confidence level. ;

[0013] S4: Dynamic weight fusion of RFID and visual recognition results: Real-time calculation of RFID read quality score Quality score of visual recognition Based on the current confidence level of metallic interference in the book Historical identification variance and dynamic calculation of fusion weights and and satisfy Final fusion recognition result The weighted decision formula is used to determine the status of the books by comparing it with the background database.

[0014] S5: Online autonomous learning and optimization of system performance: During the inventory process, the system automatically collects metal interference features and diagnostic results, integrates weights and actual identification success and failure cases, and miss scan verification results as training samples; through online learning or periodic batch learning, it updates the metal interference diagnostic model, weight adjustment strategy parameters and signal attenuation prediction model, so as to achieve the system's adaptation and continuous performance optimization for specific library environments.

[0015] Preferably, the active probing mechanism in S2 includes two operations:

[0016] The first method: If the RFID misses a scan but the visual system identifies a suspicious book, the system controls the robot to pause its movement and also controls the lifting device to raise or lower it, changing the angle and distance between the RFID antenna and the book tag. The RFID reader then rescans the area with higher power.

[0017] The second method: If the visual scanner misses a scan but the RFID identifies a suspicious book, the system controls the robot to fine-tune its own posture or adjust the status of the supplementary lighting. The visual sensor captures images of the suspicious missing position from multiple angles and identifies it.

[0018] Preferably, the concavity of the RSSI curve in S3 is... The calculation formula is:

[0019]

[0020] in, It is the expected strength predicted based on the signal attenuation model in an interference-free environment. It is the minimum value in the measured profile curve. It is the standard deviation of the overall signal profile of the book;

[0021] The disorder degree of the phase curve is calculated by the number of phase jumps and the variance of the local gradient in the system statistical profile. A phase jump refers to a phase change that exceeds π radians. The disorder degree is then quantified using these two data.

[0022] Preferably, the fusion weights in S4 The dynamic calculation is specifically as follows:

[0023]

[0024] in, The basic weights are calculated based on the confidence variance of RFID and visual historical recognition. The output of step S3 is the confidence level of the metal interference. It is based on the current real-time quality difference ( - Small adjustments were made. =1- .

[0025] Preferably, the online autonomous learning in S5 uses an incremental learning algorithm to update a lightweight neural network classifier. The input of the classifier is the RFID signal profile feature vector extracted in step S3, and the output is the probability of metal interference. The learning samples are automatically labeled by the system, and the labeling criteria are as follows: when the dynamic weight fusion result is highly dependent on vision and the final recognition is successful, the RFID signal feature at that moment is marked as "interference"; otherwise, if it depends on RFID and the recognition is successful, it is marked as "normal".

[0026] This application also provides an adaptive book inventory system based on the fusion of RFID and visual dual modes, applied to the inventory method described above. The inventory system includes an inventory robot body, on which are mounted a four-wheel omnidirectional mobile chassis, a controllable lifting device, and a laser navigation sensor.

[0027] Preferably, the inventory robot body is also equipped with a book collection module and a data processing and control core.

[0028] Preferably, the data acquisition module includes an RFID reader and its connected antenna array and high-definition vision sensor, wherein the antenna array is mounted on a controllable lifting device.

[0029] Preferably, the data processing and control core includes an embedded all-in-one machine, which runs the following units:

[0030] Environmental modeling and localization unit: integrates laser, vision and UWB data for SLAM mapping and real-time localization;

[0031] Sequence cross-validation and missed detection handling unit: performs timing alignment and active detection control of S2;

[0032] Signal profile analysis unit: performs feature extraction and metal interference diagnosis for S3;

[0033] Dynamic quality assessment and fusion unit: performs S4 weight calculation and decision fusion;

[0034] Online learning optimization unit: Performs S5 self-learning model updates;

[0035] Path planning and control unit: Generates the globally optimal path based on the inventory task and drives the robot's movement.

[0036] Preferably, the antenna array consists of two RFID antennas arranged vertically on the guide rail of the controllable lifting device for simultaneously scanning bookshelf shelves of different heights; the visual sensor is a global shutter camera deployed on the surface of the RFID antennas and equipped with an adjustable brightness fill light.

[0037] Compared with the prior art, this application has at least the following beneficial effects:

[0038] 1. This application uses software algorithms to dynamically diagnose interference without requiring hardware modifications. It has a high diagnostic accuracy and can use the interference information for subsequent fusion decisions, turning "failure" into "effective information".

[0039] 2. This application can detect missed scans in real time during the scanning process through sequence cross-validation, and actively explore and recover by combining the mobility of the robot platform, which greatly improves the integrity and accuracy of single inventory data and reduces rework.

[0040] 3. This application ensures that the system tends to trust the more reliable data source at any time through a dynamic weight fusion mechanism, so that the overall recognition rate remains stable and at a high level under complex scenarios such as changes in lighting, partial occlusion, and metal interference.

[0041] 4. This application enables the system to accumulate localized knowledge through an online learning mechanism, becoming "smarter" the more it is used, and adapting to the unique environments of different libraries, thereby reducing the debugging and maintenance costs after deployment.

[0042] 5. This invention upgrades the algorithm based on existing mature inventory robot hardware, resulting in low modification costs and easy promotion and application, providing a complete solution with high precision and high intelligence for smart libraries. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the overall inventory process of an adaptive book inventory method based on RFID and visual dual-modal fusion in one embodiment of this application.

[0044] Figure 2 This is a flowchart illustrating the real-time detection and recovery process for missed scans in an adaptive book inventory method based on RFID and visual dual-modal fusion, as described in one embodiment of this application.

[0045] Figure 3 This is a flowchart illustrating the bimodal fusion decision-making process of an adaptive book inventory method based on RFID and visual bimodal fusion in one embodiment of this application. Detailed Implementation

[0046] Please see Figure 1 This application provides an adaptive book inventory method based on RFID and visual bimodal fusion, using a mobile inventory robot. The steps of the inventory method are as follows:

[0047] S1: Dynamic environment perception and synchronous data acquisition: The robot moves on the planned inventory path, and performs precise positioning by fusing laser navigation sensors with UWB / SLAM. It also triggers RFID readers and vision sensors simultaneously to collect continuous signal streams (including RSSI and phase information) from the book RFID tags and bookshelf image streams, respectively.

[0048] In one implementation, an RFID reader continuously reads tags at a high frequency (100Hz), recording streaming data including timestamps, tag EPC, Received Signal Strength Indication (RSSI), and phase. A camera captures a video stream of the bookshelf at a fixed frame rate (15fps), with each frame accompanied by a timestamp and the corresponding robot pose. Hardware synchronization or timestamp alignment ensures that the RFID data stream and the image stream are spatiotemporally correlated.

[0049] S2: Real-time detection and recovery of missed scans based on dual-modal sequence cross-validation: The RFID tag ID sequence obtained in the same bookshelf channel and the book title sequence recognized by vision are time-series aligned and associated through a dynamic time warping algorithm; anomalies that exist only in a single modality are identified and marked to generate a list of suspected missed scans; the robot's active exploration mechanism is triggered to perform a secondary scan of the suspicious area by adjusting the RFID antenna power, the height of the lifting device, or the robot's micro-motion pose to confirm or recover the missed scan information;

[0050] The active detection mechanism includes: for suspicious books that are missed by RFID scanning but are visually present, controlling the robot to pause, raise or lower the controllable lifting device, change the angle and distance of the RFID antenna relative to the book tag, and rescan the area with higher power; for suspicious books that are missed by visual scanning but are RFID present, controlling the robot to fine-tune its posture or adjust the supplementary light, and perform multi-angle image capture and identification of the suspicious missing positions.

[0051] For details, please refer to Figure 2 In one embodiment, the specific steps of S2 are as follows:

[0052] Timing alignment: Treat a bookshelf aisle as a scanning sequence. Obtain the EPC sequence read by RFID and the book title sequence recognized by vision.

[0053] RFID Sequence: A list of EPCs sorted by read time [EPC1, EPC2, ..., EPC] m ].

[0054] Visual sequence: A list of recognized book titles ordered by spatial location in an image [Title1, Title2, ..., Title...] n ].

[0055] Since the two sensors have different acquisition frequencies and response times, a dynamic time warping algorithm is used to flexibly align the two sequences and establish a matching pair of "RFID tag-visual book".

[0056] The dynamic event warping algorithm can overcome the differences in length and acquisition time between the two, find the optimal matching path, and establish the correspondence between RFID sequence and visual sequence.

[0057] Anomaly detection: After alignment, three types of cases were identified:

[0058] (A) Successful match: Both modalities are recognized and their contents have a corresponding relationship in the database;

[0059] (B) Visual Missed Items: There is no corresponding matching item for a certain EPC in the RFID sequence in the visual sequence;

[0060] (C) RFID Missed Scan Items: A book title in the visual sequence has no corresponding matching item in the RFID sequence.

[0061] B and C are suspected missed scans.

[0062] Active detection and recovery: The system immediately marks these suspicious locations. The robot is then controlled to execute a recovery strategy: For Category B (visually missed scans), which may be due to slight obstruction by the book or poor lighting, the robot can fine-tune its posture and retake the image. Specifically, in one implementation, the robot's pose is slightly adjusted (e.g., translated 10 cm or rotated 5 degrees), changing the camera's angle of view; the brightness of the supplementary lighting is adjusted to the optimal level. Two to three images are retaken and then recognized. If recognition is successful, the visual sequence is updated.

[0063] For Category C (RFID missed scans), which may be due to metallic interference or damaged tags, the robot pauses its movement, controls the lifting device to move the antenna up and down, or increases the reader power for targeted enhanced scanning. This process involves millisecond-level decision-making, allowing for local verification and information completion before moving to the next bookshelf. Specifically, in one implementation, the robot pauses its movement, and the path planning unit marks the current position. The control unit drives the lifting device to move the RFID antenna up and down (e.g., ±5 cm) within the estimated height range of the suspected book. Simultaneously, the RFID reader increases its transmission power to 120% of the maximum allowable value (for a short time) for focused scanning of that area. If a tag is successfully read, the RFID sequence is updated, and the pair is marked as a successful match.

[0064] S3: Metal Interference Diagnosis Based on Mobile Scanning RFID Signal Profile Analysis: For each continuously read book tag, a spatiotemporal profile curve of its RSSI and phase signals as a function of robot displacement is constructed; feature values ​​characterizing metal interference are extracted from the profile curve, including RSSI curve concavity, phase curve disorder, and profile noise level; by comparing the feature values ​​of adjacent book tags, a statistical outlier detection algorithm is used to dynamically diagnose whether the current book is in a metal interference area and output the interference confidence level. ;

[0065] Specifically, in one embodiment, S3 includes the following steps:

[0066] Profile generation: For each tag that is continuously read, the RSSI and phase values ​​are plotted as a function of robot displacement (converted from laser navigation data) using its EPC as an index, i.e., the “signal space profile”.

[0067] In one implementation method, taking a continuously read book as an example, for the tag EPC A During the robot's movement, 10 valid data points were continuously read.

[0068] Using the robot's travel distance (obtained by integrating laser positioning data) as the abscissa and the RSSI value (dBm) and phase value (radians) as the ordinates, two discrete points are plotted to form a curve, known as the "signal space profile".

[0069] Feature extraction:

[0070] RSSI curve concavity: Metal reflections cause deep signal fading. Calculate the degree of deviation between the lowest point of the curve and the expected smooth attenuation curve.

[0071] The concavity of the RSSI curve The calculation formula is:

[0072]

[0073] in, It is the expected strength predicted based on the signal attenuation model in an interference-free environment. It is the minimum value in the measured profile curve. It is the standard deviation of the overall signal profile of the book.

[0074] Phase curve disorder: Metals cause multipath effects, leading to phase disorder. Calculate the phase curve's transition frequencies and local variance; phase curve disorder. The calculation is quantified by the number of phase jumps (changes exceeding π radians) in the statistical profile and the variance of the local gradient.

[0075] In one embodiment, the calculation method is as follows: traverse the phase profile points and count the number of "jumps" where the absolute value of the phase difference between adjacent points exceeds π radians. Simultaneously, the variance of the first-order difference (gradient) of the phase curve is calculated. .

[0076] Disorder Quantifiable as (α, β are preset weighting coefficients).

[0077] Noise level: directly calculate the variance of the entire RSSI profile data. The specific formula is:

[0078]

[0079] in, For the first The received signal strength indication value at each sampling point This is the arithmetic mean of all RSSI samples for this label. This represents the total number of valid sample points read for this label in the current profile.

[0080] Local contrast diagnosis: Extract common features from several adjacent books to the left and right of the target book. If the feature value of the target book is a significant outlier (the indentation degree is more than twice the standard deviation of the neighbor mean), it is determined to be affected by metallic interference. Output an interference confidence score between 0 and 1. .

[0081] Specifically, in one implementation, the current book (EPC) is extracted. A The common characteristics (depression, disorder, noise level) of the three books on each side.

[0082] Calculate the mean and standard deviation of each characteristic of these 6 adjacent books.

[0083] Determine EPC A We determine whether each feature value is an outlier. For example, if its concavity is greater than "neighbor mean + 2 * neighbor standard deviation", then the feature is considered abnormal.

[0084] By considering the anomalies of various features, and using a pre-defined logical rule or a lightweight classifier, a metal interference confidence level between 0 and 1 is output. For example, if two of the three features are abnormal, then The output is 0.8.

[0085] S4: Dynamic weight fusion of RFID and visual recognition results: Real-time calculation of RFID read quality score Quality score of visual recognition Based on the current confidence level of metallic interference in the book Historical identification variance and dynamic calculation of fusion weights and and satisfy Final fusion recognition result The weighted decision formula is used to derive the data and compare it with the background database to determine the status of the book.

[0086] For details, please refer to Figure 3 In one embodiment, S4 includes the following steps:

[0087] Real-time quality assessment:

[0088] RFID quality score Calculated based on the current RSSI intensity, stability, and continuity of phase changes. High-intensity, highly stable signals score higher.

[0089] The target tag's mean RSSI, RSSI stability (inverse variance), and continuity of phase change (low disorder) are calculated. For example:

[0090] ,

[0091] in This is the normalization function.

[0092] Visual quality score The confidence score is given by the OCR model for the text identified in the current image, and is calculated in combination with the image's contrast, sharpness, etc.

[0093] OCR recognition is performed on the target spine region in the current frame image to obtain the text content and its confidence level. Simultaneously, the contrast and sharpness (Laplacian variance) of the image in this region are calculated. For example: .

[0094] Dynamic weight calculation: The weights are not fixed. Base weights The weighting is determined by the long-term historical variance of each modality (smaller variance means greater weight). The key innovation lies in real-time adjustment: It will be based on S2 It is reduced, and fine-tuned according to the current instantaneous mass difference.

[0095] Fusion weights The dynamic calculation is specifically as follows:

[0096]

[0097] in, The basic weights are calculated based on the confidence variance of RFID and visual historical recognition. The output of step S3 is the confidence level of the metal interference. It is based on the current real-time quality difference ( - Small adjustments were made. =1- .

[0098] In one implementation, for the basic weights The variance of the independent identification results from RFID and visual methods was calculated based on the identification records of the past 100 books. The method with smaller variance was assigned a higher base trust. For example, if the historical variance of RFID was 0.1 and that of visual methods was 0.2, then... .

[0099] Weighted decision-making and state determination: final identification result = * + * The result is compared with a preset threshold. If it is greater than or equal to the threshold, the book is confirmed as the result and compared with the database to determine if the book is in the correct position. If it is less than the threshold, the book is marked and awaits manual review.

[0100] Set the threshold to 0.7, if If the value is ≥0.7, the fusion result (such as the book title recognized by OCR or the book title corresponding to the EPC of RFID) will be adopted.

[0101] The adopted book titles will be compared with those in the database.

[0102] Matching and in the correct position: Recorded as "Correctly in stock".

[0103] Matched but the location does not match the database record: the record is "wrong shelf".

[0104] Mismatch or If the value is below the threshold: Record as "Pending review".

[0105] S5: Online autonomous learning and optimization of system performance: During the inventory process, the system automatically collects metal interference features and diagnostic results, integrates weights and actual identification success and failure cases, and miss scan verification results as training samples; through online learning or periodic batch learning, it updates the metal interference diagnostic model, weight adjustment strategy parameters and signal attenuation prediction model, so as to achieve the system's adaptation and continuous performance optimization for specific library environments.

[0106] Specifically, in one implementation, the system uses the decision-making process (features, weights, and results) of each inventory count as a learning sample.

[0107] Interference model learning: When the system finally successfully identifies a book diagnosed as "potential interference" using vision, the RFID signal profile features of the book are reinforced and marked as "interference sample" to update the metal interference classifier.

[0108] Online self-learning employs an incremental learning algorithm to update a lightweight neural network classifier. The input to this classifier is the RFID signal profile feature vector extracted in step S3, and the output is the probability of metal interference. The learning samples are automatically labeled by the system, with the labeling criteria being: when the dynamic weight fusion result is highly dependent on vision (…). If the RFID signal characteristics at that moment are marked as "interference" when the threshold is reached and the identification is successful, then the RFID signal characteristics at that moment are marked as "interference"; otherwise, if relying on RFID ( If the threshold is exceeded and the identification is successful, it is marked as "normal".

[0109] In one implementation, the specific steps are as follows: An online incremental learning algorithm (such as online random forest or Naive Bayes) is employed. When the system finally bases its learning on the visual results (…),… When a book is successfully identified (>0.9), the corresponding feature vector of that book is automatically marked as a positive sample (interference). When the system uses the RFID result ( If a sample with a value >0.9 is successfully identified, it is marked as a negative sample (normal). The classifier model parameters are then periodically updated using the new sample.

[0110] Weighting strategy optimization: In one implementation, historical samples are analyzed in the background to statistically analyze different ( , , Under different weight allocations, the recognition success rate is calculated. Automatic optimization is achieved through regression analysis. The proportionality coefficient in the calculation formula, and The influence factor in the weighting formula.

[0111] Signal attenuation learning: Learning the attenuation patterns of RFID signals under different bookshelf positions and spine materials to more accurately calculate... This improves the accuracy of indentation diagnosis. In one embodiment, successful read data from different bookshelf material areas (wood, metal) are collected to update the empirical model parameters of RSSI decay with distance, thereby improving the accuracy of indentation diagnosis. The predictions are more accurate.

[0112] Upon completion of the task, the system automatically generates an inventory report, which includes:

[0113] Total inventory count, number of items correctly listed, number of items incorrectly listed, and number of items suspected of being lost.

[0114] Detailed information about the misplaced book (correct location, current location).

[0115] Mark the areas of the bookshelf affected by metal interference.

[0116] Provide a list of books "pending review" and possible reasons (such as low fusion score, missed scan recovery failure).

[0117] In addition, this application also provides an adaptive book inventory system based on RFID and visual dual-modal fusion, which is applied to the inventory method described above. The inventory system includes an inventory robot body, on which are provided a four-wheel omnidirectional mobile chassis, a controllable lifting device, and a laser navigation sensor.

[0118] In one embodiment, the inventory robot body is also equipped with a book collection module and a data processing and control core.

[0119] The data acquisition module includes an RFID reader and its connected antenna array and high-definition vision sensor. The antenna array is mounted on a controllable lifting device and consists of two RFID antennas arranged vertically on the guide rail of the controllable lifting device for simultaneously scanning bookshelf shelves of different heights. The vision sensor is a global shutter camera deployed on the surface of the RFID antenna and equipped with an adjustable brightness fill light.

[0120] The data processing and control core includes an embedded all-in-one machine, which runs the following units:

[0121] Environmental modeling and localization unit: integrates laser, vision and UWB data for SLAM mapping and real-time localization;

[0122] Sequence cross-validation and missed detection handling unit: performs timing alignment and active detection control of S2;

[0123] Signal profile analysis unit: performs feature extraction and metal interference diagnosis for S3;

[0124] Dynamic quality assessment and fusion unit: performs S4 weight calculation and decision fusion;

[0125] Online learning optimization unit: Performs S5 self-learning model updates;

[0126] Path planning and control unit: Generates the globally optimal path based on the inventory task and drives the robot's movement.

[0127] This application uses software algorithms to dynamically diagnose interference without requiring hardware modifications, achieving high diagnostic accuracy and enabling the use of interference information for subsequent fusion decisions. Furthermore, sequence cross-validation can detect missed scans in real time during the scanning process and, combined with the mobility of the robot platform, can actively explore and recover, greatly improving the completeness and accuracy of single inventory data.

[0128] The system provided in this application also has long-term learning capabilities. As it performs multiple inventory tasks within the same library, its accuracy in diagnosing metal interference, its dynamic weight allocation strategy, and its success rate in recovering missed scans will continue to improve, gradually adapting to the unique environmental characteristics of the library and achieving the self-evolution goal of becoming "more accurate with use".

[0129] This application also uses a dynamic weight fusion mechanism to ensure that at any given time, the system tends to trust the more reliable data source, so that the overall recognition rate remains stable and at a high level under complex scenarios such as changes in lighting, partial occlusion, and metal interference.

Claims

1. An adaptive book inventory method based on RFID and visual dual-modal fusion, characterized in that: S1: Dynamic environment perception and synchronous data acquisition: The robot moves along the planned inventory path, uses laser navigation sensors and UWB / SLAM fusion for precise positioning, and simultaneously acquires continuous signal streams from book RFID tags and bookshelf image streams. S2: Real-time detection and recovery of missed scans based on dual-modal sequence cross-validation: The RFID tag ID sequence obtained in the same bookshelf channel and the book title sequence identified by vision are time-series aligned and associated with each other through a dynamic time warping algorithm. Identify and mark anomalies that exist only in a single modality, and generate a list of suspected missed scans; trigger the robot's active detection mechanism, and perform a secondary scan of the suspicious area by adjusting the RFID antenna power, the height of the lifting device, or the robot's micro-motion pose, in order to confirm or recover the missed scan information; S3: Metal Interference Diagnosis Based on Mobile Scanning RFID Signal Profile Analysis: For each continuously read book tag, a spatiotemporal profile curve of its RSSI and phase signals as a function of robot displacement is constructed; feature values ​​characterizing metal interference are extracted from the profile curve, including RSSI curve concavity, phase curve disorder, and profile noise level; by comparing the feature values ​​of adjacent book tags, a statistical outlier detection algorithm is used to dynamically diagnose whether the current book is in a metal interference area and output the interference confidence level. ; S4: Dynamic weight fusion of RFID and visual recognition results: Real-time calculation of RFID read quality score Quality score of visual recognition Based on the current confidence level of metallic interference in the book Historical identification variance and dynamic calculation of fusion weights and and satisfy ; Final fusion recognition result The weighted decision formula is used to determine the status of the books by comparing it with the background database. S5: Online autonomous learning and optimization of system performance: During the inventory process, the system automatically collects metal interference features and diagnostic results, integrates weights and actual identification success and failure cases, and miss scan verification results as training samples; through online learning or periodic batch learning, it updates the metal interference diagnostic model, weight adjustment strategy parameters and signal attenuation prediction model, so as to achieve the system's adaptation and continuous performance optimization for specific library environments.

2. The adaptive book inventory method based on RFID and visual dual-modal fusion as described in claim 1, characterized in that: The active probing mechanism in S2 includes two operations: The first method: If the RFID misses a scan but the visual system identifies a suspicious book, the system controls the robot to pause its movement and also controls the lifting device to raise or lower it, changing the angle and distance between the RFID antenna and the book tag. The RFID reader then rescans the area with higher power. The second method: If the visual scanner misses a scan but the RFID identifies a suspicious book, the system controls the robot to fine-tune its own posture or adjust the status of the supplementary lighting. The visual sensor captures images of the suspicious missing position from multiple angles and identifies it.

3. The adaptive book inventory method based on RFID and visual dual-modal fusion as described in claim 1, characterized in that: The concavity of the RSSI curve in S3 The calculation formula is: in, It is the expected strength predicted based on the signal attenuation model in an interference-free environment. It is the minimum value in the measured profile curve. It is the standard deviation of the overall signal profile of the book; The disorder degree of the phase curve is calculated by the number of phase jumps and the variance of the local gradient in the system statistical profile. A phase jump refers to a phase change that exceeds π radians. The disorder degree is then quantified using these two data.

4. The adaptive book inventory method based on RFID and visual bimodal fusion as described in claim 1, characterized in that: The fusion weight in S4 The dynamic calculation is specifically as follows: in, The basic weights are calculated based on the confidence variance of RFID and visual historical recognition. The output of step S3 is the confidence level of the metal interference. It is based on the current real-time quality difference ( - Small adjustments were made. =1- .

5. The adaptive book inventory method based on RFID and visual bimodal fusion according to claim 1, characterized in that: The online autonomous learning in step S5 uses an incremental learning algorithm to update a lightweight neural network classifier. The input of the classifier is the RFID signal profile feature vector extracted in step S3, and the output is the probability of metal interference. The learning samples are automatically labeled by the system. The labeling criteria are as follows: when the dynamic weight fusion result is highly dependent on vision and the final recognition is successful, the RFID signal feature at that moment is marked as "interference"; otherwise, if it depends on RFID and the recognition is successful, it is marked as "normal".

6. An adaptive book inventory system based on RFID and visual dual-modal fusion, characterized in that: The inventory system, which is applied to the inventory method according to any one of claims 1-5, includes an inventory robot body and is equipped with a four-wheel omnidirectional mobile chassis, a controllable lifting device, and a laser navigation sensor.

7. An adaptive book inventory system based on RFID and visual dual-modal fusion as described in claim 6, characterized in that: The inventory robot is also equipped with a book collection module and a data processing and control core.

8. An adaptive book inventory system based on RFID and visual dual-modal fusion as described in claim 6, characterized in that: The data acquisition module includes an RFID reader and its connected antenna array and high-definition vision sensor. The antenna array is mounted on a controllable lifting device.

9. The adaptive book inventory system integrating RFID and visual dual-modal fusion according to claim 6, characterized in that: The data processing and control core includes an embedded all-in-one machine, which runs the following units: Environmental modeling and localization unit: integrates laser, vision and UWB data for SLAM mapping and real-time localization; Sequence cross-validation and missed detection handling unit: performs timing alignment and active detection control of S2; Signal profile analysis unit: performs feature extraction and metal interference diagnosis for S3; Dynamic quality assessment and fusion unit: performs S4 weight calculation and decision fusion; Online learning optimization unit: Performs S5 self-learning model updates; Path planning and control unit: Generates the globally optimal path based on the inventory task and drives the robot's movement.

10. An adaptive book inventory system integrating RFID and visual dual-modal fusion as described in claim 9, characterized in that: The antenna array consists of two RFID antennas, arranged vertically on the guide rail of the controllable lifting device, for simultaneously scanning bookshelf shelves of different heights; the visual sensor is a global shutter camera, deployed on the surface of the RFID antennas, and equipped with an adjustable brightness fill light.

Images