A self-service book return and sorting system and method based on multimodal recognition

Abstract

This invention discloses a self-service book return and sorting system and method based on multimodal recognition, including an information acquisition module, a feature fusion module, a sorting rule module, and a feedback optimization module. It collects multi-dimensional data such as visual information, text information, and RFID information from books. After feature extraction, reliability assessment, and cross-modal semantic alignment, the data is dynamically weighted and fused to generate a fused feature vector. A multi-objective sorting decision function combined with topic clustering determines the sorting target area. Based on the sorting results and feedback from manual verification, the model parameters are iteratively optimized. This invention achieves accurate, efficient, and automatic book sorting, improving sorting flexibility and adaptability, and meeting the dynamic management needs of libraries.

Description

Technical Field

[0001] This invention belongs to the field of book management technology and relates to a self-service book return and sorting system and method based on multimodal recognition. Background Technology

[0002] In the daily operation of libraries, the sorting of books after self-service book returns is a crucial step. Currently, most self-service book return systems rely on RFID tags / barcodes or simple visual recognition technology for sorting. RFID tags and barcodes can only obtain basic metadata such as unique book identifiers, and cannot extract deeper information such as the book's content and theme. Sorting rules are limited to preset location mappings in the library's collection, lacking flexibility.

[0003] Simple visual recognition technologies often focus on single features of book covers or spines, which are easily affected by page wear and tear or printing quality, limiting their accuracy and failing to integrate relevant information about the book content. Existing sorting systems have relatively simple logic, only associating the location of the collection with basic metadata, without considering multi-dimensional factors such as the relevance of book content, real-time borrowing data, the density of the collection area, and the physical condition of the books.

[0004] This leads to a disconnect between sorting results and actual borrowing needs. Popular books are difficult to shelve quickly, similar books are scattered, some collection areas are heavily stocked, and damaged or overdue books cannot be sorted and processed in a timely manner. Ultimately, this affects the accuracy and efficiency of sorting and makes it difficult to meet the needs of automated and dynamic library management. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention proposes a self-service book return and sorting system and method based on multimodal recognition.

[0006] The first aspect of this application provides a self-service book return and sorting system based on multimodal recognition, including: an information collection module, a feature fusion module, a sorting rule module, and a feedback optimization module;

[0007] The information acquisition module is used to collect visual information, text information, radio frequency identification information, borrowing data, physical status information, and real-time dynamic data of the library collection.

[0008] The feature fusion module is used to extract and fuse features from the collected multimodal information to generate a fused feature vector;

[0009] The sorting rules module is used to construct a multi-objective weighted optimization sorting decision function based on the fused feature vector, combine topic clustering to realize dynamic collection area mapping, and output sorting instructions;

[0010] The feedback optimization module is used to iteratively optimize model parameters based on sorting results and manual review feedback through reinforcement learning algorithms.

[0011] Optionally, the information acquisition module includes: a high-definition image acquisition unit, an optical character recognition unit, an RFID reader, a sensor group, and a data linkage unit;

[0012] The high-definition image acquisition unit captures images of the book cover, spine, copyright page, table of contents, and preface; the optical character recognition unit identifies the text content in the images; the radio frequency identification reader obtains the book's unique identifier, historical collection location, and borrowing history; the sensor group detects the book's physical integrity and binding type; and the data linkage unit retrieves real-time collection dynamic data and borrowing data.

[0013] Optionally, the feature fusion module includes: a feature extraction submodule, a reliability assessment submodule, a cross-modal semantic alignment submodule, and a weighted fusion submodule;

[0014] The feature extraction submodule extracts visual feature vectors through a convolutional neural network, extracts text feature vectors through a text encoding model, and normalizes the structured data composed of RFID information and borrowing data to obtain structured feature vectors.

[0015] The reliability assessment submodule calculates the reliability of visual features, text features, and structured features.

[0016] The cross-modal semantic alignment submodule achieves semantic matching between visual and textual data, as well as textual and structured data; the weighted fusion submodule dynamically calculates weights based on the aforementioned reliability, using the formula... Generate a fused feature vector, where, , , V is the visual feature vector, T is the text feature vector, and S is the structured feature vector. For the reliability of visual features, For text feature reliability, For the reliability of structured features.

[0017] Optionally, the sorting rules module includes: a decision function construction submodule and a topic clustering submodule;

[0018] The decision function construction submodule constructs the sorting decision function. ,in, To determine the similarity between the categories of books and the collection areas, To boost book borrowing popularity, To ensure the balance of collection density, Prioritize special statuses of books. , , , The target weight is used; the topic clustering submodule extracts the core topic words of each collection area through an incremental clustering algorithm, calculates the similarity between books and the core topic words of each area, and determines the sorting target area.

[0019] Optionally, borrowing popularity ,in, This refers to the number of times a book is borrowed in a year. This is the normalized value of the number of days since the book was last borrowed, where a and b are weighting coefficients and a+b=1.

[0020] Optionally, the feedback optimization module includes: a reinforcement learning submodule and a human feedback submodule;

[0021] The reinforcement learning submodule models the sorting process as a reinforcement learning environment, with the agent representing the sorting rules module. The state space includes fused feature vectors, real-time collection status, and borrowing data, while the action space is for selecting the sorting target area, and the reward function is... ,in, The borrowing rate is preset within a certain period after the books are sorted. The time required for sorting individual books The sorting error rate is manually verified. , , The reward coefficient is used; the manual feedback submodule receives the error types marked by manual review, adjusts the parameters of the feature fusion module and the sorting rule module accordingly, and expands the error cases to the model training set.

[0022] A second aspect of this application provides a self-service book return and sorting method based on multimodal recognition, comprising:

[0023] The information collection module collects visual information, text information, radio frequency identification information, borrowing data, physical status information, and real-time dynamic data of the library collection.

[0024] The feature fusion module extracts features from the collected multimodal information. After reliability assessment and cross-modal semantic alignment, the weights are dynamically calculated based on visual feature reliability, text feature reliability, and structured feature reliability, using the formula... Generate fused feature vectors;

[0025] The sorting decision function is constructed based on the fused feature vector through the sorting rule module. The system combines topic clustering to determine the sorting target area and outputs sorting instructions. To determine the similarity between the categories of books and the collection areas, To boost book borrowing popularity, To ensure the balance of collection density, Prioritize special statuses of books. , , , The target weight;

[0026] The feedback optimization module optimizes the model parameters iteratively using reinforcement learning algorithms based on the sorting results and manual review feedback.

[0027] Optionally, physical status information includes: physical integrity and binding type of the book; real-time collection dynamic data includes remaining capacity of the target collection area, real-time borrowing status of books on the same topic, and librarian work status; text information includes subject keywords of the table of contents, core research areas of the preface, edition and print quantity of the copyright page.

[0028] Optionally, visual feature vectors are extracted using a convolutional neural network, text feature vectors are extracted using a text encoding model, and structured feature vectors are obtained by normalizing the structured data composed of RFID information and borrowing data; formula Generate a fused feature vector, where, , , V is the visual feature vector, T is the text feature vector, and S is the structured feature vector. For the reliability of visual features, For text feature reliability, For the reliability of structured features.

[0029] Optionally, the core keywords of each collection area are periodically updated using an incremental clustering algorithm, supporting the dynamic creation of temporary thematic areas; sorting decision function. middle, To determine the similarity between the categories of books and the collection areas, , Prioritize special statuses of books. , , , The target weights are a and b, which are weighting coefficients and a + b = 1. This refers to the number of times a book is borrowed in a year. This is the normalized value of the number of days since the book's last borrowing; after each preset batch of books is sorted, a reward function is applied. The cumulative reward is calculated, and the feature fusion weights and sorting decision target weights are updated using a reinforcement learning algorithm. The borrowing rate is preset within a certain period after the books are sorted. The time required for sorting individual books The sorting error rate is manually verified. , , This is the reward coefficient.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] First, improve sorting accuracy. By acquiring multi-dimensional data such as visual information, text information, and RFID information of books through a multi-modal information acquisition module, a comprehensive fusion feature vector is generated through cross-modal semantic alignment and dynamic weighted fusion. Combined with topic clustering and multi-objective sorting decision functions, accurate matching of books with collection areas is achieved, effectively avoiding classification bias caused by single recognition technology.

[0032] Second, improve sorting efficiency. The dynamic sorting rules module dynamically adjusts sorting priorities based on real-time collection data and borrowing popularity. Popular books can be quickly assigned to convenient areas, while damaged or overdue books are directly diverted to the corresponding processing areas, reducing manual intervention, shortening the sorting cycle, and improving the overall smoothness of the sorting process.

[0033] Third, the sorting flexibility and adaptability are enhanced. The topic clustering submodule supports dynamic updates of core topic terms for collection areas and the creation of temporary topic areas, which can adapt to library collection structure adjustments and new book acquisition needs. The feedback optimization module continuously optimizes model parameters through reinforcement learning and human feedback, enabling the system to adapt to the operational needs of different types of libraries and changes in book characteristics.

[0034] Fourth, optimize collection management. The sorting process takes into account both category relevance and the balance of collection density, with similar books stored together and the collection distributed reasonably in each area, making it convenient for readers to find and borrow books, while reducing the workload of librarians and helping the library achieve dynamic and efficient management. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a self-service book return and sorting system based on multimodal recognition in one embodiment of the present invention;

[0036] Figure 2 This is a flowchart of a self-service book return and sorting method based on multimodal recognition in one embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In one embodiment, such as Figure 1As shown, a self-service book return and sorting system based on multimodal recognition is provided. This self-service book return and sorting system based on multimodal recognition corresponds one-to-one with the self-service book return and sorting method based on multimodal recognition in the above embodiments. The self-service book return and sorting system based on multimodal recognition includes: an information acquisition module, a feature fusion module, a sorting rule module, and a feedback optimization module. The detailed description of each functional module is as follows:

[0039] The information acquisition module collects visual, textual, RFID, borrowing, physical status, and real-time collection data of books. The feature fusion module extracts and fuses features from the collected multimodal information to generate a fused feature vector. The sorting rule module constructs a multi-objective weighted optimization sorting decision function based on the fused feature vector, combines topic clustering to achieve dynamic collection area mapping, and outputs sorting instructions. The feedback optimization module iteratively optimizes model parameters using reinforcement learning algorithms based on sorting results and human review feedback.

[0040] The information acquisition module is a component that enables the acquisition of multi-dimensional information from books. It consists of a high-definition image acquisition unit, an optical character recognition unit, an RFID reader, a sensor group, and a data linkage unit. All units work together to complete the acquisition of information in all dimensions.

[0041] The high-definition image acquisition unit is equipped with high-resolution imaging equipment, which automatically starts the shooting process after the book enters the designated acquisition area of ​​the self-service return terminal. This unit sequentially acquires images of the book's cover, spine, table of contents, copyright page, and preface, ensuring that the acquired images clearly display the text, patterns, and layout of each page, providing high-quality image data for subsequent text recognition and visual feature extraction. During the acquisition process, automatic supplemental lighting and angle adjustment mechanisms prevent image blurring caused by page reflections or wear due to book tilting, ensuring the integrity and clarity of the image information.

[0042] After receiving image data from the high-definition image acquisition unit, the optical character recognition unit initiates the text recognition process. This unit scans and parses the text content line by line in the image, accurately identifying key information such as book title, author, publisher, ISBN, classification number, table of contents, chapters, keywords, preface, core research areas, copyright page, edition, and print run. During the recognition process, an adaptive recognition algorithm is employed for text with different fonts, sizes, and printing clarity to improve the accuracy of text recognition, ensuring the completeness and reliability of the extracted text information, and providing fundamental data support for text feature extraction and semantic analysis.

[0043] The RFID reader is integrated into the identification area of ​​the self-service return terminal. When a book enters this area, the reader automatically sends an RFID signal to activate the RFID tag embedded in the book. Through the interaction of the RFID signal, the reader accurately obtains metadata such as the book's unique identifier, historical location in the library, and borrowing history. This reader has a fast identification capability, and can complete the information reading without the book being fully unfolded, without affecting the smoothness of book return, while ensuring the accuracy of the acquired metadata, providing a basis for book identification and historical information tracing.

[0044] The sensor array, comprising infrared and pressure sensors, is installed at key locations along the book transport path. The infrared sensors emit and receive infrared signals to scan the page edges and overall shape of the books, detecting damage such as missing pages, folds, and water stains, thus determining the books' physical integrity. The pressure sensors sense the distribution and changes in contact pressure as the books pass by, distinguishing between different binding types such as paperback, hardcover, and loose-leaf based on differences in pressure characteristics. The sensor array transmits the detected physical integrity and binding type information to the system in real time, providing data support for identifying special book conditions and adjusting the force of the sorting robotic grippers.

[0045] The data linkage unit establishes a real-time communication connection with the library management system, synchronously retrieving two types of key data during the book return process. One type is real-time collection dynamic data, including the current remaining capacity of each target collection area, the real-time borrowing status of books on the same theme, and librarian activity status, ensuring the system promptly grasps dynamic changes in collection distribution. The other type is borrowing data, including information such as the number of times a book has been borrowed annually and the number of days since the most recent borrowing, providing data support for assessing borrowing activity. The data linkage unit has a data synchronization and update mechanism to ensure that the retrieved data is consistent with the data in the library management system, providing real-time and accurate dynamic data support for sorting decisions.

[0046] The feature fusion module is the core component for achieving deep integration of multimodal information. It consists of a feature extraction submodule, a reliability assessment submodule, a cross-modal semantic alignment submodule, and a weighted fusion submodule. Each submodule collaborates in an orderly manner according to a predetermined process to complete feature fusion.

[0047] After receiving various types of data from the information acquisition module, the feature extraction submodule performs feature extraction operations on each. For image data acquired by the high-definition image acquisition unit, this submodule extracts color, texture, and layout features of the image layer by layer using a convolutional neural network, transforming this visual information into a visual feature vector with unified dimensions. For text content output by the optical character recognition unit, the text sequence is semantically encoded using a text encoding model to capture the topic information and related features in the text, generating a text feature vector. For RFID information acquired by the RFID reader and borrowing data retrieved by the data linkage unit, this submodule first standardizes these structured data to eliminate the dimensional differences between different data types, and then maps them into structured feature vectors through feature transformation, ensuring that the three types of feature vectors are fusionable.

[0048] The reliability assessment submodule calculates the reliability of each of the three types of extracted feature vectors. For visual feature vectors, this submodule quantifies the reliability by analyzing the sharpness and occlusion of the corresponding image and the proportion of effective information. For text feature vectors, the reliability is calculated based on the accuracy of optical character recognition, the completeness of text information, and the coverage of key information. For structured feature vectors, the reliability is determined based on the completeness of RFID information, the timeliness of borrowed data, and the accuracy of data transmission. All three reliability values ​​reflect the credibility of the corresponding feature vectors, providing a basis for subsequent weighted fusion.

[0049] The cross-modal semantic alignment submodule is responsible for eliminating semantic differences between features from different modalities. This submodule first establishes a semantic mapping relationship between visual features and text features, matching the visual features extracted from the image with the semantic features obtained from text encoding, so that the visual feature vectors carry the corresponding semantic information. Then, through semantic association analysis between text features and structured features, it maps information such as topic categories in the text to collection classification labels in the structured data, achieving semantic alignment between text features and structured features. Through bidirectional semantic matching, it ensures that the three types of feature vectors maintain consistency within the same semantic space, avoiding semantic conflicts during the fusion process.

[0050] The weighted fusion submodule dynamically calculates the fusion weights of each feature vector based on the reliability values ​​of visual features, text features, and structured features output by the reliability assessment submodule. The weight calculation follows the principle that higher reliability results in a larger weight allocation, specifically determined by a predetermined formula. Generate a fused feature vector, where, , , V is the visual feature vector, T is the text feature vector, and S is the structured feature vector. For the reliability of visual features, For text feature reliability, To ensure the reliability of structured features, this fused feature vector comprehensively integrates multi-dimensional information, including the visual text structure of the book, providing comprehensive feature support for subsequent sorting decisions.

[0051] The sorting rules module is the core module for making accurate book sorting decisions. It consists of a decision function construction submodule and a topic clustering submodule. The two submodules work together to determine the sorting target area.

[0052] The decision function construction submodule is responsible for building a multi-objective weighted optimization sorting decision function. This submodule first defines four core objectives for the sorting decision: category similarity, borrowing popularity, collection density balance, and priority of special book statuses. Based on the degree of fit between the textual semantic information in the fused feature vector and the subject terms of the collection area, the function is calculated using a cosine similarity algorithm. Borrowing popularity is obtained by weighted summation of the book's annual borrowing frequency and the normalized value of the number of times it was borrowed since the last borrowing. ,in, This refers to the number of times a book is borrowed in a year. This is the normalized value of the number of days since the book's last borrowing, with 'a' and 'b' being weighting coefficients, and 'a+b=1'. Collection density balance is calculated by subtracting the ratio of the current number of books in a region to the region's maximum capacity, reflecting the region's availability. Book special status priorities are set based on the book's physical condition and borrowing status; damaged or overdue books have higher priority than intact books within their borrowing period.

[0053] The decision function construction submodule assigns target weights to the four objectives. The values ​​of these weights are set according to the library's operational needs and can be flexibly adjusted. Then, the decision function construction submodule multiplies the category similarity by the corresponding target weight, i.e. ,in, To determine the similarity between the categories of books and the collection areas, To boost book borrowing popularity, To ensure the balance of collection density, Prioritize special statuses of books. , , , Using the target weights, a sorting decision function is constructed. The output value of this function directly reflects the degree of matching between books and each collection area; a higher output value indicates a higher degree of matching.

[0054] The topic clustering submodule is responsible for dynamically determining the topic attributes of collection areas and calculating the matching degree between books and areas. This submodule uses an incremental clustering algorithm to periodically perform cluster analysis on the existing book fusion feature vectors of each collection area. During the clustering process, core topic words are extracted for each cluster, which collectively reflect the topic direction of the corresponding collection area. For newly returned books, the topic clustering submodule extracts topic-related information from their fusion feature vectors and compares it with the core topic words of each collection area. By calculating the similarity between the book's topic information and the core topic words of each area, the collection area with the highest similarity is selected. Combining the output value of the sorting decision function, the topic clustering submodule finally determines the optimal sorting target area for the book, providing clear instructions for sorting execution.

[0055] The feedback optimization module is the module that enables continuous iterative upgrades of the sorting model. It consists of a reinforcement learning submodule and a human feedback submodule. The two submodules work together to dynamically optimize the model parameters.

[0056] The reinforcement learning submodule models the entire book sorting process as a reinforcement learning environment, clearly defining the core elements within it. The agent is defined as the sorting rule module, responsible for executing actions related to sorting decisions. The state space encompasses three types of key information: the fused feature vector output by the feature fusion module, the real-time collection status retrieved by the data linkage unit, and the borrowing data from the library management system. This information collectively forms the basis for the agent's decisions. The action space is defined as the selection of the sorting target area, whereby the agent selects the optimal sorting area from all available collection areas based on its current state.

[0057] The reinforcement learning submodule constructs a reward function to evaluate the effectiveness of actions. This reward function includes three evaluation metrics: Borrowing rate (the ratio of the number of books borrowed within a preset period after sorting to the total duration of that period), reflecting the rationality of the sorting location; Sorting efficiency (the total time it takes for a single book to travel from entering the sorting process to reaching its target area), reflecting the smoothness of the sorting process; and Sorting error rate (the ratio of the number of books identified as having sorting errors during manual review to the total number of books sorted), measuring the accuracy of sorting decisions. The reward function assigns reward coefficients to each of the three metrics, i.e. ,in, The borrowing rate is preset within a certain period after the books are sorted. The time required for sorting individual books The sorting error rate is manually verified. , , This represents the reward coefficient. The reinforcement learning submodule adjusts the parameters of the sorting rule module based on the reward value using a reinforcement learning algorithm, ensuring that subsequent sorting actions yield higher rewards.

[0058] The manual feedback submodule receives sorting error information marked by the manual review process and identifies the error type. Error types include various situations such as misclassification and unreasonable region allocation, with different error types corresponding to different model parameter adjustment directions. For each marked error type, the manual feedback submodule adjusts the weight parameters of the feature fusion module and the target weight parameters of the sorting rule module to directly correct model biases. Simultaneously, the manual feedback submodule compiles these error cases and expands them into the model training set. Through incremental training, it optimizes the feature extraction and sorting decision-related models, improving the model's adaptability to complex scenarios and achieving continuous performance optimization.

[0059] Specific limitations regarding the self-service book return and sorting system based on multimodal recognition can be found in the limitations of the self-service book return and sorting method based on multimodal recognition mentioned above, and will not be repeated here. Each module in the aforementioned self-service book return and sorting system based on multimodal recognition can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0060] In one embodiment, such as Figure 2 As shown, a self-service book return and sorting method based on multimodal recognition is provided, including the following specific steps:

[0061] S10: Collect visual information, text information, radio frequency identification information, borrowing data, physical status information and real-time collection dynamic data of books through the information collection module.

[0062] S20: The feature fusion module extracts features from the collected multimodal information. After reliability assessment and cross-modal semantic alignment, the weights are dynamically calculated based on the reliability of visual features, text features, and structured features to generate a fused feature vector.

[0063] S30: The sorting decision function is constructed based on the fused feature vector through the sorting rule module, and the sorting target area is determined by combining topic clustering and sorting instructions are output.

[0064] S40: The feedback optimization module optimizes the model parameters iteratively through reinforcement learning algorithms based on the sorting results and manual review feedback.

[0065] Specifically, the execution process of the self-service book return and sorting method proceeds in the following steps, with each step closely connected and precisely matched with the corresponding module functions.

[0066] The first step is to initiate the multimodal information acquisition process. The high-definition image acquisition unit of the information acquisition module captures images of the book's cover, spine, copyright page, table of contents, and preface to obtain clear visual information. The optical character recognition unit performs text recognition on the acquired images, extracting text information such as the book title, author, classification number, table of contents, chapters, and keywords. The RFID reader interacts with the RFID tag embedded in the book to obtain RFID information such as the book's unique identifier, historical location in the library, and borrowing history. The sensor group detects the physical integrity and binding type of the book through infrared scanning and pressure sensing, forming physical status information. The data linkage unit communicates in real time with the library management system, retrieving borrowing data such as the number of borrowings per year and the most recent borrowing time, while also acquiring real-time dynamic data on the library's collection, including remaining capacity in each area, borrowing status of books on the same theme, and librarian activity status.

[0067] The second step involves multimodal feature fusion. After receiving various types of information, the feature fusion module uses a convolutional neural network to extract features such as color, texture, and layout from the visual information, generating visual feature vectors. Text information is semantically encoded using a text encoding model to obtain text feature vectors. The structured data, composed of RFID information and borrowing data, is normalized and transformed into structured feature vectors. The reliability assessment module analyzes the clarity of visual information, the accuracy of text recognition, and the completeness and timeliness of structured data, calculating the reliability of visual features, text features, and structured features. The cross-modal semantic alignment module establishes a semantic mapping between visual and text features, achieving semantic matching between them. Simultaneously, it associates text features with information such as classification labels in the structured data, completing the semantic alignment between text and structured data. The weighted fusion submodule dynamically calculates weights based on three types of reliability values. The weight of the visual feature vector is the ratio of visual feature reliability to the sum of the three types of reliability; the weight of the text feature vector is the ratio of text feature reliability to the sum of the three types of reliability; and the weight of the structured feature vector is the ratio of structured feature reliability to the sum of the three types of reliability. Then, each feature vector is multiplied by its corresponding weight according to a predetermined formula, and the results are summed to generate the fused feature vector.

[0068] The third step involves sorting decisions and target area determination. After receiving the fused feature vector, the sorting rules module first calculates four core decision indicators in the decision function construction submodule. Category similarity is calculated using cosine similarity between the textual semantic information in the fused feature vector and the subject terms of each collection area. Borrowing popularity is obtained by weighted summation of the normalized values ​​of the annual borrowing frequency and the number of days since the last borrowing, with the weighting coefficients of the two parameters summing to one. Collection density balance is calculated by subtracting the ratio of the current number of books in a region to the region's maximum capacity from one. Priority for special book conditions is set based on whether the book is damaged or overdue. The decision function construction submodule assigns target weights to the four indicators, multiplies each indicator by its corresponding weight, and sums the results to construct the sorting decision function. The subject clustering submodule uses an incremental clustering algorithm to cluster the fused feature vectors of existing books in each collection area, extracting the core subject terms for each region. Calculate the similarity between the fusion feature vector of newly returned books and the core keywords of each region. Combine this with the output value of the sorting decision function to select the region with the highest matching degree as the sorting target region and output the corresponding sorting instructions.

[0069] The fourth step involves feedback optimization and parameter iteration. The feedback optimization module continuously collects sorting result data and manual review feedback information. The reinforcement learning submodule models the sorting process as a reinforcement learning environment, using the sorting rule module as the agent, the real-time library collection status and borrowing data of the fused feature vectors as the state space, and the sorting target area selection as the action space. A reward function is constructed based on the borrowing rate within a preset period, the sorting time per book, and the sorting error rate confirmed by manual review. The three indicators are multiplied by their corresponding reward coefficients, and the reward value is calculated according to the reward function formula. The reinforcement learning algorithm adjusts the weight parameters of the feature fusion module and the target weight parameters of the sorting rule module based on the reward value. The manual feedback submodule receives the error types marked by manual review, corrects the relevant module parameters accordingly, and expands the error cases to the model training set. Through incremental training, the model performance is continuously optimized to ensure that sorting accuracy and efficiency are gradually improved.

[0070] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0071] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A self-service book return and sorting system based on multimodal recognition, characterized in that, include: Information collection module, feature fusion module, sorting rule module, and feedback optimization module; The information acquisition module is used to collect visual information, text information, radio frequency identification information, borrowing data, physical status information and real-time dynamic data of the collection of books; The feature fusion module is used to extract and fuse features from the collected multimodal information to generate a fused feature vector; The sorting rules module is used to construct a multi-objective weighted optimization sorting decision function based on fused feature vectors, combine topic clustering to realize dynamic collection area mapping, and output sorting instructions; The feedback optimization module is used to iteratively optimize model parameters based on sorting results and manual review feedback through reinforcement learning algorithms.

2. The self-service book return and sorting system based on multimodal recognition according to claim 1, characterized in that, The information acquisition module includes: a high-definition image acquisition unit, an optical character recognition unit, an RFID reader, a sensor group, and a data linkage unit; The high-definition image acquisition unit captures images of the book cover, spine, copyright page, table of contents, and preface; the optical character recognition unit identifies the text content in the images; the radio frequency identification reader obtains the book's unique identifier, historical collection location, and borrowing history; the sensor group detects the book's physical integrity and binding type; and the data linkage unit retrieves real-time collection dynamic data and borrowing data.

3. The self-service book return and sorting system based on multimodal recognition according to claim 1, characterized in that, The feature fusion module includes: a feature extraction submodule, a reliability assessment submodule, a cross-modal semantic alignment submodule, and a weighted fusion submodule; The feature extraction submodule extracts visual feature vectors through a convolutional neural network, extracts text feature vectors through a text encoding model, and normalizes the structured data composed of RFID information and borrowing data to obtain structured feature vectors. The reliability assessment submodule calculates the reliability of visual features, text features, and structured features. The cross-modal semantic alignment submodule achieves semantic matching between visual and textual data, as well as textual and structured data; the weighted fusion submodule dynamically calculates weights based on the aforementioned reliability, using the formula... Generate a fused feature vector, where, , , V is the visual feature vector, T is the text feature vector, and S is the structured feature vector. For the reliability of visual features, For text feature reliability, For the reliability of structured features.

4. The self-service book return and sorting system based on multimodal recognition according to claim 1, characterized in that, The sorting rules module includes: a decision function construction submodule and a topic clustering submodule; The decision function construction submodule constructs the sorting decision function. ,in, To determine the similarity between the categories of books and the collection areas, To boost book borrowing popularity, To ensure the balance of collection density, Prioritize special statuses of books. , , , The target weight is used; the topic clustering submodule extracts the core topic words of each collection area through an incremental clustering algorithm, calculates the similarity between books and the core topic words of each area, and determines the sorting target area.

5. The self-service book return and sorting system based on multimodal recognition according to claim 4, characterized in that, Borrowing popularity ,in, This refers to the number of times a book is borrowed in a year. This is the normalized value of the number of days since the book was last borrowed, where a and b are weighting coefficients and a+b=1.

6. The self-service book return and sorting system based on multimodal recognition according to claim 1, characterized in that, The feedback optimization module includes: a reinforcement learning submodule and a human feedback submodule; The reinforcement learning submodule models the sorting process as a reinforcement learning environment, with the agent representing the sorting rules module. The state space includes fused feature vectors, real-time collection status, and borrowing data, while the action space is for selecting the sorting target area, and the reward function is... ,in, The borrowing rate is preset within a certain period after the books are sorted. The time required for sorting individual books The sorting error rate is manually verified. , , The reward coefficient is used; the manual feedback submodule receives the error types marked by manual review, adjusts the parameters of the feature fusion module and the sorting rule module accordingly, and expands the error cases to the model training set.

7. A self-service book return and sorting method based on multimodal recognition, characterized in that, include: The information collection module collects visual information, text information, radio frequency identification information, borrowing data, physical status information, and real-time dynamic data of the library's collection. The feature fusion module extracts features from the collected multimodal information. After reliability assessment and cross-modal semantic alignment, the weights are dynamically calculated based on visual feature reliability, text feature reliability, and structured feature reliability, using the formula... Generate fused feature vectors; The sorting decision function is constructed based on the fused feature vector through the sorting rule module. The system combines topic clustering to determine the sorting target area and outputs sorting instructions. To determine the similarity between the categories of books and the collection areas, To boost book borrowing popularity, To ensure the balance of collection density, Prioritize special statuses of books. , , , The target weight; The feedback optimization module optimizes the model parameters iteratively using reinforcement learning algorithms based on the sorting results and manual review feedback.

8. The self-service book return and sorting method based on multimodal recognition according to claim 7, characterized in that, Physical status information includes: physical integrity and binding type of books; real-time collection dynamic data includes remaining capacity of the target collection area, real-time borrowing status of books on the same topic, and librarian work status; text information includes subject keywords of the table of contents, core research areas of the preface, edition and print quantity of the copyright page.

9. The self-service book return and sorting method based on multimodal recognition according to claim 7, characterized in that, Visual feature vectors are extracted using a convolutional neural network, and text feature vectors are extracted using a text encoding model. Structured feature vectors are obtained by normalizing the structured data composed of RFID information and borrowing data. (Formula) Generate a fused feature vector, where, , , V is the visual feature vector, T is the text feature vector, and S is the structured feature vector. For the reliability of visual features, For text feature reliability, For the reliability of structured features.

10. The self-service book return and sorting method based on multimodal recognition according to claim 7, characterized in that, The core thematic terms of each collection area are updated periodically using an incremental clustering algorithm, supporting the dynamic creation of temporary thematic areas; a sorting decision function. middle, To determine the similarity between the categories of books and the collection areas, , Prioritize special statuses of books. , , , The target weights are a and b, which are weighting coefficients and a + b = 1. This refers to the number of times a book is borrowed in a year. This is the normalized value of the number of days since the book's last borrowing; after each preset batch of books is sorted, a reward function is applied. The cumulative reward is calculated, and the feature fusion weights and sorting decision target weights are updated using a reinforcement learning algorithm. The borrowing rate is preset within a certain period after the books are sorted. The time required for sorting individual books The sorting error rate is manually verified. , , This is the reward coefficient.

Images