A multilingual book title recognition and dynamic optimization method for high-density complex bookshelves

Abstract

This invention discloses a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves, including: Step 1, collecting visual observation data and RFID observation data from multiple bookshelf scenarios; Step 2, constructing training samples according to field definitions and structured output protocols; Step 3, employing a dynamic weighted fine-tuning strategy based on cross-domain dual validation set monitoring and historical performance feedback on an open-source pre-trained visual language model to obtain a preliminary fine-tuned model with stable multilingual structured prediction output capabilities; Step 4, aligning the output behavior and format of the preliminary fine-tuned model based on direct preference optimization to obtain an aligned model with consistent format and behavior; Step 5, performing edge-oriented AWQ quantization on the aligned model; inputting the visual observation data collected in Step 1 into the AWQ-quantized model to output structured visual recognition results; Step 6, performing cross-modal fusion and physical order diagnosis.

Description

Technical Field

[0001] This invention relates to the field of book title recognition, and in particular to a method for multilingual book title recognition and dynamic optimization for high-density complex bookshelves. Background Technology

[0002] With the development of the Internet of Things (IoT), Artificial Intelligence (AI), and robotics, library inventory and shelving management are evolving from manual verification to "automated inventory + precise positioning + proactive correction." Existing solutions mainly include two approaches: RFID and vision-based methods.

[0003] 1) RFID route: RFID can efficiently provide tag identification IDs and improve borrowing and returning efficiency. However, in dense shelving environments, UHF signals are affected by multipath, obstruction, adjacent coupling, and antenna directivity, resulting in cross-reading, missed readings, and unstable readings. This makes it difficult to achieve stable spatial resolution at the "layer / column / grid" level, making it difficult to directly give conclusions such as "whether it is misplaced or should be returned to its original position".

[0004] 2) Visual approach: Visual information can provide fine-grained spatial information, but dense bookshelves often have a large number of book spines on the same screen (e.g., more than 20 books in a single frame), small text scale, reflection and blurring, occlusion, vertical / horizontal mixed layout, multilingual mixed layout, and other complex factors. Traditional OCR (Optical Character Recognition) is difficult to reliably output structured results that can be used in business systems (database entry / retrieval / comparison).

[0005] In recent years, visual language models (VLMs) have improved multimodal understanding and generation capabilities, but their application in edge computing robots still faces challenges:

[0006] Long-tail multilingual imbalance: High-resource languages ​​tend to dominate the gradient, while low-resource languages ​​are prone to underfitting or regression in the later stages.

[0007] Insufficient structured controllability: Generative output is prone to missing fields, quantity drift, illusions, and unparseable JSON;

[0008] Heterogeneous conflicts are difficult to resolve: visual and RFID sampling are not synchronized and have different noise mechanisms, which can easily lead to "seeing but not reading / reading but not seeing", requiring computable fusion and consistency constraints.

[0009] Therefore, there is an urgent need for an integrated mis-shelf detection method that is suitable for dense bookshelves, can be deployed at the edge, and can perform structured identification, cross-frame stabilization, database matching, RFID decision-level fusion, and global consistency constraints. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves, which addresses the shortcomings of the existing technology. This method aims to solve the problems of multilingual recognition imbalance, uncontrollable output format of large visual models, and difficulty in aligning visual and RFID cross-modal data in complex bookshelf scenarios.

[0011] To address the aforementioned technical problems, this invention discloses a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves:

[0012] Step 1: Multi-source observation and unified preprocessing: Collect RFID observation data of bookshelf scenes, and obtain spine image blocks based on different library bookshelves for language recognition and dual verification set division as visual observation data;

[0013] Step 2: Construct training samples according to field definitions and structured output protocols: Define a JSON format output protocol containing information such as main and subtitles, and generate structured annotations using automated inference or multi-stage correction strategies for medium-high resource languages ​​and low resource languages ​​respectively.

[0014] Step 3: Dynamic weighted fine-tuning based on cross-domain dual validation set monitoring and historical performance feedback: The training loss of each language is monitored using a dual validation set. Once the performance of a low-resource language is found to be declining, the historical penalty weight is triggered by exponential amplification. This is used to dynamically weight and fine-tune the visual language model.

[0015] Step 4: Behavior and format alignment based on Direct Preference Optimization (DPO): Generate a comparison dataset that conforms to and does not conform to the protocol format using the initial fine-tuning model, and obtain a model that is aligned in terms of structured format and output behavior through Direct Preference Optimization;

[0016] Step 5, Model Quantization and Inference for Edge Deployment: Perform AWQ (Activation-aware Weight Quantization) quantization on the aligned model to adapt to edge computing, and use it to infer the collected visual observation data to output structured visual recognition results.

[0017] Step 6: Visual and RFID cross-modal fusion and physical order diagnosis: Combine visual unified identity confidence and RFID radio frequency confidence to construct a cost matrix for Hungarian optimal matching, eliminate cross-modal conflict nodes, and realize fine-grained misalignment diagnosis within the layer based on the lexicographical monotonicity of the rack index according to the matching results.

[0018] Step 1, which involves collecting visual observation data, specifically includes:

[0019] Step 1-1: Collect visual observation data: Obtain spine image patches based on bookshelves in different libraries as the original image dataset, and divide the dataset into the current library's validation set and other libraries' validation sets according to the different libraries;

[0020] Steps 1-2: Process each image block using an OCR engine to identify the text information and thus obtain the language corresponding to the original image.

[0021] Step 2 specifically includes:

[0022] Step 2-1: Define the structured output protocol as the JSON field protocol, and split the book title information into main title, subtitle, language tag, and confidence score;

[0023] Step 2-2: Automated construction of raw image data in medium and high resource languages, generating field-level structured annotations through automated methods, and reducing the uncontrollability of output distribution through consistency verification;

[0024] Steps 2-3: Perform multi-stage construction on the raw image data of low-resource languages ​​to generate field-level structured annotations.

[0025] The automated construction described in step 2-2 includes the following steps:

[0026] Step 2-2-1: Use the field corresponding to the book title as data to generate prompt words for the large model;

[0027] Step 2-2-2: The large data generation model takes the prompt words from Step 2-2-1 and the original images from Step 1-1 to perform reasoning in order to identify the main title and subtitle in the text;

[0028] Step 2-2-3: Structured annotation generation. Based on the identified fields of the book title, this information is converted into a structured annotation format, and the language tags are annotated using the language of each image.

[0029] Steps 2-3 specifically include:

[0030] Step 2-3-1: Visual Enhancement: Perform visual enhancement processing on the original image data;

[0031] Step 2-3-2: Multi-source recognition: Apply multiple OCR engines to the enhanced spine image for preliminary recognition to obtain multiple candidate book title texts. By combining multi-source OCR, different character representations and language forms are covered.

[0032] Step 2-3-4: Semantic Adjudication: Perform semantic-level correction and fusion on the multi-round OCR results in Step 2-3-2, perform consistency fusion, remove garbled characters, merge erroneously split fragments, and finally output a structured book title field. Also, use the language of each image obtained in Step 1-1 to label the language tag.

[0033] The dynamic weighted fine-tuning strategy based on cross-domain dual-validation set monitoring and historical performance feedback described in step 3 is as follows:

[0034] Step 3-1, Data Preparation: Intervene on the distribution of data quantity for different language labels in the training samples, and adopt oversampling and undersampling data processing strategies for long-tail languages ​​and high-frequency languages; obtain a balanced dataset with different language quantities for different libraries.

[0035] Step 3-2: Calculate the basic cross-entropy loss for each language using the balanced dataset. :

[0036] Basic cross-entropy loss definition:

[0037]

[0038] The specific meanings of each parameter in the formula in our bookshelf recognition scenario are as follows: Represents a single real multilingual spine image patch as input; This refers to the target structured JSON text (containing main and subtitle fields) constructed for the image in step 2. This represents the network parameters that need to be optimized within the current visual language model; This represents the total probability that, under the current model parameters, the model can accurately and completely output the target JSON format and book title text based on the input spine image. A lower probability indicates greater difficulty in recognition and a higher baseline error. The larger.

[0039] Step 3-3: Calculate the cross-entropy loss of each language on the two validation sets using the basic cross-entropy formula for each language, and then merge them into a unified cross-entropy loss. :

[0040] Meanwhile, caching languages The optimal cross-entropy loss value during training If the current training cycle measures This indicates that the data for this language experienced performance degradation during training; the performance difference values ​​were extracted and recorded. ;

[0041] Step 3-4: Calculate the dynamically weighted total loss based on the cross-entropy loss of the fused validation set.

[0042] Using the cross-entropy loss results for each language calculated in step 3-3, dynamic weights are assigned to each language. And calculate the final optimization objective for the current batch. :

[0043]

[0044]

[0045] The parameters in the formula are: Language The dynamic penalty weight is directly determined by the feedback from step 3-2: The larger the value (indicating a more severe underfit), the higher the weight. Historical rollback penalty items (among others) (This is a preset exponential amplification factor). Once step 3-2 detects a performance degradation in this language (i.e., ... This penalty term is amplified exponentially, forcing the model's computing power to be reallocated to this low-resource language; The first in the current batch The language of each image sample ( The corresponding dynamic weights mentioned above; The final total loss is the sum of the basic cross-entropy loss of all samples in the current batch, after being dynamically weighted according to the corresponding language.

[0046] Steps 3-5, updating LoRA parameters based on dynamic weighted objective: The dynamic weighted total loss is used as the loss function for training the visual language model. The internal weights of the visual language model are updated using the backpropagation algorithm and low-rank adaptation technique, and finally the fine-tuned model is obtained.

[0047] Step 4 specifically includes:

[0048] Step 4-1: Use the inference results of the fine-tuned model on the original image as candidate samples, and use the outputs that meet the structured output protocol format as the preferred outputs. The output that does not meet the requirement will be treated as the non-preference output. This allows us to construct a high-quality contrastive dataset for direct preference optimization.

[0049] Step 4-2: Based on the high-quality comparison dataset, use direct preference optimization to align the fine-tuned model to obtain the aligned model.

[0050] Step 6 specifically involves: performing cross-frame clustering on the visual recognition results; performing cross-modal global optimal allocation and isolated node adjudication on the visually standardized book titles after cross-frame clustering and semantic matching, and finally accurately determining fine-grained misalignments within the layer based on the sequence monotonicity of the shelving index.

[0051] Step 6-1: The visual recognition results are sorted by deduplication using a sliding window method. Then, a text similarity algorithm is used to retrieve and match the real book titles in the database, thereby obtaining the visually standardized book titles and their unified identity confidence scores. ;

[0052] Step 6-2: Perform cross-modal global optimal allocation and isolated node adjudication on the visually standardized book titles to obtain the final truly sorted book titles:

[0053] Step 6-3: Fine-grained diagnosis of mis-shelf and disordered order within a shelf: Extract the book names obtained in Step 6-2 and search for their call numbers in the library database. Based on their actual physical order sequence on the current physical bookshelf shelf, check whether the shelf indexes of adjacent books strictly satisfy the monotonically increasing property of lexicographical order. If not, output the mis-shelf and the correct position candidate.

[0054] Step 6-2 specifically includes:

[0055] Step 6-2-1: Extract RFID observation data within the time window (corresponding to the sliding window in 6-1), and construct a normalized radio frequency confidence score based on the returned signal strength indication and the reading frequency. ;

[0056] Step 6-2-2: Using the visual entity as one end node and the RFID entity as the other end node, utilize the unified identity confidence level of the visual side. The radio frequency confidence level calculated above The difference is used to construct the matching cost, and the Hungarian algorithm is used to solve the minimum cost matching problem to obtain the optimal one-to-one correspondence between the visual and RFID entities of the current bookshelf layer.

[0057] Step 6-2-3: Mechanism for handling isolated nodes; For RFID results that failed to pair successfully due to an imbalance in the number of nodes after solving the Hungarian algorithm, the radio frequency confidence level is considered. It is compared with the global uniqueness constraint to determine whether to remove it.

[0058] Beneficial effects:

[0059] 1. Overcoming Multilingual Training Degradation: Through a dynamic backoff penalty mechanism, once performance degradation of low-resource languages ​​is detected, an exponential weight amplification is triggered, forcing the model's computing power to tilt, thereby achieving balanced and synchronous convergence of long-tail multilingual languages.

[0060] 2. Completely eliminate cross-modal quantity conflicts: Abandoning traditional hard alignment, it accurately removes RFID cross-reading noise through bipartite graph matching and global unique physical adjudication, and effectively finds entities with severe visual obstruction.

[0061] 3. Achieve fine-grained mis-shelf diagnosis: It perfectly integrates the one-dimensional spatial topology provided by vision with the unique identity provided by RFID, which not only solves the problem of inconsistent sensor counts, but also achieves accurate and automatic error correction for local disorder within the bookshelf layers. Attached Figure Description

[0062] Figure 1 This is a flowchart of the method.

[0063] Figure 2 This is a schematic diagram of the input scenario in the embodiment.

[0064] Figure 3 This is the high-precision intermediate recognition result in the example.

[0065] Figure 4 This is the final physical diagnostic output of the embodiment.

[0066] Figure 5 The training loss convergence comparison curve is shown in the example. Detailed Implementation

[0067] The technical problem this invention aims to solve is to address the shortcomings of existing technologies: fine-tuning of multilingual long-tail data easily leads to overfitting of visual language models to high-frequency languages ​​and performance degradation of less common languages; furthermore, in actual inventory checks, visual solutions are prone to missed detections due to occlusion, and RFID solutions are highly susceptible to cross-layer cross-reading, resulting in conflicts in multi-source statistical quantities and the inability to detect disordered order within layers. This invention provides a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves.

[0068] To address the aforementioned technical issues, this invention discloses a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves. During the model training phase, this method introduces a dynamic weighting strategy based on independent monitoring of cross-domain dual validation sets and historical performance backtracking penalties. This dynamically allocates loss weights and replaces static cross-entropy loss for low-rank fine-tuning of the model, ensuring balanced convergence across languages. In the cross-modal inventory phase, confidence scores are calculated for visually recognized entities and RFID entities, and a cost matrix is ​​constructed. A bipartite graph is used for globally optimal matching. Isolated, unpaired nodes are adjudicated using RF attenuation characteristics and the library's uniqueness constraint. Finally, fine-grained mis-shelf determination is achieved based on the physical spatial topological sequence of the entities.

[0069] A method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves, comprising the following steps:

[0070] Step 1: Multi-source observation and unified preprocessing: Collect visual observation data and RFID observation data of the bookshelf scene;

[0071] Step 2: Construct training samples according to the field definitions and structured output protocol;

[0072] Step 3: For the open-source pre-trained visual language model (Qwen3-vl), a dynamic weighted fine-tuning strategy based on cross-domain dual validation set monitoring and historical performance feedback is adopted to overcome the bottleneck of multilingual long-tail distribution and obtain a preliminary fine-tuned model with stable multilingual structured prediction output capability.

[0073] Step 4: Align the output behavior and format of the initial fine-tuned model with those of Direct Preference Optimization (DPO) to obtain an aligned model that is consistent in both format and behavior;

[0074] In this embodiment, specifically, a preference comparison sample of "correct structure / parsable output" and "incorrect structure / illusion / parsing failure output" is constructed, and DPO is used to align and optimize the output behavior to suppress field drift, splitting errors and cross-frame inconsistencies and improve the parsability of JSON format;

[0075] Step 5: Perform edge-oriented AWQ quantization on the aligned model; input the visual observation data collected in Step 1 into the AWQ quantized model obtained in Step 5, and output structured visual recognition results;

[0076] In this embodiment, specifically, Activation-Aware Weighted Quantization (AWQ) is used for 4-bit quantization; by calibrating the data selection strategy to cover visual diversity and ensure language quotas, the risk of performance collapse of low-resource languages ​​by quantization is reduced, and low memory and low latency inference are achieved.

[0077] Step 6: Perform cross-frame clustering, database matching, and cross-modal misalignment detection on the visual recognition results;

[0078] The visual recognition results are clustered across frames. The visual standard book title after clustering and semantic matching of multiple consecutive frames is then used for cross-modal global optimal allocation and isolated node adjudication with the underlying RFID radio frequency confidence. Finally, based on the sequence monotonicity of the shelving index, fine-grained mis-shelving within the layer is accurately determined.

[0079] The visual output is modeled in a field and fused across frames to obtain standardized book titles; the standardized book titles are matched with the library's collection database using Top-K recall and fine-tuning, and a unified confidence level is established on the visual side; the confidence level on the visual side is dynamically weighted and fused with the confidence level on the RFID side, and deduplication, capacity / inventory and space consistency constraints are introduced to output the fusion result; when the fusion confirms that the identity is inconsistent with the book that should be stored on the bookshelf, the confidence level of the misplaced bookshelf and the possible candidates for the correct bookshelf are output.

[0080] Step 1, which involves collecting visual observation data, specifically includes:

[0081] Step 1-1: Collect visual observation data: acquire images / video frames of bookshelves from different libraries and record the timestamp and spine frame position of each frame. Perform spine detection on each frame to obtain a set of spine frames. Crop each spine frame to obtain spine image blocks as the original image dataset. Divide the dataset into the current library's validation set and other libraries' validation sets according to the different libraries.

[0082] To verify the feasibility and effectiveness of the multilingual book title recognition and dynamic optimization method for high-density complex bookshelves described in this invention, this embodiment was fully deployed and tested in a library scenario with realistic high density, multilingual mixed layout, and complex lighting conditions. The following, in conjunction with the appendix... Figures 2 to 4 The implementation process and results are described in detail.

[0083] In this embodiment, the data collection work covered two libraries with significantly different physical environments (designated as "Library A" and "Library B").

[0084] Description of scene complexity: (e.g.) Figure 2 As shown, the image displays real bookshelf images from Hall A (left) and Hall B (right), respectively. Both scenes exemplify the technical challenges this invention aims to address: severe glare and shadow occlusion, mixed text in multiple languages ​​(Chinese / English / Malay, etc.), and extremely narrow spines. Traditional OCR technology... Figure 2 It is highly prone to failure in the scenarios shown.

[0085] Dataset Scale: Following steps 1-1, this embodiment constructed an original image dataset containing approximately 20,000 high-quality spine image patches (approximately 10,000 from Library A and approximately 10,000 from Library B). Preliminary language labeling was completed in steps 1-2 for subsequent model training and validation. Structured book title recognition results (corresponding to steps 2-5 and appendix) Figure 3 )

[0086] For the aforementioned complex dataset, the system executed the core processes from steps 2 to 5. In particular, the "dynamic weighted fine-tuning strategy based on cross-domain dual validation set monitoring and historical performance feedback" described in step 3 was adopted, effectively balancing the learning weights of medium-to-high resource languages ​​and low-resource long-tail languages; subsequently, strict alignment of the output format was achieved through DPO optimization in step 4.

[0087] Recognition effect description: such as Figure 3As shown, this figure demonstrates the recognition results of the edge-side model after step 5AWQ quantization for some book spines in the picture collected in step 1-1. Despite the severe interference in the original input image, the model still stably outputs structured and standardized text information. For example, it accurately recognizes bilingual titles (such as "BADGES OFFURY 不二神探"), pure Chinese titles (such as "家有囍事 2009"), and texts with complex punctuation, and all outputs strictly conform to the predefined JSON protocol format. This result fully proves the high robustness and controllability of the present invention in the front-end visual recognition link.

[0088] Based on obtaining high-quality visual recognition results, the system executed the cross-modal fusion and physical order diagnosis in step 6. The system Figure 3 matched the recognized "visually standardized book title" with the library collection database to obtain a unique call number (shelf arrangement index), and performed a globally optimal match of the visual entity with the RFID read data based on radio frequency confidence. Finally, a diagnosis was made according to the rule of "lexicographic monotonic increase".

[0089] Explanation of mis-shelving diagnosis: As Figure 4 shown, this figure intuitively presents the fusion of the final visual result and the RFID result of this method, as well as the output interface of the final mis-shelving diagnosis.

[0090] As shown in the figure, the RFID result is disordered and remembers the book information on other bookshelves, but for visual recognition, there are usually cases of missed reading and incorrect matching in step 6-1. Therefore, the results of the two need to be fused in step 6-3 to obtain the final correct and completely sorted book title information.

[0091] For the recognition of mis-shelving, it is necessary to rely on the final result recognized in 6-3: As shown in the figure, the system constructs the book sequence of the current physical layer shelf and its corresponding call number (such as the call number sequence of the current shelf should be `791.4372.1`).

[0092] Using step 6-4 to check the monotonicity of adjacent book indexes, the system keenly captures the logical break points in the physical sequence (that is, the non-increasing change of the 4 call numbers on the left (730.9512.1; 745924.5, etc.)).

[0093] Attached Figure 4 In the figure, the mis-shelved books on the current bookshelf are marked with red arrows.

[0094] To sum up, combining the extreme input scenario in Attachment Figure 2 Attachment Figure 3 the high-precision intermediate recognition results in Attachment Figure 4The final physical diagnostic output fully demonstrates that the method of the present invention can achieve a complete closed loop from multilingual visual perception to fine-grained physical correction in real and complex high-density bookshelf environments, and has extremely high feasibility and practical application value.

[0095] Steps 1-2: Process each image block using an OCR engine to identify the text information and thus obtain the language corresponding to the original image; form a unified data representation for subsequent structured modeling and fusion decision-making.

[0096] The OCR engine processes each image block to identify the text information within it, including book titles in multiple languages ​​such as Chinese, English, Malay, and Tamil. Through the processing in steps 1-1-2, the text becomes more consistent, ultimately forming a unified data representation.

[0097] Step 2 specifically includes:

[0098] Step 2-1: Define the structured output protocol as JSON and the field protocol as follows;

[0099] y=Title,Subtitle,Lang,Conf

[0100] This involves breaking down the book title information into the main title (Title), subtitle (Subtitle, optional), language ID, and confidence score.

[0101] The goal of this step is to obtain a structured output protocol: JSON format data containing a main title (Title) and an optional subtitle (Subtitle). The final recognition output must meet this unified structure (JSON field protocol). Step 2-2: Automated construction of original image data in medium to high resource languages ​​(Chinese, English, Malay) is performed. Field-level structured annotations are generated automatically, and consistency checks are used to reduce the uncontrollability of output distribution.

[0102] Steps 2-3: Perform multi-stage construction on the raw image data of low-resource languages ​​(Tamil) to generate field-level structured annotations;

[0103] The book title text corresponding to the image is generated through steps 2-2 and 2-3, and will be used for training in step 3.

[0104] The automated construction described in step 2-2 includes the following steps:

[0105] Step 2-2-1: Use the fields corresponding to the book title (such as main title, subtitle, language tags, etc.) as prompt words for generating the large data model;

[0106] The large data generation model described in this embodiment is the Gemini-3-Flash model, which can accurately extract relevant information about book titles through optimization for specific tasks;

[0107] Step 2-2-2: The large data generation model takes the prompt words from Step 2-2-1 and the original images from Step 1-1-1 to perform reasoning in order to identify the main title and subtitle in the text.

[0108] Step 2-2-3: Structured annotation generation. After identifying the various fields of the book title in step 2-2-2, this information is converted into a structured annotation format (JSON). The language tags are annotated using the language of each image obtained in step 1-1.

[0109] Steps 2-3 specifically include:

[0110] Step 2-3-1: Visual Enhancement: Use the cropped spine image data from Step 1-1-1 to perform visual enhancement processing. This stage uses image enhancement techniques (such as contrast stretching, sharpening, perspective correction, etc.) to improve the visual effect of low-quality images, thereby improving the recognition accuracy of OCR.

[0111] Step 2-3-2: Multi-source recognition: Apply multiple open-source OCR engines to the enhanced spine image in Step 2-3-1 for preliminary recognition to obtain multiple candidate book title texts. By combining multi-source OCR, different character representations and language forms are covered as much as possible to ensure the full extraction of book title information in low-resource languages.

[0112] Step 2-3-4: Semantic Adjudication: The data generation model (Gemini) is used to perform semantic-level correction and fusion of the multi-round OCR results in Step 2-3-2. The Gemini model uses the recognition results of multi-source OCR as evidence set and relies on the original images collected in Step 1-1-1 to perform consistency fusion, remove garbled characters, merge erroneously split fragments, and finally output a structured book title field. The language label is also labeled using the language of each image obtained in Step 1-1.

[0113] The dynamic weighted fine-tuning strategy based on cross-domain dual-validation set monitoring and historical performance feedback described in step 3 is as follows: The specific steps for receiving the multilingual structured training samples constructed in step 2 (i.e., the input is a spine image patch, and the labels are structured JSON text containing fields such as Title) and the visual data collected from different libraries in steps 1-1 are as follows:

[0114] Step 3-1, Data Preparation: Intervene on the distribution and quantity of data for different language labels, and adopt oversampling and undersampling data processing strategies for long-tail languages ​​and high-frequency languages; obtain a balanced dataset with different numbers of languages ​​under different libraries; the long-tail languages ​​refer to language categories that have a very small proportion of samples and a significantly lower frequency of occurrence than mainstream languages ​​in the actual library collection distribution and model training dataset; in this embodiment, the long-tail language is Tamil.

[0115] Step 3-2: Calculate the corresponding cross-entropy loss using the data from the balanced dataset.

[0116] Basic cross-entropy loss definition:

[0117]

[0118] The specific meanings of each parameter in the formula in our bookshelf recognition scenario are as follows: Represents a single real multilingual spine image patch as input; This refers to the target structured JSON text (containing main and subtitle fields) constructed for the image in step 2. This represents the network parameters that need to be optimized within the current visual language model; This represents the total probability that, under the current model parameters, the model can accurately and completely output the target JSON format and book title text based on the input spine image. A lower probability indicates greater difficulty in recognition and a higher baseline error. The larger.

[0119] For each language ( Using the basic cross-entropy formula and the corresponding data from step 2, calculate the cross-entropy loss on the two validation sets respectively, and then merge them into a unified cross-entropy loss. :

[0120]

[0121] The parameters in the formula have the following meanings: Language Cross-entropy loss on the current library's validation set; Language Cross-entropy loss on the validation set of other partitioned pavilions; : Scene weighting coefficient, which adjusts the trade-off between in-venue accuracy and cross-venue generalization ability.

[0122] Meanwhile, during the training process, the cached language is... The optimal cross-entropy loss value during training If the current training cycle measures This indicates that the data for that language experienced performance degradation during training, and the system immediately extracts and records the performance difference value. ;

[0123] Step 3-3: Based on the cross-entropy loss derived from the fusion of the validation sets in Step 3-2, design a dynamically weighted total loss:

[0124] In processing the current training batch (using the total number of images from step 1-1), At this stage, all samples are no longer treated equally. Instead, the cross-entropy loss results for each language calculated in step 3-2 are used to assign dynamic weights to each language. And calculate the final optimization objective for the current batch. :

[0125]

[0126]

[0127] The parameters in the formula are: Language The dynamic penalty weight is directly determined by the feedback from step 3-2: The larger the value (indicating a more severe underfit), the higher the weight. Historical rollback penalty items (among others) (This is a preset exponential amplification factor). Once step 3-2 detects a performance degradation in this language (i.e., ... This penalty term is amplified exponentially, forcing the model's computing power to be reallocated to this low-resource language; The first in the current batch The language of each image sample ( The corresponding dynamic weights mentioned above; The final total loss is the sum of the basic cross-entropy loss of all samples in the current batch, after being dynamically weighted according to the corresponding language.

[0128] Step 3-4, updating LoRA parameters based on dynamic weighted objective: Replace the traditional cross-entropy loss function with the dynamic weighted total loss constructed in step 3-3, and update the internal weights of the visual language model through backpropagation algorithm and low-rank adaptation (LoRA) technique to prevent gradient explosion and finally complete the model fine-tuning loop that takes into account multilingual balance and cross-domain generalization ability, and finally obtain the fine-tuned model.

[0129] To further verify the significant technical advantages of the dynamic fine-tuning strategy described in this invention compared to traditional visual recognition methods, combined with the appendix... Figure 5 (Training Loss Convergence Comparison Curve) and Appendix Figure 3 (Example image of structured book title recognition results) provides a comprehensive explanation.

[0130] 1. Superiority analysis at the Loss curve level (mathematical and training evidence)

[0131] As mentioned above, Appendix Figure 5 This clearly demonstrates the significant advantages of the dynamic weighted fine-tuning strategy of this invention compared to the traditional baseline method in terms of cross-entropy loss (the smaller the value, the better). Thanks to the "dual validation set fusion mechanism" and "historical backoff penalty" introduced in this invention, the solid curve (of this invention) maintains a downward trend throughout the entire training period, mitigating the rebound and degradation phenomenon of cross-entropy loss in the dashed curve of the baseline. More importantly, the method of this invention ultimately achieves the optimal cross-entropy loss point in terms of overall performance.

[0132] 2. Recognition Results

[0133] The advantages of the aforementioned underlying algorithms ultimately translate into a noticeable improvement in recognition performance in high-density bookshelf recognition tasks in the physical world. This is powerfully demonstrated in the attached... Figure 3 The structured book title recognition results.

[0134] In handling the attached Figure 3 When dealing with complex, mixed-layout bookshelf images exhibiting severe glare, the output of traditional visual recognition methods (Baseline) without fine-tuning using the specific strategies of this invention typically suffers from serious defects. Specifically:

[0135] Multilingual recognition crashes and garbled text: When faced with spines containing mixed Chinese and English text or long-tailed languages ​​(such as Malay and Tamil), traditional methods are prone to "missing some while focusing on others." For example, they can only recognize the dominant Chinese characters, while misidentifying adjacent foreign characters as garbled symbols, or even missing them altogether.

[0136] Uncontrollable format: Traditional methods typically output unstructured raw string fragments, making it impossible to distinguish between main titles and subtitles, and there is no unified JSON protocol format, which makes it impossible for subsequent steps to achieve accurate database matching.

[0137] (Concrete evidence of the method of this invention: Appendix) Figure 3 result)

[0138] In comparison, attached Figure 3 The final output result after applying the complete identification link of this invention is visually demonstrated. Thanks to the balanced optimization of different languages ​​by the "dynamic weighted fine-tuning strategy" in step 3, the invented model still maintains extremely high robustness under extremely complex input conditions.

[0139] Combined with appendix Figure 3 Specific examples prove:

[0140] As shown in the figure, for a typical Chinese-English bilingual mixed book, the model of the present invention successfully outputs the complete main title field containing "BADGES OF FURY 不二神探". However, the traditionally trained model will finally output the title of this book in two parts: "BADGES OF FURY" and "不二神探". This shows the robustness and advantages of our method.

[0141] Step 4 specifically includes:

[0142] Step 4-1: To align the model output with the structured protocol, construct preference comparison samples, and screen the model output by introducing an automated verification mechanism: Extract the inference result of the fine-tuned model on the original image in Step 3 as the candidate sample, and take the output that meets the structured output protocol format (structured output protocol: json format) as the preferred output , and take the output that does not meet as the non-preferred output , and thus construct a high-quality comparison data set for direct preference optimization (DPO);

[0143] Step 4-2: Based on the high-quality comparison data set, use direct preference optimization (DPO) to align and optimize the fine-tuned model, improve the format parseability, make the output meet the structured protocol and improve the JSON format legal rate, and obtain the model of the aligned model.

[0144] To achieve the efficient deployment of the multi-lingual vision-language model optimized by DPO in Step 4 on resource-constrained devices, this study introduces the activation-sensitive weight quantization (AWQ) technology to perform 4-bit low-bit quantization processing on the aligned model. The specific steps are as follows:

[0145] Step 5-1: Construction of a quantization calibration set for multi-dimensional robustness. For the model trained by DPO, randomly select 512 of the image data collected in Step 1-1 as the calibration data set;

[0146] Step 5-2: Based on the constructed calibration data set, perform quantization on the model aligned by DPO using the existing AWQ algorithm to obtain the quantized model.

[0147] The cross-modal global optimal allocation and isolated node adjudication described in Step 6: Specifically include:

[0148] After inputting the visual data collected in Step 1 into the quantized model in Step 5 to obtain the recognition result, perform cross-modal fusion and physical order diagnosis on the multi-source data of vision and RFID. Specifically include the following sub-steps:

[0149] Step 6-1: Visual Standardized Book Title Extraction and Database Mapping: Using the visual recognition results generated by the model in Step 5, multiple consecutive images are processed using an existing sliding window deduplication method to sort and merge the book titles in a specific order. These book titles are then compared to the library's collection database using an existing embedding text similarity algorithm. The final output is the actual book title in the collection database, i.e., the visually standardized book title. The text similarity retrieval score is defined as the entity's visually standardized identity confidence score. This allows us to obtain the actual book title and the confidence level of the visual results.

[0150] Step 6-2: In real-world high-density bookshelf environments, the total number of visual entities extracted from the same bookshelf layer often does not match the total number of RFID entities (i.e., visual missed detections or RFID cross-reads). To completely resolve this cross-modal conflict, this method constructs the following decision-making steps:

[0151] Step 6-2-1: Radio Frequency Confidence Quantization calculation:

[0152] Extract the RFID observation data within the time window in step 1, and construct a normalized radio frequency confidence score based on the returned signal strength indication (RSSI) and the reading frequency (Count). The specific calculation formula is as follows:

[0153]

[0154] In the formula, Normalized radio frequency confidence score, with a value range of: ; : The signal strength returned by the current RFID tag; The system's preset upper limit reference value for extremely strong near-field signals; The system's preset lower limit reference value for long-distance or wall-penetrating noise signals; : Truncation constraint function, used to forcibly limit the signal strength score after linear mapping to a certain value. Within the range; : The total number of times the RFID tag under test has actually been read; : The threshold for the expected frequency of tags reaching a stable reading state; Take the smaller function; : These are the physical weighting coefficients for signal strength characteristics and readout frequency characteristics, respectively, and they strictly satisfy... By adjusting this coefficient, the system can adapt to the attenuation characteristics of radio frequency signals on bookshelves made of different materials.

[0155] Step 6-2-2: Hungarian Algorithm Matching Solution: Using the visual entity as one end node and the RFID entity as the other end node, utilize the unified identity confidence level of the visual side. The radio frequency confidence level calculated above The difference is used to construct the matching cost, and the Hungarian algorithm is used to solve the minimum cost matching problem to obtain the optimal one-to-one correspondence between the visual and RFID entities of the current bookshelf layer.

[0156] Step 6-2-3: Handling mechanism for isolated nodes; For RFID results that failed to pair successfully due to an unequal number of nodes after solving the Hungarian algorithm in step 6-2-2, the radio frequency confidence level is considered. Determine whether to use the global uniqueness constraint:

[0157] Scenario A: If the radio frequency confidence level of the single node is... Signals below the set environmental attenuation threshold are physically identified as long-distance signals that cross floors / through walls and are directly removed as "serialized noise".

[0158] Scenario B: If the radio frequency confidence level of the single node is... If the value exceeds the threshold, but the system finds that the entity's unique identifier has been visually confirmed with high confidence in other physical bookshelf locations throughout the library, then a global exclusive deduplication constraint is triggered. The strong signal here is determined to be "strong signal crosstalk noise" caused by wall reflection and is removed (a book cannot be in two locations at the same time).

[0159] Scenario C: If the radio frequency confidence level of the single node is... If the value is above the threshold and meets the condition of not being visually detected globally, then the actual physical determination is "the real book that is severely obscured", and it is retained as a hidden node and forcibly inserted into the current bookshelf entity observation queue.

[0160] The final result is a properly sorted set of books.

[0161] To further illustrate the operational effectiveness of the "cross-modal fusion and physical sequence diagnosis" mechanism described in step 6 of this invention in real-world scenarios, in conjunction with the appendix... Figure 4 (A schematic diagram of the cross-modal matching fusion and misalignment diagnosis process) will be provided for detailed explanation.

[0162] 1. Initial cross-modal data collisions and noise ( Figure 4 (Top right part)

[0163] As attached Figure 4 As shown in the upper right corner, the system first retrieves the original "visual results" list and "RFID results" list for the same bookshelf layer. In real-world, high-density, complex scenarios, these two modalities of data naturally exhibit significant conflicts and noise, and each has its own typical drawbacks:

[0164] Typical noise in visual results—missing books: Due to severe mutual occlusion, reflections, or shadows in dense bookshelves, visual algorithms often fail to detect some books that are actually present (such as those in the appendix). Figure 4 The visual list omits books such as "The Assassin Nie Yinniang" from the RFID list. This results in the visual list often having fewer books than the actual physical shelves.

[0165] Typical noise in RFID results – cross-reading: RFID readers often read tag signals from adjacent shelves or even opposite shelves (as shown in the attached image). Figure 4 The RFID list includes terms like "the art of communication for outstanding leaders" and "leadership charisma," which are not currently visible in the visual field. This often results in the RFID list containing more books than the actual physical shelves.

[0166] The mismatch between quantity and content: This combination of visual omission and RFID cross-reading results in significant differences in the number and content of items between the two lists. Traditional simple alignment methods are completely ineffective and are prone to causing inaccurate matching problems.

[0167] Because visual recognition is sequential but can lead to errors, while RFID is correct but lacks order, this method maps them together: "obtain the optimal one-to-one correspondence between the visual recognition and RFID entities of the current bookshelf layer;" forming a final, ordered, and correct system for mis-shelf identification.

[0168] 2. Matching fusion and noise filtering based on the Hungarian algorithm ( Figure 4 (Mid-matching and fusion process)

[0169] To address the aforementioned severe cross-modal conflicts, the system executed the "matching and fusion" process described in step 6-2. As shown by the solid black lines connecting the visual list and the RFID list in the figure, these lines visually represent the globally optimal "one-to-one correspondence" calculated by the "Hungarian matching algorithm based on confidence difference" in this invention, and further processed using the rules in step 6-2-3 to obtain the final list of book titles.

[0170] Through this algorithm:

[0171] Precise matching: The system successfully found the most matching RFID entity (such as "The Invincible Detective" in the RFID list) for high-confidence visual entities (such as "The Invincible Detective" in the visual list) in a complex candidate pool.

[0172] Noise Removal and Missing Book Marking: The algorithm automatically identifies and removes noisy RFID reading nodes that cannot be matched (such as "The Art of Communication for Outstanding Leaders"). At the same time, for books that are visually missed but read by RFID (such as "The Assassin Nie Yinniang"), the system marks them as "visually missed" instead of forcibly performing incorrect matching.

[0173] Ultimately, the system successfully extracted a clean "final fusion list" from the noisy raw data, one that was highly bound to physical location and identity information. Figure 4 (As shown in the lower right part).

[0174] 3. Physical misalignment diagnosis based on shelving index direction ( Figure 4 (Bottom and left side indicators)

[0175] After obtaining a clean fusion list, the system performs the "physical sequence diagnosis" step 6-3.

[0176] Shelf arrangement rules and standards: as attached Figure 4 As shown by the long red arrow below the original image on the left, the library's bookshelf arrangement rule is defined as "call number size direction" increasing from right to left (that is, the further to the left the physical location, the larger the call number should be).

[0177] Monotonicity check and diagnosis: Based on this rule, the system performs a monotonically increasing check on the physical location sequence of the call number corresponding to each book in the fusion list.

[0178] Visualization of diagnostic results: During the verification process, the system accurately detected breakpoints in the sequence that violated the monotonicity rule. (See attached image.) Figure 4 As shown in the lower right corner, the system uses red vertical arrows and the text "Misplaced Books" to accurately mark those books whose actual physical location does not match the correct order of their call numbers in the final list (such as "Original People's Songs", "Land Praise Songs", and "Dragon and Phoenix Shop" whose location sequence is logically disordered).

[0179] In summary, Appendix Figure 4 This invention clearly and completely presents the entire process of how it transforms multimodal sensor data (visual missing books and RFID cross-reading) that is originally highly conflicting and noisy into accurate and executable physical mis-shelf diagnosis conclusions. Compared with most current simple matching or single-modal results, this invention fully demonstrates the technical advancement and practical application value of the method.

[0180] Step 6-3: Fine-grained diagnosis of mis-shelf and intra-shelf disorder: Extract the book entities confirmed by cross-modal fusion in Step 6-2 and search for their call numbers in the library's collection database. Based on their actual physical order sequence on the current physical bookshelf layer (from left to right), verify whether the shelving indexes of adjacent books strictly satisfy the monotonically increasing lexicographical order. If the call number sequence breaks the monotonically increasing rule, it is determined that a fine-grained "intra-shelf disorder" has occurred, and the mis-shelf and correct position candidates are output.

[0181] This invention provides a method for multilingual book title recognition and dynamic optimization in high-density complex bookshelves. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for multi-lingual title recognition and dynamic optimization of high-density complex bookshelves, characterized in that, The steps include the following: Step 1: Multi-source observation and unified preprocessing: Collect visual observation data and RFID observation data of the bookshelf scene; Step 2: Construct training samples according to the field definitions and structured output protocol; Step 3: The open-source pre-trained visual language model is subjected to a dynamic weighted fine-tuning strategy based on cross-domain dual validation set monitoring and historical performance feedback, thereby obtaining a preliminary fine-tuned model with stable multilingual structured prediction output capability. Step 4: Align the output behavior and format of the initial fine-tuned model with those of the direct preference optimization to obtain an aligned model that is consistent in both format and behavior; Step 5: Perform AWQ quantization on the aligned model for edge deployment; The visual observation data collected in step 1 is input into the AWQ quantized model to output structured visual recognition results; Step 6: Perform cross-modal fusion and physical sequence diagnosis of the visual recognition results and RFID multi-source data.

2. The method for multi-lingual book title recognition and dynamic optimization of high-density complex bookshelves according to claim 1, characterized in that, Step 1, which involves collecting visual observation data, specifically includes: Step 1-1: Obtain spine image patches from bookshelves in different libraries as the original image dataset, and divide the dataset into the current library validation set and other library validation sets according to the different libraries. Steps 1-2 involve processing each image block to identify the text information and thus obtain the language corresponding to the original image.

3. The method of claim 1, wherein the method is characterized by, Step 2 specifically includes: Step 2-1: Define the structured output protocol as the JSON field protocol, and split the book title information into main title, subtitle, language tag, and confidence score; Step 2-2: Automated construction of original image data in medium and high resource languages, generating field-level structured annotations through automated methods, and reducing the uncontrollability of output distribution through consistency verification; Steps 2-3: Perform multi-stage construction on the original image data of low-resource languages ​​to generate field-level structured annotations.

4. The method for multi-lingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 3, characterized in that, The automated construction described in step 2-2 includes the following steps: Step 2-2-1: Use the field corresponding to the book title as data to generate prompt words for the large model; Step 2-2-2: The data generation model accepts the prompts from Step 2-2-1 and the original images from Step 1-1 to perform reasoning in order to identify the main title and subtitle in the text. Step 2-2-3: Structured annotation generation. Based on the identified fields of the book title, this information is converted into a structured annotation format, and the language tags are annotated using the language of each image.

5. The method for multi-lingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 3, wherein, Steps 2-3 specifically include: Step 2-3-1, Visual Enhancement: Perform visual enhancement processing on the original image data; Step 2-3-2, Multi-source recognition: Apply multiple OCR engines to the enhanced spine image for preliminary recognition to obtain multiple candidate book title texts. By combining multi-source OCR, different character representations and language forms are covered. Step 2-3-4, Semantic Adjudication: Perform semantic-level correction and fusion on the multi-round OCR results in Step 2-3-2, perform consistency fusion, remove garbled characters, merge erroneously split fragments, and finally output a structured book title field. Also, use the language of each image obtained in Step 1-1 to label the language tag.

6. The method of claim 1, wherein the method of multi-lingual book title recognition and dynamic optimization of high-density complex bookshelves is characterized by, The dynamic weighted fine-tuning strategy based on cross-domain dual-validation set monitoring and historical performance feedback described in step 3 is as follows: Step 3-1, Data Preparation: Intervene on the distribution of data quantity for different language labels in the training samples, and adopt oversampling and undersampling data processing strategies for long-tail languages ​​and high-frequency languages; Obtain a balanced dataset with varying numbers of languages ​​across different libraries; Step 3-2, calculate the base cross-entropy loss of each language using the data of the balanced data set : Step 3-3: Calculate the cross-entropy loss of each language on the two validation sets using the basic cross-entropy formula, and then merge them into a unified cross-entropy loss. Meanwhile, caching languages The optimal cross-entropy loss value during training And extract and record performance difference values. ; Steps 3-4: Calculate the dynamically weighted total loss based on the cross-entropy loss of the fusion of the validation set; Steps 3-5: Using the dynamically weighted total loss as the loss function for training the visual language model, the internal weights of the visual language model are updated through backpropagation and low-rank adaptation techniques to finally obtain the finely tuned model.

7. The method for multilingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 6, characterized in that, The dynamic weighted total loss described in steps 3-4 : In the formula: For language Dynamic penalty weights; This is a penalty item for historical regression, among which This is the preset exponential amplification factor. The first in the current batch Language of the image sample The corresponding dynamic weights mentioned above; This represents the basic cross-entropy loss for all samples in the current batch.

8. The method for multilingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 1, characterized in that, Step 4 specifically includes: Step 4-1: Use the inference results of the fine-tuned model on the original image as candidate samples, and use the output that meets the structured output protocol format as the preferred output. The output that does not meet the requirement will be treated as the non-preference output. This allows us to construct a high-quality contrastive dataset for direct preference optimization. Step 4-2: Based on the high-quality comparison dataset, use direct preference optimization to align the fine-tuned model to obtain the aligned model.

9. The method for multilingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 1, characterized in that, Step 6 specifically involves: Step 6-1: The visual recognition results are sorted by deduplication using a sliding window method. Then, the real book titles in the database are retrieved and matched using a text similarity algorithm to obtain the visually standardized book titles and their unified identity confidence scores. ; Step 6-2: Perform cross-modal global optimal allocation and isolated node adjudication on the visually standardized book titles to obtain the final truly sorted book titles: Step 6-3, Fine-grained diagnosis of mis-shelf and disordered order within a shelf: Extract the book names obtained in Step 6-2 and search for their call numbers in the library database. Based on their actual physical order sequence on the current physical bookshelf shelf, check whether the shelf indexes of adjacent books strictly satisfy the monotonically increasing property of lexicographical order. If not, output the mis-shelf and correct position candidates.

10. The method for multilingual book title recognition and dynamic optimization of a high-density complex bookshelf according to claim 9, characterized in that, Step 6-2 specifically includes: Step 6-2-1: Extract RFID observation data within the time window, and construct a normalized radio frequency confidence score based on the returned signal strength indication and reading frequency. ; Step 6-2-2: Using the visual entity as one end node and the RFID entity as the other end node, utilize the unified identity confidence level of the visual side. With radio frequency confidence The difference is used to construct the matching cost, and the Hungarian algorithm is used to solve the minimum cost matching problem to obtain the optimal one-to-one correspondence between the visual and RFID entities of the current bookshelf layer. Step 6-2-3, the mechanism for handling isolated nodes: For RFID results that failed to pair successfully due to an imbalance in the number of nodes after solving the Hungarian algorithm, the mechanism is combined with their radio frequency confidence level. It is compared with the global uniqueness constraint to determine whether to remove it.

Images