Intelligent Tool Cabinet System and Control Method Based on Multimodal Large Language Model

Abstract

This invention belongs to the field of artificial intelligence and intelligent systems technology, and particularly relates to an intelligent tool cabinet system and control method based on a multimodal large language model. It includes: a core processing module that acquires a status image of the intelligent tool cabinet and inputs the status image to a multimodal recognition module; a prompt word engineering module that inputs prompt words to the multimodal recognition module, the prompt words including role information, output format information, and spatial distribution guidance words; the multimodal recognition module integrates a multimodal large language model, which, based on the status image and prompt words of the intelligent tool cabinet, realizes the recognition of tool names and borrowing / returning status; and a data processing module receives the recognition results from the multimodal recognition module and completes the extraction and storage of tool retrieval and / or return records. This invention, based on a multimodal large language model and prompt word engineering, solves the problems of high initial investment cost and limited accuracy in existing technologies.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and intelligent systems technology, and particularly relates to an intelligent tool cabinet system and control method based on a multimodal large language model. Background Technology

[0002] Intelligent tool cabinets have multiple uses, such as intelligent bookshelves and intelligent snack cabinets. Taking intelligent bookshelves as an example, with the rapid development of artificial intelligence technology, people's demand for intelligent and personalized reading spaces is increasing. Traditional bookshelves only serve the function of storing books. Therefore, the traditional library model, which relies on a large number of people for book management and lending services, has exposed drawbacks such as high labor costs and low management efficiency. The emergence of intelligent bookshelves has broken this traditional model, promoting the development of book lending towards intelligence and automation, and becoming an important breakthrough in upgrading library services.

[0003] The existing technological approaches to smart bookshelves can be mainly categorized into two types: RFID radio frequency identification and computer vision algorithms. RFID-based smart bookshelves have several limitations: firstly, each book requires an additional RFID tag, resulting in high initial investment costs; secondly, RFID tags are prone to identification failures or missed reads due to physical damage, close proximity of tags between adjacent books, and other factors. Furthermore, communication between the tag and the reader is highly susceptible to environmental electromagnetic interference, severely impacting the system's stability and reliability. On the other hand, smart bookshelves based on current mainstream computer vision algorithms require collecting image datasets of massive amounts of books and training models, which is extremely costly. Moreover, the dense and compact arrangement of books in smart bookshelves, with similar spine shapes and a lack of significant patterns or visual features, further increases the difficulty of computer vision methods in book recognition. Summary of the Invention

[0004] In view of this, the present invention aims to provide an intelligent tool cabinet system and control method based on a multimodal large language model, so as to solve the problems of high initial investment cost and limited accuracy of the existing technology. The intelligent tool cabinet designed based on the multimodal large language model of the present invention effectively avoids the above problems and provides a new direction for the development of intelligent tool cabinets.

[0005] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0006] A smart tool cabinet system based on a multimodal large language model includes: a core processing module, and a prompt word engineering module, a multimodal recognition module, and a data processing module connected to the core processing module. The core processing module acquires a status image of the smart tool cabinet and inputs the status image to the multimodal recognition module. The prompt word engineering module inputs prompt words to the multimodal recognition module, and the prompt words include role information, output format information, and spatial distribution guidance words. The multimodal recognition module integrates a multimodal large language model, which, based on the status image and prompt words of the smart tool cabinet, recognizes the tool name and its borrowing / returning status. The data processing module receives the recognition results from the multimodal recognition module and extracts and stores the tool retrieval and / or return records.

[0007] Furthermore, the core processing module is based on an edge computing platform and is connected to an authentication module, an image acquisition module, a lock control module, a Hall sensor module, a status sensing module, and a user interaction module.

[0008] Furthermore, the identity verification module completes user identity verification through facial recognition, RFID card recognition, or fingerprint recognition.

[0009] Furthermore, the prompt word engineering module inputs prompt words into the multimodal large language model through preset prompt strategies, which include:

[0010] The role-based prompting strategy involves the prompting word engineering module inputting role information into the multimodal large language model. The multimodal large language model then sets the tool administrator role based on the role information to match the tool management scenario.

[0011] Based on a template-based prompting strategy, the prompting word engineering module inputs formatted prompting words into the multimodal large language model, and the multimodal large language model outputs them in a fixed format according to the formatted information.

[0012] Based on the task decomposition strategy, the prompt words input by the prompt word engineering module to the multimodal large language model are spatial distribution guide words. The spatial distribution guide words guide the multimodal large language model to compare and analyze the changes in the spatial distribution of tools before and after the smart tool cabinet door is opened and closed.

[0013] Furthermore, the multimodal large language model is GPT-4o.

[0014] Furthermore, the core processing module acquires the status images of the smart tool cabinet, including the status image of the smart tool cabinet before user operation and the status image of the smart tool cabinet after user operation.

[0015] A control method for an intelligent tool cabinet based on a multimodal large language model is disclosed, which is implemented using an intelligent tool cabinet system based on a multimodal large language model. The method includes the following steps:

[0016] S1: The core processing module acquires the status image of the intelligent tool cabinet and inputs the status image into the multimodal recognition module;

[0017] S2: The prompt word engineering module inputs prompt words into the multimodal recognition module. The prompt words include role information, output format information, and spatial distribution guidance words.

[0018] S3: The multimodal recognition module integrates a multimodal large language model. Based on the status images and prompts of the smart tool cabinet, the multimodal large language model realizes the recognition of tool names and borrowing / returning status.

[0019] S4: The data processing module receives the recognition results from the multimodal recognition module and completes the extraction and storage of tool retrieval and / or return records.

[0020] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0021] (1) The intelligent tool cabinet system and control method based on multimodal large language model created by the present invention analyzes the differences in the bookshelf images before and after borrowing and returning through multimodal large language model. It does not require RFID tags or massive image training data, but only uses the model's image and text understanding ability to identify books, saving the cost of purchasing and maintaining RFID tags. At the same time, it does not require collecting image datasets for thousands of books, reducing the cost of data annotation and model training. In addition, the present invention can also avoid the problem of missed reading caused by physical damage to RFID tags, electromagnetic interference and visual algorithms due to similar spine features.

[0022] (2) The intelligent tool cabinet system and control method based on the multimodal large language model created by the present invention improves the recognition efficiency and standardization based on the multi-strategy interaction mechanism optimized by prompt word engineering. The present invention improves the consistency of output format from 80% to 100% through template design, solving the problem that traditional methods require manual correction due to chaotic format and have a large time consumption. The present invention improves data processing efficiency by directly reducing manual intervention.

[0023] (3) The intelligent tool cabinet system and control method based on a multimodal large language model described in this invention integrates modules such as face recognition, magnetic lock, and Hall sensor on the NVIDIA Jetson Orin NX platform, automatically completing the entire process of "identity verification, image acquisition, model inference, and record storage" without manual supervision. This invention can replace the repetitive work of 2-3 administrators in a traditional library, reducing manpower expenditure. Furthermore, it supports the borrowing and returning of multiple books in a single operation. The Hall sensor and magnetic lock are linked to control the cabinet door, avoiding recognition errors caused by users not closing the door. In other words, this edge computing-driven, fully automated hardware system achieves unmanned management and efficient borrowing and returning.

[0024] (4) The intelligent tool cabinet system and control method based on a multimodal large language model described in this invention uses hardware interfaces (USB, GPIO) and software prompt word modules to support flexible adjustments. By replacing prompt words (such as changing from "book recognition" to "product recognition") and adapting domain data, it can be migrated to scenarios such as smart retail. This invention adopts a cross-scenario portable modular architecture, which can solve the problems of technology reuse and scenario adaptation. Using this invention, there is no need to redevelop hardware or train models, and the deployment cycle is greatly reduced in time cost compared to traditional methods. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0026] Figure 1 A schematic diagram of the structure of the intelligent tool cabinet system based on a multimodal large language model as described in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the workflow of the intelligent tool cabinet based on a multimodal large language model as described in the embodiments of the present invention;

[0028] Figure 3 This is a flowchart illustrating the control method for an intelligent tool cabinet based on a multimodal large language model, as described in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0031] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0032] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0033] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] like Figure 1 As shown, this invention proposes an intelligent tool cabinet system based on a multimodal large language model, comprising: a core processing module, a prompt word engineering module, a multimodal recognition module, and a data processing module. The core processing module acquires a status image of the intelligent tool cabinet and inputs the status image into the multimodal recognition module. The prompt word engineering module inputs prompt words into the multimodal recognition module, the prompt words including role information, output format information, and spatial distribution guidance words. The multimodal recognition module integrates a multimodal large language model, which, based on the status image and prompt words of the intelligent tool cabinet, recognizes the tool name and its borrowing / returning status. The data processing module receives the recognition results from the multimodal recognition module and extracts and stores the tool retrieval and / or return records.

[0035] It should be noted that the intelligent tool cabinet proposed in this invention does not require 24 / 7 human supervision. It can efficiently complete functions such as borrowing and returning books through a self-service system. Simultaneously, it utilizes the powerful understanding and recognition capabilities of a multimodal large language model, and through optimization of prompt words, accurately identifies books, significantly reducing library manpower costs and freeing staff from repetitive, low-value-added tasks. Furthermore, through targeted adjustments to modules such as prompt words, it can be flexibly applied to scenarios such as smart retail and smart bookshelves.

[0036] Specifically, this invention introduces a multimodal large language model to construct an intelligent tool cabinet system that requires no additional physical tags, is unaffected by environmental interference, and does not require massive image training. Specifically, it achieves the following technological breakthroughs: 1) Reduced initial costs: Eliminating the need for RFID tags on books or pre-collecting large amounts of image data avoids hardware tag costs and data training costs; 2) Improved recognition stability: Book recognition is achieved based on text-based information interaction, avoiding problems such as physical damage to RFID tags, electromagnetic interference, and visual feature similarity that affect recognition, thus improving recognition accuracy and system reliability; 3) Simplified technical path: Book management is completed through natural language interaction and semantic understanding, facilitating subsequent information extraction, breaking through the technical bottleneck of traditional visual algorithms relying on complex image feature extraction, and reducing the difficulty of system deployment and maintenance.

[0037] In some embodiments, the core processing module is based on an edge computing platform and is connected to an authentication module, an image acquisition module, a lock control module, a Hall sensor module, a status sensing module, and a user interaction module.

[0038] It should be noted that the multimodal interaction interface of this invention supports data access such as image acquisition (USB camera), user touch (HDMI touch screen), and authentication (face recognition); the core processing module is based on the NVIDIA Jetson OrinNX edge computing platform, integrating a multimodal large language model and prompt word engineering optimization components, and can realize tool status recognition and borrowing / returning record generation through standardized API interfaces; the hardware expansion capability supports the connection of external peripheral modules such as magnetic locks and Hall sensors through GPIO interfaces, so as to meet the hardware configuration requirements of different scenarios through modular design.

[0039] Furthermore, the hardware system architecture of this invention is based on the core hardware platform NVIDIA Jetson Orin NX, and combines an authentication module, an image acquisition module, a lock control module, a status sensing module, and a user interaction module to realize the corresponding functions. The authentication module can acquire the user's facial image through a camera and compare it with the database to achieve authentication; the image acquisition module has a built-in high-definition camera to capture the status of the tools in the tool cabinet; the lock control module receives system commands to control the cabinet door opening and closing, and uses an electromagnetic lock to achieve contactless unlocking; the Hall sensor module triggers image acquisition and lock control logic through distance sensing; the user interaction module uses a touch screen for operation guidance (such as clicking the "borrow / return" button) and result display.

[0040] This invention constructs a fully automated system encompassing user authentication, cabinet door control, image acquisition, model inference, and data storage, achieving an intelligent borrowing and returning experience with "unattended operation and real-time response." To enhance understanding of the entire process, using an intelligent tool cabinet as an example of an intelligent bookcase, the detailed borrowing and returning process is illustrated: The user initiates an operation request by clicking the "Borrow / Return Books" button through a user interaction module (such as a touchscreen). The user interaction module transmits the signal to the system core, triggering subsequent processes and simultaneously prompting the user to authenticate. After receiving the signal, the authentication module (such as using facial recognition) acquires the user's facial image through a camera and compares it with the database: if the verification fails, the result is fed back through the user interaction module (such as "identity not matched"), the process terminates, and the user waits to try again; if the verification succeeds, the image acquisition module automatically captures the initial state before the cabinet door opens, recorded as Image1, the authentication module sends an unlock command to the lock control module, and simultaneously prompts "cabinet door unlocked, please operate" through the user interaction module. After the user completes the operation and closes the cabinet door, they click the "Process Complete" button through the user interaction module to confirm the end of the operation. The Hall sensor module detects the change in the cabinet door's state from "open" to "closed" and immediately sends a trigger signal to the image acquisition module (such as a high-definition camera). The image acquisition module responds to the trigger signal, automatically captures the final state of the cabinet door after it is closed, and records it as Image2. The state image (including Image1 and Image2) is temporarily stored in the system. The image acquisition module uploads the state image to a multimodal large language model guided by prompts to analyze the differences in the operation (such as the addition or removal of items). The analysis results (including the names of borrowed and returned books) are displayed through the user interaction module for user confirmation. After user confirmation, the data (including timestamp, username, and operation content) is stored in the database (MySQL database can be used), the user interaction module displays "Operation Complete," and the system resets to its initial state.

[0041] In some embodiments, the authentication module completes user authentication through facial recognition, RFID card recognition, or fingerprint recognition.

[0042] It should be noted that if fingerprint recognition is used, a fingerprint sensor needs to be integrated to verify the user's identity through fingerprint comparison.

[0043] In some embodiments, the prompt word engineering module inputs prompt words into the multimodal large language model through a preset prompt strategy, which includes:

[0044] The role-based prompting strategy involves the prompting word engineering module inputting role information into the multimodal large language model. The multimodal large language model then sets the tool administrator role based on the role information to match the tool management scenario.

[0045] Based on a template-based prompting strategy, the prompting word engineering module inputs formatted prompting words into the multimodal large language model, and the multimodal large language model outputs them in a fixed format according to the formatted information.

[0046] Based on the task decomposition strategy, the prompt words input by the prompt word engineering module to the multimodal large language model are spatial distribution guide words. The spatial distribution guide words guide the multimodal large language model to compare and analyze the changes in the spatial distribution of tools before and after the smart tool cabinet door is opened and closed.

[0047] It should be noted that this invention employs a multimodal large language model and cue word engineering in tool recognition. GPT-4o, developed by OpenAI, is an advanced multimodal large language model capable of simultaneously processing text and image information. This invention utilizes its powerful visual recognition capabilities to analyze the input image content, while simultaneously understanding and processing related text instructions and information through text interaction functions. Cue word engineering refers to the careful design and optimization of cue words input into the multimodal large language model to guide the model to better understand task requirements and generate expected output. After receiving the image and cue words, the GPT-4o model utilizes its multimodal processing capabilities to perform in-depth analysis of the image, combining the cue words to understand the task requirements and accurately identify the book title in the image. The cue word engineering introduced in this invention specifically employs three cue word acquisition methods: role-based, template-based design, and task decomposition-based.

[0048] Furthermore, taking a smart tool cabinet as an example of a smart bookcase, a role-based prompting strategy assigns a specific role to the multimodal large language model, making it more targeted in task processing. For example, "You are a librarian." A template-based prompting strategy designs a fixed template to guide the multimodal large language model to output according to a specific format. For example, "The borrowed books are: 'Recognition Result 1', 'Recognition Result 2'; the returned books are: 'Recognition Result 1', 'Recognition Result 2'." A task decomposition-based prompting strategy guides the model to improve its image analysis. For example, "Observe the distribution of books in the bookcase and compare the changes inside the bookcase before and after opening the door." By comparing the basic requirements with prompts based on role setting, template design, and task decomposition strategies, the effectiveness of different strategies in guiding the multimodal large language model to process the "smart tool cabinet (simulated smart bookcase) book borrowing and returning recognition" task is evaluated. The impact of each strategy on output accuracy, structure, and scene adaptability is clarified. Experiments show that by using a multimodal large language model and prompt engineering, book borrowing and returning information can be accurately obtained. At the same time, this information is output according to the required template, which facilitates the extraction of information by subsequent code.

[0049] In some embodiments, the multimodal large language model is GPT-4o.

[0050] In some embodiments, the state images of the smart tool cabinet obtained by the core processing module include the state image of the smart tool cabinet before user operation and the state image of the smart tool cabinet after user operation.

[0051] This invention also provides a control method for an intelligent tool cabinet based on a multimodal large language model, implemented using an intelligent tool cabinet system based on a multimodal large language model, specifically including the following steps:

[0052] S1: The core processing module acquires the status image of the intelligent tool cabinet and inputs the status image into the multimodal recognition module;

[0053] S2: The prompt word engineering module inputs prompt words into the multimodal recognition module. The prompt words include role information, output format information, and spatial distribution guidance words.

[0054] S3: The multimodal recognition module integrates a multimodal large language model. Based on the status images and prompts of the smart tool cabinet, the multimodal large language model realizes the recognition of tool names and borrowing / returning status.

[0055] S4: The data processing module receives the recognition results from the multimodal recognition module and completes the extraction and storage of tool retrieval and / or return records.

[0056] It should be noted that this invention employs label-free book recognition technology based on a multimodal large language model. By integrating the multimodal large language model and combining image data before and after borrowing and returning, the model's image and text understanding capabilities are utilized to achieve accurate recognition of tool names and statuses. This invention also proposes a multi-strategy interaction mechanism optimized by prompt word engineering. Through the innovative introduction of prompt word engineering, and strategies such as template design and task decomposition, the multimodal large language model is guided to quickly understand scene information and generate borrowing and returning records that meet the requirements, improving recognition accuracy and reasoning efficiency, and reducing reliance on manually labeled data. Furthermore, the hardware interface and software model used in this invention support modular adjustments. By replacing prompt word templates and adapting domain data (such as converting book images to product packaging images), it can be quickly migrated to scenarios such as smart retail and logistics warehousing, achieving functional reuse of "smart bookshelves - vending machines - inventory management systems," reducing cross-domain deployment costs.

[0057] Finally, it should be noted that the multimodal large language model is constantly evolving and is not limited to using the GPT-4o model. The edge computing platform utilizes domestically produced edge computing equipment, which can further reduce costs. Furthermore, in book recognition, a book knowledge base (containing structured data such as book title, author, and spine features) is constructed. Knowledge base references are added to prompts (e.g., "Based on knowledge base ID: BN001, identify the book title corresponding to this spine") to guide the multimodal large language model in combining prior knowledge to analyze images.

[0058] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0059] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An intelligent tool cabinet system based on a multimodal large language model, characterized in that: include: The system comprises a core processing module, and connected to it a prompt word engineering module, a multimodal recognition module, and a data processing module. The core processing module acquires a status image of the smart tool cabinet and inputs it into the multimodal recognition module. The prompt word engineering module inputs prompt words into the multimodal recognition module, including role information, output format information, and spatial distribution guidance words. The multimodal recognition module integrates a multimodal large language model, which, based on the status image and prompt words of the smart tool cabinet, identifies tool names and their borrowing / returning status. The data processing module receives the recognition results from the multimodal recognition module and extracts and stores tool retrieval and / or return records.

2. The intelligent tool cabinet system based on a multimodal large language model according to claim 1, characterized in that: The core processing module is based on an edge computing platform and is connected to an authentication module, an image acquisition module, a lock control module, a Hall sensor module, a status sensing module, and a user interaction module.

3. The intelligent tool cabinet system based on a multimodal large language model according to claim 2, characterized in that: The identity verification module verifies user identity through facial recognition, RFID card recognition, or fingerprint recognition.

4. The intelligent tool cabinet system based on a multimodal large language model according to claim 1, characterized in that: The prompt word engineering module inputs prompt words into the multimodal large language model through preset prompt strategies. The preset prompt strategies include: The role-based prompting strategy involves the prompting word engineering module inputting role information into the multimodal large language model. The multimodal large language model then sets the tool administrator role based on the role information to match the tool management scenario. Based on a template-based prompting strategy, the prompting word engineering module inputs formatted prompting words into the multimodal large language model, and the multimodal large language model outputs them in a fixed format according to the formatted information. Based on the task decomposition strategy, the prompt words input by the prompt word engineering module to the multimodal large language model are spatial distribution guide words. The spatial distribution guide words guide the multimodal large language model to compare and analyze the changes in the spatial distribution of tools before and after the smart tool cabinet door is opened and closed.

5. The intelligent tool cabinet system based on a multimodal large language model according to claim 1, characterized in that: The multimodal large language model is GPT-4o.

6. The intelligent tool cabinet system based on a multimodal large language model according to claim 1, characterized in that: The core processing module acquires the status images of the smart tool cabinet, including the status image of the smart tool cabinet before user operation and the status image of the smart tool cabinet after user operation.

7. A control method for an intelligent tool cabinet based on a multimodal large language model, implemented using the intelligent tool cabinet system based on a multimodal large language model as described in any one of claims 1-6, characterized in that: Specifically, the steps include the following: S1: The core processing module acquires the status image of the intelligent tool cabinet and inputs the status image into the multimodal recognition module; S2: The prompt word engineering module inputs prompt words into the multimodal recognition module. The prompt words include role information, output format information, and spatial distribution guidance words. S3: The multimodal recognition module integrates a multimodal large language model. Based on the status images and prompts of the smart tool cabinet, the multimodal large language model realizes the recognition of tool names and borrowing / returning status. S4: The data processing module receives the recognition results from the multimodal recognition module and completes the extraction and storage of tool retrieval and / or return records.

Images