Industrial scene-oriented multi-modal retrieval enhanced model fine-tuning method
By constructing a multimodal knowledge base and target segmentation network, and combining a dynamic dual-path retrieval mechanism and knowledge distillation technology, the multimodal retrieval enhancement model is optimized, solving the problems of difficult model deployment and visual feature omission in industrial scenarios, and achieving efficient and accurate multimodal retrieval and generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV OF SCI & TECH
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-09
AI Technical Summary
Existing multimodal language models are difficult to deploy in industrial scenarios, have high computing costs, lack professional knowledge, are prone to illusions and miss key visual features, and existing multimodal retrieval enhancement technologies deviate from the key points in complex backgrounds.
We construct a multimodal retrieval enhancement model for industrial scenarios. By building a multimodal knowledge base, we use a target segmentation network to accurately locate key regions in images, fuse multi-resolution visual features, and combine a fine-tuning strategy with extremely low computational consumption. We design a dynamic dual-path retrieval mechanism and knowledge distillation technology to optimize the model's multimodal feature alignment and retrieval generation in vertical domains.
It enables efficient deployment on the industrial edge computing side, improves the model's inference accuracy and local detail capture capability, eliminates illusion phenomena, ensures data security, and meets the high-frequency real-time requirements of industrial production.
Smart Images
Figure CN122173672A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal large model technology, and more specifically, to a method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios. Background Technology
[0002] In recent years, multimodal language models have demonstrated powerful cross-modal understanding and generation capabilities. However, their massive parameter size and high computational cost severely limit their deployment on industrial terminals and edge devices with limited computing power. Simultaneously, due to confidentiality requirements for core production data, enterprises typically cannot upload sensitive internal text and image data to large cloud models for inference. To address these deployment challenges, lightweight multimodal models with fewer parameters are often deployed locally. While this effectively reduces model size, it results in a severe lack of domain-specific expertise when dealing with complex industrial scenarios, making them prone to illusions and failing to meet the stringent requirements of high accuracy and reliability in industrial production. Furthermore, on-site images in industrial scenarios often have cluttered backgrounds and minute key details (such as minute cracks in equipment or complex instrument scales). Conventional lightweight models that have not undergone deep adaptation are prone to losing these crucial visual features during processing.
[0003] Multimodal retrieval enhancement technology offers an effective way to address the aforementioned problems of lack of professional knowledge and illusion. This technology introduces external textual and graphical knowledge to support the multimodal language model in generating correct answers, improving output accuracy without significantly increasing model parameters. However, existing technologies still face several technical bottlenecks when applied to industrial scenarios: First, current open-source corpora used to train retrieval machines largely rely on general internet data, with extremely low coverage of non-public data in industrial scenarios (such as process flow diagrams, quality inspection defect diagrams, and equipment maintenance manuals), resulting in inaccurate knowledge recall by the retrieval machine. Second, if the entire multimodal language model and retrieval machine are jointly and fully fine-tuned to adapt to private industrial scenarios, the high computational cost becomes a significant obstacle, making it difficult to complete on local enterprise equipment. Finally, existing multimodal retrieval enhancement technologies typically extract the entire image for global retrieval, which, when faced with complex industrial textual and graphical backgrounds, is easily affected by redundant background information, causing the retrieval to deviate from its focus and thus misleading the language model's output.
[0004] To address the core pain points of general-purpose large models in industrial scenarios, such as data leakage risks and high computing costs, as well as the lack of professional knowledge, susceptibility to illusions, and tendency to miss key visual features in lightweight models, this invention proposes a multimodal retrieval enhancement model fine-tuning method for industrial scenarios, based on actual industrial needs. This method not only aims to construct a dedicated multimodal knowledge base using private industrial image and text data to compensate for the lack of professional knowledge, but also hopes to accurately locate key regions of images and fuse multi-resolution visual features through a lightweight target segmentation network. Combined with a fine-tuning strategy with extremely low computing power consumption, it significantly optimizes the model's multimodal feature alignment and retrieval generation quality in vertical domains. Ultimately, it achieves a model architecture that balances computing power, inference accuracy, local detail capture, and data security, meeting the requirements for offline and efficient deployment on industrial edge computing sides. Summary of the Invention
[0005] The present invention aims to provide a multimodal retrieval enhancement model that balances computational cost, inference accuracy, local detail capture, and data security.
[0006] To achieve the above objectives, this invention provides a method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios. The specific technical solution adopted by this invention is as follows: A method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios includes the following steps: Step A: Construct a multimodal retrieval enhancement model based on a multimodal language model and a multimodal retrieval system; Step B: Collect and clean the graphic and textual data in the industrial scenario, and construct a multimodal knowledge base and a fine-tuning dataset, wherein the fine-tuning dataset contains at least graphic and textual instructions; Step C: Fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset, and freeze the network parameters of the multimodal language model during the fine-tuning process. The fine-tuning includes at least the step of processing image and text instructions, wherein processing image and text instructions includes the following steps: S1: Obtain image-text instructions from the fine-tuning dataset, wherein the image-text instructions include an image, a corresponding text query, and a standard answer; S2: Use a target segmentation network to locate and segment key regions in the image; S3: Obtain high-resolution visual features of the key region; S4: Obtain the global visual features of the image; S5: Combine the high-resolution visual features with the global visual features to construct a fused visual feature that includes local enhancement information; S6: Call the multimodal retrieval device to retrieve multiple relevant text knowledge entries for the key region in the multimodal knowledge base, and concatenate each relevant text knowledge entry after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement; S7: Input the fused visual features and the multiple sets of fused text features into the multimodal language model for feature calculation; the multimodal language model calculates and assigns attention weights for the key regions based on the high-resolution visual features, and generates multiple prediction results corresponding to the image and text instructions by combining the multiple relevant text knowledge; S8: Obtain multiple prediction results from the multimodal language model, compare them with the standard answer to obtain multiple corresponding feedback signals, and update the network parameters of the multimodal retrieval system based on the multiple feedback signals.
[0007] Furthermore, in step A, the internal architecture of the multimodal retrieval enhancement model includes: The multimodal language model employs a pre-trained multimodal architecture combining a visual encoder and a language model; the visual encoder is used to extract visual features from images, and the language model is used to receive text features and visual features and generate results. The multimodal retrieval device employs a dual encoder architecture that supports multimodal feature alignment and unified representation. The dual encoder includes a text encoder for encoding text content and a visual encoder for encoding image content, thereby enabling similarity retrieval between text and images in a unified vector space.
[0008] Furthermore, in step B, text and technical documents in industrial scenarios are obtained as raw text data, cleaned, and then a plain text knowledge base is constructed; image data of industrial scenarios are obtained, and a generative artificial intelligence model is used to generate text descriptions of the images; the images and the corresponding text descriptions are combined to construct a graphic knowledge base. The multimodal knowledge base includes a plain text knowledge base and a graph-text knowledge base. The plain text knowledge base is converted into a plain text vector library by the text encoder of the multimodal retrieval device, and the graph-text knowledge base is converted into a graph-text vector library by the multimodal retrieval device.
[0009] Furthermore, in step B, the fine-tuning dataset is constructed based on the actual needs of the industrial scenario; the fine-tuning dataset includes plain text instructions and image-text instructions, the plain text instructions contain only text queries, the image-text instructions contain images and corresponding text queries, and both the plain text instructions and the image-text instructions contain standard answers.
[0010] Furthermore, in step C, the target segmentation network in step S2 is configured to locate key regions in the image and perform image segmentation; the key regions are visual feature regions in an industrial scene that represent industrial production elements or safety supervision objects; the number of key regions is not unique and is determined according to the content of the image.
[0011] Furthermore, in step S2, the target segmentation network uses a zero-shot image segmentation and detection network as the teacher model and is obtained using a knowledge distillation method. The knowledge distillation method specifically involves simultaneously inputting training samples into the target segmentation network and the teacher model, optimizing the network parameters of the target segmentation network so that its output results are close to those of the teacher model. The training samples originate from image data in a multimodal knowledge base.
[0012] Furthermore, in steps S3-S5, the key region is encoded by the visual encoder to obtain the high-resolution visual features of the key region; the image is encoded by the visual encoder to obtain the global visual features of the image.
[0013] Furthermore, in step S6, the mechanism for calling the multimodal retrieval device to retrieve relevant text knowledge is specifically as follows: Based on the key region extracted in step S2 as the query vector, the multimodal retrieval device performs image-to-text retrieval in the plain text vector library and returns the first M terms; Simultaneously, the multimodal retrieval device performs graph-to-graph retrieval in the image-text vector library, returning the first K images; extracts the text descriptions corresponding to the first K images, and combines them with the first M terms retrieved from the plain text vector library to form the multiple related text knowledge; the multiple related text knowledge are then concatenated after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement.
[0014] Furthermore, in step S8, the parameter update strategy for fine-tuning the multimodal retrieval machine based on the standard answer is as follows: During the fine-tuning process, the network parameters of the multimodal language model are completely frozen; multiple prediction results of the multimodal language model are compared with the standard answer, and the cross-entropy loss between the prediction results and the standard answer is calculated. The multiple cross-entropy losses are used as the multiple feedback signals. Based on the multiple feedback signals, the quality of multiple relevant text knowledge retrieved by the multimodal retrieval machine is evaluated. Text knowledge with cross-entropy loss less than a preset threshold is constructed as positive samples, and text knowledge with cross-entropy loss greater than or equal to the preset threshold is constructed as negative samples. Based on the positive and negative samples, a contrastive loss function is constructed to directly and independently update the network parameters of the multimodal retrieval machine, thereby optimizing the relevance between the knowledge retrieved by the multimodal retrieval machine and the standard answer.
[0015] Furthermore, in step C, when fine-tuning is performed using plain text instructions from the fine-tuning dataset, the fine-tuning method further includes: S9: Obtain plain text instructions and standard answers from the fine-tuning dataset; S10: Call the multimodal retrieval machine to retrieve the top N terms most relevant to the plain text instruction from the plain text vector library, and concatenate the top N terms to the plain text instruction to construct multiple sets of fused text features based on retrieval knowledge enhancement; S11: Input the fused text features into the multimodal language model; S12: The multimodal language model generates multiple output results and compares them with the standard answer to obtain multiple feedback signals. The network parameters of the multimodal retrieval machine are updated based on the multiple feedback signals.
[0016] This invention also provides a multimodal retrieval enhancement model fine-tuning device for industrial scenarios, specifically comprising: The model building module is used to combine a multimodal language model with a multimodal retrieval system to build an initial multimodal retrieval enhancement model. The data construction module is used to collect raw graphic and textual data from industrial scenarios, perform cleaning and format conversion to build a multimodal knowledge base and a fine-tuned dataset containing standard answers; A model fine-tuning module is used to fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset, and is configured with processing logic for handling image / text commands and plain text commands respectively; the model fine-tuning module includes: The visual feature extraction submodule is used to locate key regions in the image and obtain their high-resolution visual features when processing image and text instructions, and to construct fused visual features by combining global visual features. The knowledge retrieval and fusion submodule is used to call the multimodal retrieval device to retrieve multiple relevant text knowledge from the knowledge base, and then concatenate them to construct multiple sets of fused text features. The model prediction submodule is used to input the fused visual features and the multiple sets of fused text features into the multimodal language model respectively, and generate multiple corresponding prediction results respectively; The evaluation and parameter update submodule is used to compare the multiple prediction results with the standard answer to obtain multiple feedback signals, divide positive and negative samples based on the multiple feedback signals to construct a contrast loss function, and update the network parameters of the multimodal retrieval device.
[0017] The present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, wherein when the instructions are executed by the at least one processor, the at least one processor enables the at least one processor to implement the multimodal retrieval enhancement model fine-tuning method for industrial scenarios as described above.
[0018] The present invention also provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the multimodal retrieval enhancement model fine-tuning method for industrial scenarios as described above.
[0019] Compared with the prior art, the present invention has the following significant advantages.
[0020] By innovatively employing an asymmetric parameter fine-tuning strategy, the network parameters of the multimodal language model backbone are completely frozen during training, and gradient backpropagation and parameter updates are performed only on the lightweight multimodal retrieval system. This invention significantly reduces memory usage and computational overhead, making it possible to fine-tune and deploy the model on limited industrial edge devices. Furthermore, updating the model's knowledge base is sufficient to update the model's knowledge, without needing to readjust the multimodal retrieval enhancement model.
[0021] This invention designs a dynamic dual-path retrieval mechanism for image-text commands. It utilizes the image features of key areas to achieve accurate cross-modal knowledge retrieval from image to image and from image to text. Furthermore, by extracting the text description corresponding to the image, it avoids the distraction caused by directly inputting redundant multimodal features and forces the model to generate answers based on the professional text retrieved by the retrieval, thus fundamentally eliminating the illusion phenomenon of small edge models.
[0022] To address the challenge of cluttered backgrounds and minute details (such as fine cracks or instrument markings) in images collected from industrial sites, a fusion and stitching mechanism combining high-resolution local features and global visual features was designed. Without significantly increasing the model's computational load, a target segmentation network is used to accurately extract key regions. By stitching together high-resolution visual features, the attention weight of the language model to local details is effectively strengthened, overcoming the pain point of models frequently neglecting minute details in industrial applications.
[0023] This invention utilizes knowledge distillation technology to obtain a target segmentation network. During actual inference, this network can quickly locate industrial production elements or key areas for safety supervision and extract features, significantly reducing the latency of visual processing and meeting the stringent requirements of high-frequency, real-time industrial production.
[0024] The lightweight multimodal retrieval enhancement model finely tuned in this invention supports fully localized and offline deployment in private cloud environments or edge computing device nodes without external networks, eliminating the risk of cloud leakage of sensitive data such as enterprise production specifications and drawings from the physical architecture perspective. Attached Figure Description
[0025] Figure 1 The flowchart illustrates the method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios, as provided in this embodiment of the invention.
[0026] Figure 2 This is a schematic diagram of the global-local dual-resolution visual token fusion mechanism and attention weight enhancement provided in an embodiment of the present invention.
[0027] Figure 3 This is a schematic diagram of the architecture for the reasoning and generation process of a multimodal retrieval enhancement model for industrial scenarios provided in an embodiment of the present invention. Detailed Implementation
[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0029] Example 1: This example provides a method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios. For example... Figure 1 The flowchart shown contains the core steps S1 to S8 of the multimodal retrieval enhancement model fine-tuning method. The overall implementation framework of this method includes three core stages (corresponding to steps A to C): model base construction, data and knowledge base preparation, and joint fine-tuning training.
[0030] Step A: Construct a multimodal retrieval enhancement model based on a multimodal language model and a multimodal retrieval system.
[0031] Given the constraints of limited computing power and video memory in industrial edge computing devices, this embodiment prefers a lightweight vision-language pre-trained model with no more than 7B parameters as the multimodal language model (e.g., the Qwen-2.5-VL-7B model). This model adopts an architecture that combines a visual encoder with a language model, and has excellent native multimodal understanding capabilities and low video memory requirements.
[0032] In conjunction with the language model, the multimodal retrieval system preferably adopts a dual encoder architecture that supports cross-modal feature alignment (e.g., the BGE-Visualized model). This retrieval system includes independent text encoders and visual encoders, which can map image features and text features in industrial scenarios to a unified dense vector space to achieve cross-modal similarity calculation.
[0033] Step B involves collecting and cleaning text and image data from industrial scenarios to build a multimodal knowledge base and fine-tune the dataset.
[0034] To address real-world industrial needs (such as quality defect detection and safety monitoring), private data is collected and a multimodal knowledge base is constructed. This multimodal knowledge base comprises two types of knowledge bases: Plain text knowledge base: Obtain compliance texts and technical documents in industrial scenarios as raw text data (such as operating specifications, safety rules, technical requirements, etc.), clean them and build the plain text knowledge base, and then use a multimodal retrieval device to transform them into a plain text vector library; Image and text knowledge base: Acquire image data of industrial scenarios (such as abnormal images taken on site, violations of regulations, etc.), and obtain text descriptions of the images (such as "The image shows that the bearing of conveyor belt No. 2 is severely worn and needs to be replaced immediately") through manual compilation or generative artificial intelligence model generation. Combine the images and corresponding text descriptions to build an image and text knowledge base, and then use a multimodal retrieval device to convert it into an image and text vector library.
[0035] Based on actual needs in industrial scenarios (such as accurate readings of complex instruments, quality inspection of minor cracks, and safety warnings for violations), we construct targeted query prompts through manual compilation or generation using large language models, and match them with accurate answers based on industrial standards. This results in graphic and plain text instructions for model fine-tuning.
[0036] Step C: Fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset. During the fine-tuning process, freeze the network parameters of the multimodal language model. The fine-tuning includes at least the step of processing image and text instructions. When processing image and text instructions, the fine-tuning includes the following steps: S1: Obtain image-text instructions from the fine-tuning dataset, wherein the image-text instructions include an image, a corresponding text query, and a standard answer; S2: Use a target segmentation network to locate and segment key regions in the image; S3: Obtain high-resolution visual features of the key region; S4: Obtain the global visual features of the image; S5: Combine the high-resolution visual features with the global visual features to construct a fused visual feature that includes local enhancement information; S6: Call the multimodal retrieval device to retrieve multiple relevant text knowledge entries for the key region in the multimodal knowledge base, and concatenate each relevant text knowledge entry after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement; S7: Input the fused visual features and the multiple sets of fused text features into the multimodal language model for feature calculation; the multimodal language model calculates and assigns attention weights for the key regions based on the high-resolution visual features, and generates multiple prediction results corresponding to the image and text instructions by combining the multiple relevant text knowledge; S8: Obtain multiple prediction results from the multimodal language model, compare them with the standard answer to obtain multiple corresponding feedback signals, and update the network parameters of the multimodal retrieval system based on the multiple feedback signals.
[0037] Furthermore, to achieve the segmentation of key regions in step S2, this embodiment uses a knowledge distillation method to train a target segmentation network: a network with zero-sample segmentation and detection capabilities (such as Grounded-Segment-Anything, or GSA for short) is selected as the teacher model; in order to make the distilled target segmentation network more in line with industrial scenarios, image data in the graph and text knowledge base is used as training samples.
[0038] The training samples are input into the GSA to obtain accurate bounding boxes. Images from the same batch are input into the target segmentation network (e.g., YOLO-Seg). By calculating the bounding box regression loss, the output of the target segmentation network is forced to approximate the GSA network. Finally, a target segmentation network adapted to the training samples is obtained. After distillation, the network is configured to quickly segment the visual regions of industrial production elements or safety supervision objects (such as workers, instruments, and defective parts) in the image.
[0039] Furthermore, in steps S3-S5, considering that industrial scene images typically have extremely high resolution and contain minute key details (such as instrument scales and minor flaws), this embodiment proposes a global-local dual-resolution visual feature fusion mechanism.
[0040] The input raw industrial image is downsampled or coarsely segmented, and a low-resolution global visual feature is generated by a visual encoder to preserve the macroscopic positional relationships of the image. Key regions are obtained using a target segmentation network, maintaining the original high resolution or using finer segmentation, and high-resolution local visual features are generated by a visual encoder. In the sequence dimension, the high-resolution local visual features are fused into the global visual features to construct a fused visual feature containing local enhancement information, in which the image focus information carried by the high-resolution features receives higher attention weight.
[0041] Combination Figure 2 The computational process of the global-local dual-resolution visual feature fusion mechanism is as follows: Let the input industrial original image be... I The result is obtained by scaling down to the default low-resolution scale of the multimodal language model using a downsampling function. I global Input it into the visual encoder E v (.) Global visual features 110 were extracted. T global ), can be represented as:
[0042] in, N g denoted as the sequence length of the global visual features, and D as the dimension of the hidden layer vector.
[0043] Assume the target segmentation network is in the original image I Detected in m The key areas have bounding box sets B={B1,B2,...,B m}, based on this set, the original graph I Segmentation is performed to obtain m A set of local image patches , The above m Each local image patch is input into the visual encoder. E v (.) The corresponding high-resolution visual feature set 120 is obtained:
[0044] in, This represents the feature sequence of the j-th key region. The length of the visual features of this region.
[0045] Along the sequence dimension, the feature fusion module 130 fuses these m high-resolution visual features into a global visual feature. T global In the process, 140 final fused visual features were constructed. T vision ):
[0046] Furthermore, in step S6, the mechanism for calling the multimodal retrieval device to retrieve relevant text knowledge is specifically as follows: Based on the key region extracted in step S2 as the query vector, a multimodal retrieval tool is used to perform image-to-text retrieval in the plain text vector library and return the first M terms; simultaneously, an image-to-image retrieval is performed in the image-text vector library and return the first K images; the text descriptions corresponding to the images are extracted and combined with the first M terms to form multiple related text knowledge; the multiple related text knowledge are concatenated after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement.
[0047] Furthermore, in step S8, during the training process of fine-tuning the parameters of the multimodal retrieval machine based on the standard answer, this embodiment constructs an asymmetric gradient update method oriented towards the retrieval machine. The system calculates the generation error (such as cross-entropy loss) based on the output text of the multimodal language model and the real annotations in the fine-tuning dataset. That is, it uses the degree of dependence of the language model on the recalled documents as a supervision signal to calculate the contrastive learning loss. Finally, the parameters of the multimodal retrieval machine are updated.
[0048] Specifically: Let θ be the set of network parameters for the multimodal language model. MLLM The network parameter set of the multimodal retrieval system is θ. Retriever ， During the fine-tuning of this invention, the parameters of the multimodal language model are frozen.
[0049] For the first in the fine-tuning dataset i A sample of graphic instructions, assuming its input instruction is... x i The actual answer is marked as y i The search engine is based on x i The set of Top-k texts retrieved from the knowledge base is denoted as . D top-k ={d 1 ,d 2 ...d k } .
[0050] To construct contrastive learning positive and negative samples, this invention utilizes a frozen multimodal language model to independently assess the quality of the recalled text. Specifically: [The text then describes a process involving...] D top-k Each text in d j With input commands x i After concatenation, the data is input into a multimodal language model, which then calculates the model's predicted results and compares them with the actual answers. y i error ej :
[0051] Based on the calculated error set {e 1 ,e 2 ...e k } Segmenting the recalled text: dividing the error e j Smaller texts are divided into positive sample sets. D + This indicates that the text has a positive enhancement effect on the model's generation of the correct answer; the error... e j Larger texts are divided into negative sample sets. D - This indicates that the text contains noise or is unhelpful in answering the question.
[0052] The system calculates the generation loss and the retrieval matching loss:
[0053] in, f MLLM (.) This represents the forward prediction function of a multimodal language model. L CE The cross-entropy loss between the frozen language model output and the true answer; L InfoNCE The retrieval system is based on the set of positive samples. D + and negative sample set D - Calculate the contrastive learning loss; For hyperparameter weights.
[0054] During the backpropagation phase, the contrastive learning loss term is used. L InfoNCE Dominant, thus only updating parameters of the retriever:
[0055] By using the aforementioned asymmetric objective function and update formula, the powerful generalization and generation capabilities of the multimodal language model are guaranteed. This enables the multimodal retrieval system to fit the spatial feature distribution of text and image data in specific industrial scenarios, reducing training memory consumption while achieving accurate knowledge retrieval.
[0056] Further, in step C, the multimodal retrieval enhancement model is fine-tuned based on the fine-tuning dataset. During the fine-tuning process, the network parameters of the multimodal language model are frozen. The fine-tuning includes steps for processing image-text instructions and plain text instructions. When processing plain text instructions, the fine-tuning further includes the following steps: S9: Obtain plain text instructions and standard answers from the fine-tuning dataset; S10: Call the multimodal retrieval machine to retrieve the top N terms most relevant to the plain text instruction from the plain text vector library, and concatenate the top N terms to the plain text instruction to construct multiple sets of fused text features based on retrieval knowledge enhancement; S11: Input the fused text features into the multimodal language model; S12: The multimodal language model generates multiple output results and compares them with the standard answer to obtain multiple feedback signals. The network parameters of the multimodal retrieval machine are updated based on the multiple feedback signals.
[0057] Example 2: Based on the same inventive concept as Example 1, this example provides a multimodal retrieval enhancement model fine-tuning device for industrial scenarios. The logical architecture of this device includes: Model building module: Used to combine a multimodal language model with a multimodal retrieval system with a dual encoder structure to initialize and build a multimodal retrieval enhancement model; Data building module: used to collect raw image and text data from industrial scenarios, perform cleaning and format conversion to build a multimodal knowledge base and fine-tune the dataset; A model fine-tuning module is used to fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset, and is configured with processing logic for handling image / text commands and plain text commands respectively; the model fine-tuning module includes: The visual feature extraction submodule is used to locate key regions in the image and obtain their high-resolution visual features when processing image and text instructions, and to construct fused visual features by combining global visual features. The knowledge retrieval and fusion submodule is used to call the multimodal retrieval device to retrieve multiple relevant text knowledge from the knowledge base, and then concatenate them to construct multiple sets of fused text features. The model prediction submodule is used to input the fused visual features and the multiple sets of fused text features into the multimodal language model respectively, and generate multiple corresponding prediction results respectively; The evaluation and parameter update submodule is used to compare the multiple prediction results with the standard answer to obtain multiple feedback signals, divide positive and negative samples based on the multiple feedback signals to construct a contrast loss function, and update the network parameters of the multimodal retrieval device.
[0058] Example 3: Based on Example 1, this example demonstrates the process of a retrieval enhancement language model for industrial scenarios when processing image and text commands, combined with... Figure 3 The architecture diagram.
[0059] First, input an image 10 of an industrial scene. This image is processed by a target segmentation network 30 to segment three key regions: key region 11 (worker), key region 12 (forklift), and key region 13 (goods). On the other hand, the image is processed by a visual encoder 40 to obtain global visual features.
[0060] The key regions 11-13 are input into the visual encoder 40 to obtain high-resolution visual features. Furthermore, the global visual features and the high-resolution visual features are combined to obtain fused visual features 41 containing local enhancement information.
[0061] The key regions 11-13 are input into the multimodal retrieval device 50, and image-to-text retrieval and image-to-image retrieval are performed in the multimodal knowledge base 51 respectively. For the results of image-to-image retrieval, the text description corresponding to the retrieved image is extracted. Furthermore, the retrieved text knowledge is concatenated with the text query 20 for the image to obtain the fusion text feature 52 with enhanced retrieval knowledge.
[0062] The fused visual features 41 and fused text features 52 are input into the multimodal language model 60. The model fuses the input information and outputs the final result through the output module 70.
[0063] Example 4: This example provides an electronic device designed to execute the multimodal retrieval enhancement model fine-tuning method for industrial scenarios described in Example 1 above.
[0064] The electronic device can be an industrial control computer, an edge computing gateway, or a private cloud server node. The device includes: at least one processor (such as a CPU, GPU, or NPU); and a memory (such as RAM, ROM, or NVMe solid-state drive) communicatively connected to said at least one processor.
[0065] The memory stores computer program instructions that can be executed by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor can implement the various steps described in detail in Embodiment 1, including core operational logic such as knowledge distillation, dual-path retrieval enhancement, visual token dual-resolution fusion calculation, and asymmetric model fine-tuning.
[0066] Example 5: This example provides a non-volatile computer-readable storage medium on which computer-executable instructions are stored.
[0067] The non-volatile computer-readable storage medium can be a physical medium capable of persistently storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk. When the computer-executable instructions in the storage medium are read and executed by the computer's processor, the entire process of the multimodal retrieval enhancement model fine-tuning method for industrial scenarios described in Embodiment 1 above can be realized.
[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention shall be determined by the scope defined in the claims.
Claims
1. A method for fine-tuning a multimodal retrieval enhancement model for industrial scenarios, characterized in that, Includes the following steps: Step A: Construct a multimodal retrieval enhancement model based on a multimodal language model and a multimodal retrieval system; Step B: Collect and clean the graphic and textual data in the industrial scenario, and construct a multimodal knowledge base and a fine-tuning dataset, wherein the fine-tuning dataset contains at least graphic and textual instructions; Step C: Fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset, and freeze the network parameters of the multimodal language model during the fine-tuning process. The fine-tuning includes at least the step of processing image and text instructions, wherein processing image and text instructions includes the following steps: S1: Obtain image-text instructions from the fine-tuning dataset, wherein the image-text instructions include an image, a corresponding text query, and a standard answer; S2: Use a target segmentation network to locate and segment key regions in the image; S3: Obtain high-resolution visual features of the key region; S4: Obtain the global visual features of the image; S5: Combine the high-resolution visual features with the global visual features to construct a fused visual feature that includes local enhancement information; S6: Call the multimodal retrieval device to retrieve multiple relevant text knowledge entries for the key region in the multimodal knowledge base, and concatenate each relevant text knowledge entry after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement; S7: Input the fused visual features and the multiple sets of fused text features into the multimodal language model for feature calculation; the multimodal language model calculates and assigns attention weights for the key regions based on the high-resolution visual features, and generates multiple prediction results corresponding to the image and text instructions by combining the multiple relevant text knowledge; S8: Obtain multiple prediction results from the multimodal language model, compare them with the standard answer to obtain multiple corresponding feedback signals, and update the network parameters of the multimodal retrieval system based on the multiple feedback signals.
2. The method according to claim 1, characterized in that, In step A, the internal architecture of the multimodal retrieval enhancement model includes: The multimodal language model employs a pre-trained multimodal architecture combining a visual encoder and a language model; the visual encoder is used to extract visual features from images, and the language model is used to receive text features and visual features and generate results. The multimodal retrieval device employs a dual encoder architecture that supports multimodal feature alignment and unified representation. The dual encoder includes a text encoder for encoding text content and a visual encoder for encoding image content, thereby enabling similarity retrieval between text and images in a unified vector space.
3. The method according to claim 2, characterized in that, In step B, the method for constructing the multimodal knowledge base and fine-tuning the dataset is as follows: Text and technical documents from industrial scenarios are acquired as raw text data, cleaned, and then used to construct a plain text knowledge base; image data from industrial scenarios are acquired, and a generative artificial intelligence model is used to generate text descriptions of the images. The images and corresponding text descriptions are combined to construct a graphic knowledge base. The multimodal knowledge base includes a plain text knowledge base and a graph-text knowledge base. The plain text knowledge base is converted into a plain text vector library by the text encoder of the multimodal retrieval device, and the graph-text knowledge base is converted into a graph-text vector library by the multimodal retrieval device. The fine-tuning dataset is constructed based on the actual needs of the industrial scenario; the fine-tuning dataset includes plain text instructions and image-text instructions, the plain text instructions contain only text queries, the image-text instructions contain images and corresponding text queries, and both the plain text instructions and the image-text instructions contain standard answers.
4. The method according to claim 1, characterized in that, The target segmentation network in step S2 is configured to locate the key regions in the image and perform image segmentation; the key regions are visual feature regions that represent industrial production elements or safety supervision objects in an industrial scene; the number of key regions is determined according to the content of the image. The target segmentation network uses a zero-shot image segmentation and detection network as the teacher model and is obtained using a knowledge distillation method. The knowledge distillation method is as follows: training samples are simultaneously input into the target segmentation network and the teacher model, and the network parameters of the target segmentation network are optimized so that its output results are close to the output results of the teacher model. The training samples are derived from the multimodal knowledge base, and the training samples use image data from the text and image data collected in step B.
5. The method according to claim 3, characterized in that, In steps S3-S6: the high-resolution visual features are obtained by encoding the key region through the visual encoder in the multimodal language model, and the global visual features are obtained by encoding the image through the visual encoder; The mechanism for calling the multimodal retrieval device to retrieve relevant textual knowledge of the key area in the multimodal knowledge base is as follows: based on the key area extracted in step S2 as the query vector, the multimodal retrieval device performs graph-to-text retrieval in the plain text vector library and returns the first M terms; Simultaneously, the multimodal retrieval device performs graph-to-graph retrieval in the image-text vector library, returning the first K images; extracts the text descriptions corresponding to the first K images, and combines them with the first M terms retrieved from the plain text vector library to form the multiple related text knowledge; the multiple related text knowledge are then concatenated after the text query to construct multiple sets of fused text features based on retrieval knowledge enhancement.
6. The method according to claim 1, characterized in that, In step S8, the parameter update strategy for fine-tuning the multimodal retrieval machine based on the standard answer is as follows: During the fine-tuning process, the network parameters of the multimodal language model are completely frozen; multiple prediction results of the multimodal language model are compared with the standard answer, and the cross-entropy loss between the prediction results and the standard answer is calculated. The multiple cross-entropy losses are used as the multiple feedback signals. Based on the multiple feedback signals, the quality of multiple relevant text knowledge retrieved by the multimodal retrieval machine is evaluated. Text knowledge with cross-entropy loss less than a preset threshold is constructed as positive samples, and text knowledge with cross-entropy loss greater than or equal to the preset threshold is constructed as negative samples. Based on the positive and negative samples, a contrastive loss function is constructed to directly and independently update the network parameters of the multimodal retrieval machine, thereby optimizing the relevance between the knowledge retrieved by the multimodal retrieval machine and the standard answer.
7. The method according to claim 3, characterized in that, In step C, the multimodal retrieval enhancement model is fine-tuned based on the fine-tuning dataset. When using plain text instructions, the fine-tuning further includes: S9: Obtain plain text instructions and standard answers from the fine-tuning dataset; S10: Call the multimodal retrieval machine to retrieve the top N terms most relevant to the plain text instruction from the plain text vector library, and concatenate the top N terms to the plain text instruction to construct multiple sets of fused text features based on retrieval knowledge enhancement; S11: Input the multiple sets of fused text features into the multimodal language model; S12: The multimodal language model generates multiple output results and compares them with the standard answer to obtain multiple feedback signals. The network parameters of the multimodal retrieval machine are updated based on the multiple feedback signals.
8. A multimodal retrieval enhancement model fine-tuning device for industrial scenarios, characterized in that, include: The model building module is used to combine a multimodal language model with a multimodal retrieval system to build an initial multimodal retrieval enhancement model. The data construction module is used to collect raw graphic and textual data from industrial scenarios, perform cleaning and format conversion to build a multimodal knowledge base and a fine-tuned dataset containing standard answers; The model fine-tuning module is used to fine-tune the multimodal retrieval enhancement model based on the fine-tuning dataset, and is configured with processing logic for handling image and text commands and plain text commands respectively. The model fine-tuning module includes: The visual feature extraction submodule is used to locate key regions in the image and obtain their high-resolution visual features when processing image and text instructions, and to construct fused visual features by combining global visual features. The knowledge retrieval and fusion submodule is used to call the multimodal retrieval device to retrieve multiple relevant text knowledge from the knowledge base, and then concatenate them to construct multiple sets of fused text features. The model prediction submodule is used to input the fused visual features and the multiple sets of fused text features into the multimodal language model respectively, and generate multiple corresponding prediction results respectively; The evaluation and parameter update submodule is used to compare the multiple prediction results with the standard answer to obtain multiple feedback signals, divide positive and negative samples based on the multiple feedback signals to construct a contrast loss function, and update the network parameters of the multimodal retrieval device.
9. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the multimodal retrieval enhancement model fine-tuning method for industrial scenarios as described in any one of claims 1 to 7.
10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the multimodal retrieval enhancement model fine-tuning method for industrial scenarios as described in any one of claims 1 to 7.