Power industry image-text multi-modal model lightweight method and related system

Abstract

The invention belongs to the field of image-text models, and discloses a power industry image-text multi-modal model lightweight method and related systems.According to the method, a full connection layer in an original image-text multi-modal model is replaced with a convolution layer with the required step length, lightweight is carried out in combination with a grouping convolution mode, and the model parameter quantity and the calculation quantity are greatly reduced. The convolution is equivalent to a full connection layer in calculation, but a convolution acceleration structure of the GPU or the NPU can be utilized more efficiently, so that the calculation overhead is remarkably reduced while the feature expression capability is kept, and the model can be efficiently operated on edge terminal equipment with limited calculation power. According to the method, the similarity matrix between the image token and the text token is constructed, the correlation between the image feature and the text feature is calculated, and only the image token and the text token which are most related to the current input text are reserved, so that the redundant information of the input feature is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of graphic and textual models, specifically relating to a lightweight method and related system for graphic and textual multimodal models in the power industry. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology has profoundly impacted power industry-related operations such as drone inspections of power transmission and distribution lines, substation inspections, and violation identification. Defect detection and violation identification algorithms based on deep neural networks have gradually replaced the original manual review workflow, significantly improving work efficiency while reducing the workload of staff. However, existing detection algorithms are typically based on a purely visual architecture, failing to effectively integrate power industry-related business knowledge to aid reasoning. Furthermore, existing algorithms often only learn the complex mapping from input images to output results, lacking logical thinking and reasoning abilities, resulting in poor interpretability of their predictions. These shortcomings lead to a high false detection rate for existing defect detection and violation identification algorithms when dealing with complex defects requiring the integration of power industry-related business knowledge and logic, preventing them from reaching a practical level.

[0003] In recent years, large-scale modeling technologies, represented by large language models, have developed rapidly, demonstrating significant advantages in logical thinking and reasoning abilities. Based on this, the image-text multimodal model organically combines visual and large language models. The visual model extracts visual features from images and aligns them to the semantic space, while the large language model analyzes the aligned image features and related input text, returning results. Due to the introduction of large language models, the image-text multimodal model can effectively address the problems of existing detection algorithms failing to effectively integrate power industry business knowledge and lacking logical reasoning capabilities, thus improving the model's detection accuracy for complex defects. However, as crucial components of the image-text multimodal model, both the visual and large language models typically have a large number of parameters, making it difficult to deploy on edge devices. Furthermore, the high resolution of power industry-related image data results in long feature sequences after visual model feature extraction, further increasing the model's computational complexity and slowing down inference speed, failing to meet the real-time requirements of related business operations.

[0004] Existing image-text multimodal models still have a series of problems in areas such as drone inspection of power transmission and distribution lines, substation inspection, and violation identification: (1) Due to limitations in size, weight, power consumption, etc., the built-in edge computing devices of drones, substation inspection robots, etc. have significant differences in computing power and memory compared with the cloud environment that runs large models, making it difficult to deploy multimodal models of graphics and text on edge devices.

[0005] (2) Most of the image data in the power industry has a high resolution. After the visual model features are extracted, a long feature sequence will be generated, which further increases the computational complexity of the model and results in a slow inference speed, which cannot meet the real-time requirements of related businesses.

[0006] How to lightweight the text-image multimodal model to adapt it to edge devices and meet the real-time requirements of related businesses, and improve the inference efficiency of the model while minimizing the loss of inference accuracy, has important practical value. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of difficulty in deploying models on edge devices and slow inference speed, and to provide a lightweight method and related system for multimodal graph models in the power industry.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a lightweight method for creating a multimodal graphic model in the power industry, comprising the following steps: Obtain the fully connected layer in the image-text multimodal model, and perform lightweight processing on the fully connected layer to obtain a lightweight fully connected layer; Obtain the image token and text token from the image-text multimodal model; Select the image relevance value of the desired image based on the image token and text token; Based on the image token and text token, select the text relevance value of the desired text; Image tokens and text tokens are aligned based on text relevance and image relevance values, and combined with a lightweight fully connected layer to obtain a lightweight image-text multimodal model.

[0009] A further improvement of this invention lies in obtaining the fully connected layer in the image-text multimodal model and performing lightweight processing on the fully connected layer to obtain the lightweight fully connected layer. The specific method is as follows: Obtain the linear mapping relationship between the input feature matrix, weight matrix, and bias terms of the fully connected layer in the image-text multimodal model; The graph-text multimodal model is adjusted according to the linear mapping relationship, so that the adjusted fully connected layer is uniformly replaced with the target step size, and the preliminary adjusted fully connected layer is obtained. The convolutional layers of the pre-adjusted fully connected layers are grouped into convolutional layers. The independent sub-convolutions after grouping are then concatenated along the channel dimension to form a complete output feature map. Knowledge distillation is performed on the complete output feature map to obtain a lightweight fully connected layer.

[0010] A further improvement of this invention lies in the following method for performing knowledge distillation on the complete output feature map to obtain a lightweight fully connected layer: Obtain the fully connected layer in the text-image multimodal model and use it as the teacher model; use the complete output feature map as the student model. Obtain the output vectors of each fully connected layer in the teacher model and the output vectors of each fully connected layer in the student model; The output vectors of each fully connected layer in the teacher model are used to update the output vectors of each fully connected layer in the student model, resulting in a lightweight fully connected layer.

[0011] A further improvement of this invention lies in the following method for selecting the image relevance value of the desired image based on the image token and text token: Using image tokens as queries and text tokens as keys, we can calculate the similarity matrix between each image token and all text tokens. Process all elements in each row of the similarity matrix to obtain the overall relevance value between each image token and the entire input text sequence; A preset image correlation threshold range is used to obtain the overall correlation value within the image correlation threshold range, which is then used as the image correlation value.

[0012] A further improvement of this invention lies in the following method for selecting the text relevance value of the desired text based on the image token and the text token: Using text tokens as queries and image tokens as keys, calculate the similarity matrix between each text token and all image tokens; Process all elements in each row of the similarity matrix to obtain the text relevance value between each text token and the entire input image sequence; Preset a text relevance threshold range, obtain the overall relevance value within the text relevance threshold range, and use it as the text relevance value; A preset text relevance threshold range is defined, and the overall relevance value within the text relevance threshold range is obtained as the text relevance value.

[0013] Secondly, the present invention provides a lightweight system for graphic multimodal models in the power industry, comprising: The fully connected layer processing module is used to obtain the fully connected layer in the image-text multimodal model, perform lightweight processing on the fully connected layer, and obtain a lightweight fully connected layer. The token acquisition module is used to obtain the image token and text token in the image-text multimodal model; The image relevance value selection module is used to select the image relevance value of the desired image based on the image token and text token; The text relevance value selection module is used to select the text relevance value of the desired text based on the image token and the text token. The module is used to align image tokens and text tokens based on text relevance values ​​and image relevance values. Combined with a lightweight fully connected layer, a lightweight image-text multimodal model is obtained.

[0014] A further improvement of this invention is that the functionality of the fully connected layer processing module is implemented through the following method: Obtain the linear mapping relationship between the input feature matrix, weight matrix, and bias terms of the fully connected layer in the image-text multimodal model; The graph-text multimodal model is adjusted according to the linear mapping relationship, so that the adjusted fully connected layer is uniformly replaced with the target step size, and the preliminary adjusted fully connected layer is obtained. The convolutional layers of the pre-adjusted fully connected layers are grouped into convolutional layers. The independent sub-convolutions after grouping are then concatenated along the channel dimension to form a complete output feature map. Knowledge distillation is performed on the complete output feature map to obtain a lightweight fully connected layer.

[0015] A further improvement of this invention is that the functionality of the fully connected layer processing module is implemented through the following method: Using image tokens as queries and text tokens as keys, we can calculate the similarity matrix between each image token and all text tokens. Process all elements in each row of the similarity matrix to obtain the overall relevance value between each image token and the entire input text sequence; A preset image correlation threshold range is used to obtain the overall correlation value within the image correlation threshold range, which is then used as the image correlation value.

[0016] A further improvement of this invention is that the functionality of the fully connected layer processing module is implemented through the following method: Using text tokens as queries and image tokens as keys, calculate the similarity matrix between each text token and all image tokens; Process all elements in each row of the similarity matrix to obtain the text relevance value between each text token and the entire input image sequence; Preset a text relevance threshold range, obtain the overall relevance value within the text relevance threshold range, and use it as the text relevance value; A preset text relevance threshold range is defined, and the overall relevance value within the text relevance threshold range is obtained as the text relevance value.

[0017] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a lightweight method for a multimodal graphic model in the power industry.

[0018] Fourthly, the present invention provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a lightweight method for a multimodal graphic model in the power industry.

[0019] Compared with the prior art, the present invention has the following beneficial effects: This invention significantly reduces the number of model parameters and computational cost by replacing the fully connected layers in the original image-text multimodal model with convolutional layers of the required stride, and combining this with grouped convolution for lightweighting. Since convolution is computationally equivalent to a fully connected layer but can more efficiently utilize the convolutional acceleration structure of GPUs or NPUs, it significantly reduces computational overhead while maintaining feature representation capabilities, enabling the model to run efficiently on edge devices with limited computing power. This invention constructs a similarity matrix between image tokens and text tokens, calculates the correlation between image features and text features, and retains only the image tokens and text tokens most relevant to the current input text, effectively reducing redundant information in the input features. This semantic relevance-based filtering mechanism allows the model to significantly compress the input dimension without losing key semantic information, thereby further reducing memory usage and bandwidth consumption during inference. This invention achieves image-text token alignment based on lightweight fully connected layers, ensuring semantic consistency between images and text in the feature space, and improving the model's expressive efficiency in the multimodal information fusion process. Because the aligned feature representations are more compact and more relevant, the model's response time in generation or classification tasks is significantly shortened, and inference speed is significantly improved. This invention does not disrupt the original model's multimodal structure and cross-modal association mechanisms during the lightweighting process, maintaining stable accuracy in semantic matching, image / text retrieval, or question answering tasks. Compared to traditional model pruning or quantization methods, this method significantly improves the model's deployability and real-time performance on edge devices while maintaining high performance. In conclusion, this invention achieves lightweight, high-speed, and low-energy deployment of image / text multimodal models on edge devices, effectively solving the technical problems of large size, slow inference, and difficulty in edge deployment of existing multimodal models, and has significant engineering practical value and promotional significance. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention; Figure 2 This is a system diagram of the present invention; Figure 3 This is a structural diagram of the graphic multimodal model in this invention; Figure 4 This is a flowchart of the heterogeneous distillation process of the image-text multimodal model based on grouped convolution in this invention; Figure 5 This is a flowchart of the token compression process based on bidirectional visual text compression in this invention; Figure 6 This is a system diagram of Example 5. Detailed Implementation

[0021] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0022] Example 1: See Figure 1 A lightweight method for creating a multimodal graph model in the power industry includes the following steps: S1, obtain the fully connected layer in the image-text multimodal model, perform lightweight processing on the fully connected layer to obtain a lightweight fully connected layer.

[0023] S2, obtain the image token and text token from the image-text multimodal model.

[0024] S3: Select the image relevance value of the desired image based on the image token and text token.

[0025] S4. Based on the image token and text token, select the text relevance value of the desired text.

[0026] S5 aligns the image token and text token based on the text relevance value and the image relevance value, and combines them with a lightweight fully connected layer to obtain a lightweight image-text multimodal model.

[0027] Example 2: See Figure 2 A lightweight system for graphic multimodal models in the power industry, comprising: The fully connected layer processing module is used to obtain the fully connected layer in the image-text multimodal model, perform lightweight processing on the fully connected layer, and obtain a lightweight fully connected layer.

[0028] The token acquisition module is used to obtain the image token and text token in the image-text multimodal model.

[0029] The image relevance value selection module is used to select the image relevance value of the desired image based on the image token and text token.

[0030] The text relevance value selection module is used to select the text relevance value of the desired text based on the image token and the text token.

[0031] The module is used to align image tokens and text tokens based on text relevance values ​​and image relevance values. Combined with a lightweight fully connected layer, a lightweight image-text multimodal model is obtained.

[0032] Example 3: See Figure 3 and Figure 4 In this embodiment, the fully connected layers in the original text-image multimodal model are replaced with grouped convolutional layers with a kernel size of 1×1 and the same output dimension. The original text-image multimodal model serves as the teacher model, and the replaced model serves as the student model. The output of the fully connected layer in the teacher model is used as the supervision signal, and the output of the grouped convolutional layers in the student model is used as the prediction value. The parameters of the grouped convolutional layers in the student model are updated using the cross-entropy loss function, resulting in a lightweight fully connected layer with fewer parameters. The details are as follows: Step 1: Addressing the numerous fully connected layers present in the text-image multimodal model. Consider input Weight bias By leveraging the characteristics of 1×1 convolutional layers in convolution operations, all fully connected layers in the image-text multimodal model are replaced with layers having a stride of 1 and an input channel count of [missing information]. The number of output channels is A 1×1 convolutional layer. Among them, Let be the set of real numbers. For input dimensions, For output dimensions.

[0033] Step two, using a group size of... The grouped convolutions are lightweighted across all 1×1 convolutional layers to obtain... The group step size is 1, and the number of input channels is The number of output channels is A 1×1 convolutional layer. Compared to the original fully connected layer or a 1×1 convolutional layer, the number of parameters... , The total number of parameters in the group of convolutional layers is This significantly reduces the number of parameters in the model.

[0034] Step 3, The outputs of the convolutional layers are concatenated to obtain a network output with the same dimensions as the original fully connected network output.

[0035] Step four: Use the original text-image multimodal model as the teacher model, and the model after grouped convolutional replacement as the student model. Let... As the output vector of a fully connected layer in the teacher model As the output vector of a grouped convolutional layer in the student model, the cross-entropy loss function is used. The parameters of the grouped convolutional layers in the student network are updated to obtain a lightweight fully connected layer.

[0036] Example 3: See Figure 5 This embodiment further explains the method of step S3 and the function of the image correlation value selection module as follows: Step 1: Tokenize the image As a query, a text token Used as the key, with a similarity metric function Calculate the similarity matrix between each image token and all text tokens. .in, For image feature token set, A set of text feature tokens. For feature dimension, For the number of image tokens, This represents the number of text tokens.

[0037] Step 2, for the similarity matrix The sum of each row of elements is used as the correlation between each image token and the input text sequence.

[0038] Step 3: Sort all image tokens by text relevance from highest to lowest, and then apply a threshold. Before screening Each image token serves as the input image token for the final language model.

[0039] Example 4: See Figure 5 This embodiment further explains the method of step S4 and the function of the text relevance value selection module as follows: Step 1, transfer the text token As a query, image token Used as a key, a similarity metric function Calculate the similarity matrix between each text token and all image tokens. .

[0040] Step 2, for the similarity matrix The sum of each row of elements is used as the correlation magnitude between each text token and the sequence of input image patches.

[0041] Step 3: Sort all text tokens by image relevance from highest to lowest, and then use a threshold to determine the relevance. Before screening Each text token serves as the input text token for the final language model.

[0042] The above embodiments utilize raw image data captured by drones during power distribution line inspections. and text instructions The initial visual token and text token are obtained by inputting them into the visual encoder and text encoder of the lightweight image-text multimodal model described above, respectively. The visual token is processed by a visual token compression module and a visual text alignment module to obtain a compressed visual token. .

[0043] The above embodiments compress both the visual tokens and text tokens of the input language model, significantly improving the inference speed of the model.

[0044] Example 5: Please see Figure 6 As shown, the present invention also provides an electronic device 100 for a lightweight method of graphic multimodal model in the power industry; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0045] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the lightweight method for a multimodal graphic model in the power industry described in Embodiment 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0046] The at least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor. The processor 102 is the control center of the electronic device 100, connecting various parts of the electronic device 100 via various interfaces and lines.

[0047] The memory 101 in the electronic device 100 stores multiple instructions to implement a lightweight method for a multimodal graphic model in the power industry, and the processor 102 can execute the multiple instructions to achieve the following: Obtain the fully connected layer in the image-text multimodal model, and perform lightweight processing on the fully connected layer to obtain a lightweight fully connected layer; Obtain the image token and text token from the image-text multimodal model; Select the image relevance value of the desired image based on the image token and text token; Based on the image token and text token, select the text relevance value of the desired text; Image tokens and text tokens are aligned based on text relevance and image relevance values, and combined with a lightweight fully connected layer to obtain a lightweight image-text multimodal model.

[0048] Example 6: If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0049] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0051] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0052] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A lightweight method for creating a multimodal graphic model in the power industry, characterized in that, Includes the following steps: Obtain the fully connected layer in the image-text multimodal model, and perform lightweight processing on the fully connected layer to obtain a lightweight fully connected layer; Obtain the image token and text token from the image-text multimodal model; Select the image relevance value of the desired image based on the image token and text token; Based on the image token and text token, select the text relevance value of the desired text; Image tokens and text tokens are aligned based on text relevance and image relevance values, and combined with a lightweight fully connected layer to obtain a lightweight image-text multimodal model.

2. The lightweight method for a multimodal graphic model in the power industry according to claim 1, characterized in that, The specific method for obtaining the fully connected layer in the image-text multimodal model and then performing lightweight processing on the fully connected layer is as follows: Obtain the linear mapping relationship between the input feature matrix, weight matrix, and bias terms of the fully connected layer in the image-text multimodal model; The graph-text multimodal model is adjusted according to the linear mapping relationship, so that the adjusted fully connected layer is uniformly replaced with the target step size, and the preliminary adjusted fully connected layer is obtained. The convolutional layers of the pre-adjusted fully connected layers are grouped into convolutional layers. The independent sub-convolutions after grouping are then concatenated along the channel dimension to form a complete output feature map. Knowledge distillation is performed on the complete output feature map to obtain a lightweight fully connected layer.

3. The lightweight method for a multimodal graphic model in the power industry according to claim 2, characterized in that, The specific method for performing knowledge distillation on the complete output feature map to obtain a lightweight fully connected layer is as follows: Obtain the fully connected layer in the text-image multimodal model and use it as the teacher model; use the complete output feature map as the student model. Obtain the output vectors of each fully connected layer in the teacher model and the output vectors of each fully connected layer in the student model; The output vectors of each fully connected layer in the teacher model are used to update the output vectors of each fully connected layer in the student model, resulting in a lightweight fully connected layer.

4. The lightweight method for a multimodal graphic model in the power industry according to claim 1, characterized in that, The specific method for selecting the image relevance value of the desired image based on the image token and text token is as follows: Using image tokens as queries and text tokens as keys, we can calculate the similarity matrix between each image token and all text tokens. Process all elements in each row of the similarity matrix to obtain the overall relevance value between each image token and the entire input text sequence; A preset image correlation threshold range is used to obtain the overall correlation value within the image correlation threshold range, which is then used as the image correlation value.

5. A lightweight method for a multimodal graphic model in the power industry according to claim 1, characterized in that, The specific method for selecting the text relevance value of the desired text based on the image token and text token is as follows: Using text tokens as queries and image tokens as keys, calculate the similarity matrix between each text token and all image tokens; Process all elements in each row of the similarity matrix to obtain the text relevance value between each text token and the entire input image sequence; Preset a text relevance threshold range, obtain the overall relevance value within the text relevance threshold range, and use it as the text relevance value; A preset text relevance threshold range is defined, and the overall relevance value within the text relevance threshold range is obtained as the text relevance value.

6. A lightweight system for graphic multimodal models in the power industry, characterized in that, include: The fully connected layer processing module is used to obtain the fully connected layer in the image-text multimodal model, perform lightweight processing on the fully connected layer, and obtain a lightweight fully connected layer. The token acquisition module is used to obtain the image token and text token in the image-text multimodal model; The image relevance value selection module is used to select the image relevance value of the desired image based on the image token and text token; The text relevance value selection module is used to select the text relevance value of the desired text based on the image token and the text token. The module is used to align image tokens and text tokens based on text relevance values ​​and image relevance values. Combined with a lightweight fully connected layer, a lightweight image-text multimodal model is obtained.

7. A lightweight system for a multimodal graphic model in the power industry according to claim 6, characterized in that, The functionality of the fully connected layer processing module is implemented through the following methods: Obtain the linear mapping relationship between the input feature matrix, weight matrix, and bias terms of the fully connected layer in the image-text multimodal model; The graph-text multimodal model is adjusted according to the linear mapping relationship, so that the adjusted fully connected layer is uniformly replaced with the target step size, and the preliminary adjusted fully connected layer is obtained. The convolutional layers of the pre-adjusted fully connected layers are grouped into convolutional layers. The independent sub-convolutions after grouping are then concatenated along the channel dimension to form a complete output feature map. Knowledge distillation is performed on the complete output feature map to obtain a lightweight fully connected layer.

8. A lightweight system for a multimodal graphic model in the power industry according to claim 6, characterized in that, The functionality of the fully connected layer processing module is implemented through the following methods: Using image tokens as queries and text tokens as keys, we can calculate the similarity matrix between each image token and all text tokens. Process all elements in each row of the similarity matrix to obtain the overall relevance value between each image token and the entire input text sequence; A preset image correlation threshold range is used to obtain the overall correlation value within the image correlation threshold range, which is then used as the image correlation value.

9. A lightweight system for a multimodal graphic model in the power industry according to claim 6, characterized in that, The functionality of the fully connected layer processing module is implemented through the following methods: Using text tokens as queries and image tokens as keys, calculate the similarity matrix between each text token and all image tokens; Process all elements in each row of the similarity matrix to obtain the text relevance value between each text token and the entire input image sequence; Preset a text relevance threshold range, obtain the overall relevance value within the text relevance threshold range, and use it as the text relevance value; A preset text relevance threshold range is defined, and the overall relevance value within the text relevance threshold range is obtained as the text relevance value.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the lightweight method for a graphic multimodal model in the power industry as described in any one of claims 1 to 5.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the lightweight method for a graphic multimodal model in the power industry as described in any one of claims 1 to 5.

Images