Electric power multi-modal sample knowledge augmented question and answer method, system, device and medium

Abstract

The application relates to the technical field of power knowledge question and answer, and provides a power multi-modal sample knowledge enhancement question and answer method, system, device and medium, which comprises the following steps: in response to a knowledge question and answer request of each user terminal device, target power question images and target power question text prompts are obtained by preprocessing corresponding question images and question texts, the target power question images and the target power question text prompts are input into a power knowledge question and answer model comprising an image feature extraction module, a category guide enhancement module used for embedding classification probability labels of the target power question images into image features, a visual language conversion module, a text embedding extraction module and a fusion analysis module, fusion analysis processing is carried out on the power knowledge question and answer model to obtain a question and answer result, and the question and answer result is sent to a corresponding terminal device. The application can enhance the relevance of image features and power scenes, ensure effective alignment and fusion of image text information, realize fine perception and accurate understanding of complex power scenes, and improve the efficiency and accuracy of multi-modal sample question and answer processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power knowledge question and answer, in particular to a power multi-modal sample knowledge enhanced question and answer method, system, device and medium. BACKGROUND

[0002] Based on artificial intelligence technology, constructing a power knowledge question and answer model has gradually become a problem that is popular in the power grid field. The existing power knowledge question and answer method mainly focuses on improving the understanding ability of the power knowledge question and answer model to the power scene by fusing multi-modal data such as text, image and sensor data, and using a federated learning framework to construct a cross-region multi-modal power question and answer model to improve the accuracy of the model question and answer. However, the existing multi-modal data fusion method used for constructing the power knowledge question and answer model is relatively complex and cannot guarantee the effective alignment between the modal data, there is a problem of information redundancy or loss, which easily leads to insufficient generalization ability, and cannot realize comprehensive and accurate understanding of complex power scenes; at the same time, when constructing the power knowledge question and answer model based on federated learning training, there is a problem of large communication overhead and sensitivity to non-independent and identically distributed data, which leads to slow model convergence speed, cannot effectively adapt to complex data distribution across regions, and is difficult to guarantee the training efficiency and training effect of the power knowledge question and answer model, thereby reducing the practical value of the power knowledge question and answer model. SUMMARY

[0003] The purpose of the present application is to provide a power multi-modal sample knowledge enhanced question and answer method, which can enhance the association between image features and power scenes while ensuring the effective alignment and fusion of image text information by using a multi-modal sample analysis processing mechanism based on image classification label guided image feature and embedded representation conversion of image feature, realizing fine perception and accurate understanding of complex power scenes, and effectively improving the efficiency and accuracy of power multi-modal sample question and answer processing.

[0004] In order to achieve the above purpose, it is necessary to provide a power multi-modal sample knowledge enhanced question and answer method, system, device and medium in view of the above technical problems.

[0005] In the first aspect, the present application embodiment provides a power multi-modal sample knowledge enhanced question and answer method, which comprises:

[0006] In response to the knowledge question and answer request of each user terminal device, the problem image and the problem text corresponding to the knowledge question and answer request are obtained, and the problem image and the problem text are preprocessed respectively to obtain the corresponding target power problem image and target power problem text prompt;

[0007] The target power problem image and the target power problem text prompt corresponding to each of the knowledge question and answer requests are input into a pre-constructed power knowledge question and answer model for fusion analysis and processing to obtain a corresponding question and answer result; the power knowledge question and answer model comprises an image embedding extraction module, a text embedding extraction module and a fusion analysis module; the image embedding extraction module comprises an image feature extraction module, a category guide enhancement module and a visual language conversion module connected in sequence; the category guide enhancement module is configured to embed a category probability label of the target power problem image into an image feature corresponding to the target power problem image;

[0008] Each of the question and answer results is sent to the corresponding terminal device.

[0009] Further, the image feature extraction module is configured to obtain an image feature of the target power problem image based on a visual large model of a self-attention mechanism; the category guide enhancement module comprises a classification layer, a label embedding layer and an embedding splicing layer connected in sequence; the classification layer is configured to convert each image feature output by the image feature extraction module into a corresponding category probability label; the label embedding layer is configured to convert each category probability label output by the classification layer into a corresponding conditional embedding; and the embedding splicing layer is configured to splice each conditional embedding output by the label embedding layer with a corresponding image feature to obtain a corresponding category fusion image feature.

[0010] Further, the classification layer comprises a fully connected layer and a ReLU activation function connected in sequence.

[0011] Further, the conditional embedding is obtained by linearly mapping a category probability label using a preset parameter matrix and a bias matrix.

[0012] Further, the fusion analysis module comprises an embedding fusion module and a preset autoregressive large language model connected in sequence; the embedding fusion module is configured to splice a text embedding output by the text embedding extraction module to an image embedding output by the image embedding extraction module to obtain an image-text fusion embedding; and the preset autoregressive large language model is configured to analyze and process the image-text fusion embedding output by the embedding fusion module to obtain a corresponding question and answer result.

[0013] Further, the construction steps of the power knowledge question and answer model comprise:

[0014] An initial model is sent to each federated learning training node, so that each federated learning training node trains the initial model according to local data and a preset frozen fine-tuning principle to obtain a corresponding local model training result; the local model training result comprises a local optimized model parameter and corresponding model Fisher information;

[0015] receive local model training results sent by all the federated learning training nodes, filter and aggregate local optimization model parameters in the local model training results according to model Fisher information in each of the local model training results, and obtain corresponding global aggregated model parameters;

[0016] determine whether the global aggregated model parameters meet a preset iteration termination condition, if yes, obtain the power knowledge question and answer model according to the global aggregated model parameters, and if not, send the global aggregated model parameters to each of the federated learning training nodes for next round of training until the preset iteration termination condition is met.

[0017] Further, the local optimization model parameters include image feature extraction module parameters, category guide enhancement module parameters, visual language conversion module parameters and text embedding extraction module parameters; and the model Fisher information includes module Fisher information corresponding to each group of module parameters in the local optimization model parameters.

[0018] The step of filtering and aggregating the local optimization model parameters in the local model training results according to the model Fisher information in each of the local model training results to obtain the corresponding global aggregated model parameters comprises:

[0019] comparing each module Fisher information in all the model Fisher information to obtain maximum module Fisher information corresponding to each group of module parameters in the local optimization model parameters;

[0020] obtain target module parameters corresponding to each of the maximum module Fisher information respectively, and perform global aggregation update according to each of the target module parameters to obtain the global aggregated model parameters.

[0021] In a second aspect, an embodiment of the present application provides a power multi-modal sample knowledge enhanced question and answer system, which comprises:

[0022] a preset processing module configured to, in response to a knowledge question and answer request of each user terminal device, obtain a question image and a question text corresponding to the knowledge question and answer request, and pre-process the question image and the question text respectively to obtain a target power question image and a target power question text prompt;

[0023] The question and answer analysis module is configured to input the target power problem image and the target power problem text prompt corresponding to each knowledge question and answer request into a pre-constructed power knowledge question and answer model for fusion analysis processing to obtain a corresponding question and answer result.

[0024] The result feedback module is configured to send each question and answer result to the corresponding terminal device.

[0025] In a third aspect, an embodiment of the present application also provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0026] In a fourth aspect, an embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of the above method.

[0027] The present application provides a power multi-modal sample knowledge enhanced question and answer method, system, device and medium, which realizes responding to knowledge question and answer requests of each user terminal device, obtaining problem images and problem texts corresponding to the knowledge question and answer requests, pre-processing the problem images and the problem texts to obtain target power problem images and target power problem text prompts, inputting the target power problem images and the target power problem text prompts corresponding to each knowledge question and answer request into a pre-constructed power knowledge question and answer model comprising an image feature extraction module, a category guide enhancement module for embedding classification probability labels of the target power problem images into image features, a visual language conversion module, a text embedding extraction module and a fusion analysis module for fusion analysis processing to obtain corresponding question and answer results, and sending each question and answer result to a corresponding terminal device. Compared with the prior art, the power multi-modal sample knowledge enhanced question and answer method adopts a multi-modal sample analysis processing mechanism based on image classification label guided sample knowledge enhancement combined with embedded representation conversion of image features, which can enhance the association between image features and power scenarios, ensure effective alignment and fusion of image text information, realize fine perception and accurate understanding of complex power scenarios, and effectively improve the efficiency and accuracy of power multi-modal sample question and answer processing. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1is a flowchart of the power multimodal sample knowledge enhanced question and answer method in the embodiment of the present application;

[0029] Figure 2 is a structural diagram of the power knowledge question and answer model in the embodiment of the present application;

[0030] Figure 3 is a structural diagram of the category guide enhancement module in the embodiment of the present application;

[0031] Figure 4 is an image feature processing flowchart of the category guide enhancement module in the embodiment of the present application;

[0032] Figure 5 is a question image in one knowledge question and answer request in the embodiment of the present application;

[0033] Figure 6 is a question image in another knowledge question and answer request in the embodiment of the present application;

[0034] Figure 7 is a question image in still another knowledge question and answer request in the embodiment of the present application;

[0035] Figure 8 is a structural diagram of the power multimodal sample knowledge enhanced question and answer system in the embodiment of the present application;

[0036] Figure 9 is an internal structure diagram of the computer device in the embodiment of the present application;

[0037] Among them, the reference signs are:

[0038] 1, preset processing module; 2, question and answer analysis module; 3, result feedback module. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. Obviously, the following described embodiments are part of the embodiments of the present application, and are only used to illustrate the present application, but not to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] The power multimodal sample knowledge enhanced question and answer method provided by the application can be understood as being based on the existing power knowledge question and answer model. Due to the fact that the effective alignment between the multimodal data cannot be guaranteed, information redundancy or information loss is extremely likely to occur during the analysis of the multi-source data fusion, and the fine perception and accurate understanding of the complex power scene cannot be ensured. Therefore, the application status is proposed. A power knowledge question and answer method capable of guiding the image features based on the image classification labels to enhance the relevance of the image features and the power scene, combining the embedding representation conversion of the image features to ensure the effective alignment and fusion analysis of the power image text information can be deployed on a server providing power knowledge question and answer services. The following embodiments will describe the power terminal comprehensive access control power multimodal sample knowledge enhanced question and answer method in detail.

[0041] In one embodiment, as shown in Figure 1 A power multimodal sample knowledge enhanced question and answer method is provided, comprising:

[0042] S11, in response to the knowledge question and answer request of each user terminal device, acquiring the problem image and the problem text corresponding to the knowledge question and answer request, and pre-processing the problem image and the problem text to obtain the corresponding target power problem image and target power problem text prompt.

[0043] The problem image and the problem text can be understood as the power-related image and the problem text related to the power-related image included in the power knowledge question and answer request submitted by the user through the terminal device. The specific problem image and the problem text vary depending on the knowledge question and answer request submitted by the user terminal device in actual use, which is not limited here. The target power problem image can be understood as image data that meets the size and format required for subsequent power knowledge question and answer model analysis after denoising, size normalization, and pixel normalization of the problem image. The target power problem text prompt can be understood as text data that meets the use requirements of subsequent power knowledge question and answer model analysis after denoising (filtering special characters, irregular spaces, or other text format information unrelated to the analysis content), sensitive word processing, and integration and association with historical dialogue text. It should be noted that the pre-processing process of the problem image and the problem text can be realized by referring to the existing related image processing and text processing technology according to the data input requirements of the constructed power knowledge question and answer model.

[0044] S12, the target power problem image and the target power problem text prompt corresponding to each of the knowledge question and answer request are input into a pre-constructed power knowledge question and answer model for fusion analysis processing to obtain a corresponding question and answer result; wherein the power knowledge question and answer model can be understood as a multi-modal image and text question and answer model capable of fusion analysis based on the target power problem image and the corresponding target power problem text prompt input by the user to finally generate the required question and answer result.

[0045] In order to effectively integrate the image and text multi-source heterogeneous data in the power field and realize fine perception and accurate understanding of complex power scenarios, as shown in Figure 2 As shown in the figure, the power knowledge question and answer model is preferably provided with an image embedding extraction module, a text embedding extraction module and a fusion analysis module, wherein the image embedding extraction module can be understood as a processing module capable of feature extraction of an image and mapping high-dimensional image features to a low-dimensional dense vector space to obtain image embedding representation after class label guided enhancement of image features; specifically, the image embedding extraction module includes an image feature extraction module, a class guided enhancement module and a visual language conversion module connected in sequence, and the class guided enhancement module is used to embed the classification probability label of the target power problem image into the image feature corresponding to the target power problem image; that is, the image feature extraction module can be understood as a pre-trained large language model for visual image feature extraction of the target power problem image, the class guided enhancement module can be understood as a processing module for embedding the feature classification probability label corresponding to each image feature obtained by the image feature extraction module for visual image feature extraction of the target power problem image, and the visual language conversion module can be understood as a processing module for converting the image feature into an image embedding that can be processed by the large language model.

[0046] In order to ensure the comprehensiveness and fineness of image information extraction of complex power scenarios, the image feature extraction module is preferably provided with a visual large model based on a self-attention mechanism to obtain the image feature of the target power problem image, that is, the image feature is extracted by the visual large model based on the self-attention mechanism; in principle, the visual large model based on the self-attention mechanism can adopt any large language model capable of realizing the required visual image feature extraction function, in order to guarantee the efficiency, comprehensiveness and accuracy of image feature extraction, the EVA-ViT-G model is preferably used as the image feature extraction module.

[0047] In order to ensure the reliability of class guided enhancement of image features, as shown in Figure 3As shown, the embodiment preferably provides that the category guidance enhancement module comprises a classification layer, a label embedding layer and an embedding concatenation layer connected in sequence; wherein the classification layer is used to convert each image feature output by the image feature extraction module into a corresponding category probability label, in order to improve the representation ability of each category probability label as much as possible, the classification layer is preferably provided to comprise a fully connected layer and a ReLU activation function connected in sequence, through the processing of the fully connected layer, all local features in the image feature can be effectively captured, and complex global combination relationships among these local features can be learned, and through the introduction of the nonlinear activation processing of the ReLU activation function, the model expression ability of the classification layer can be effectively improved, and rich image features can be efficiently mapped to the category space.

[0048] The label embedding layer is used to convert each category probability label output by the classification layer into a corresponding conditional embedding, that is, to realize reliable dimension reduction of different image features to retain important label information; specifically, the conditional embedding can be understood as reducing all feature elements in each image feature to feature information of a preset dimension to represent corresponding category information, and preferably, the category probability label is linearly mapped by using a preset parameter matrix and a bias matrix to obtain, which can be represented as:

[0049]

[0050] wherein, represents the category probability label of the i-th image feature; represents the conditional embedding of the category probability label of the i-th image feature, and the feature dimension of the conditional embedding can be set according to actual application requirements; and respectively represent the parameter matrix and the bias matrix, and the dimensions of the parameter matrix and the bias matrix are determined based on the dimensions of the image feature and the conditional embedding.

[0051] The embedding concatenation layer is used to concatenate each conditional embedding output by the label embedding layer with the corresponding image feature to obtain a corresponding category fusion image feature.

[0052] In actual applications, for example, Figure 4As shown, assuming that the dimension of the image features of the target power problem image is 32x768, i.e., there are 32 channels, and each channel contains 768 feature information, the image features are input to the category guided enhancement module, first processed by the classification layer to obtain the corresponding category probability label, and then the label embedding layer is used to perform embedding conversion processing on the corresponding category probability label through a parameter matrix with a dimension of 768x8 and a bias matrix with a dimension of 32x8 to obtain a condition embedding with a dimension of 32x8, and then the embedding splicing layer is used to splice the condition embedding with a dimension of 32x8 and the image features with a dimension of 32x768 to obtain a category fusion image feature with a dimension of 32x776. It should be noted that the embedding splicing layer splices the feature information and the condition embedding in the same channel when splicing the condition embedding and the image features.

[0053] The visual language conversion module in the image embedding extraction module can in principle adopt any functional module that can convert image features into embedding representation. In order to minimize the gap between visual image information and text language features and provide an analysis basis for effective alignment and fusion of image information and text information, the Query-Transformer module in the VisualGLM model is preferably used as the visual language conversion module to convert the category fusion image features obtained by the category guided enhancement module into image embeddings that can be analyzed and used by the large language model. The network structure of the Query-Transformer module can refer to the network structure of the existing VisualGLM model, which will not be described in detail here.

[0054] The present embodiment not only can effectively enhance the relevance of image features and power scenarios by guiding image features with image classification labels, facilitate to improve the accurate understanding ability of image information in complex power scenarios, but also can minimize the gap between visual image information and text language features through visual language conversion processing, provide a reliable analysis basis for effective alignment and fusion of multi-source heterogeneous image information and text information, and further provide reliable technical support for improving the understanding ability and question and answer ability of the power multi-modal question and answer model for image and text data.

[0055] The fusion analysis module in the power knowledge question and answer model can be understood as being used for fusion analysis of the image embedding output by the image embedding extraction module and the text embedding output by the text embedding extraction module to obtain reliable question and answer results. In order to ensure the efficiency and reliability of the fusion analysis of image information and text information, the embodiment preferably sets the fusion analysis module to include an embedding fusion module and a preset autoregressive large language model connected in sequence, and the embedding fusion module is used to splice the text embedding output by the text embedding extraction module to the image embedding output by the image embedding extraction module to obtain image-text fusion embedding, and the preset autoregressive large language model is used to analyze and process the image-text fusion embedding output by the embedding fusion module to obtain the corresponding question and answer results. It should be noted that the preset autoregressive large language model can in principle use any large language model in the existing GLM series model, GPT series model, Claude series model and Gemini series model, but in order to ensure the efficiency and accuracy of the Chinese question and answer scene processing while reducing the model deployment cost and improving the universality of the application scene, the embodiment preferably sets the preset autoregressive large language model in the fusion analysis module to be a ChatGLM-6B model, and the process of using the ChatGLM-6B model to analyze the image-text fusion embedding to obtain the result text can refer to the implementation of the existing ChatGLM-6B model, which will not be described in detail here.

[0056] By constructing a cross-regional power knowledge question and answer model in a distributed training manner using a federated learning framework, the multi-application scenario intelligent interaction capability of the power knowledge question and answer model can be improved by effectively utilizing more data sample resources to train the model. However, considering that the existing federated learning training does not focus on solving the problem of large communication overhead and sensitivity to non-independent and identically distributed data when constructing a cross-regional power knowledge question and answer model, the model convergence speed is slow, and it cannot effectively adapt to complex data distribution across regions, making it difficult to ensure the training efficiency and training effect of the power knowledge question and answer model. The embodiment preferably adopts a federated learning training strategy of freezing fine-tuning training of the power knowledge question and answer model by a federated learning training node and aggregating distributed model parameters to generate a global model based on Fisher information of each group of module parameters. Specifically, the construction steps of the power knowledge question and answer model include:

[0057] sending an initial model to each federated learning training node, so that each federated learning training node trains the initial model according to local data and a preset frozen fine-tuning principle to obtain a corresponding local model training result; the local model training result includes local optimized model parameters and corresponding model Fisher information. Wherein, the initial model can be understood as having the aforementioned image embedding extraction module, text embedding extraction module and fusion analysis module, and the image embedding extraction module includes an image feature extraction module (a pre-trained visual large language model), a class guided enhancement module and a visual language conversion module connected in sequence, and the fusion analysis module includes an embedding fusion module and a preset autoregressive large language model (a pre-trained text autoregressive large language model) intelligent question and answer model, and the parameters of each functional module need to be obtained through distributed federated learning training.

[0058] In actual application, after each federated learning training node receives the initial model sent by the central server, the initial model is trained and optimized according to the local data of the node and based on a preset loss function (for example, the cross-entropy loss is calculated by comparing the difference between the text probability distribution of the model output result and the real text label), according to the preset frozen fine-tuning principle, to obtain the required local optimized model parameters and corresponding model Fisher information, and generate the local model training result and send it to the central server to complete a round of federated training. It should be noted that the preset frozen fine-tuning principle in the present embodiment can be different based on the characteristics of the functional modules in the actual power knowledge question and answer model and the model application scenario requirements. For example, for the image embedding extraction module, in order to make it more suitable for the extraction of power image features, to provide rich image information for the subsequent question and answer model, to ensure that the question and answer model can effectively process complex power images and improve the image understanding ability of the question and answer system, the parameters of the other layers in the image embedding extraction module except the last feature summary layer can be frozen during model training, and only the last feature summary layer is trained to make it more accurately capture power image features. When the image embedding extraction module adopts the EVA-ViT-G model, during the model training process, the parameters of other layers of the pre-trained EVA-ViT-G model need to be frozen, and only the last layer of Transformer is fine-tuned and trained; for the class guided enhancement module and the visual language conversion module, the parameters of all modules need to be updated and trained; for the fusion analysis module, the parameters that need to be trained and updated are the parameters of the preset autoregressive large language model in principle, but considering that this part of processing only involves the processing of text data, directly using the related pre-trained model parameters can meet the application requirements, and in order to improve the training efficiency of the entire model, the parameters of the preset autoregressive large language model can be frozen during the model training process.

[0059] The local optimization model parameters in the embodiment include image feature extraction module parameters, category guide enhancement module parameters, visual language conversion module parameters and text embedding extraction module parameters; the corresponding model Fisher information includes module Fisher information corresponding to each group of module parameters in the local optimization model parameters. In actual training, after each federal learning training node completes a round of model training, the Fisher information matrix corresponding to each group of module parameters is calculated, that is, the model Fisher information sent by each federal learning training node to the central server includes the module Fisher information matrix corresponding to the image feature extraction module parameters, the category guide enhancement module parameters, the visual language conversion module parameters and the text embedding extraction module parameters; the Fisher information is used to evaluate the information amount of a random variable for an unknown parameter estimate, and can be calculated by the following formula:

[0060]

[0061]

[0062] Wherein, D represents the local data of the federal learning training node; represents a preset loss function; represents the parameter of the trainable module j of the federal learning training node, and represent the weight matrix and the bias matrix of the trainable module j respectively; d and r represent two dimensions of the matrix respectively; represents the expected value of the Fisher information corresponding to each matrix parameter; represents the Fisher information of the trainable module j; represents the normalized Fisher information of the trainable module j.

[0063] In each round of model training process, the federal learning training node fine-tunes the model parameters using the local data, calculates the Fisher information matrix of each module parameter, takes the average of the matrix elements as the corresponding module Fisher information, and uploads it to the central server together with the local optimization model parameters obtained by the current round of federal training.

[0064] Receiving all the local model training results sent by the federal learning training nodes, screening and aggregating the local optimization model parameters in the local model training results according to the model Fisher information in each of the local model training results, obtaining the corresponding global aggregated model parameters; wherein the model Fisher information can be used to evaluate the importance of the model parameters, in order to ensure the reliability of the aggregation of each local optimization model parameter, the embodiment preferably screens out the optimal module parameters of each trainable module based on the comparison results of each module Fisher information respectively, and combines the optimal module parameters of each trainable module to obtain the required global aggregated model parameters.

[0065] Specifically, the step of screening and aggregating the local optimization model parameters in the local model training results according to the model Fisher information in each of the local model training results to obtain the corresponding global aggregated model parameters comprises:

[0066] comparing each module Fisher information in all the model Fisher information to obtain the maximum module Fisher information corresponding to each group of module parameters in the local optimization model parameters;

[0067] obtaining the target module parameters corresponding to each of the maximum module Fisher information respectively, and performing global aggregated updating according to each of the target module parameters to obtain the global aggregated model parameters.

[0068] determining whether the global aggregated model parameters satisfy a preset iteration termination condition, if yes, obtaining the power knowledge question and answer model according to the global aggregated model parameters, otherwise, sending the global aggregated model parameters to each of the federated learning training nodes to continue the next round of training until the preset iteration termination condition is reached; wherein the preset iteration termination condition can be set according to actual application requirements, such as whether the text generation quality corresponding to the global aggregated model parameters reaches the target quality requirement, or whether the number of rounds of federated learning training reaches the maximum training round number, etc., which is not limited here.

[0069] The method for training the power knowledge question and answer model by adopting the federated learning training node to perform frozen fine-tuning training and combining the federated learning training strategy of aggregating the distributed model parameters to generate the global model based on the Fisher information of each group of module parameters can not only reduce the communication overhead, improve the adaptability of complex data distribution across regions and maintain privacy security, but also effectively improve the convergence speed of model training by screening the optimal module parameters for global model parameter aggregated updating, and reliably improve the generalization ability of model application and the accuracy of question and answer result text generation.

[0070] S13, sending each of the question and answer results to the corresponding terminal device.

[0071] The embodiment of the application provides a response to the knowledge question and answer request of each user terminal device, obtains a question image and a question text corresponding to the knowledge question and answer request, respectively pre-processes the question image and the question text to obtain a corresponding target power question image and a target power question text prompt, and then inputs the target power question image and the target power question text prompt corresponding to each knowledge question and answer request into a pre-constructed power knowledge question and answer model including an image feature extraction module, a category guide enhancement module for embedding a classification probability label of the target power question image into an image feature, a visual language conversion module, a text embedding extraction module and a fusion analysis module for fusion analysis processing to obtain a corresponding question and answer result, and sends each question and answer result to a scheme of a corresponding terminal device. The multi-modal sample analysis processing mechanism based on the image classification label guided image feature sample knowledge enhancement and the embedded representation conversion of the image feature can enhance the association between the image feature and the power scene, ensure the effective alignment and fusion of the image text information, realize fine perception and accurate understanding of the complex power scene, and effectively improve the efficiency and accuracy of the power multi-modal sample question and answer processing.

[0072] In order to verify the effectiveness of the power multi-modal sample knowledge enhancement question and answer method provided by the application, the embodiment analyzes the intelligent reply of a plurality of knowledge question and answer requests proposed by different user terminal devices: for the knowledge question and answer request including the question image and the corresponding question text as shown in the figure, the answer result is "this picture shows a group of electric power workers installing or maintaining high-voltage electrical equipment during the day. A worker stands on a ladder and operates a large black cylindrical device, which may be part of a transformer, and other colleagues in blue work clothes are assisting on the ground. There is a crane lifting equipment on the scene, indicating that it is a task that requires heavy machinery cooperation." Figure 5 For the knowledge question and answer request including the question image and the corresponding question text as shown in the figure, the answer result is "this picture shows a worker in a yellow reflective vest crossing a construction fence. In the background, there are green plants and many buildings, and there is a white zebra crossing on the ground, indicating that it is an ongoing urban road construction area." Figure 6 For the knowledge question and answer request including the question image and the corresponding question text as shown in the figure, the answer result is "this picture shows a worker in a yellow reflective vest crossing a construction fence. In the background, there are green plants and many buildings, and there is a white zebra crossing on the ground, indicating that it is an ongoing urban road construction area." Figure 7The question image and the corresponding question text shown are "Describe the content of the scene in this image." The answer result given is "This is a photo taken inside a substation, where two staff members in blue clothes are talking. Neither of them is wearing a safety helmet. It is important to note that substation is involved in the transmission and distribution of high-voltage electricity in the power system, so wearing safety measures such as safety helmets is crucial when working in a substation." Based on the analysis of the answer result of the above knowledge question and answer request, the power multi-modal sample knowledge enhanced question and answer method provided by the application can perceive and accurately understand the key information of the complex power scene based on the multi-modal data input of the user, and then give accurate and reliable answer results.

[0073] At the same time, based on the same test set, the application of the power multi-modal sample knowledge enhanced question and answer method provided by the application and other different power knowledge question and answer models are compared and verified, and the scoring results shown in Table 1 are obtained. Among them, the BLEU (Bilingual Evaluation Understudy) index is used to automatically evaluate the quality of machine translation and text generation task, and the method of comparing the n-gram overlap between the generated text and the reference text is used to evaluate the quality of the generated text; the METEOR index is a text generation quality evaluation index that considers word matching, synonym matching and word sequence information; the CIDEr index is an index used to evaluate the image description generation task, which calculates the TF-IDF weighted n-gram between the text description and the image to evaluate the consistency of the description.

[0074] Table 1 Scoring performance of different models on test set

[0075]

[0076] Based on the results in Table 1, the power multi-modal sample knowledge enhanced question and answer method provided by the application has higher scoring advantages compared with the existing LLava-1.6 model, QwenVL and GPT-4 model.

[0077] It should be noted that although each step in the above flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified in this document, the execution of these steps has no strict order limitation, and these steps can be executed in other order.

[0078] In one embodiment, as Figure 8 shown, a power multi-modal sample knowledge enhanced question and answer system is provided, which comprises:

[0079] A preset processing module 1 is configured to, in response to a knowledge question and answer request of each user terminal device, acquire a question image and a question text corresponding to the knowledge question and answer request, and pre-process the question image and the question text to obtain a target power question image and a target power question text prompt, respectively.

[0080] A question and answer analysis module 2 is configured to input the target power question image and the target power question text prompt corresponding to each knowledge question and answer request into a pre-constructed power knowledge question and answer model for fusion analysis processing to obtain a corresponding question and answer result. The power knowledge question and answer model includes an image embedding extraction module, a text embedding extraction module, and a fusion analysis module. The image embedding extraction module includes an image feature extraction module, a category guided enhancement module, and a visual language conversion module connected in sequence. The category guided enhancement module is configured to embed a classification probability label of the target power question image into an image feature corresponding to the target power question image.

[0081] A result feedback module 3 is configured to send each question and answer result to a corresponding terminal device.

[0082] The specific limitations of the power multi-modal sample knowledge enhanced question and answer system can refer to the limitations of the power multi-modal sample knowledge enhanced question and answer method in the above text, and the corresponding technical effects can also be obtained equally, which will not be repeated here. Each module in the above power multi-modal sample knowledge enhanced question and answer system can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0083] Figure 9 An internal structure diagram of a computer device in an embodiment is shown, which can be a terminal or a server. As shown in FIG. 6, the computer device includes a processor 61, a memory 62, a communication interface 63, and a communication bus 64. The communication bus 64 is configured to connect the processor 61, the memory 62, and the communication interface 63 to each other. Figure 9As shown, the computer device includes a processor, a memory, a network interface, a display, a camera and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program, when executed by the processor, can implement the power multi-modal sample knowledge enhancement question and answer method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0084] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0085] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the steps of the above method.

[0086] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by the processor, implements the steps of the above method.

[0087] In summary, the power multimodal sample knowledge enhanced question answering method, system, device and medium provided by the embodiment of the present application can ensure the effective alignment and fusion of image text information while enhancing the relevance of image features and power scenarios, realize fine perception and accurate understanding of complex power scenarios, by adopting the multimodal sample analysis processing mechanism of sample knowledge enhancement based on image classification label guiding image features combined with embedded representation conversion of image features. In addition, the federated learning training strategy of freezing fine-tuning training of the federated learning training node combined with the aggregation of distributed model parameters based on the Fisher information of each group of module parameters to generate a global model can not only reduce communication overhead, improve the adaptability of complex data distribution across regions and maintain privacy security, but also effectively improve the convergence speed of model training by screening the optimal module parameters for global model parameter aggregation and updating, and also reliably improve the generalization ability of model application and the accuracy of question and answer result text generation, thereby significantly improving the efficiency and accuracy of power multimodal sample question and answer processing.

[0088] Each embodiment in the specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, the system embodiment is described relatively simply because it is basically similar to the method embodiment, and the relevant parts can be referred to the part of the method embodiment. It should be noted that the technical features of the above embodiments can be combined arbitrarily, and in order to make the description simple, not all possible combinations of the technical features of the above embodiments are described, but as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the specification.

[0089] The above-described embodiments only express several preferred embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and replacements can be made, and these improvements and replacements should be considered as the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the protection scope of the claims.

Claims

1. A power multi-modal sample knowledge augmented question answering method, characterized in that, The method comprises: In response to a knowledge question and answer request of each user terminal device, obtaining a question image and a question text corresponding to the knowledge question and answer request, and preprocessing the question image and the question text to obtain a corresponding target power question image and a target power question text prompt; The target power question image and the target power question text prompt corresponding to each knowledge question and answer request are input into a pre-constructed power knowledge question and answer model for fusion analysis and processing to obtain a corresponding question and answer result; the power knowledge question and answer model comprises an image embedding extraction module, a text embedding extraction module and a fusion analysis module; the image embedding extraction module comprises an image feature extraction module, a category guide enhancement module and a visual language conversion module connected in sequence; the category guide enhancement module is used for embedding the classification probability label of the target power question image into the image feature corresponding to the target power question image; wherein the image feature extraction module is used for obtaining the image feature of the target power question image based on a self-attention mechanism visual large model; the category guide enhancement module comprises a classification layer, a label embedding layer and an embedding splicing layer connected in sequence; the classification layer is used for converting each image feature output by the image feature extraction module into a corresponding category probability label; the label embedding layer is used for converting each category probability label output by the classification layer into a corresponding conditional embedding; the embedding splicing layer is used for splicing each conditional embedding output by the label embedding layer with the corresponding image feature to obtain a corresponding category fusion image feature; Each question and answer result is sent to the corresponding terminal device.

2. The power multimodal sample knowledge augmentation question answering method of claim 1, wherein, The classification layer comprises a fully connected layer and a ReLU activation function connected in sequence.

3. The power multimodal sample knowledge augmentation question answering method of claim 1, wherein, The conditional embedding is obtained by linear mapping of the category probability label using a preset parameter matrix and a bias matrix.

4. The power multimodal sample knowledge augmentation question answering method of claim 1, wherein, The fusion analysis module comprises an embedding fusion module and a preset autoregressive large language model connected in sequence; the embedding fusion module is used for splicing the text embedding output by the text embedding extraction module to the image embedding output by the image embedding extraction module to obtain an image-text fusion embedding; the preset autoregressive large language model is used for analyzing and processing the image-text fusion embedding output by the embedding fusion module to obtain a corresponding question and answer result.

5. The power multimodal sample knowledge augmentation question answering method of claim 1, wherein, The construction steps of the power knowledge question and answer model comprise: An initial model is sent to each federated learning training node, so that each federated learning training node trains the initial model according to local data and a preset frozen fine-tuning principle to obtain a corresponding local model training result; the local model training result comprises a local optimized model parameter and corresponding model Fisher information; The local model training results sent by all the federated learning training nodes are received, and the local optimized model parameters in the local model training results are screened and aggregated according to the model Fisher information in each local model training result to obtain corresponding global aggregated model parameters; determine whether the global aggregated model parameters meet a preset iteration termination condition, and if so, obtain the power knowledge question and answer model according to the global aggregated model parameters, and if not, send the global aggregated model parameters to each of the federated learning training nodes to continue the next round of training until the preset iteration termination condition is reached.

6. The power multimodal sample knowledge augmentation question answering method of claim 5, wherein, The local optimization model parameters include image feature extraction module parameters, category guided enhancement module parameters, visual language conversion module parameters, and text embedding extraction module parameters; and the model Fisher information includes module Fisher information corresponding to each group of module parameters in the local optimization model parameters. The step of filtering and aggregating the local optimization model parameters in the local model training results according to the model Fisher information in each of the local model training results to obtain corresponding global aggregated model parameters comprises: comparing each module Fisher information in all the model Fisher information to obtain maximum module Fisher information corresponding to each group of module parameters in the local optimization model parameters; obtaining target module parameters corresponding to each of the maximum module Fisher information respectively, and performing global aggregation update according to each of the target module parameters to obtain the global aggregated model parameters.

7. An electric power multimodal sample knowledge augmented question answering system, characterized in that, The system comprises: a preset processing module configured to, in response to a knowledge question and answer request of each user terminal device, acquire a problem image and a problem text corresponding to the knowledge question and answer request, and pre-process the problem image and the problem text respectively to obtain a target power problem image and a target power problem text prompt; a question and answer analysis module configured to input the target power problem image and the target power problem text prompt corresponding to each of the knowledge question and answer requests into a pre-constructed power knowledge question and answer model for fusion analysis processing to obtain a corresponding question and answer result; the power knowledge question and answer model comprises an image embedding extraction module, a text embedding extraction module, and a fusion analysis module; the image embedding extraction module comprises an image feature extraction module, a category guided enhancement module, and a visual language conversion module connected in sequence; the category guided enhancement module is configured to embed a classification probability label of the target power problem image into an image feature corresponding to the target power problem image; wherein the image feature extraction module is configured to acquire an image feature of the target power problem image based on a self-attention mechanism visual large model; the category guided enhancement module comprises a classification layer, a label embedding layer, and an embedding splicing layer connected in sequence; the classification layer is configured to convert each image feature output by the image feature extraction module into a corresponding classification probability label; the label embedding layer is configured to convert each classification probability label output by the classification layer into a corresponding conditional embedding; and the embedding splicing layer is configured to splice each conditional embedding output by the label embedding layer with a corresponding image feature to obtain a corresponding category fusion image feature; a result feedback module configured to send each of the question and answer results to a corresponding terminal device.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images