Power grid operation scene classification and identification method, system, equipment and medium

Abstract

The invention discloses a power grid operation scene classification and recognition method and system and a medium, and belongs to the field of laser radar ranging, and the method comprises the steps: carrying out the classification and recognition of a power grid operation scene, carrying out the image recognition of an input construction image through a scene recognition model, and outputting a classification result; the scene recognition model is trained according to a historical industry data set and is obtained through LoRa fine tuning; according to a classification result, determining and selecting a cue word, obtaining a text semantic vector corresponding to the cue word, carrying out image analysis on the construction image according to the cue word, and obtaining a construction image semantic vector and the text semantic vector corresponding to the cue word; and performing multi-modal alignment on the construction image semantic vector and the text semantic vector to obtain a classification and recognition result of the power grid operation scene, so that by implementing the method and the device, the accuracy of intelligent question answering in scene classification and recognition can be improved, and high-precision power grid operation scene classification and recognition can be realized.

Description

Technical Field

[0001] This invention relates to the field of power grid safety monitoring, and in particular to a method, system, equipment, and medium for classifying and identifying power grid operation scenarios. Background Technology

[0002] Operational risk management is a crucial aspect of ensuring the safety of employees during daily outdoor operations for power grid companies. Its core lies in the real-time identification and analysis of personnel's behavior during key procedures. To accelerate the digital transformation of safety production supervision towards pre-emptive prevention, power grid companies are continuously increasing the promotion and application of "Internet + Law Enforcement," promoting the use of non-site enforcement methods to achieve "remote + on-site" enforcement. Accurate classification and identification of power grid operation scenarios is fundamental to achieving these goals. Existing technical solutions typically only utilize multimodal models for representation learning, extracting image environmental feature vectors and then inputting them into a classification module to achieve scene classification. This only achieves image-level classification and is insufficient to handle complex power grid operation procedures, mainly exhibiting the following two shortcomings: First, the high-dimensional semantic vectors of the images are not saved, resulting in the failure to effectively capture some key operational procedures and the inability to promptly correct and stop operational violations; Second, the lack of cue word engineering technology, relying solely on the model's free interpretation, makes it difficult to guarantee the accuracy and reliability of the output results. Summary of the Invention

[0003] This invention provides a method, system, device, and medium for classifying and recognizing power grid operation scenarios, which can improve the accuracy of intelligent question answering in scenario classification and recognition, and achieve highly accurate classification and recognition of power grid operation scenarios.

[0004] This invention provides a method for classifying and recognizing power grid operation scenarios, including: The scene recognition model performs image recognition on the input construction images and outputs classification results; the scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa. Based on the classification results, select prompt words, obtain the text semantic vector corresponding to the prompt words, and perform image analysis on the construction image based on the prompt words to obtain the construction image semantic vector; The semantic vectors of the construction images and the semantic vectors of the text are aligned in a multimodal manner to obtain the classification and recognition results of the power grid operation scene.

[0005] This invention utilizes prompt word engineering technology to call preset prompt word templates tailored to business needs based on different classification results. This deeply integrates scene classification and language generation, avoiding unstable output caused by large models operating freely and improving the quality and controllability of model responses. By acquiring the semantic vector of the construction image and aligning it with the textual semantic vector of the prompt words in a multimodal manner, accurate classification and recognition results are obtained. The image semantic vector can also be used for subsequent key process tracing and analysis. Compared to existing technologies that only utilize multimodal models for representation learning, this application significantly improves the accuracy of intelligent question answering in the field of power grid safety supervision by extracting image environment feature vectors and then connecting them to the classification module to achieve scene classification. It also strengthens the effective control of operational safety risks.

[0006] Furthermore, the scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa, specifically as follows: Construct an initial model for image recognition, and set the weights and prediction vectors of the initial model; The historical industry dataset is input into the initial model, and the data is processed by softmax normalization to obtain the true label vector. Calculate the cross-entropy loss between the true label vector and the predicted vector, and calculate the image classification loss. Then, calculate the joint loss based on the cross-entropy loss and the image classification loss. The loss gradient is calculated based on the joint loss, cross-entropy loss, and image classification loss, and backpropagation is performed on the initial model. The weight parameters of the initial model are then fine-tuned using LoRa until the Softmax loss is optimized to the optimal solution, and the scene recognition model is output.

[0007] This approach involves constructing an initial model for image recognition and setting its weights and prediction vectors. By introducing softmax normalization to process historical industry datasets and obtain real label vectors, the model's semantic understanding accuracy in complex scenes is enhanced. By calculating the joint loss of cross-entropy loss and image classification loss, and performing gradient backpropagation based on this joint loss, the overall performance and generalization ability of the model in scene recognition are significantly improved. LoRa fine-tuning technology is used to update the initial model weights efficiently and with low parameter count. While maintaining the model's original powerful feature extraction capabilities, it quickly adapts to industry-specific scenarios until the softmax loss is optimized to its optimal value, ultimately resulting in a LoRa model for vertical safety supervision.

[0008] Furthermore, the weight parameters of the initial model are fine-tuned and updated using LoRa, as shown in the following formula: Where W is the original weight matrix; A is the reduced-dimensional matrix; B is the increased-dimensional matrix; r is the rank; d and k are the dimensions of the original weight matrix; This is the updated weight matrix; This is the incremental update of the weights; R is the real number field.

[0009] This approach allows for efficient data processing and model understanding guidance by fine-tuning the input data format through pre-defined dictionary instructions. The LoRa fine-tuning mechanism updates the initial model's weight parameters in a low-rank adaptive manner, introducing trainable dimensionality reduction and expansion matrices on top of the original weight matrix to achieve efficient parameter updates and quickly adapt to specific recognition task scenarios.

[0010] Further, based on the classification results, prompt words are selected to obtain the text semantic vectors corresponding to the prompt words, specifically as follows: Based on the classification results, prompt words are selected. By segmenting the prompt words, word segments are obtained. The word segments are mapped to obtain the corresponding discrete token ID sequence. A semantic vector is generated based on the token ID sequence to obtain the text semantic vector corresponding to the prompt word.

[0011] By selecting prompt words that match the scene based on the classification results, segmenting them into discrete token ID sequences, and then generating corresponding text semantic vectors based on the token ID sequences, the structured and semantic encoding of prompt words is achieved. By introducing a prompt word segmentation and token mapping mechanism, the model's ability to understand technical terms and complex semantics is enhanced, semantic bias is avoided, and the accuracy and consistency of subsequent multimodal alignment are improved.

[0012] Furthermore, image analysis is performed on the construction image based on the prompt words to obtain the semantic vector of the construction image, specifically as follows: The construction image is segmented and linearly mapped according to the prompt words to obtain the matrix representation of the construction image. Based on the matrix representation, a semantic vector is generated through MLP to obtain the semantic vector of the construction image.

[0013] By segmenting and linearly mapping construction images, they are converted into structured matrix representations, enabling efficient and accurate encoding of image information. This allows for the extraction and fusion of deep visual semantic features, enhancing the model's semantic understanding of complex construction environments and details, thereby improving the accuracy and consistency of subsequent multimodal alignment.

[0014] Furthermore, the semantic vectors of the construction images and the semantic vectors of the text are multimodally aligned to obtain the classification and recognition results of the power grid operation scene, specifically: The construction image semantic vector and the text semantic vector are multimodally aligned using the following formula to obtain the fused multimodal feature representation: in, For multimodal feature representation; For text semantic vectors; For construction image semantic vectors; , and for After passing through the linear layer, the results represent the query intent, image index, and image content, respectively. for transpose; for The dimension; , and These are learnable linear coefficients; Based on the multimodal feature representation, the classification and recognition results of power grid operation scenarios are obtained.

[0015] By introducing a multimodal attention alignment mechanism, the semantic vectors of construction images and text are deeply fused. A learnable weight matrix is ​​used to generate query, key, and value vectors, and the fused multimodal feature representation is obtained through scaling dot product attention calculation. This achieves accurate alignment and interactive enhancement of image information and text semantics, thereby enhancing the ability to analyze images of power grid operation scenarios.

[0016] Further, the step of generating a semantic vector based on the token ID sequence to obtain a text semantic vector corresponding to the prompt word includes: The scene recognition model is fused with the preset initial language model using the following formula; in, The initial language model weights; Assign weights to the scene recognition model; The weights of the fused language model; The token ID sequence is input into the fused language model to obtain the text semantic vector corresponding to the prompt word.

[0017] By weighting and fusing the scene recognition model with the initial language model, the fused language model gains stronger scene perception and semantic understanding capabilities, thereby enabling the generated text semantic vector to more accurately reflect the scene semantic features corresponding to the prompt words.

[0018] Another embodiment of the present invention provides a power grid operation scene classification and recognition system, including: an image recognition module, an image analysis module, and a multimodal alignment module; The image recognition module is used to perform image recognition on the input construction image and output the classification result; The image analysis module is used to determine prompt words based on the classification results, and to perform image analysis on the construction image based on the prompt words to obtain the semantic vector of the construction image and the text semantic vector corresponding to the prompt words; The multimodal alignment module is used to perform multimodal alignment between the semantic vector of the construction image and the semantic vector of the text to obtain the classification and recognition results of the power grid operation scenario.

[0019] This invention trains the scene recognition model based on historical industry datasets and incorporates LoRa fine-tuning, enabling the model to possess a deep understanding of the details of power grid construction sites, thereby improving the accuracy and reliability of image recognition. Through prompt word engineering technology, scene classification and language generation are deeply integrated, avoiding unstable outputs caused by large models operating freely, and improving the quality and controllability of model responses. By acquiring the semantic vector of the construction image and the text semantic vector corresponding to the prompt words, and performing multimodal alignment of the semantic vectors, the analysis results of the construction image are obtained, significantly improving the accuracy of intelligent question answering in the field of power grid safety supervision and strengthening the effective management of operational safety risks.

[0020] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of the power grid operation scenario classification and identification method of the present invention.

[0021] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the power grid operation scenario classification and identification method of the present invention. Attached Figure Description

[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating an embodiment of the power grid operation scenario classification and identification method provided by the present invention; Figure 2This is an overall flowchart of another embodiment of the power grid operation scenario classification and recognition method provided by the present invention; Figure 3 This is a flowchart illustrating another embodiment of the LoRa fine-tuning model provided by the present invention; Figure 4 This is a schematic diagram of the process for obtaining semantic vectors of construction images according to another embodiment of the present invention; Figure 5 This is a flowchart illustrating the multimodal alignment process according to another embodiment of the present invention. Figure 6 This is a schematic diagram of a module of another embodiment of the power grid operation scenario classification and recognition system provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0029] See Figure 1 To address the phase ranging problem in existing technologies, an embodiment of the present invention provides a method for classifying and recognizing power grid operation scenarios, comprising steps S1 to S3, the specific details of which are as follows: S1. The scene recognition model is used to perform image recognition on the input construction image and output the classification result; the scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa. S2. Select prompt words based on the classification results, obtain the text semantic vector corresponding to the prompt words, and perform image analysis on the construction image based on the prompt words to obtain the construction image semantic vector; S3. Perform multimodal alignment on the semantic vector of the construction image and the semantic vector of the text to obtain the classification and recognition results of the power grid operation scene.

[0030] As an example of an embodiment of the present invention, such as Figure 2 As shown, by inputting a construction image into a preset scene recognition model, the scene recognition model performs image recognition on the construction image and outputs a classification result. Based on the classification result, image analysis is performed based on scene selection prompts. Tokenizer encoding and image input encoding are used to obtain the semantic vector of the construction image and the text semantic vector corresponding to the prompts. The text semantic vector and the construction image semantic vector are then multimodally aligned to obtain a multimodal fusion representation result. Industry knowledge is injected into the model through LoRa fine-tuning. Based on the multimodal representation result and the text semantic vector, language text is gradually generated to obtain the analysis result of the construction image and output the industry analysis conclusion.

[0031] In one embodiment, see Figure 3 The scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa, including steps S201 to S204, each step of which is as follows: S201. Construct an initial model for image recognition, and set the weights and prediction vectors of the initial model; S202. Input the historical industry dataset into the initial model, and process the historical industry dataset through softmax normalization to obtain the true label vector. Specifically, based on the text in the historical industry dataset, the text is converted into word vectors through an encoding layer, and then input into a multi-layer Transformer Encoder module to output the sentence vector. ,Will The input is fed into a fully connected layer, and finally normalized by softmax to obtain the true label vector. The formula is as follows: S203. Calculate the cross-entropy loss between the true label vector and the predicted vector, and calculate the image classification loss. Calculate the joint loss based on the cross-entropy loss and the image classification loss. The calculation is performed using the following formula: scene = Where N is the number of samples in the dataset; y^i is the prediction vector of the initial model; The i-th sample belongs to the true label of the c-th category (0 / 1); To predict the logits probability of the i-th sample belonging to the c-th class; λ1 and λ2 are user-defined parameters, usually set to λ1=0.7 and λ2=0.3.

[0032] S204. Calculate the loss gradient based on the joint loss, cross-entropy loss, and image classification loss, and perform backpropagation on the initial model. Fine-tune the input data format using preset dictionary instructions, and fine-tune and update the weight parameters of the initial model using LoRa, as shown in the following formula: Where W is the original weight matrix, A is the reduced-dimensional matrix, B is the increased-dimensional matrix, r is the rank, and d is the dimension of the original weight matrix; The scene recognition model is output after optimizing the Softmax loss to the optimal solution.

[0033] The preset dictionary instructions consist of five parts: Instruction: A fixed text reads, "You are a professional construction site analyst. Please analyze and describe the image according to the following aspects: 1. Scene Type: Specify whether it is an indoor or outdoor scene; 2. Work Environment Classification: Clearly state which type of construction it is (rural road construction, urban road construction, etc.); 3. Personnel Information: Number of workers, current worker status, worker attire description; 4. Large Operating Equipment: Excavator, boom truck, etc.; Please only focus on the above points in your description and do not include other irrelevant information." Data: Image category ID. Image: Path to the on-site construction drawing. Bbox: Pixel center coordinates, width, and height of personnel and tools in the image. Conversations: Description of personnel status, operational compliance; placement status of items and tools, corresponding to the Bbox. Before LoRa fine-tuning, the initial model weights are fixed. The dimensionality reduction matrix is ​​initialized using a Gaussian distribution, and the dimensionality increase matrix is ​​initialized to all zeros. The parameter updates of the initial model are simulated through low-rank decomposition of the dimensionality increase and reduction matrices. r is the rank, typically 4-64, and d is the last dimension of the W matrix shape.

[0034] This embodiment constructs an initial model for image recognition and sets its weights and prediction vectors. By introducing softmax normalization to process historical industry datasets and obtain real label vectors, the model's semantic understanding accuracy in complex scenes is enhanced. By calculating the joint loss of cross-entropy loss and image classification loss, and performing gradient backpropagation based on this joint loss, the overall performance and generalization ability of the model in scene recognition are significantly improved. The input data format is fine-tuned through preset dictionary instructions, achieving efficient data processing and model understanding guidance. Through the LoRa fine-tuning mechanism, the weight parameters of the initial model are updated in a low-rank adaptive manner. Trainable dimensionality reduction and dimensionality increase matrices are introduced on the basis of the original weight matrix to achieve efficient parameter updates, which can quickly adapt to specific recognition task scenarios such as outdoor road construction, indoor maintenance operations, and substation construction.

[0035] In one embodiment, a prompt word is selected based on the classification result, and a text semantic vector corresponding to the prompt word is obtained, including steps S301 to S302, each of which is as follows: S301. Select prompt words according to the classification results, obtain word segments of the prompt words by segmenting the prompt words, and obtain the corresponding discrete token ID sequence by mapping the word segments; Taking the classification result of "outdoor road construction scene" as an example, the scene recognition model selects the following prompt words: "Please focus on analyzing the following: 1. Road type (urban road / rural road / highway / factory road); 2. Whether the road is closed or whether there is vehicle interference; 3. Whether power workers are carrying out trenching, cable laying, road restoration, hoisting, etc.; 4. Whether traffic safety barriers, cones, and warning lights are used; 5. Tool and equipment status (cables, manhole covers, cranes, excavators, etc.); 6. Potential risks." The prompt word text is segmented by the BPE word segmenter, such as "Please analyze the environment and people's status in the picture..." being segmented into "Please, segment, analyze, ..., status", and then the segmented words are mapped to a discrete token ID sequence, such as "101, 404, 523, 1134, ...".

[0036] S302. The scene recognition model is fused with the preset initial language model using the following formula: in, The initial language model weights; Assign weights to the scene recognition model; The weights of the fused language model; The token ID sequence is input into the fused language model to obtain the text semantic vector corresponding to the prompt word.

[0037] The text semantic vector is obtained using the following formula: For text semantic vectors; This embodiment selects prompt words that match the scene based on the classification results, segments them into word segments, maps them to discrete token ID sequences, and then generates corresponding text semantic vectors based on the token ID sequences. This achieves structured and semantic encoding of prompt words. By introducing a prompt word segmentation and token mapping mechanism, the model's ability to understand technical terms and complex semantics is enhanced, avoiding semantic bias and thus improving the accuracy and consistency of subsequent multimodal alignment. By weighting and fusing the scene recognition model with the initial language model, the fused language model has a stronger scene perception and semantic understanding ability, which makes the generated text semantic vectors more accurately reflect the scene semantic features corresponding to the prompt words.

[0038] In one embodiment, see Figure 4 The construction image is analyzed based on the prompt words to obtain the semantic vector of the construction image, including step S401, as follows: S401. By segmenting and linearly mapping the construction image, a matrix representation corresponding to the construction image is obtained. Based on the matrix representation, a semantic vector is generated through MLP to obtain the semantic vector of the construction image.

[0039] Taking a histogram RGB image with a shape of H×W×3 as an example, the process first involves patching and linear mapping, meaning the image is divided into N patches. The image (height 224, width 224, 3 channels) is patched using a 16x16 convolution kernel with a stride of 16, resulting in a 3D matrix of size (14, 14, 768). Then, the H and W dimensions are linearly flattened to obtain a 2D matrix of size (196, 7680). Finally, semantic vectors are generated using an MLP, as shown in the following formula: For construction image semantic vectors; This embodiment segments and linearly maps construction images, converting them into structured matrix representations. This enables efficient and accurate encoding of image information, allowing for the extraction and fusion of deep visual semantic features from the images. This enhances the model's semantic understanding of complex construction environments and details, thereby improving the accuracy and consistency of subsequent multimodal alignment.

[0040] In one embodiment, see Figure 5 The semantic vectors of the construction images and the semantic vectors of the text are multimodally aligned to obtain the classification and recognition results of the power grid operation scene, including step S501, as follows: S501. The construction image semantic vector and the text semantic vector are multimodally aligned using the following formula to obtain the fused multimodal feature representation: in, For multimodal feature representation; For text semantic vectors; For construction image semantic vectors; , and for After passing through the linear layer, the results represent the query intent, image index, and image content, respectively. for transpose; for The dimension; , and These are learnable linear coefficients; Based on the multimodal feature representation, the classification and recognition results of power grid operation scenarios are obtained.

[0041] in, After passing through the decoder and autoregressive layer of the scene recognition model, text is gradually generated, and finally the classification and recognition results of the power grid operation scene are obtained.

[0042] This embodiment introduces a multimodal attention alignment mechanism to deeply fuse the semantic vectors of construction images and text semantic vectors. It uses a learnable weight matrix to generate query, key, and value vectors, and obtains the fused multimodal feature representation through scaling dot product attention calculation. This achieves accurate alignment and enhanced interaction between image information and text semantics, thereby improving the ability to analyze images of power grid operation scenarios.

[0043] like Figure 6 As shown in the power grid operation scenario classification and identification, based on the above method implementation examples, corresponding system implementation examples are provided; This invention provides a power grid operation scenario classification and recognition system, including: an image recognition module 601, an image analysis module 602, and a multimodal alignment module 603; The image recognition module 601 is used to perform image recognition on the input construction image and output the classification result; The image analysis module 602 is used to determine prompt words based on the classification results, and to perform image analysis on the construction image based on the prompt words to obtain the semantic vector of the construction image and the text semantic vector corresponding to the prompt words; The multimodal alignment module 603 is used to perform multimodal alignment between the construction image semantic vector and the text semantic vector to obtain the classification and recognition results of the power grid operation scenario.

[0044] The image recognition module 601 of this embodiment is trained based on historical industry datasets and fine-tuned using LoRa, enabling the scene recognition model to have a deep understanding of the details of power grid construction sites, thereby improving the accuracy and reliability of image recognition. The image analysis module 602 uses prompt word engineering technology to deeply integrate scene classification and language generation, avoiding unstable output caused by large models operating freely, and improving the quality and controllability of model responses. The multimodal alignment module 603 obtains the semantic vector of the construction image and the semantic vector of the text corresponding to the prompt word, and performs multimodal alignment of the semantic vector of the construction image and the semantic vector of the text to obtain the analysis results of the construction image, which significantly improves the accuracy of intelligent question answering in the field of power grid safety supervision and strengthens the effective control of operational safety risks.

[0045] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement the power grid operation scenario classification and recognition method provided by any of the above method embodiments of the present invention.

[0046] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0047] For ease of description and brevity, the system embodiments of the present invention include all the implementation methods described in the above embodiments of the power grid operation scenario classification and identification method, and will not be repeated here.

[0048] Based on the above embodiments of the power grid operation scenario classification and identification method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the power grid operation scenario classification and identification method of any embodiment of the present invention.

[0049] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0050] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0051] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0052] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the power grid operation scenario classification and identification method described in any of the above-described method embodiments of the present invention.

[0053] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0054] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for classifying and recognizing power grid operation scenarios, characterized in that, include: The scene recognition model is used to perform image recognition on the input construction images and output classification results. The scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa. Based on the classification results, select prompt words, obtain the text semantic vector corresponding to the prompt words, and perform image analysis on the construction image based on the prompt words to obtain the construction image semantic vector; The semantic vectors of the construction images and the semantic vectors of the text are aligned in a multimodal manner to obtain the classification and recognition results of the power grid operation scene.

2. The power grid operation scenario classification and identification method as described in claim 1, characterized in that, The scene recognition model is trained based on historical industry datasets and fine-tuned using LoRa, specifically: Construct an initial model for image recognition, and set the weights and prediction vectors of the initial model; The historical industry dataset is input into the initial model, and the data is processed by softmax normalization to obtain the true label vector. Calculate the cross-entropy loss between the true label vector and the predicted vector, and calculate the image classification loss. Then, calculate the joint loss based on the cross-entropy loss and the image classification loss. The loss gradient is calculated based on the joint loss, cross-entropy loss, and image classification loss, and backpropagation is performed on the initial model. The weight parameters of the initial model are then fine-tuned using LoRa until the Softmax loss is optimized to the optimal solution, and the scene recognition model is output.

3. The power grid operation scenario classification and identification method as described in claim 2, characterized in that, The weight parameters of the initial model are fine-tuned and updated using LoRa, as follows: Where W is the original weight matrix; A is the reduced-dimensional matrix; B is the increased-dimensional matrix; r is the rank; d and k are the dimensions of the original weight matrix; This is the updated weight matrix; This is the incremental update of the weights; R is the real number field.

4. The power grid operation scenario classification and identification method as described in claim 1, characterized in that, The step of selecting prompt words based on the classification results and obtaining the text semantic vector corresponding to the prompt words specifically involves: Based on the classification results, prompt words are selected. By segmenting the prompt words, word segments are obtained. The word segments are mapped to obtain the corresponding discrete token ID sequence. A semantic vector is generated based on the token ID sequence to obtain the text semantic vector corresponding to the prompt word.

5. The power grid operation scenario classification and identification method as described in claim 1, characterized in that, The step of performing image analysis on the construction image based on the prompt words to obtain the semantic vector of the construction image specifically involves: The construction image is segmented and linearly mapped according to the prompt words to obtain the matrix representation of the construction image. Based on the matrix representation, a semantic vector is generated through MLP to obtain the semantic vector of the construction image.

6. The power grid operation scenario classification and identification method as described in claim 1, characterized in that, The step of performing multimodal alignment between the semantic vector of the construction image and the semantic vector of the text to obtain the classification and recognition results of the power grid operation scene is as follows: The construction image semantic vector and the text semantic vector are multimodally aligned using the following formula to obtain the fused multimodal feature representation: in, For multimodal feature representation; For text semantic vectors; The semantic vector of the construction image; , and for After passing through the linear layer, the results represent the query intent, image index, and image content, respectively. for transpose; for The dimension; , and These are learnable linear coefficients; Based on the multimodal feature representation, the classification and recognition results of the power grid operation scenario are obtained.

7. The power grid operation scenario classification and recognition method as described in claim 4, characterized in that, The step of generating a semantic vector based on the tokenID sequence to obtain the text semantic vector corresponding to the prompt word includes: The scene recognition model is fused with the preset initial language model using the following formula; in, The initial language model weights; Assign weights to the scene recognition model; The weights of the fused language model; The token ID sequence is input into the fused language model to obtain the text semantic vector corresponding to the prompt word.

8. A power grid operation scenario classification and recognition system, characterized in that, include: Image recognition module, image analysis module, and multimodal alignment module; The image recognition module is used to perform image recognition on the input construction image and output the classification result; The image analysis module is used to determine prompt words based on the classification results, and to perform image analysis on the construction image based on the prompt words to obtain the semantic vector of the construction image and the text semantic vector corresponding to the prompt words; The multimodal alignment module is used to perform multimodal alignment between the semantic vector of the construction image and the semantic vector of the text to obtain the classification and recognition results of the power grid operation scenario.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the power grid operation scenario classification and identification method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the power grid operation scenario classification and identification method as described in any one of claims 1-7.

Images