Power image text multi-modal understanding method, system and equipment and medium

Abstract

The invention discloses an electric power image text multi-modal understanding method, system and device and a medium. The method comprises the steps that electric power image text information to be understood is acquired; inputting the to-be-understood electric power image text information into a trained electric power image-text large model to obtain an understanding result of the to-be-understood electric power image text information, the electric power image-text large model being sequentially trained through an electric power image-text sample, a positive and negative sample pair and a mixed sample to obtain an understanding result of the to-be-understood electric power image text information; according to the method, the system, the equipment and the medium, the accuracy of multi-modal understanding of the electric power image text can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power image and text semantic understanding, and relates to an electric power image and text multimodal understanding method, system, device and medium. Background Art

[0002] With the construction of new power systems, the scale of equipment and the frequency of operations have increased significantly. Intelligent analysis is an important means to timely discover, analyze and eliminate defects in power equipment, violations of personnel regulations and environmental hazards. At present, the target detection and behavior recognition algorithms widely used in power systems are only based on single visual data, and the model has high false positives and false negatives when deep semantic analysis is required. A large amount of multimodal data such as images and texts is generated every day in power scenarios. How to use these multimodal data to realize intelligent analysis of typical application scenarios such as power transmission, substation, distribution, and safety supervision is crucial.

[0003] The current large-scale model of electric power graphics and text is committed to aligning the spatial information of graphics and text. Its main tasks include mutual generation of graphics and text, cross-modal retrieval, visual positioning, reasoning question and answer, etc. According to the interaction mode of the graphics and text modalities in the pre-trained model, it can be mainly divided into a single-tower structure and a dual-tower structure. The single-tower structure maps the data of two different modalities into the same network for processing. The cross-modal alignment is difficult and is not suitable for retrieval tasks. The dual-tower structure uses two independent networks to map the data of each modality separately, and finally fuses them at a certain layer or output stage. However, when training the large-scale model of electric power graphics and text, the training method is relatively single, so the accuracy of multimodal understanding of electric power image text needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, system, device and medium for multimodal understanding of power image text, which can improve the accuracy of multimodal understanding of power image text.

[0005] To achieve the above object, the present invention discloses a method for multimodal understanding of electric power image text, comprising:

[0006] Obtaining the power image text information to be understood;

[0007] The electric power image text information to be understood is input into the trained electric power image and text large model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text large model is trained with electric power image and text samples, positive and negative sample pairs, and mixed samples in sequence.

[0008] The further improvement of the multimodal understanding method of power image text of the present invention is:

[0009] Furthermore, before inputting the power graphic information to be identified into the trained power graphic model, the process further includes:

[0010] Obtain an electric power image and text dataset, wherein the electric power image and text dataset includes but is not limited to: unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs;

[0011] Constructing a large electric power graph and text model, wherein the large electric power graph and text model comprises a connected electric power visual feature extraction module and a large electric power graph and text model language module;

[0012] Using the electric power graph and text data set to train the electric power graph and text large model, to obtain the electric power graph and text large model after the first stage of training;

[0013] Construct positive and negative sample data sets;

[0014] Using the positive and negative sample data sets to train the electric power graph-text large model trained in the first stage, to obtain the electric power graph-text large model trained in the second stage;

[0015] Construct a mixed sample training set;

[0016] The mixed sample training set is used to train the electric power graph-text large model trained in the second stage to obtain the trained electric power graph-text large model.

[0017] Furthermore, the process of using the power graph-text data set to train the power graph-text large model to obtain the power graph-text large model after the first stage of training is:

[0018] The unlabeled power image samples in the power image and text dataset are used to perform unsupervised training on the power visual feature extraction module to obtain the power visual feature extraction module after unsupervised training;

[0019] Using the annotated power image samples in the power image and text data set to train the power visual feature extraction module after the unsupervised training, to obtain the power visual feature extraction module after the first stage of training;

[0020] Using unlabeled electric power text samples in the electric power image and text data set to perform unsupervised training on the language model to obtain a language module after unsupervised training;

[0021] The language module after the unsupervised training is trained using the manually annotated power text negative sample pairs and the manually annotated power text positive sample pairs in the power graphic and text data set to obtain the language module after the first stage of training.

[0022] Furthermore, the process of constructing the positive and negative sample data sets is:

[0023] Extracting a number of power image samples X with a label I from the power image and text dataset;

[0024] Generate antonym description language O and synonym description language S for each power image sample X;

[0025] Construct negative sample pairs (X, O), construct positive sample pairs (X, I), (X, O);

[0026] The positive and negative sample data sets are constructed according to the negative sample pairs (X, O) and the positive sample pairs (X, I) and (X, O).

[0027] Furthermore, the process of constructing the mixed sample training set is:

[0028] Randomly sampling the electric power graphic data set to obtain a random sampling result;

[0029] Randomly obtain image and text samples from general fields;

[0030] The mixed sample training set is constructed using the random sampling results and randomly acquired graphic and text samples of a general field.

[0031] The present invention discloses a multimodal understanding system of electric power image text, comprising:

[0032] A first acquisition module is used to acquire the power image text information to be understood;

[0033] The understanding module is used to input the electric power image text information to be understood into the trained electric power image and text model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text model is trained in turn with electric power image and text samples, positive and negative sample pairs, and mixed samples.

[0034] The further improvement of the electric power image text multimodal understanding system of the present invention is:

[0035] Furthermore, it also includes:

[0036] A second acquisition module is used to acquire an electric power image and text data set, wherein the electric power image and text data set includes but is not limited to: unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs;

[0037] A first construction module is used to construct a large electric power graph and text model, wherein the large electric power graph and text model includes a connected electric power visual feature extraction module and a large electric power graph and text model language module;

[0038] A first training module is used to train the power graph-text big model using the power graph-text data set to obtain the power graph-text big model after the first stage of training;

[0039] The second building module is used to build a positive and negative sample data set;

[0040] A second training module is used to train the electric power graph-text large model trained in the first stage using the positive and negative sample data sets to obtain the electric power graph-text large model trained in the second stage;

[0041] The third building module is used to build a mixed sample training set;

[0042] The third training module is used to train the electric power graph and text large model trained in the second stage by using the mixed sample training set to obtain the trained electric power graph and text large model.

[0043] Furthermore, the first training module includes:

[0044] A fourth training module is used to perform unsupervised training on the power visual feature extraction module using unlabeled power image samples in the power image and text data set to obtain the power visual feature extraction module after unsupervised training;

[0045] A fifth training module is used to train the power visual feature extraction module after the unsupervised training using the annotated power image samples in the power image and text data set to obtain the power visual feature extraction module after the first stage of training;

[0046] A sixth training module, used to perform unsupervised training on the language model using unlabeled power text samples in the power graphic and text data set to obtain a language module after unsupervised training;

[0047] The seventh training module is used to use the manually labeled power text negative sample pairs and the manually labeled power text positive sample pairs in the power graphic and text data set to train the language module after unsupervised training to obtain the language module after the first stage of training.

[0048] Furthermore, the second building block includes:

[0049] A third acquisition module is used to extract a number of power image samples X with a label I from the power image and text data set;

[0050] A generating module, used for generating an antonym description language O and a synonym description language S for each power image sample X;

[0051] The fourth construction module is used to construct negative sample pairs (X, O) and construct positive sample pairs (X, I) and (X, O);

[0052] The fifth construction module is used to construct the positive and negative sample data sets according to the negative sample pairs (X, O) and the positive sample pairs (X, I) and (X, O).

[0053] Furthermore, the third building block includes:

[0054] A sampling module, used for randomly sampling the electric power graphic data set to obtain a random sampling result;

[0055] The fourth acquisition module is used to randomly acquire image and text samples in a general field;

[0056] The sixth construction module is used to construct the mixed sample training set using the random sampling results and randomly acquired general domain graphic and text samples.

[0057] The present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for multimodal understanding of electric power image text are implemented.

[0058] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for multimodal understanding of electric power image text are implemented.

[0059] The present invention has the following beneficial effects:

[0060] During specific operation, the method, system, device and medium for multimodal understanding of electric power image text described in the present invention input the electric power image text information to be understood into the trained electric power image and text large model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text large model is trained in sequence with electric power image and text samples, positive and negative sample pairs and mixed samples, wherein comparative training using positive and negative sample pairs can align the features of electric power element image texts, and in addition, training using mixed samples can achieve fine-tuning of the electric power image and text large model while improving the understanding ability of the electric power image and text large model in general and electric power fields, thereby improving the accuracy of multimodal understanding of electric power image and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2is a system structure diagram of the present invention; DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0066] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0067] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0068] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0069] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0070] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0071] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0072] refer to Figure 1 The method for multimodal understanding of electric power image text of the present invention comprises the following steps:

[0073] 1) Construct an electric power image-text dataset, wherein the electric power image-text dataset includes unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, electric power image-text sample pairs, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs.

[0074] It should be noted that the power business scenario is complex, and massive images and video data are generated every day that need to be annotated with corresponding texts. In order to reduce the amount of manual annotation, for image data, the image deduplication method is used to remove images with high similarity before annotation. For video data, a frame is extracted every t time and compared with the historical frame sequence, and the frames with low similarity to all historical frames are added to the historical frame sequence. The historical frame sequence collected every T time is annotated, where t and T can be adjusted according to the on-site conditions.

[0075] Cooperate with front-line business personnel to formulate power graphic annotation specifications. Due to the complex business logic in the power field, the advanced context semantics are more than general fields. For example, the category of "losing protection at high altitude" requires judging whether the operator is in a high-altitude working scene, whether the belt is fastened, whether the back protection rope is hung and higher than the operator, and multiple factors are combined to judge whether the operator is not wearing or not wearing a safety belt. Taking this category of annotation specifications as an example, after determining the key business points with front-line business personnel, the business is disassembled based on the thinking chain. The annotation is divided into the following steps:

[0076] 11) Identify the object;

[0077] Visual coordinates: mark the positions of safety belts and workers in the image.

[0078] Attribute description: Personal characteristics, such as clothing, body shape, etc., seat belt characteristics, such as color, material, etc.

[0079] 12) Determine the status;

[0080] Safety belt status: Determine whether the safety belt is worn correctly, for example, whether the safety belt belt is fixed on the power pole tower and whether the back protection rope is hung at a high place.

[0081] Personnel status: Determine whether the operator is working at a height. According to the safety operating procedures, the operator must be 1.5 meters or more above the ground.

[0082] 13) Reasoning relationship;

[0083] Spatial relationship: Analyze the positional relationship between the safety belt and the operator to ensure that the safety belt is correctly fixed on the operator.

[0084] Multi-object interaction: Determine whether the operator's movements and positions affect the normal use of the safety belt.

[0085] Through the above steps, we ensure that the annotation process has high accuracy and consistency to improve the performance of large multimodal models in complex tasks.

[0086] 2) Construct a large model of power graphics and text;

[0087] The electric power graphic model includes an electric power visual feature extraction module and a language module.

[0088] Specifically, the implementation process of 21) the power visual feature extraction module is as follows:

[0089] Image features include global and local features. The global feature directly cuts the input image into P*P pieces, and obtains the global feature f through ViT extraction after general data set training. Due to the large size difference and different resolution of power targets, the global feature is difficult to help the model accurately learn the visual features of the core elements of power. Therefore, multiple target box annotation bboxes in pure image annotation are used to crop the image in each target box annotation bbox and extract the feature bbox. i When annotating, the global feature is combined with each local feature (bbox i ,f i ) splicing as visual features.

[0090] 22) Language module;

[0091] The language module may be a QwenLM model, and the input may be a pure image, pure text, or an image-text pair.

[0092] 3) Using the electric power graph and text data set to train the electric power graph and text large model;

[0093] The process of step 3) is:

[0094] 31) Train the power visual feature extraction module, specifically:

[0095] The power visual feature extraction module in the present invention is trained using power image samples in the power image and text dataset. Specifically, first, the power visual feature extraction module is unsupervisedly trained using unlabeled power image samples by means of random mask image blocks to obtain the power visual feature extraction module after unsupervised training, wherein the random mask image blocks are trained using the image feature encoder V e The masked image sequence blocks are encoded into 2048-dimensional image features and passed through the decoder V d Decode and restore the image; secondly, use the labeled power image samples to train the power visual feature extraction module after unsupervised training, so as to improve the visual feature encoder V in downstream tasks such as target detection and image classification. e The ability to extract image features of relevant elements such as equipment and personnel in the power field is used to obtain the power visual feature extraction module after the first stage of training.

[0096] 32) training the language module;

[0097] The process of step 32) is:

[0098] It should be noted that the parameters of the language module are initialized by the basic language model trained on a large-scale general text dataset, and incremental training is performed on the power text dataset to enhance the ability to understand power knowledge. Specifically, due to the large number of power documents and the high annotation cost, an unsupervised training method is first used to randomly mask the input power text, requiring the model to predict the masked words using the context unmasked text to obtain the language module after unsupervised training; secondly, a certain number of power text pairs are annotated, I is used as the original power text description, (I, O) is the manually annotated power text negative sample pair, (I, S) is the manually annotated power text positive sample pair, and the instruction fine-tuning method is used to fine-tune the part of the large language model, with I as the model input. The training task includes instructions such as "generate antonym description for this sentence" and "generate synonymous description for this sentence", which minimizes the distance between the text generated by the model and the true value, improves the model's ability to understand power text, and understands the synonymous and antonym concepts in the power field, and obtains the language module after the first stage of training.

[0099] 33) Power vision text feature alignment training;

[0100] In the alignment stage, in order to enable the large model of electric power graphics and text to better capture the abnormal features of the core target of electric power and distinguish abnormal features from normal features at the semantic level, the present invention adopts a contrastive learning method for training. At the same time, since there are fewer abnormal samples such as personnel violations and equipment defects in the electric power graphics and text data set, a small amount of data sets is difficult to meet the training requirements of the large model of electric power graphics and text in the alignment stage. In order to enable the large model of electric power graphics and text to enhance the image and text feature alignment capabilities of equipment defects, illegal behaviors, and environmental anomalies, for abnormal categories with fewer samples, image-text negative sample pairs are constructed. Secondly, the positive and negative samples are combined and sent to the large model of electric power graphics and text. The training goal is that the large language model of electric power can find the text description that is closest to the input image from the description.

[0101] The process of step 33) is:

[0102] 331) Construct positive and negative sample data sets;

[0103] Based on the trained language module, image X, and annotated text I, a multi-dimensional description language such as antonym O and synonym S is generated for the image. For the antonym description, a negative sample pair (X, O) is constructed. For the annotated text I and the synonym description S, a positive sample pair (X, I) and (X, O) is constructed. A positive and negative sample dataset is constructed based on the negative sample pair (X, O) and the positive sample pair (X, I) and (X, O).

[0104] 332) using the positive and negative sample data sets to train the electric power graph-text large model trained in the first stage, to obtain the electric power graph-text large model trained in the second stage;

[0105] Among them, in the model training stage, the visual part uses the visual module obtained in the power visual feature extraction module training stage to extract global or local features for the input image X and splice it into<vision_start> f<vision_end> or<vision_start> f i <vision_end> ; At the same time, in order to make the model distinguish the starting position of the text description, for all text description languages ​​I, O, S, tokens are added before and after:<text_start> ,<text_end> , after calculating the loss between the model output and the labeled true value, the model is trained by minimizing the loss.

[0106] 33) Power and general mixed training.

[0107] The process of step 33) is:

[0108] 331) Construct a mixed sample training set;

[0109] After aligning the two different spatial features of power images and texts, the power image and text large model is further trained on downstream tasks such as image description, target positioning, and image and text question and answer. In order to ensure that the general ability is not lost, the power image and text dataset is randomly sampled to obtain the results of random sampling, and the image and text samples of the general field are randomly obtained. The mixed sample training set is constructed according to the preset mixing ratio using the results of random sampling and the randomly obtained image and text samples of the general field, wherein the preset mixing ratio is determined through a specific experimental process.

[0110] The mixed sample training set is used to train the large electric power graph and text model trained in the second stage to obtain the trained large electric power graph and text model.

[0111] 4) Obtaining the power image text information to be understood;

[0112] 5) Inputting the electric power image text information to be understood into the trained electric power image-text large model to obtain the understanding result of the electric power image text information to be understood.

[0113] Embodiment 2

[0114] refer to Figure 2 The electric power image text multimodal understanding system of the present invention comprises:

[0115] A first acquisition module is used to acquire the power image text information to be understood;

[0116] The understanding module is used to input the electric power image text information to be understood into the trained electric power image and text model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text model is trained in turn with electric power image and text samples, positive and negative sample pairs, and mixed samples.

[0117] As an implementation mode of the present invention, this embodiment also includes:

[0118] A second acquisition module is used to acquire an electric power image and text data set, wherein the electric power image and text data set includes but is not limited to: unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs;

[0119] A first construction module is used to construct a large electric power graph and text model, wherein the large electric power graph and text model includes a connected electric power visual feature extraction module and a large electric power graph and text model language module;

[0120] A first training module is used to train the power graph-text big model using the power graph-text data set to obtain the power graph-text big model after the first stage of training;

[0121] The second building module is used to build a positive and negative sample data set;

[0122] A second training module is used to train the electric power graph-text large model trained in the first stage using the positive and negative sample data sets to obtain the electric power graph-text large model trained in the second stage;

[0123] The third building module is used to build a mixed sample training set;

[0124] The third training module is used to train the electric power graph and text large model trained in the second stage by using the mixed sample training set to obtain the trained electric power graph and text large model.

[0125] As an embodiment of the present invention, the first training module includes:

[0126] A fourth training module is used to perform unsupervised training on the power visual feature extraction module using unlabeled power image samples in the power image and text data set to obtain the power visual feature extraction module after unsupervised training;

[0127] A fifth training module is used to train the power visual feature extraction module after the unsupervised training using the annotated power image samples in the power image and text data set to obtain the power visual feature extraction module after the first stage of training;

[0128] A sixth training module, used to perform unsupervised training on the language model using unlabeled power text samples in the power graphic and text data set to obtain a language module after unsupervised training;

[0129] The seventh training module is used to use the manually labeled power text negative sample pairs and the manually labeled power text positive sample pairs in the power graphic and text data set to train the language module after unsupervised training to obtain the language module after the first stage of training.

[0130] As an embodiment of the present invention, the second building block includes:

[0131] A third acquisition module is used to extract a number of power image samples X with a label I from the power image and text data set;

[0132] A generating module, used for generating an antonym description language O and a synonym description language S for each power image sample X;

[0133] The fourth construction module is used to construct negative sample pairs (X, O) and construct positive sample pairs (X, I) and (X, O);

[0134] The fifth construction module is used to construct the positive and negative sample data sets according to the negative sample pairs (X, O) and the positive sample pairs (X, I) and (X, O).

[0135] As an embodiment of the present invention, the third building block includes:

[0136] A sampling module, used for randomly sampling the electric power graphic data set to obtain a random sampling result;

[0137] The fourth acquisition module is used to randomly acquire image and text samples in a general field;

[0138] The sixth construction module is used to construct the mixed sample training set using the random sampling results and randomly acquired general domain graphic and text samples.

[0139] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0140] Embodiment 3

[0141] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of implementing the method for multimodal understanding of electric power image texts include, for example, obtaining electric power image text information to be understood; inputting the electric power image text information to be understood into a trained electric power image text model to obtain an understanding result of the electric power image text information to be understood, wherein the electric power image text model is trained sequentially with electric power image text samples, positive and negative sample pairs, and mixed samples. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, the network interface, and the memory are interconnected through an internal bus, which may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0142] Embodiment 4

[0143] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of implementing the method for multimodal understanding of electric power image text are implemented, for example, including: obtaining electric power image text information to be understood; inputting the electric power image text information to be understood into a trained electric power image and text large model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text large model is trained with electric power image and text samples, positive and negative sample pairs, and mixed samples in sequence. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0144] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0149] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0150] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for multimodal understanding of electric power image text, characterized in that: include: Obtaining the power image text information to be understood; The electric power image text information to be understood is input into the trained electric power image and text large model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text large model is trained with electric power image and text samples, positive and negative sample pairs, and mixed samples in sequence.

2. The method for multimodal understanding of electric power image text according to claim 1 is characterized in that: Before inputting the power graphic information to be identified into the trained power graphic model, the method further includes: Obtain an electric power image and text dataset, wherein the electric power image and text dataset includes but is not limited to: unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs; Constructing a large electric power graph and text model, wherein the large electric power graph and text model comprises a connected electric power visual feature extraction module and a large electric power graph and text model language module; Using the electric power graph and text data set to train the electric power graph and text large model, to obtain the electric power graph and text large model after the first stage of training; Construct positive and negative sample data sets; Using the positive and negative sample data sets to train the electric power graph-text large model trained in the first stage, to obtain the electric power graph-text large model trained in the second stage; Construct a mixed sample training set; The mixed sample training set is used to train the large electric power graph and text model trained in the second stage to obtain the trained large electric power graph and text model.

3. The method for multimodal understanding of electric power image text according to claim 2 is characterized in that: The process of using the power graph-text data set to train the power graph-text large model to obtain the power graph-text large model after the first stage of training is: The unlabeled power image samples in the power image and text dataset are used to perform unsupervised training on the power visual feature extraction module to obtain the power visual feature extraction module after unsupervised training; Using the annotated power image samples in the power image and text data set to train the power visual feature extraction module after the unsupervised training, to obtain the power visual feature extraction module after the first stage of training; Using unlabeled electric power text samples in the electric power image and text data set to perform unsupervised training on the language model to obtain a language module after unsupervised training; The language module after the unsupervised training is trained using the manually annotated power text negative sample pairs and the manually annotated power text positive sample pairs in the power graphic and text data set to obtain the language module after the first stage of training.

4. The method for multimodal understanding of electric power image text according to claim 2, characterized in that: The process of constructing the positive and negative sample data sets is as follows: Extracting a number of power image samples X with a label I from the power image and text dataset; Generate antonym description language O and synonym description language S for each power image sample X; Construct negative sample pairs (X, O), construct positive sample pairs (X, I), (X, O); The positive and negative sample data sets are constructed according to the negative sample pairs (X, O) and the positive sample pairs (X, I) and (X, O).

5. The method for multimodal understanding of electric power image text according to claim 2, characterized in that: The process of constructing the mixed sample training set is: Randomly sampling the electric power graphic data set to obtain a random sampling result; Randomly obtain image and text samples from general fields; The mixed sample training set is constructed using the random sampling results and randomly acquired graphic and text samples of a general field.

6. A multimodal understanding system for power image text, characterized in that: include: A first acquisition module is used to acquire the power image text information to be understood; The understanding module is used to input the electric power image text information to be understood into the trained electric power image and text model to obtain the understanding result of the electric power image and text information to be understood, wherein the electric power image and text model is trained in turn with electric power image and text samples, positive and negative sample pairs, and mixed samples.

7. The electric power image text multimodal understanding system according to claim 6, characterized in that: Also includes: A second acquisition module is used to acquire an electric power image and text data set, wherein the electric power image and text data set includes but is not limited to: unlabeled electric power image samples, labeled electric power image samples, unlabeled electric power text samples, manually labeled electric power text negative sample pairs, and manually labeled electric power text positive sample pairs; A first construction module is used to construct a large electric power graph and text model, wherein the large electric power graph and text model includes a connected electric power visual feature extraction module and a large electric power graph and text model language module; A first training module is used to train the power graph-text big model using the power graph-text data set to obtain the power graph-text big model after the first stage of training; The second building module is used to build a positive and negative sample data set; A second training module is used to train the electric power graph-text large model trained in the first stage using the positive and negative sample data sets to obtain the electric power graph-text large model trained in the second stage; The third building module is used to build a mixed sample training set; The third training module is used to train the electric power graph and text large model trained in the second stage by using the mixed sample training set to obtain the trained electric power graph and text large model.

8. The electric power image text multimodal understanding system according to claim 7, characterized in that: The first training module includes: A fourth training module is used to perform unsupervised training on the power visual feature extraction module using unlabeled power image samples in the power image and text data set to obtain the power visual feature extraction module after unsupervised training; A fifth training module is used to train the power visual feature extraction module after the unsupervised training using the annotated power image samples in the power graphic data set to obtain the power visual feature extraction module after the first stage of training; A sixth training module, used to perform unsupervised training on the language model using unlabeled power text samples in the power graphic and text data set to obtain a language module after unsupervised training; The seventh training module is used to use the manually labeled power text negative sample pairs and the manually labeled power text positive sample pairs in the power graphic and text data set to train the language module after unsupervised training to obtain the language module after the first stage of training.

9. The electric power image text multimodal understanding system according to claim 7, characterized in that: The second building block comprises: A third acquisition module is used to extract a number of power image samples X with a label I from the power image and text data set; A generating module, used for generating an antonym description language O and a synonym description language S for each power image sample X; The fourth construction module is used to construct negative sample pairs (X, O) and construct positive sample pairs (X, I) and (X, O); The fifth construction module is used to construct the positive and negative sample data sets according to the negative sample pairs (X, O) and the positive sample pairs (X, I) and (X, O).

10. The electric power image text multimodal understanding system according to claim 7, characterized in that: The third building block comprises: A sampling module, used for randomly sampling the electric power graphic data set to obtain a random sampling result; The fourth acquisition module is used to randomly acquire image and text samples in a general field; The sixth construction module is used to construct the mixed sample training set using the random sampling results and randomly acquired general domain graphic and text samples.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for multimodal understanding of power image text as described in any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for multimodal understanding of power image text as described in any one of claims 1 to 5 are implemented.

Images