Electric power multi-modal sample labeling method and related device

Abstract

The invention discloses an electric power multi-modal sample labeling method and a related device, and aims to unify labels of labeling files corresponding to image samples and ensure labeling standardization of image samples from different sources; comprehensively marking all related objects and defects based on the obtained image sample labels; converting the annotation format of the annotated image sample; designing a target detection task question-answer pair template, and converting the annotated data into a question-answer pair form; aiming at an image understanding task, constructing a multi-round question and answer pair data set based on the obtained question and answer pair form, and training a large language model by utilizing the constructed multi-round question and answer pair data set for identifying and analyzing safety behaviors in the image; expression of question and answer pairs is enriched through an open source large language model, language diversity is increased while semantic equivalence is kept, and generalization ability and adaptability of the large language model are improved.

Description

Background Art

[0002] The image-text multimodal large model technology is a deep learning model that combines image and text information. It can accurately identify and understand the content in the image through deep learning technology. Whether it is a landscape photo, a portrait photo or a product photo, it can quickly extract key information and give corresponding feedback. The development of this technology makes the application of AI systems in complex scenarios more powerful and flexible, and can process and understand various types of data, such as text, images, audio and video. The technical evolution of the multimodal large model follows the overall scheme of pre-training-fine-tuning. According to the interaction mode of the image and text modality in the pre-training model, it is mainly divided into two types: double tower structure and single tower structure. The representative architecture of the double tower structure is CLIP, which models image information through a visual encoder and a text encoder. The feature vectors of the image and text can be pre-calculated and stored, and the modal interaction is processed through the cosine similarity of the image and text feature vectors. The representative architecture of the single tower structure is ViLT, which uses the Transformer model to interact with image and text features, can fully integrate multimodal information, and is better at multimodal classification tasks.

[0003] With the development of big model technology, people gradually realize that multimodality will be the only way for the development of big models. Big multimodal models of pictures and texts have been initially applied in many fields such as finance, medical care, education, automobiles, film and television, and have produced obvious economic and social benefits. These models are based on powerful large language models (LLMs) and can process and understand multimodal information such as text, images, videos and audio. Big multimodal models usually consist of three modules: a pre-trained modal encoder, a pre-trained LLM and a modal interface connecting them. The modal encoder compresses the original information (such as images or audio) into a more compact representation, while the pre-trained LLM is responsible for understanding and reasoning the processed signal. The research and application of big multimodal models are constantly expanding from text to more modalities, and have shown their strong potential in many fields. These models can not only improve the understanding and reasoning ability of AI, but also play an important role in the integration of multimodal data, the unified multimodal modeling of algorithms and cross-modal semantic alignment, and the application of multimodal AI in office, e-commerce, entertainment, education and other fields. With the continuous advancement of technology, the high demand for computing power of multimodal large models has also promoted the development of chip manufacturing and cloud service industries. In general, the technology of multimodal large models of images and texts is an important development direction in the field of AI. By combining image and text information, it greatly expands the application scope and capabilities of AI and provides strong technical support for the realization of more intelligent and flexible AI systems.

[0004] In the current field of artificial intelligence, the development of image sample annotation tools has made significant progress, especially in traditional image recognition tasks such as object detection and image segmentation. However, with the rise of multimodal large models, these tools are insufficient in supporting image-text pair annotation. Existing image sample annotation tools mainly target single-modal data, namely image data, and lack annotation support for multimodal data such as image-text pairs (image and text). These tools perform well in tasks such as object detection and image segmentation, but their functions are stretched when faced with the annotation needs of image-text pairs. Multimodal large models are able to handle multiple types of data including images and text. These models have demonstrated powerful capabilities in tasks such as image annotation, text generation, and image-text matching. However, in order to train these models, a large amount of image-text pair annotation data is required, and existing annotation tools cannot meet this demand. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and related device for labeling electric power multimodal samples in view of the deficiencies in the above-mentioned prior art, so as to solve the technical problem that the existing image sample labeling method cannot meet the requirements of multimodal data processing.

[0006] The present invention adopts the following technical solutions: A method for labeling electric power multimodal samples comprises the following steps: Unify the labels of the annotation files corresponding to the image samples; Annotate all relevant objects and defects based on the labels of the obtained annotation files to obtain annotation data; Design a question-answer pair template for the target detection task and convert the obtained labeled data into a question-answer pair format; For the image understanding task, a multi-round question-answer pair dataset is constructed based on the obtained question-answer pair format. The constructed multi-round question-answer pair dataset is used to train a large language model for identifying and analyzing safety behaviors in images. S6. Enrich the expression of question-answer pairs through open source large language models, increase language diversity while maintaining semantic equivalence, and improve the generalization ability and adaptability of large language models.

[0007] Preferably, the annotation file is used to record the labels of target objects to be identified in the image and their precise coordinates, and the naming of the labels in the annotation file follows the same rules and standards.

[0008] Preferably, the annotation data is specifically: Use the conversion script to convert the annotated VOC format image sample XML file into a COCO format JSON file, convert the image sample annotation format, and obtain the annotation data.

[0009] Preferably, the conversion script is used to convert the labeled VOC format image sample XML file into a COCO format JSON file as follows: A JSON object is created for each image, including the image's file name, ID, width, and height. For each object in the image, the conversion script generates a label object, including the object's category ID, bounding box coordinates, area, and whether it belongs to a crowd. All information is organized into a JSON file, which is directly used to train deep learning models that support the COCO format, so that the labeled data designed for the VOC format can be adapted to the COCO format.

[0010] Preferably, the question-answer pair template for the target detection task is designed as follows: Question: Please describe the electrical equipment in the image and give its coordinates. If there are any defects in the equipment, please explain them. Answer: There are X types of equipment in the figure, including transmission towers and transmission lines. Specifically, transmission tower 1: coordinates (x1, y1, x2, y2), transmission tower 2: coordinates (x3, y3, x4, y4; After observing the figure, x equipment defects were found, specifically the tower bird's nest: coordinates (x5, y5, x6, y6) and the anti-vibration hammer slip: coordinates (x7, y7, x8, y8).

[0011] Preferably, according to the designed question-answer pair template for the target detection task, the annotation files in the COCO format are batch converted through script code to generate multiple question-answer pairs for training a large multimodal model.

[0012] Preferably, an open source large language model is used to regenerate and organize the content of image-text question and answer. The multimodal large model uses natural language processing technology to answer questions related to the image content and achieve a deep understanding of the image.

[0013] Preferably, the natural language understanding and generation capabilities of the open source large language model are used to fine-tune the existing question-answer pairs to generate new question-answer pairs that are semantically equivalent but expressed differently. By introducing different language expressions, the open source large language model can learn more generalized feature representations and show better adaptability and accuracy when faced with new scenarios and problems.

[0014] In a second aspect, an embodiment of the present invention provides a power multimodal sample annotation system, including: Label module, unifies the labels of image samples corresponding to annotation files; The annotation module annotates all relevant objects and defects based on the labels of the obtained annotation file to obtain annotation data; The conversion module designs the question-answer pair template for the target detection task and converts the obtained labeled data into the form of question-answer pairs; The recognition module builds a multi-round question-answer pair dataset based on the obtained question-answer pair format for image understanding tasks, and uses the constructed multi-round question-answer pair dataset to train a large language model for identifying and analyzing safety behaviors in images. The output module enriches the expression of question-answer pairs through an open source large language model, increases language diversity while maintaining semantic equivalence, and improves the generalization ability and adaptability of the large language model.

[0015] In a third aspect, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned power multimodal sample labeling method when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned power multimodal sample labeling method.

[0017] In a fifth aspect, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned power multimodal sample labeling method when executing the computer program.

[0018] In a sixth aspect, an embodiment of the present invention provides an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-mentioned power multimodal sample labeling method.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: A method for labeling multimodal samples in the power industry improves data quality through standardized labels and comprehensive annotations, and provides deep semantic support for image understanding tasks and target detection tasks in the form of question-answer pairs. Training a large language model with a multi-round question-answering dataset can not only achieve more accurate target recognition and safety behavior analysis, but also improve the application effect of the model in the actual power field by increasing language diversity and improving generalization ability.

[0020] Furthermore, by unifying the image sample labels and comprehensively annotating all relevant objects and defects, the consistency and comprehensiveness of the data are ensured, providing a high-quality, standardized data foundation for model training.

[0021] Furthermore, by converting the VOC format into the COCO format and designing a question-answer pair template, the present invention improves the compatibility and usability of the data, while enhancing the contextual understanding ability of the model, providing structured data support for the training of large multimodal models.

[0022] Furthermore, converting VOC format image annotation data to COCO format can improve the standardization of data, increase data compatibility and scalability, and facilitate the training of deep learning models, especially complex tasks such as multi-task learning and instance segmentation. This conversion can not only support more efficient data processing, but also utilize the rich features of the COCO format to optimize the training effect in computer vision tasks.

[0023] Furthermore, by constructing a multi-round question-answer pair dataset and using an open source large language model to enrich the expression of question-answer pairs, the present invention improves the model's deep understanding and generalization capabilities, enabling the model to better adapt to changing practical application scenarios.

[0024] Furthermore, the combination of open source large language models and multimodal technology not only makes the understanding of image content more accurate and profound, but also enhances the flexibility, intelligence, automation and cross-domain application capabilities of the system. This brings more application potential to all walks of life, thereby improving data processing efficiency, user experience and the feasibility of cross-industry solutions.

[0025] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0026] In summary, the method of the present invention improves data quality and model performance, enhances the contextual understanding and adaptability of the model, and improves data processing efficiency and model generalization ability.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 3The present invention is a block diagram of an electronic device provided according to an embodiment.

[0030] Among them, 60. computer equipment; 61. processor; 62. memory; 63. computer program; 600. electronic device; 610. processing unit; 620. storage unit; 6201. random access storage unit; 6202. cache storage unit; 6203. read-only storage unit; 6204. program / utility; 6205. program module; 630. bus; 640. display unit; 650. input / output interface; 660. network adapter; 700. external device. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0033] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0034] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0035] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0036] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0037] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0038] The present invention provides a method for labeling electric power multimodal samples, which adopts standardized and comprehensive labeling to improve data quality, thereby improving the recognition accuracy and generalization ability of the model, and optimizing the model training and application effects; through the construction of question-answer pair templates and multi-round question-answer pair data sets, the model can have a deeper understanding of the image content, and at the same time, the rich question-answer pair expression enhances the adaptability and accuracy of the model when facing new scenarios; the conversion of the labeling format and the application of automated scripts improve the efficiency of data processing, and the rich expression of the question-answer pair enhances the generalization ability of the model, so that the model can better cope with the diversity and complexity in practical applications.

[0039] Aims to improve the accuracy and depth of image recognition in the power field and optimize model training and application effects.

[0040] Example 1 The present invention provides a method for labeling electric power multimodal samples, comprising the following steps: S1. Unify image sample labels to ensure standardization of labels for samples from different sources, improve data consistency and model training efficiency; In the field of power artificial intelligence, the collection and annotation of image samples are crucial for target detection tasks. They are widely used in intelligent inspection of power transmission and transformation equipment, safety behavior monitoring at work sites, and many other aspects. Image samples are mainly divided into two parts: original image samples and corresponding annotation files. Original samples are usually stored in common image formats such as JPEG and PNG, while annotation files record in detail the labels and precise coordinates of the target objects to be identified in the image. Annotation files have various formats, including JSON and XML, which provide the necessary structured information for the training of deep learning models.

[0041] The premise of large-scale model training is to collect and organize a large number of image samples. Due to the diversity of sample sources, samples from different sources may use different annotation standards, resulting in inconsistencies in the annotation of the same type of target objects in different samples. This inconsistency may have a negative impact on the generalization ability and recognition accuracy of the model. Therefore, in order to ensure the effectiveness of model training and the accuracy of the results, the labels in the annotation files must be standardized to achieve unified label naming. Taking the transmission equipment defect recognition scenario as an example, the unified description is shown in the following table:

[0042] Unified label naming not only helps improve data consistency and comparability, but also optimizes the model training process and reduces the complexity of data preprocessing. It ensures that image samples collected from different sources follow the same rules and standards in their label naming in the annotation files before entering the training process.

[0043] In addition, automated scripts or professional image annotation tools can be used to assist in the renaming of large-scale image samples and the standardization of annotation files, thereby improving the efficiency of the entire data processing process.

[0044] S2. Label all relevant objects and defects as much as possible to improve the recognition accuracy and generalization ability of the large model; In the field of image recognition in the power industry, traditional algorithm models usually focus on specific tasks, such as only identifying transmission towers but not ground wires. This specificity leads to the annotation of training image sample sets only for specific tasks, failing to fully cover all objects and defects in the power field, thus causing the problem of insufficient annotation. Under the application of large model methods, this traditional annotation strategy will severely limit the effectiveness of the model, because large models require more extensive data to achieve their generalization capabilities. In order to give full play to the potential of large models in the power field, all relevant objects and defects must be fully annotated to ensure the comprehensiveness of the annotations, so that the large model can have complete power field object recognition capabilities. This comprehensive annotation strategy can not only improve the recognition accuracy of the model, but also enhance its generalization ability, enabling it to adapt to the changing power business scenarios.

[0045] S3, convert the image sample annotation format, convert the VOC format XML file into the COCO format JSON file to adapt to different target detection frameworks; The annotated image samples are usually saved in XML format, following the organization specifications of the VOC (Visual Object Classes) dataset. The VOC format is well-known for its XML annotation files, which describe the object categories and locations in the image in detail. However, in order to adapt to different object detection frameworks and algorithms, such as Mask-RCNN, these XML annotation files need to be converted into JSON files in the COCO (Common Objects in Context) format. The COCO format uses JSON files to store images, annotations, and object location information. This format is widely adopted due to its flexibility and rich annotation information. To achieve the conversion from VOC to COCO format, developers usually write or use existing scripts to automate the process. These scripts parse the bounding box coordinates and category information in the XML file and write this information into the JSON file according to the requirements of the COCO format. During the conversion process, the script iterates over each image and its corresponding XML annotation file, extracting the image's file name, size, and annotation information, including the category ID and bounding box (bbox). Specifically, the conversion script creates a JSON object for each image, containing information such as the image's file name, ID, width, and height. For each target in the image, the script generates a label object containing the target's category ID, bounding box coordinates, area, and whether it belongs to a crowd. Ultimately, all this information is organized into a JSON file that can be used directly to train deep learning models that support the COCO format. In this way, the label data originally designed for the VOC format can be adapted to the COCO format, thereby expanding the availability and compatibility of the data, allowing a wider range of tools and frameworks to be used for image recognition and object detection tasks.

[0046] S4. Design a question-answer pair template for target detection tasks, convert the labeled data into question-answer pairs, and enhance the model's context understanding ability. During the fine-tuning phase of a large multimodal model, training samples are usually presented in the form of question-answer pairs. Although different models may have different organizational forms, they generally follow the question-answer structure. For example, a typical question-answer pair may be: "Question: What objects are there in the image?", "Answer: The image contains objects such as transmission towers and transmission lines." In order to make full use of the existing annotated data, a question-answer pair template for the object detection task is first designed.

[0047] The designed template is as follows: Question: Please describe the electrical equipment in the image and give its coordinates. If there are any defects in the equipment, please explain.

[0048] Answer: There are X types of equipment in the figure, including transmission towers, transmission lines, etc. Specifically, transmission tower (1): coordinates (x1, y1, x2, y2), transmission tower (2): coordinates (x3, y3, x4, y4). In addition, x equipment defects are found in the figure, specifically the bird's nest of the tower: coordinates (x5, y5, x6, y6), and the slip of the anti-vibration hammer: coordinates (x7, y7, x8, y8). According to the designed template, the COCO format annotation files can be batch converted through script code to quickly generate multiple question-answer pairs for training multimodal large models. This process involves converting the COCO format annotation data, a JSON file containing images, categories, annotations, and target location information, into the form of question-answer pairs. This conversion not only improves the availability of data, but also provides rich contextual information for the model, which helps to improve the performance of the model in power equipment identification and defect detection tasks.

[0049] S5. For image understanding tasks, we build a multi-round question-answer dataset to train the model to identify and analyze safety behaviors in images. In the field of artificial intelligence, the capabilities of large multimodal models are not limited to traditional target detection tasks. Their more critical capabilities are reflected in image understanding and in-depth analysis of image content. These models can answer questions related to image content through natural language processing technology, thereby achieving a deep understanding of the image. Taking the power operation scene as an example, the large multimodal model can use its multi-round dialogue ability and image understanding ability to conduct a detailed analysis of safe behaviors in the image and identify whether there are potential unsafe behaviors. In order to train the model to have the above capabilities, it is necessary to build a dataset containing multiple rounds of question and answer pairs. The following is an example showing how to build such a dataset: { "id": "identity_1", "name": "picture_1.jpg", "conversations": [ { "question": "Please briefly describe the scene shown in the image", "answer": "Please briefly describe the scene shown in the image", "question":"What electrical equipment is there in the picture?", "answer": "The power equipment in the picture includes "pole tower" and "transmission line". "question":"How many people are there in the picture? Are they performing tasks?", "answer":"There are 2 people in the picture. Personnel 1: coordinates "x, x, x, x" is not doing any work; Personnel 2: coordinates "x, x, x, x" is doing a high-altitude work, he is climbing the tower to perform equipment maintenance." "question":"Is Person 2 wearing the safety belt correctly?", "answer":"Yes, Person 2 wore the seat belt correctly.", "question":"Is Person 2 wearing the belt correctly?", "answer":"Yes, Agent 2 wore the belt correctly.", "question":"Is Person 2 wearing the safety rope correctly?" "answer":"No, person 2 did not hang the safety rope above the waist belt, which poses a safety hazard" }] } In order to improve the efficiency of personnel annotating image-text pairs, a table template is designed to integrate the above key information. Different template definitions can be provided for different tasks to facilitate the quick and accurate completion of the annotation work. Through this structured data set and template, a multimodal large model can be effectively trained so that it can accurately identify and analyze safe behaviors in images in practical applications, thereby improving the safety of the work site. The following is an example template:

[0050] S6. Enrich the expression of question-answer pairs through open source large language models, increase language diversity while maintaining semantic equivalence, and improve the generalization ability and adaptability of the model.

[0051] In the application of multimodal large models, although the templated annotation method can significantly improve the efficiency of sample annotation, it may cause the problem of high similarity of question-answer pairs, which is not conducive to the generalization ability and learning effect of the model. In order to solve this challenge, after constructing the question-answer pair dataset, an open source large language model is used to regenerate and organize the content of the picture-text question-answer pairs. Through this strategy, while keeping the original meaning of the question-answer pairs unchanged, the diversity and richness of the question-answer pair language can be effectively enhanced, thereby improving the training effect of the model. Specifically, by using the natural language understanding and generation capabilities of the open source large language model, the existing question-answer pairs can be fine-tuned to generate new, semantically equivalent but differently expressed question-answer pairs. This method can not only increase the diversity of the dataset, but also simulate the natural language variability in the real world, which is crucial for training a more robust and adaptable multimodal large model. In addition, by introducing different language expressions, the model can learn more generalized feature representations, thereby showing better adaptability and accuracy when facing new scenarios and problems.

[0052] By restating, the previous example is modified as shown below.

[0053] { "id": "identity_1", "name": "picture_1.jpg", "conversations": [ { "question": "Can you briefly describe the scene presented in the picture?", "answer": "The picture shows an electrical operation site where workers are performing related tasks.", "question": "What types of power facilities can be identified in the image?", "answer": "The picture contains power facilities such as towers and transmission lines.", "question": "How many workers are shown in the image and are they performing their tasks?", "answer": "There are two workers in the picture. Worker 1 is at coordinates (x1, y1, x2, y2) and is not involved in the operation; while worker 2 is at coordinates (x3, y3, x4, y4) and is performing high-altitude operations, specifically climbing the tower to perform equipment maintenance.", "question": "Did staff 2 wear the safety belt as required?", "answer": "Yes, worker 2 has worn the safety belt correctly.", "question": "Did worker 2 wear the belt correctly?", "answer": "Yes, worker 2 has worn the belt correctly.", "question": "Did staff 2 use the safety rope correctly?", "answer": "No, Officer 2 did not secure the safety line in place above the waist belt, which posed a safety hazard." } ]}.

[0054] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0055] Example 2 The present invention provides an electric power multimodal sample annotation system, which can be used to implement the above-mentioned electric power multimodal sample annotation method. Specifically, the electric power multimodal sample annotation system includes a label module, an annotation module, a conversion module, an identification module and an output module.

[0056] Among them, the label module unifies the labels of the image samples corresponding to the annotation files; The annotation module annotates all relevant objects and defects based on the labels of the obtained annotation file to obtain annotation data; The conversion module designs the question-answer pair template for the target detection task and converts the obtained labeled data into the form of question-answer pairs; The recognition module builds a multi-round question-answer pair dataset based on the obtained question-answer pair format for image understanding tasks, and uses the constructed multi-round question-answer pair dataset to train a large language model for identifying and analyzing safety behaviors in images. The output module enriches the expression of question-answer pairs through an open source large language model, increases language diversity while maintaining semantic equivalence, and improves the generalization ability and adaptability of the large language model.

[0057] Example 3 The present invention provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, graphics processors (GPU), tensor processors (TPU), digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the power multimodal sample labeling method, including: Unify the labels of the annotation files corresponding to the image samples; annotate all relevant objects and defects based on the labels of the annotation files to obtain annotated data; design a question-answer pair template for the target detection task, and convert the obtained annotated data into a question-answer pair format; for the image understanding task, construct a multi-round question-answer pair dataset based on the obtained question-answer pair format, and use the constructed multi-round question-answer pair dataset to train a large language model for identifying and analyzing safety behaviors in images; enrich the expression of question-answer pairs through the open source large language model, increase language diversity while maintaining semantic equivalence, and improve the generalization ability and adaptability of the large language model.

[0058] See also Figure 2 , the terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the power multimodal sample annotation method of the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the power multimodal sample annotation system of the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0059] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 2 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0060] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, graphics processing units (GPU), tensor processing units (TPU), digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0061] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0062] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0063] See also Figure 3 The terminal device is an electronic device 600, which is in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0064] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0065] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0066] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0067] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0068] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems). Such communication may be performed via an input / output interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., local area networks, wide area networks, and / or public networks, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0069] Example 4 The present invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0070] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program code. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the above.

[0071] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network or a wide area network, or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0072] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for labeling electric power multimodal samples in the above embodiment; the processor may load and execute the following steps: Unify the labels of the annotation files corresponding to the image samples; annotate all relevant objects and defects based on the labels of the annotation files to obtain annotated data; design a question-answer pair template for the target detection task, and convert the obtained annotated data into a question-answer pair format; for the image understanding task, construct a multi-round question-answer pair dataset based on the obtained question-answer pair format, and use the constructed multi-round question-answer pair dataset to train a large language model for identifying and analyzing safety behaviors in images; enrich the expression of question-answer pairs through the open source large language model, increase language diversity while maintaining semantic equivalence, and improve the generalization ability and adaptability of the large language model.

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0074] 1. Simulation data design The purpose of the simulation data is to generate a dataset that can be used to train and validate object detection and multi-round question answering models. The simulation data can be expanded based on the existing power image sample library and defect scenarios.

[0075] S1. Unify the labels of the image samples corresponding to the annotation files: Data design: Collect image samples from different power equipment and detection systems to ensure consistent annotation content, for example, uniformly annotate "power poles and towers", "cables", "substation equipment", etc. Labels for each object and defect are classified using a unified standard.

[0076] Simulation data example: If a power tower appears in the image, the annotation file includes the "tower" label and coordinate information.

[0077] S2. Comprehensively label all relevant objects and defects: Data design: Label various objects and defects in the image (such as broken power lines, aging equipment, etc.), ensuring that each image contains all the content that needs to be labeled. Use bounding boxes and pixel-level annotations to describe different objects or defects.

[0078] Example of simulated data: The image contains a "broken wire" defect, and the annotation file contains the bounding box, defect location, and type of the defect.

[0079] S3. Convert annotation format: Data design: Convert the labeled data into formats that are suitable for different models, such as YOLO format, COCO format, or PASCAL VOC format. For multimodal tasks, the correlation between image and text data also needs to be reflected in the annotation.

[0080] Simulation data example: After the data of an image is converted, it may become the following format: json { "image": "image_001.jpg", "annotations": [ { "label": "Power tower", "bbox": [50, 60, 200, 250] }, { "label": "Broken wire", "bbox": [300, 400, 500, 550] } ] } S4. Design question-answer pair template for target detection task: Data design: Design a series of question-answer templates based on the task requirements of target detection. For example, input image information and generate corresponding question-answer pairs (such as question: What is the power equipment in the image? Answer: Power pole tower).

[0081] Example of simulated data: Q: What electrical equipment is shown in the image? Answer: Power poles, wires, and transformers.

[0082] Q: Is there any damage to electrical equipment in the image? A: Yes, the wire is broken.

[0083] S5. Construct a multi-round question-answer dataset: Data design: Based on the objects and defects in the power images, we designed multiple rounds of Q&A scenarios. In each round of Q&A, we generated new questions and answers based on the image content and the context of the previous Q&A. For example, after the power towers and cables in the image are identified, we can continue to ask whether the equipment is dangerous.

[0084] Example of simulated data: Round 1 Question: What electrical equipment is present in the image? First round of answers: power poles, wires, and transformers.

[0085] Second round of questions: Are the power poles and towers damaged? Second round of answers: Yes, there are slight cracks on the power towers.

[0086] S6. Enrich the expression of question-answer pairs and improve the generalization ability of language models: Data design: In order to improve the language diversity of the model, we use open source large language models to generate versions that are semantically equivalent to the question-answer pair but have different expressions. For example, for the question "there are cracks in the power pole", we can generate different expressions such as "there are cracks in the power pole" or "there are cracks in the power pole".

[0087] Example of simulated data: Q: Are the power poles and towers damaged? Answer: Yes, cracks appeared on the column of the power tower.

[0088] 2. Experimental data design The purpose of the experimental data is to verify the performance of the model trained by simulation data on real power images.

[0089] Data collection: Collect image data of power facilities, including power towers, substations, power lines, equipment failures, etc.

[0090] Experimental dataset: The number of images should cover various power equipment and defect types to ensure diversity and representativeness.

[0091] Experimental tasks: Object detection task: Perform object detection training on labeled data, and use the trained model to perform detection and evaluation on new data.

[0092] Multi-round question-answering task: Use the constructed multi-round question-answering dataset to train the question-answering model, and perform question-answering generation and accuracy evaluation on the experimental data.

[0093] Experimental indicators: Object detection task evaluation indicators: Common evaluation indicators such as mAP (mean Average Precision), precision, recall, etc. are used.

[0094] Question answering task evaluation indicators: To evaluate the accuracy of question answers generated by the model, you can use text generation evaluation indicators such as BLEU and ROUGE to ensure that the generated answers are consistent with the actual image content.

[0095] 3. Experimental results Object detection task results: Assuming that the YOLO model is used for training, the mAP value of the experimental data is 85%, which is an excellent performance.

[0096] Question-answering task results: A multi-round question-answering system trained based on a pre-trained large language model (such as GPT-3) achieves an accuracy of more than 80% in image recognition tasks, and the generated question-answering content has a high language diversity.

[0097] Therefore, through the design of simulation and experimental data, the effectiveness and reliability of the proposed power multimodal sample annotation method can be verified. The method of the invention can accurately annotate power equipment and defects, and generate effective training data through multiple rounds of question-answer pairs, thereby improving the generalization ability of the language model, thereby better serving power safety monitoring and equipment maintenance tasks.

[0098] In summary, the present invention provides a method and a related device for labeling electric power multimodal samples, which have the following effects: 1. Improve data quality and model performance Standardized and comprehensive labeling improves data quality, thereby improving the recognition accuracy and generalization ability of the model, and optimizing model training and application effects.

[0099] 2. Enhance the contextual understanding and adaptability of the model By constructing question-answer pair templates and multi-round question-answer pair datasets, the model can have a deeper understanding of image content. At the same time, the rich question-answer pair expressions enhance the adaptability and accuracy of the model when facing new scenarios.

[0100] 3. Improve data processing efficiency and model generalization capabilities The conversion of annotation formats and the application of automated scripts improve the efficiency of data processing, while the rich expression of question-answer pairs enhances the generalization ability of the model, enabling the model to better cope with the diversity and complexity of practical applications.

[0101] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0102] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0103] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0104] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0108] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for labeling electric power multimodal samples, characterized in that: The following steps are involved: Unify the labels of the annotation files corresponding to the image samples; Annotate all relevant objects and defects based on the labels of the obtained annotation file to obtain annotation data; Design a question-answer pair template for the target detection task and convert the obtained labeled data into a question-answer pair format; For image understanding tasks, a multi-round question-answer pair dataset is constructed based on the obtained question-answer pair format, and the constructed multi-round question-answer pair dataset is used to train a large language model for identifying and analyzing safety behaviors in images; the expression of question-answer pairs is enriched through the open source large language model, while maintaining semantic equivalence, language diversity is increased, and the generalization ability and adaptability of the large language model are improved.

2. The method for labeling electric power multimodal samples according to claim 1, characterized in that: The annotation file is used to record the labels of the target objects to be identified in the image and their precise coordinates. The naming of the labels in the annotation file follows the same rules and standards.

3. The method for labeling electric power multimodal samples according to claim 1, characterized in that: The specific annotation data is: Use the conversion script to convert the annotated VOC format image sample XML file into a COCO format JSON file, convert the image sample annotation format, and obtain the annotation data.

4. The method for labeling electric power multimodal samples according to claim 3, characterized in that: Use the conversion script to convert the labeled VOC format image sample XML file into a COCO format JSON file as follows: A JSON object is created for each image, including the image's file name, ID, width, and height. For each object in the image, the conversion script generates a label object, including the object's category ID, bounding box coordinates, area, and whether it belongs to a crowd. All information is organized into a JSON file, which is directly used to train deep learning models that support the COCO format, so that the labeled data designed for the VOC format can be adapted to the COCO format.

5. The method for labeling electric power multimodal samples according to claim 1, characterized in that: Design the question-answer pair template for the target detection task as follows: Question: Please describe the electrical equipment in the image and give its coordinates. If there are any defects in the equipment, please explain them. Answer: There are X types of equipment in the figure, including transmission towers and transmission lines. Specifically, transmission tower 1: coordinates (x1, y1, x2, y2), transmission tower 2: coordinates (x3, y3, x4, y4; After observing the figure, x equipment defects were found, specifically the tower bird's nest: coordinates (x5, y5, x6, y6) and the anti-vibration hammer slip: coordinates (x7, y7, x8, y8).

6. The method for labeling electric power multimodal samples according to claim 5, characterized in that: According to the designed question-answer pair template for the target detection task, the COCO format annotation files are batch converted through script code to generate multiple question-answer pairs for the training of multimodal large models.

7. The method for labeling electric power multimodal samples according to claim 1, characterized in that: An open source large language model is used to regenerate and organize the content of image and text question and answer. The multimodal large model uses natural language processing technology to answer questions related to the image content and achieve a deep understanding of the image.

8. The method for labeling electric power multimodal samples according to claim 7, characterized in that: By leveraging the natural language understanding and generation capabilities of the open source large language model, we can fine-tune the existing question-answer pairs and generate new question-answer pairs that are semantically equivalent but expressed differently. By introducing different language expressions, the open source large language model can learn more generalized feature representations and show better adaptability and accuracy when faced with new scenarios and problems.

9. A power multimodal sample annotation system, characterized in that: include: Label module, unifies the labels of image samples corresponding to annotation files; The annotation module annotates all relevant objects and defects based on the labels of the obtained annotation file to obtain annotation data; The conversion module designs the question-answer pair template for the target detection task and converts the obtained labeled data into the form of question-answer pairs; The recognition module builds a multi-round question-answer pair dataset based on the obtained question-answer pair format for image understanding tasks, and uses the constructed multi-round question-answer pair dataset to train a large language model for identifying and analyzing safety behaviors in images. The output module enriches the expression of question-answer pairs through an open source large language model, increases language diversity while maintaining semantic equivalence, and improves the generalization ability and adaptability of the large language model.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 8.

11. A computing device, characterized in that: include: One or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for executing the method according to any one of claims 1 to 8.

Images