A method for rotary switch state recognition based on multimodal model
Through the knob switch status recognition method combined with multimodal model and knowledge base, the problems of low knob switch status recognition efficiency and large resource utilization in the prior art are solved, automatic, fast and accurate identification and alarm are realized, and recognition is improved Efficiency and accuracy.
Patent Information
- Application Number
- CN202411155430.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-22
AI Technical Summary
In the prior art, knob switch status recognition relies on manual inspection efficiency and is prone to errors. The machine vision solution has the problems of complex image preprocessing, large resource occupancy, and the inability to quickly identify new states.
The knob switch state recognition method based on multimodal model is adopted, and the data set is supervised and learned through training graphic matching, and the multimodal model is constructed and improved, and the state recognition and alarm are combined with the knowledge base, and images are collected using an automatic inspection robot. Conduct identification and use the CogVLM model to accelerate the identification process.
It realizes automatic, fast and accurate identification of knob switch status, reduces resource occupation, improves identification efficiency and alarm accuracy, and avoids the defects of human inspection.
Smart Images

Figure CN118690250B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of automated inspection, and in particular relates to a method for identifying a knob switch state based on a multimodal model. Background Art
[0002] With the rapid development of the national economy, ensuring the stable supply of electricity to all walks of life has become a crucial link in the production of various industries. At present, there are a large number of knob switches in various power stations to control and switch circuits, thereby dispatching and distributing electricity.
[0003] Therefore, in order to prevent personnel from operating the knob incorrectly, and to accurately obtain the status of the knob switch at each position and notify the alarm in time, the knob switch needs to be checked regularly in real scenarios. Although some switches have a self-checking function, there are still a considerable number of knob switches that do not have a self-checking function due to cost considerations or design requirements. Therefore, the status check of the knob switch in the prior art still relies on manual switch inspections. However, the above traditional methods have problems such as slow inspection by personnel and inspection errors by personnel. Failures that occur later may cause huge economic losses and even endanger personnel safety.
[0004] On the basis of the above, the prior art also has automated inspection solutions based on machine vision, using machine inspection to replace manual inspection. For example, Chinese patent CN117036665B discloses a method for identifying the state of a knob switch based on a twin neural network. The method first performs image preprocessing on the acquired original preset position image, extracts the corrected positive angle image of the knob switch in the state to be identified, so as to reduce the influence of complex background, lighting and variable shooting angles, and then uses a similarity calculation model based on a twin neural network to learn and express the state characteristics of the knob switch and the differences between the various gear states, and determines the image similarity between the image to be identified and the standard images of each knob switch to achieve state recognition. Although the above technical solution solves the automation problem to a certain extent, it still has the following problems in common with similar machine vision inspection solutions:
[0005] (1) Image preprocessing: feature point matching and screening are performed on the preset template image and the original preset image. The original preset image is corrected using a matrix algorithm based on the selected feature points. However, in actual scenes, due to the different pointing angles of the knob switch rod, the feature matching points may be incorrectly obtained, and the image cannot be corrected correctly.
[0006] (2) Recognition method: The result is obtained by extracting the features of the original preset image and matching the features of all the templates in the image template library. It is not possible to achieve universal and fast recognition for new switch states that are not in the template library;
[0007] (3) Resource occupation. In similar schemes, each knob detection has a corresponding image processing link. In particular, knobs with multiple gears and stepless adjustment require a lot of preliminary image processing and position calculation, and are used with binocular smart cameras to obtain depth of field data. As a whole, they occupy a lot of system resources, increase costs, and are inefficient.
[0008] In summary, there is a need to provide a knob switch state recognition method that is low-cost, high-efficiency, and universal / scalable. Summary of the invention
[0009] The present invention provides a method for identifying a knob switch state based on a multi-modal model to solve the above-mentioned problem.
[0010] The present invention adopts the following technical solutions:
[0011] A method for identifying the state of a knob switch based on a multimodal model, the main steps of which include:
[0012] Based on the inspection route, training images of multiple knob switches in different switch states are collected, and category labels and target detection results are obtained through the training images;
[0013] The category labels and target detection results are combined with the training images to form a picture-text matching pair data set;
[0014] Constructing a knob switch state vertical classification dataset based on the image-text matching dataset;
[0015] Performing supervised learning training on at least one multimodal model using the knob switch state vertical classification data set, and obtaining an improved multimodal model;
[0016] The improved multimodal model is deployed after verification and a multimodal model knowledge base is configured;
[0017] The multimodal model knowledge base includes the whole machine electronic document and operation electronic document of the knob switch;
[0018] Collecting working images of a plurality of knob switches along the inspection route by an automatic inspection robot, and inputting the working images and question prompt words into the improved multimodal model to obtain switch state recognition results;
[0019] When the confidence level of the switch state recognition result is lower than a threshold, the operation system of the improved multimodal model feeds back a warning message;
[0020] When the improved multimodal model performs knowledge inconsistency detection, if there is inconsistency between the switch state recognition result and the content text of the corresponding knob switch in the multimodal model knowledge base, the improved multimodal model feedbacks prompt information.
[0021] Optionally, performing data augmentation on the training image;
[0022] The data enhancement includes rotation, cropping, flipping, scaling, color conversion, adding noise, and image synthesis;
[0023] The elements of the image-text matching data set are arranged based on the shooting order, and question-answer annotations are added to each element to form the knob switch state vertical classification data set.
[0024] Optionally, a plurality of training images of knob angles are prepared for the stepless adjustment knob, and at least include training images of the knob pointing to an extreme position and a middle position.
[0025] Optionally, the adjustment range of the stepless adjustment knob is divided into a plurality of intervals of equal angles, and the training images of the plurality of knob angles further include training images of knobs pointing to the end points of the intervals.
[0026] Optionally, knob data enhancement is performed on the training image of the stepless adjustment knob:
[0027] The step of knob data enhancement includes:
[0028] Extracting knob image elements and panel image elements from the training image of the stepless adjustment knob;
[0029] The knob image element is rotated at multiple angles, and is fused with the panel image element to obtain a fused image;
[0030] The fused image is annotated with a rotation angle and then merged with the training image of the stepless adjustment knob.
[0031] Optionally, the question and answer annotation is configured in pairs with the question prompt word text, and the question prompt word text is provided with at least the keywords of the question and answer annotation, or synonyms of the keywords, or near-synonyms of the keywords.
[0032] Optionally, the improved multimodal model is locally deployed in a distributed manner based on a service architecture of a message queue;
[0033] The improved multimodal model is accelerated by the tensor real-time engine.
[0034] Optionally, the whole device electronic document is a set of technical parameter documents of the device to which the knob switch belongs, and at least records the specification information of the knob switch;
[0035] The specification information of the knob switch includes the knob switch gear information, range information, and function information;
[0036] The operation electronic document is a set of knob switch operation information documents, and at least records the operation information of the knob switch;
[0037] The operation information includes permission information and work information.
[0038] Optionally, the multimodal model includes at least a CogVLM model;
[0039] The category labels and target detection results are combined with the training images to form a picture-text matching pair data set through visual question answering technology.
[0040] Optionally, the multimodal model is a CogVLM model;
[0041] During the supervised learning training process, the visual encoder and the pre-trained large language model are frozen, and the visual expert module is trained.
[0042] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0043] The present invention improves the fine-tuning of a multimodal large model with vertical data sets, thereby realizing automatic identification and alarm of the status of knob switches in the substation room through intelligent machine vision inspection, thereby avoiding problems such as slow efficiency, high manpower consumption, and recording errors in manual inspection; further, the corresponding knowledge base can improve the accuracy of the alarm and discover problems as early as possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0045] Figure 1 It is a flow chart of the present invention;
[0046] Figure 2 is a demonstration image of a gear knob in a specific embodiment of the present invention;
[0047] Figure 3 It is a demonstration diagram of the question and answer annotation of the gear knob in a specific embodiment of the present invention;
[0048] Figure 4 It is a demonstration image of a stepless adjustment knob in a specific embodiment of the present invention;
[0049] Figure 5 It is a question-and-answer demonstration diagram of a stepless adjustment knob in a specific embodiment of the present invention.
[0050] In the figure: 1. Stepless adjustment knob, 2. Knob panel, 3. Panel information, 4. Minimum value of the range, 5. Maximum value of the range, 6. Switch position. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0053] It should be noted that the methods used in the present invention are all conventional methods unless otherwise specified; the raw materials and devices used are all conventional commercially available products unless otherwise specified.
[0054] The present invention provides a method for identifying the state of a knob switch based on a multimodal model, such as Figure 1 As shown, the main steps include:
[0055] First, the hardware foundation of this method is prepared, and an automatic inspection robot is selected as the platform. In a preferred embodiment, its main body is an automatic path-finding robot, which is equipped with a robotic arm, a pan-tilt head and an intelligent camera. Through a preset path, it automatically captures images of knob switches at fixed points along the inspection route in the substation.
[0056] Secondly, a training data set needs to be prepared. The method is to collect training images of multiple knob switches in different switch states based on the inspection route, and obtain the corresponding category labels and target detection results by annotating the category information and gear position (or rotation angle) of the training images, so as to form a picture-text matching pair (picture-text pair) data set with the corresponding category labels and target detection results and the training images. Figure 2 The knob switch shown in the figure is produced by adding question and answer annotations to the image based on visual question answering technology (such as VQA). Figure 3As shown. The above steps can be completed by manual annotation, but it consumes human resources. If the amount of data preparation is large, it is difficult to quickly complete the data required for later training with manual preparation; therefore, it is also possible to innovatively introduce the existing complete technical solution to prepare the preliminary training set separately, and complete the preliminary work in a more efficient way. Optionally, in a specific embodiment, the training image can be input into the machine vision model to obtain the category label and target detection result. Among them, the machine vision model used in this step can rent a mature technical solution on the market, such as a target detection model based on the YOLO algorithm. After conventional image preprocessing (distortion correction, color adjustment), the knob switch is detected, and then the contour is extracted. The type is identified with the panel text information, and the gear position or deflection angle is calculated to output the detection result, so that the corresponding category label and target detection result are combined with the training image to form a picture-text matching pair (picture-text pair) data set. Referring to the above picture-text matching pair production method, the image is added with question-answering annotations based on visual question-answering technology (such as VQA).
[0057] Optionally, in order to improve the training effect, the image-text matching dataset can be expanded to perform data enhancement on the training images. The means of data enhancement include rotation, cropping, flipping, scaling, color conversion, adding noise, and image synthesis; in the specific use process, one or a combination of these means can be selected to expand the original sample image into multiple images to significantly increase the amount of training. The image generated during the enhancement retains the original text annotation. Furthermore, in a preferred embodiment, it is considered that the knob switch is a stepless adjustment knob, such as Figure 4 As shown. Therefore, in order to enable the multimodal model to give a more accurate state recognition result, it is necessary to further improve the training set, that is, to prepare training images of multiple knob angles for the stepless adjustment knob, and at least include training images of the knob pointing to the extreme position (end point of the range) and the middle position, so as to expand the subsequent training volume. Furthermore, the adjustment range of the stepless adjustment knob 1 can be divided into multiple intervals of equal angles, and the training images of multiple knob angles also include training images of the knob pointing to the end points of the interval. For example: Figure 4 The knobs shown are set to 0° (range minimum value 4), 15° (interval end point), 30° (interval end point), 45° (interval end point, Figure 4 The positions shown in the figure), 60° (interval endpoint), 75° (interval endpoint), 90° (range maximum value 5), and the corresponding question and answer annotations form a picture-text matching pair (such as Figure 5As shown, 45° is taken as an example). Furthermore, although the aforementioned machine vision model can automatically output category labels and target detection results, the multi-angle images of the stepless adjustment knob 1 still need to be manually adjusted after the knob angle is taken by the machine, which is not only inefficient but also difficult to adjust the angle. Therefore, in a preferred embodiment, only the extreme position images can be collected, and the training images of the stepless adjustment knob can be enhanced with separate knob data; the steps of knob data enhancement include:
[0058] a. Artificially extract the knob image elements from the training image of the stepless adjustment knob 1 ( Figure 4 The area within the thick circle) and the panel image elements of the knob panel 2 ( Figure 4 The area between the square thick line and the circular thick line in the figure), that is, the knob image and the panel image are deducted separately in the image processing software;
[0059] b. Rotate the knob image elements at multiple angles and fuse them with the panel image elements to obtain a fused image. That is, rotate the knob images in batches around the center according to angles (such as 15°, 30°, 45°, 60°, and 75°), and then fuse the rotated knob images with the panel image to form a fused image.
[0060] c. After annotating the above rotation angle, the fused image is merged with the training image of the extreme position of the stepless adjustment knob to form a corresponding image-text matching pair data set.
[0061] Therefore, based on the structural characteristics of the stepless adjustment knob, its image-text matching data set can be quickly expanded in batches in an artificial way, thereby providing sufficient data for subsequent training and improving recognition accuracy. The overall efficiency is higher than the method in the traditional model in the prior art that requires the calculation of the pointer position.
[0062] The elements of the image-text matching dataset are arranged based on the shooting order (which can also be understood as the path running direction) to form a vertical classification dataset of the knob switch state. Among them, the arrangement based on the shooting order can increase the accuracy of subsequent identification.
[0063] At least one multimodal model is trained through supervised learning (also known as "model fine-tuning") using a knob switch state vertical classification data set, and an improved multimodal model is obtained. A multimodal model is a model that can process and fuse multiple different types of data modalities, which usually include but are not limited to: text, images, audio, and video. The multimodal model integrates and understands information from these different modalities to achieve a more comprehensive and accurate understanding and analysis. The basis is to extract the features of each modality and then use a specific fusion strategy to combine these features together to capture the association and complementarity between different modalities. Optionally, this embodiment takes the CogVLM model as an example. The CogVLM model not only has better performance, but its technical architecture is also more in line with the use scenario of this embodiment. The model adds a trainable visual expert module to each layer of the large language model to achieve deep alignment of image features and text features, while ensuring that the performance of NLP tasks is not sacrificed while enhancing visual understanding capabilities. Therefore, in the process of supervised learning training, the CogVLM model of this embodiment needs to freeze the visual encoder and the pre-trained large language model, and only train the visual expert module to ensure that the ability of the large text model is not destroyed and that the text and visual features can be deeply jointly trained. After training, the tensor real-time engine (such as the software development kit based on tensor accelerated computing, TensorRT) is used to accelerate the reasoning speed of the improved multimodal model.
[0064] After the improved multimodal model is verified for accuracy, it is deployed and the multimodal model knowledge base is configured; the deployment method adopts distributed local deployment based on the message queue service architecture.
[0065] The multimodal model knowledge base provides a built-in retrieval space for the model. By configuring appropriate electronic text content, the improved multimodal model of this embodiment has stronger recognition capabilities, and can provide additional state analysis content through knowledge inconsistency detection, which mainly includes the whole machine electronic document and operation electronic document of the knob switch; optionally, the whole machine electronic document is a collection of technical parameter documents of the device to which the knob switch belongs, and at least records the specification information of the knob switch. In a specific embodiment, the whole machine electronic document selects the electronic document of the manual of the device to which the knob switch belongs, which contains the specification information of the knob switch, such as the knob switch gear information (how many gears there are, what marks they correspond to, such as switch gear 6 corresponds to closing or shutting down), range information (text description of the knob limit position, such as the range minimum value 4 and the range maximum value 5), and function information (knob adjustment function, such as the on / off of a specific circuit, and the speed regulation information fed back by the panel information 3). The operation electronic document is a collection of knob switch operation information documents, and at least records the operation information of the knob switch. In a specific embodiment, the operation electronic document selects the operation safety manual electronic document, wherein the operation information includes the permission information of the knob switch (the gear positions allowed to be set in a certain scenario, and a certain scenario may also be associated with other knob switch states) and the working information (the device working state corresponding to the knob gear position).
[0066] After the above work is completed, in the subsequent automatic inspection process, the automatic inspection robot collects working images of multiple knob switches along the set inspection route, and inputs the working images and question prompt word text into the improved multimodal model to obtain the switch state recognition result. Optionally, the question and answer annotation and the question prompt word text are configured in pairs, and the question prompt word text is at least provided with the keyword of the question and answer annotation, or the synonym of the keyword, or the synonym of the keyword. In this way, the improved multimodal model can obtain the switch state recognition result in the form of question and answer.
[0067] When the confidence of the switch state recognition result is lower than the preset threshold, the operating system of the improved multimodal model of this embodiment feeds back a warning message; specifically, the confidence output can be obtained directly by using the loss function in the supervised learning process, or an evaluation model can be built by itself, which is not specifically limited in this embodiment. The server is also equipped with a detection module. When the confidence of the switch state recognition result output by the detection module is low (lower than the preset threshold), a warning message is sent to the operating console to remind personnel that there may be a large deviation in the inspection result and manual re-inspection is required.
[0068] When the improved multimodal model performs knowledge inconsistency detection based on the configured multimodal model knowledge base, if there is an inconsistency between the switch state recognition result and the content text of the corresponding knob switch, the improved multimodal model will feedback prompt information. Specifically, for example: when it is found that the knob switch position output this time exceeds the limit position in the electronic document of the manual (especially the stepless adjustment knob), feedback prompt information; for another example: when it is found that the knob switch position output this time is not applicable to the current scenario in the electronic document of the safety manual, feedback prompt information.
[0069] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for identifying the state of a knob switch based on a multimodal model, characterized in that the steps include: Based on the inspection route, training images of multiple knob switches in different switch states are collected, and category labels and target detection results are obtained through the training images; The category labels and target detection results are combined with the training images to form a picture-text matching pair data set; Constructing a knob switch state vertical classification dataset based on the image-text matching dataset; Arranging the elements of the image-text matching pair data set based on the shooting order, and adding question-answer annotations to each element to form the knob switch state vertical classification data set; Performing supervised learning training on at least one multimodal model using the knob switch state vertical classification data set, and obtaining an improved multimodal model; the multimodal model is a CogVLM model; freezing the visual encoder and the pre-trained large language model during the supervised learning training, and training the visual expert module; The improved multimodal model is deployed after verification and a multimodal model knowledge base is configured; The multimodal model knowledge base includes the whole machine electronic document and operation electronic document of the knob switch; Collecting working images of a plurality of knob switches along the inspection route by an automatic inspection robot, and inputting the working images and question prompt words into the improved multimodal model to obtain switch state recognition results; When the confidence level of the switch state recognition result is lower than a threshold, the operation system of the improved multimodal model feeds back a warning message; When the improved multimodal model performs knowledge inconsistency detection, if there is inconsistency between the switch state recognition result and the content text of the corresponding knob switch in the multimodal model knowledge base, the improved multimodal model feedbacks prompt information.
2. The method for identifying the state of a knob switch based on a multimodal model according to claim 1, characterized in that: Performing data augmentation on the training image; The data enhancement includes rotation, cropping, flipping, scaling, color conversion, adding noise, and image synthesis.
3. The method for identifying the state of a knob switch based on a multimodal model according to claim 2, characterized in that: A plurality of training images of knob angles are prepared for the stepless adjustment knob, and at least the training images of the knob pointing to the extreme position and the middle position are included.
4. The method for identifying the state of a knob switch based on a multimodal model according to claim 3, characterized in that: The adjustment range of the stepless adjustment knob is divided into a plurality of intervals of equal angles, and the training images of the plurality of knob angles also include training images of knobs pointing to the end points of the intervals.
5. The method for identifying the state of a knob switch based on a multimodal model according to claim 3 or 4, characterized in that: The knob data is enhanced for the training image of the stepless adjustment knob: The step of knob data enhancement includes: Extracting knob image elements and panel image elements from the training image of the stepless adjustment knob; The knob image element is rotated at multiple angles, and is fused with the panel image element to obtain a fused image; The fused image is annotated with a rotation angle and then merged with the training image of the stepless adjustment knob.
6. The method for identifying the state of a knob switch based on a multimodal model according to claim 2, characterized in that: The question and answer annotation is configured in pairs with the question prompt word text, and the question prompt word text is at least provided with the keyword of the question and answer annotation, or the synonym of the keyword, or the near synonym of the keyword.
7. The method for identifying the state of a knob switch based on a multimodal model according to claim 1, characterized in that: Distribute and locally deploy the improved multimodal model based on a message queue-based service architecture; The improved multimodal model is accelerated by the tensor real-time engine.
8. The method for identifying the state of a knob switch based on a multimodal model according to claim 1, characterized in that: The whole device electronic document is a collection of technical parameter documents of the device to which the knob switch belongs, and at least records the specification information of the knob switch; The specification information of the knob switch includes the knob switch gear information, range information, and function information; The operation electronic document is a set of knob switch operation information documents, and at least records the operation information of the knob switch; The operation information includes permission information and work information.
9. The method for identifying the state of a knob switch based on a multimodal model according to claim 1, characterized in that: The category labels and target detection results are combined with the training images to form a picture-text matching pair data set through visual question answering technology.
