Large language model-based pointer type instrument display number identification method

Abstract

The invention relates to a pointer type instrument reading identification method applied to an unmanned inspection robot. The method comprises the following steps: acquiring the position of a pointer instrument in a shot image by using a target detection network, and removing a background irrelevant to detection; identifying four key points, namely a scale starting point and a scale ending point, a pointer tip and a pointer rotation center, of an interval in which the pointer is positioned by adopting a key point detection method; recognizing characters in the pointer type instrument image; complementing and correcting a character recognition result by using a large language model; and according to the four detected key points and the complemented and corrected character recognition result, calculating by using an angle method to obtain the reading of the instrument. The invention provides the method for complementing and correcting the character recognition result by using the large language model, so that the detection rate of the number recognition text of the pointer type instrument is greatly improved, and the inspection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial inspection, in particular to a pointer-type instrument indication number recognition method applied to an unmanned inspection system Background Art

[0002] In the field of industrial production, accurate readings of pointer instruments are of vital importance to ensure the stable and safe operation of the production system. Developing a pointer instrument reading recognition method for unmanned inspection robot system can improve inspection efficiency, ensure production safety, and improve detection accuracy.

[0003] The character recognition of pointer instruments has the following difficulties: missed detection due to partial occlusion of the text caused by the rotation of the pointer; missed detection of decimal points; mistakenly confusing the number "0" with the letter "O", and the number "1" with the lowercase letter "l". These difficulties can be overcome by current natural language processing technology in scenarios with rich contextual semantics, such as recognizing documents, bills, and license plates by relying on contextual information. However, the text in pointer instruments has the characteristics of curved structure, flexible and uncertain position, and no contextual reference, which brings great challenges to text recognition. Existing methods for recognizing numbers on pointer instruments have problems such as low detection rate and high error rate in character recognition. At present, there is an urgent need for a pointer instrument number recognition algorithm that can adapt to complex industrial environments to meet the needs of industrial inspections. Summary of the invention

[0004] The present invention proposes a method for identifying numbers indicated by pointer instruments based on a large language model, which mainly includes the following steps:

[0005] Step S1: Target detection, quickly identifying and locating pointer instruments in complex environments;

[0006] Step S2: Key point detection: perform key point detection on the instrument identified in step S1 to obtain the scale starting point (such as Figure 2 .1) The end point of the scale (such as Figure 2 .2) Pointer tip (such as Figure 2 .3) Pointer rotation center (such as Figure 2 .4), the locations of four key points in total;

[0007] Step S3: character recognition, using optical character recognition technology to recognize characters on the dial;

[0008] Step S4: using a large language model, inferring and predicting the characters detected by character recognition to obtain accurate text information on the dial of the pointer instrument;

[0009] Step S5: numerical calculation, according to the four key points detected in step S2 and the text information recognized in step S4, according to a method for calculating the readings of pointer instruments based on an angle method, the readings of the pointer instruments are calculated.

[0010] Furthermore, the S4 step includes:

[0011] Two cooperative agents in series, where agent 1 (such as Figure 3 .1) is used to predict the complete text information of the dial and complete the missed information. Agent 2 (such as Figure 3 .2) is used to correct the detection results of the key point corresponding to the text. The two agents use their own prompt words (such as Figure 4 ) and is fine-tuned using LoRA (Low-Order Adaptive Training of Large Language Models) technology to complete and correct the character recognition results.

[0012] Furthermore, the S5 step mainly includes:

[0013] A method for calculating the reading of a pointer instrument based on the angle method (such as Figure 5 ), the formula is as follows:

[0014]

[0015] Where θ (such as Figure 5 .5) indicates the angle between the pointer and the initial scale. (like Figure 5 .6) represents the angle of the measuring range, where k1 represents the starting point of the scale, k2 represents the starting point of the scale interval where the pointer is located, k3 represents the end point of the scale interval where the pointer is located, and k4 represents the end point of the scale.

[0016] The invention has the advantages of proposing a novel character recognition completion and correction method based on a large language model, realizing accurate recognition of numbers indicated by pointer-type instruments, greatly improving the text detection effect, and overcoming the shortcomings of traditional character recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific details and processes of the present invention, the drawings required for use in the method will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 The flowchart of the pointer instrument indication number recognition method based on the large language model is shown;

[0019] Figure 2A schematic diagram of a key point detection strategy used in the present invention;

[0020] Figure 3 A schematic diagram of a two-agent workflow based on a large language model used in the present invention;

[0021] Figure 4 The prompt words used by the two intelligent agents built on the basis of the large language model used in the present invention;

[0022] Figure 5 Schematic diagram of the angle reading calculation method used in the present invention (θ represents the pointer deflection angle, represents the range angle);

[0023] Figure 6 A schematic diagram of a method for constructing large language model training data used in the present invention;

[0024] Figure 7 The comparison results between the large language model used in the present invention and other large language models;

[0025] Figure 8 Comparison of text recognition effects before and after using a large language model;

[0026] Fig. 9 Reduce the effects of various text recognition anomalies for large language models;

[0027] Fig.10 Several groups of sample results are provided for the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings shown in the present invention. Obviously, some of the results shown are only part of the embodiments of the present invention. Based on the results in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The data set used in the present invention is the pointer instrument indication number recognition data set open sourced by Baidu PaddlePaddle.

[0029] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] In the field of industrial production, accurate readings of pointer instruments are of vital importance to ensure the stable and safe operation of the production system. Developing a pointer instrument reading recognition method for unmanned inspection robot system can improve inspection efficiency, ensure production safety, and improve detection accuracy.

[0031] However, character recognition of pointer instruments has the following difficulties: recognition fails because part of the text is blocked by the rotation of the pointer; the number "0" is mistakenly confused with the letter "O", and the number "1" is confused with the lowercase letter "l"; the decimal point is missed. These difficulties can be overcome by current natural language processing technology in scenarios with rich contextual semantics, such as recognizing documents, bills, and license plates by relying on contextual information. However, the text in pointer instruments has the characteristics of curved structure, flexible and uncertain position, and no contextual reference, which brings great challenges to text recognition. Existing methods for recognizing numbers on pointer instruments have problems such as low detection rate and high error rate in character recognition. At present, there is an urgent need for a pointer instrument number recognition algorithm that can adapt to complex industrial environments to meet the needs of industrial inspections.

[0032] An implementation flow of this method is as follows Figure 1 After the inspection device collects the image of the pointer instrument, the inspection device feeds back the output result to the monitoring center through the pointer instrument indication number recognition method based on key point detection proposed by the present invention, thereby realizing intelligent inspection.

[0033] Step S1 mainly includes an object detection network, which is used to quickly identify and locate pointer instruments in complex environments.

[0034] Step S2 mainly includes a key point detection network, which is used to perform key point detection on the instrument identified in step S1, and obtain the scale starting point (such as Figure 2 .1) The end point of the scale (such as Figure 2 .2) Pointer tip (such as Figure 2 .3) Pointer rotation center (such as Figure 2 .4), the locations of four key points in total;

[0035] Step S3 mainly includes a currently commonly used text recognition technology, which is used to recognize text information on the pointer instrument dial.

[0036] Step S4 mainly includes two mutually cooperating agents in series composed of a large language model, wherein agent 1 (such as Figure 3 .1) is used to predict the complete text information of the dial and complete the missed information. Agent 2 (such as Figure 3 .2) is used to correct the detection results of the key point corresponding to the text. The two agents use their own prompt words (such as Figure 4 ) and is fine-tuned using LoRA (Low-Order Adaptive Training of Large Language Models) technology to complete and correct the character recognition results.

[0037] Furthermore, agent 1 (such as Figure 3.1) is the text recognition of the entire instrument panel, which is characterized by disordered text and possible recognition errors and missed detections. Figure 4 .1) and the large language model, output 1 can accurately give the range, scale interval and unit of this pointer instrument. Agent 2 (such as Figure 3 .2) is the text recognition result near the four key points, which is characterized by possible recognition errors and missed detections. Figure 4 .2) When combined with the large language model, output 2 can accurately give the recognition results of these four key points, eliminating recognition errors and missed detections. It is worth noting that when agent 2 is running, it will still use the content of agent 1 as context as the input of the large model. After agent 2 gives an output result, it is considered the end of a process and finally clears the conversation history of the large language model.

[0038] Furthermore, the training of large language models requires rich corpus data. The present invention uses a random method to generate pre-training data, such as Figure 6 A schematic diagram of a training data generation method is given, which includes the following steps: First, a set of arithmetic progressions and a physical unit (such as Figure 6 .3); shuffle the order of these elements (such as Figure 6 .4); randomly discard some elements (such as Figure 6 .5); add random interference characters (such as Figure 6 .6). These measures simulate the possible false detection and missed detection results caused by actual text recognition. The data set generates a total of 1,000,000 data according to the above method, combined with the corresponding prompt words (such as Figure 4 ), and store the data in an Alpaca format, and then use a LoRA method to fine-tune the large model.

[0039] Step S5 introduces a method for calculating the reading of a pointer-type instrument based on the angle method. This method benefits from the angle-preserving property of linear space mapping, so that there is no need to perform complex perspective transformation to correct the image distortion caused by shooting. Figure 5 ), the formula is as follows:

[0040]

[0041] Where θ (such as Figure 5 .5) indicates the angle between the pointer and the initial scale. (like Figure 5.6) represents the angle of the measuring range, where k1 represents the starting point of the scale, k2 represents the starting point of the scale interval where the pointer is located, k3 represents the end point of the scale interval where the pointer is located, and k4 represents the end point of the scale.

[0042] Furthermore, the present invention compares the large language model introduced in step S4 with other commonly used large language models, and also gives the results of these models before and after fine-tuning (those with the suffix LoRA are fine-tuned models), such as Figure 7 shown.

[0043] The definitions of the network evaluation indicators used are as follows:

[0044] AWER stands for Average Word Error Rate and SR stands for Complete Success Rate. Figure 3 The processing flow of the large language model shown contains 8 key outputs, which are: the minimum scale value in agent 1, the maximum scale value, the minimum scale interval and the unit of the instrument (such as Figure 3 .1); the minimum scale value in agent 2, the starting scale value of the pointer interval, the maximum scale value of the pointer interval, and the maximum scale value, (such as Figure 3 .2). WER word error rate represents the ratio of wrong results to 8 results. AWER represents the average WER result of all test samples. SR represents the ratio of 8 results in the test sample that are predicted correctly.

[0045] Furthermore, in order to test the effectiveness of the large language model method proposed in the present invention, Figure 8 The results of text recognition using OCR alone and using a large language model to complete and correct text recognition (LLM) are shown. PR represents the correct recognition rate of scale values, and PU represents the correct recognition rate of units. The test experiment was conducted on a dataset with 100 samples.

[0046] Furthermore, Fig. 9 Added Figure 8 Analysis of the abnormal types identified in the experiment. As can be seen from the figure, the use of a large language model significantly reduces the probability of missed detections and also greatly reduces the probability of recognition errors.

[0047] After a series of steps mentioned above, the present invention can identify the indication of the pointer instrument image photographed by the inspection equipment and feed back the final result to the monitoring center. Fig.10 The visualization results of some embodiments are shown, wherein the numbers in the upper left corner represent the recognition results.

Claims

1. A method for identifying numbers on pointer-type instruments based on a large language model, characterized in that: The inspection equipment captures the image of the pointer instrument, and through the pointer instrument indication recognition algorithm, gives the indication and uploads the result to the monitoring center.

2. The method according to claim 1, characterized in that The pointer-type instrument indication number recognition algorithm comprises the following steps: Step S1: Target detection, quickly identifying and locating pointer instruments in complex environments; Step S2: key point detection, the instrument identified in step S1 is subjected to key point detection, and the positions of four key points are obtained, namely, the scale starting point (as shown in Figure 2.1), the scale end point (as shown in Figure 2.2), the pointer tip (as shown in Figure 2.3), and the pointer rotation center (as shown in Figure 2.4); Step S3: character recognition, using optical character recognition technology to recognize characters on the dial; Step S4: using a large language model, inferring and predicting the characters detected by character recognition to obtain accurate text information on the dial of the pointer instrument; Step S5: numerical calculation, according to the four key points detected in step S2 and the text information recognized in step S4, according to a method for calculating the readings of pointer instruments based on an angle method, the readings of the pointer instruments are calculated.

3. The method according to claim 2, characterized in that The step S1 comprises: An object detection algorithm based on deep convolutional neural networks.

4. The method according to claim 2, characterized in that: The step S2 comprises: A keypoint detection algorithm based on deep convolutional neural networks.

5. The method according to claim 2, characterized in that: The step S3 comprises: A character recognition algorithm based on deep convolutional neural networks.

6. The method according to claim 2, characterized in that The step S4 comprises: A large language model fine-tuned with LoRA (Low-Rank Adaptive Training of Large Language Models).

7. The method according to claim 6, characterized in that The characteristics of the large language model include: Two agents are connected in series and cooperate with each other, where agent 1 (as shown in Figure 3.1) is used to predict the complete text information of the dial and complete the missed information, and agent 2 (as shown in Figure 3.2) is used to correct the detection results of the text corresponding to the key points.

8. The method according to claim 7, characterized in that Features of the agent include: Specific prompt words (as shown in Figure 4), and standard input and output formats.

9. The method according to claim 2, characterized in that: The step S5 comprises: The invention discloses a method for calculating the reading of a pointer instrument based on the angle method.

Images