Tilt correction method and tilt correction device for pointer type instrument image
Through the preset segmentation model and neural network model, the accuracy and speed problems of instrument image tilt correction in the prior art are solved, and the correction accuracy and robustness are achieved. Practical application.
Patent Information
- Application Number
- CN202110844952.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-07-26
AI Technical Summary
The prior art is difficult to accurately and quickly perform tilt correction of pointer instrument images, especially in the absence of tick marks or characters, and the recognition accuracy is low, resulting in unsatisfactory correction effect.
The preset segmentation model is used to segment the instrument image, determine the area of the pointer and the instrument box, generate label images through training of the neural network model, and calculate the rotation angle of the pointer relative to the instrument box for correction.
It improves the accuracy and speed of instrument image correction, enhances robustness, meets practical application requirements, and improves the accuracy of automatic readings.
Smart Images

Figure CN113487515B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for correcting the tilt of a pointer-type instrument image and also to a corresponding tilt correction device, belonging to the technical field of image processing. Background Art
[0002] At present, it is difficult to correct the tilt of the pointer instrument image, and there is no perfect correction solution. Some people observed the instrument panel information and found that there was a string of information recorded on the instrument panel. Through the extraction and processing of this part of the string information, the tilt angle of the string is calculated by linear fitting, which is the tilt angle of the instrument, and finally the instrument image is corrected by this angle. This method can realize the tilt correction of the instrument without prior information. However, this method is more difficult in the process of string extraction, and each step of the image processing process will affect the angle of the final calculated line. If some instruments do not have such a string text for reference, this method is not feasible, and there is no guarantee that the string is corrected, so the scope of use is small.
[0003] In addition, some people input the instrument image to be corrected into the key point detection model to obtain the coordinates of multiple key scale numbers. Based on the least squares method, an ellipse is fitted to the coordinates of the key scale numbers to obtain a fitted ellipse. The ellipse parameters of the fitted ellipse are obtained to determine the coordinate points before perspective transformation. Then the coordinate points after perspective transformation are determined, thereby obtaining the perspective transformation matrix. The instrument image to be corrected is subjected to perspective transformation to obtain the original corresponding coordinates, thereby realizing the correction of the instrument image.
[0004] However, the above method requires accurate recognition of the scale on the instrument panel. In practical applications, due to the inclination of the instrument itself, changes in light caused by shooting factors, or occlusion by obstacles, as well as the very small pixels of the scale, the recognition accuracy is relatively low, making it difficult to accurately detect such a small target as the scale line, resulting in unsatisfactory results of the subsequent tilt correction algorithm based on the scale line. Summary of the invention
[0005] The primary technical problem to be solved by the present invention is to provide a method for correcting the tilt of a pointer-type instrument image, which can correct the tilt of the instrument image more accurately and quickly.
[0006] Another technical problem to be solved by the present invention is to provide a device for correcting the tilt of a pointer-type instrument image.
[0007] In order to solve the above problems, the present invention adopts the following technical solutions:
[0008] According to a first aspect of an embodiment of the present invention, a method for correcting a tilt of a pointer instrument image is provided, comprising the following steps:
[0009] Acquire an instrument image to be corrected, and input the instrument image into a preset segmentation model;
[0010] Segmenting the instrument image to be corrected by using the preset segmentation model to determine the pointer and instrument frame in the instrument image;
[0011] The angle between the pointer and the instrument frame is determined, and the instrument image to be corrected is corrected according to the angle.
[0012] Preferably, the tilt correction method further comprises the following steps:
[0013] Acquire training samples corresponding to the instrument image, wherein the training samples are instruments with different tilt angles in the instrument image;
[0014] Based on the instrument image in the training sample, annotate the area corresponding to the pointer and the area of the instrument frame in the image to generate a label image; wherein the area of the instrument frame is located below the area of the pointer;
[0015] The initial neural network model is trained according to the training samples and the label images, and the trained model is obtained as the preset segmentation model.
[0016] Preferably, the training of the initial neural network model according to the training samples and the label images comprises the following sub-steps:
[0017] Inputting the instrument image in the training sample into the initial neural network model to obtain a corresponding training output result, wherein the training output result includes an area of the pointer and an area of the instrument frame in the corresponding input instrument image;
[0018] The initial neural network model is trained according to the training output results and the label image.
[0019] Preferably, the step of inputting the instrument image in the training sample into the initial neural network model to obtain the corresponding training output result comprises the following sub-steps:
[0020] The training feature data of the pixels in the instrument image are determined by the initial neural network model, and the pixels are classified according to the training feature data. The area of the pointer and the area of the instrument frame in the input instrument image are obtained according to the training classification result.
[0021] Preferably, the method of segmenting the instrument image to be corrected by using the preset segmentation model to determine the pointer and the instrument frame in the instrument image includes the following sub-steps:
[0022] According to the preset segmentation model, feature data of pixels in the instrument image to be corrected are determined, and pixels are classified according to the feature data. The area of the pointer and the area of the instrument frame in the instrument image to be corrected are determined according to the classification result.
[0023] Preferably, the step of determining the angle between the pointer and the instrument frame comprises the following sub-steps:
[0024] Determines the rotation angle of the pointer's area relative to the area of the meter frame.
[0025] Preferably, the step of determining the rotation angle of the pointer area relative to the instrument frame area includes the following sub-steps:
[0026] Determine a first minimum bounding rectangular frame of the pointer area and a second minimum bounding rectangular frame of the instrument frame area;
[0027] Determining a first rotation angle of a first minimum circumscribed rectangular frame relative to a preset reference position, and determining a second rotation angle of a second minimum circumscribed rectangular frame relative to the preset reference position;
[0028] The rotation angle of the pointer region relative to the instrument frame region is determined according to the first rotation angle and the second rotation angle.
[0029] Preferably, correcting the instrument image to be corrected according to the angle comprises the following sub-steps:
[0030] The rotation angle of the pointer area relative to the area of the instrument frame is used as the final rotation angle of the pointer in the instrument image to be corrected;
[0031] The instrument image to be corrected is adjusted according to the final rotation angle.
[0032] According to a second aspect of an embodiment of the present invention, there is provided a device for correcting a tilt of a pointer-type instrument image, comprising:
[0033] An acquisition module, used for acquiring an instrument image to be calibrated, and inputting the instrument image into a preset segmentation model;
[0034] A determination module, used to segment the instrument image to be corrected by using the preset segmentation model, and determine the pointer and instrument frame in the instrument image;
[0035] The correction module is used to determine the angle between the pointer and the instrument frame, and correct the instrument image to be corrected according to the angle.
[0036] Compared with the prior art, the tilt correction method and device for the pointer-type instrument image provided by the present invention segment the instrument frame and the pointer through a preset segmentation model, thereby performing correction. Since the segmentation accuracy and effect of the segmentation model are significantly better than the detection of features such as scale lines, characters, and the center of the dial, it has better robustness, and the instrument image is corrected faster and more accurately, thereby improving the accuracy of the automatic reading of the instrument, and can well meet the requirements of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic flow chart of a method for correcting the tilt of a pointer instrument image provided by an embodiment of the present invention;
[0038] Figure 2 is a schematic diagram of an instrument image to be calibrated in an embodiment of the present invention;
[0039] Figure 3 This is a schematic diagram of the output results of the model in an embodiment of the present invention;
[0040] Figure 4 is a schematic diagram of a circumscribed rectangular frame of an instrument image in an embodiment of the present invention;
[0041] Figure 5 A schematic diagram of a device for correcting the tilt of a pointer instrument image provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0043] like Figure 1 As shown, the embodiment of the present invention first provides a method for tilt correction of a pointer instrument image. The tilt correction method 100 at least includes the following steps:
[0044] 101: Acquire an instrument image to be calibrated, and input the instrument image into a preset segmentation model.
[0045] 102: Segment the instrument image to be corrected by using the preset segmentation model, and determine the pointer and instrument frame in the instrument image.
[0046] 103: Determine an angle between the pointer and the instrument frame, and calibrate the instrument image to be calibrated according to the angle.
[0047] It should be noted that the execution subject of the tilt correction method 100 can be a device with computing functions, such as a server, a computer, etc. The following description is made by taking a server as an example. The embodiment of the present invention can more effectively and accurately correct the photographed image of the pointer instrument, improve the accuracy of the automatic reading of the pointer instrument, and provide a better solution for the actual application of the automatic reading of the pointer instrument.
[0048] The following is a detailed description of the above steps:
[0049] 101: Acquire an instrument image to be calibrated, and input the instrument image into a preset segmentation model.
[0050] The instrument image to be corrected refers to the image of the instrument with an inclined angle. Figure 2 As shown, in the image, there is an instrument 203 with an inclined angle, a pointer 201 and an instrument frame 202, and the instrument frame 202 is below the pointer 201. The instrument refers to an instrument with a pointer, that is, a pointer-type instrument. The instrument image to be corrected may be caused by a problem of shooting angle, in which the instrument has a certain inclination angle, or the position where the instrument entity is placed is inclined, resulting in the need to correct the instrument image.
[0051] The instrument image may be obtained by sending it to a server from a computer, a mobile phone or other device, or by obtaining it from other servers or other platforms.
[0052] After the server obtains the instrument image, it inputs it into a preset segmentation model (ie, preset segmentation model).
[0053] The specific training method of the segmentation model includes the following steps:
[0054] Acquire training samples corresponding to the instrument image, wherein the training samples are instruments with different tilt angles in the instrument image;
[0055] Based on the instrument image in the training sample, annotate the area corresponding to the pointer and the area of the instrument frame in the image to generate a label image; wherein the area of the instrument frame is located below the area of the pointer;
[0056] The initial neural network model is trained according to the training samples and the label images to obtain a trained model as the preset segmentation model.
[0057] In an embodiment of the present invention, a plurality of instrument images are collected for training a model. Each instrument image is an image of an instrument with a different tilt angle. Each collected instrument image is used as a training sample (the training sample composition may be a training set). The area of the pointer and the table frame in the pointer-type instrument image in each training sample is annotated, and the area may be annotated by coordinates, thereby constructing a corresponding training label image, i.e., a label image, to form a training label set. The training labels in the training label set correspond one-to-one to the pointer-type instrument images in the training set.
[0058] In an embodiment of the present invention, the initial neural network model may be an initial FCN (Fully Convolutional Networks) model, etc., and any initial model that can implement semantic segmentation may be used.
[0059] The instrument image in each training sample is used as the input of the initial model, and the labeled image with region annotation corresponding to the training sample in the training label set is used as the reference of the output of the initial model to complete the pre-training of the semantic segmentation model.
[0060] Specifically, the initial neural network model is trained according to the training samples and the label images, including the following steps: the instrument image in the training sample is input into the initial neural network model to obtain a corresponding training output result, wherein the training output result includes the area of the pointer and the area of the instrument frame in the corresponding input instrument image; the initial neural network model is trained according to the training output result and the label image.
[0061] For example, the server inputs the instrument image in the training sample into the initial FCN model to obtain an initial training output result, which marks the area of the pointer and the area of the instrument frame in the image by coordinates. Then it is compared with the corresponding label image to determine the loss function value, and then the training parameters in the FCN are iteratively trained to finally determine the FCN model and complete the model training.
[0062] During the training process, the instrument image in the training sample is input into the initial neural network model to obtain the corresponding training output result, including the following steps: determining the training feature data of the pixels in the instrument image through the initial neural network model, and classifying the pixels according to the training feature data, and obtaining the area of the pointer and the area of the instrument frame in the input instrument image according to the training classification result.
[0063] The training feature data may refer to a feature vector.
[0064] For example, the initial FCN model receives the instrument image in the training sample, and then can obtain the feature vector of the pixel according to the convolutional network structure inside the model, divide the image into blocks, and then determine the feature vector of the pixel in the block. Then, according to the feature vector, the pixels in each block are classified to determine the pixel points belonging to the pointer class and the instrument frame class, thereby determining the coordinates of the corresponding pointer and the coordinates of the instrument frame, and thus determining the corresponding differences.
[0065] 102: Segment the instrument image to be corrected by using the preset segmentation model, and determine the pointer and instrument frame in the instrument image.
[0066] For example, after the server inputs the instrument image to be corrected into the preset segmentation model, the model performs semantic segmentation on the image, and the output result of the model can be obtained, that is, the area of the pointer and the area of the instrument frame are marked on the instrument image, and the coordinates corresponding to the two areas can also be determined. Figure 3 As shown, a pointer area 301 is marked and an instrument frame area 302 is marked, and the instrument frame area 302 is below the pointer area 301 .
[0067] Specifically, the preset segmentation model is used to segment the instrument image to be corrected, and the pointer and the instrument frame in the instrument image are determined, including the following steps: characteristic data of pixels in the instrument image to be corrected are determined according to the preset segmentation model, and the pixels are classified according to the characteristic data, and the area of the pointer and the area of the instrument frame in the instrument image to be corrected are determined according to the classification result.
[0068] For example, when the preset segmentation model performs semantic segmentation on the image, the feature vector of the pixels in the instrument image can be obtained according to the convolutional network structure, which can be the feature vector of the pixels in the image block. Then, according to the feature vector, the pixels are classified to determine the set of pixels belonging to the pointer and the set of pixels belonging to the instrument frame, thereby determining the corresponding pointer area and instrument frame area, as well as the corresponding coordinates.
[0069] 103: Determine an angle between the pointer and the instrument frame, and calibrate the instrument image to be calibrated according to the angle.
[0070] Specifically, determining the angle between the pointer and the instrument frame includes: determining a rotation angle of an area of the pointer relative to an area of the instrument frame.
[0071] For example, the server determines the coordinates of a line representing the pointer according to the coordinates corresponding to the area passing through the pointer, and determines the coordinates of a line representing the area of the instrument frame, such as taking the bottom edge of the instrument frame as the line to determine the corresponding line coordinates. The rotation angle between the two line coordinates is determined.
[0072] In addition, determining the rotation angle of the pointer area relative to the area of the instrument frame may also include the following steps: determining a first minimum circumscribed rectangular frame of the pointer area and a second minimum circumscribed rectangular frame of the instrument frame area; determining a first rotation angle of the first minimum circumscribed rectangular frame relative to a preset reference position, and determining a second rotation angle of the second minimum circumscribed rectangular frame relative to a preset reference position; and determining the rotation angle of the pointer area relative to the instrument frame area based on the first rotation angle and the second rotation angle.
[0073] The preset reference position may refer to a horizontal position where the circumscribed rectangular frame is at an angle of 0 degrees.
[0074] For example, the server can call the contour detection and minimum rectangular frame function from the opencv image processing library to determine the first minimum bounding rectangle of the pointer area and the second minimum bounding rectangle of the instrument frame area. Then, the rotation angle of the minimum bounding rectangle of the pointer and the instrument frame relative to the horizontal position is determined. Figure 4 As shown, the rotation angle corresponding to the minimum circumscribed rectangular frame 401 of the pointer can be 124.99201965332031 degrees, and the instrument frame can be itself, and the corresponding rotation angle can be 1.01297705078125. The actual rotation angle of the pointer is determined, that is, the final rotation angle is: 124.99201965332031-1.01297705078125=123.97804260252106 degrees.
[0075] Specifically, the instrument image to be corrected is corrected according to the angle, including the following steps: taking the rotation angle of the pointer area relative to the instrument frame area as the final rotation angle of the pointer in the instrument image to be corrected; and adjusting the instrument image to be corrected according to the final rotation angle.
[0076] For example, after the final rotation angle of the pointer is determined, the pointer can be adjusted to the instrument image to be calibrated according to this angle, such as adjusting the first minimum circumscribed rectangular frame of the pointer area to the final rotation angle, and using this as a reference to drive the image to rotate together. Alternatively, the image can be directly rotated to 0 degrees at the first rotation angle of the instrument frame area.
[0077] In addition, the tilt correction method 100 may further include the following step: determining the value pointed to by the pointer in the instrument image according to the final rotation angle.
[0078] After adjusting the instrument image to be calibrated, the instrument image can be automatically read to determine the value pointed to by the pointer.
[0079] In an embodiment of the present invention, a segmentation model based on a deep neural network is used to segment the area of the pointer and the meter frame, and then image processing technology is used to detect its contour and obtain the rotation angle of the minimum circumscribed rectangular frame, and then the pointer and the image are corrected based on the rotation angle of the instrument frame. Since the detection of the pointer area and the meter frame area both belong to large target detection, and the features are significantly different from other areas of the instrument image, the accuracy and effect of the segmentation model in processing these two detection categories are significantly better than the detection of features such as scale lines, characters, and the center of the dial. The tilt correction method for the pointer instrument provided in the embodiment of the present invention has better robustness, faster correction speed, and higher accuracy, and can well meet the requirements of practical applications.
[0080] Furthermore, an embodiment of the present invention also provides a device for correcting the tilt of a pointer-type instrument image, which can be applied to smart devices such as servers or computers. Figure 5 As shown, the tilt correction device 500 at least includes:
[0081] An acquisition module 501 is used to acquire an instrument image to be calibrated and input the instrument image into a preset segmentation model;
[0082] A determination module 502 is used to segment the instrument image to be corrected by using the preset segmentation model, and determine the pointer and the instrument frame in the instrument image;
[0083] The correction module 503 is used to determine the angle between the pointer and the instrument frame, and correct the instrument image to be corrected according to the angle.
[0084] In addition, the acquisition module 501 is further used to acquire training samples corresponding to the instrument image, where the training samples are instruments with different tilt angles in the instrument image.
[0085] The tilt correction device 500 may also include: a generation module, which is used to mark the area corresponding to the pointer and the area of the instrument frame in the image based on the instrument image in the training sample, and generate a label image, wherein the area of the instrument frame is located below the area of the pointer; a training module, which is used to train the initial neural network model according to the training sample and the label image to obtain a trained model as the preset segmentation model.
[0086] Specifically, the training module may include: an input unit, used to input the instrument image in the training sample into the initial neural network model to obtain the corresponding training output result, and the training output result includes the area of the pointer and the area of the instrument frame in the corresponding input instrument image; a training unit, used to train the initial neural network model according to the training output result and the label image.
[0087] In an embodiment of the present invention, the input unit is used to determine the training feature data of pixels in the instrument image through the initial neural network model, and classify the pixels according to the training feature data, and obtain the area of the pointer and the area of the instrument frame in the input instrument image according to the training classification results.
[0088] In an embodiment of the present invention, the determination module 502 is used to determine the feature data of the pixels in the instrument image to be corrected according to the preset segmentation model, classify the pixels according to the feature data, and determine the area of the pointer and the area of the instrument frame in the instrument image to be corrected according to the classification result.
[0089] In the embodiment of the present invention, the correction module 503 is used to determine the rotation angle of the pointer area relative to the area of the instrument frame.
[0090] In an embodiment of the present invention, the correction module 503 includes: a first determination unit, used to determine the first minimum circumscribed rectangular frame of the pointer area and the second minimum circumscribed rectangular frame of the instrument frame area; a second determination unit, used to determine the first rotation angle of the first minimum circumscribed rectangular frame relative to a preset reference position, and determine the second rotation angle of the second minimum circumscribed rectangular frame relative to the preset reference position; a third determination unit, used to determine the rotation angle of the pointer area relative to the instrument frame area based on the first rotation angle and the second rotation angle.
[0091] In an embodiment of the present invention, the correction module 503 may also include: a fourth determination unit, used to use the rotation angle of the pointer area relative to the area of the instrument frame as the final rotation angle of the pointer in the instrument image to be corrected; and an adjustment unit, used to adjust the instrument image to be corrected according to the final rotation angle.
[0092] In addition, the determination module 502 is further used to determine the value pointed to by the pointer in the adjusted instrument image.
[0093] The above is a detailed description of the tilt correction method and device for the pointer instrument image provided by the present invention. For those skilled in the art, any obvious changes made to it without departing from the essential spirit of the present invention will constitute an infringement of the patent right of the present invention and will bear corresponding legal responsibilities.
Claims
1. A method for correcting the tilt of a pointer instrument image, characterized in that The steps include: Acquire an instrument image to be corrected, and input the instrument image into a preset segmentation model; Segmenting the instrument image to be corrected by using the preset segmentation model to determine the pointer and instrument frame in the instrument image; Determine the angle between the pointer and the instrument frame, and correct the instrument image to be corrected according to the angle; Acquire training samples corresponding to the instrument image, wherein the training samples are instruments with different tilt angles in the instrument image; Based on the instrument image in the training sample, annotate the area corresponding to the pointer and the area of the instrument frame in the image to generate a label image; wherein the area of the instrument frame is located below the area of the pointer; According to the training samples and the label images, an initial neural network model is trained to obtain a trained model as the preset segmentation model; The initial neural network model is trained according to the training samples and the label images, comprising the following sub-steps: Inputting the instrument image in the training sample into the initial neural network model to obtain a corresponding training output result, wherein the training output result includes an area of the pointer and an area of the instrument frame in the corresponding input instrument image; Training the initial neural network model according to the training output results and the label image; The step of inputting the instrument image in the training sample into the initial neural network model to obtain the corresponding training output result includes the following sub-steps: Determine the training feature data of the pixels in the instrument image through the initial neural network model, classify the pixels according to the training feature data, and obtain the area of the pointer and the area of the instrument frame in the input instrument image according to the training classification result; The step of determining the angle between the pointer and the instrument frame comprises the following sub-steps: Determining the rotation angle of the pointer's region relative to the region of the meter frame; The step of determining the rotation angle of the pointer area relative to the instrument frame area includes the following sub-steps: Determine a first minimum bounding rectangular frame of the pointer area and a second minimum bounding rectangular frame of the instrument frame area; Determining a first rotation angle of a first minimum circumscribed rectangular frame relative to a preset reference position, and determining a second rotation angle of a second minimum circumscribed rectangular frame relative to the preset reference position; The rotation angle of the pointer region relative to the instrument frame region is determined according to the first rotation angle and the second rotation angle.
2. The tilt correction method according to claim 1, characterized in that: The method of segmenting the instrument image to be corrected by using the preset segmentation model and determining the pointer and the instrument frame in the instrument image includes the following sub-steps: According to the preset segmentation model, feature data of pixels in the instrument image to be corrected are determined, and pixels are classified according to the feature data. The area of the pointer and the area of the instrument frame in the instrument image to be corrected are determined according to the classification result.
3. The tilt correction method according to claim 1, wherein: Correcting the instrument image to be corrected according to the angle includes the following sub-steps: The rotation angle of the pointer area relative to the area of the instrument frame is used as the final rotation angle of the pointer in the instrument image to be corrected; The instrument image to be corrected is adjusted according to the final rotation angle.
4. The tilt correction method according to claim 3, characterized in that Also includes: Determine the value that the needle in the adjusted instrument image points to.
5. A device for correcting the tilt of a pointer instrument image, characterized in that include: An acquisition module, used for acquiring an instrument image to be calibrated, and inputting the instrument image into a preset segmentation model; A determination module, used to segment the instrument image to be corrected by using the preset segmentation model, and determine the pointer and instrument frame in the instrument image; A correction module, used for determining the angle between the pointer and the instrument frame, and correcting the instrument image to be corrected according to the angle; The acquisition module is also used to acquire training samples corresponding to the instrument image, wherein the training samples are instruments with different tilt angles in the instrument image; The tilt correction device further includes a generation module, which is used to annotate the area corresponding to the pointer and the area of the instrument frame in the image based on the instrument image in the training sample to generate a label image, wherein the area of the instrument frame is located below the area of the pointer; A training module, used to train the initial neural network model according to the training samples and the label images to obtain a trained model as the preset segmentation model; The determining the angle between the pointer and the instrument frame comprises determining a rotation angle of a region of the pointer relative to a region of the instrument frame; The correction module includes: a first determination unit, used to determine the first minimum circumscribed rectangular frame of the pointer area and the second minimum circumscribed rectangular frame of the instrument frame area; a second determination unit, used to determine the first rotation angle of the first minimum circumscribed rectangular frame relative to a preset reference position, and determine the second rotation angle of the second minimum circumscribed rectangular frame relative to the preset reference position; a third determination unit, used to determine the rotation angle of the pointer area relative to the instrument frame area based on the first rotation angle and the second rotation angle.
