Program, information processing apparatus, and information processing method

Abstract

To improve the accuracy of reading numerical values in reading an analog meter.SOLUTION: A program, according to an embodiment, causes a computer to function as a storage unit, an inclination information acquisition unit, a correction unit, a numeric character recognition unit, an indicator detection unit, and a measurement value detection unit. The storage unit stores an inclination recognition model. The inclination information acquisition unit acquires, regarding multiple pieces of numeric character image data extracted from image data obtained by imaging an analog meter, inclination information of each of the pieces of numeric character image data from the inclination recognition model. The correction unit corrects by rotating the numeric character image data on the basis of the inclination information. The numeric character recognition unit recognizes a value of a numeric character from the rotated and corrected numeric character image data. The indicator detection unit detects a position of a tip of an indicator of the analog meter. The measurement value detection unit detects a measurement value indicated by the indicator, on the basis of positions of the numeric characters displayed on a display board of the analog meter and the values indicated by the numeric characters, and the positions of the tip of the indicator.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] Recently, services that assist in facility inspections using tablets have been provided. For example, a service that assists in the needle inspection of meters indicating the state of facilities is known. In this type of service, software that processes image data obtained by photographing a meter and converts it into a numerical value is used to read the meter value. For numerical reading, for example, AI (Artificial Intelligence) is utilized. On the other hand, meter types are roughly classified into numerical meters in which the meter value is directly indicated by a numerical value and analog meters in which the measured value is indicated by the position of a pointer.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are also various types of analog meters, and some have numerical values on the scale plate printed in a substantially concentric circle pattern. Since each of these numerical values has a different inclination, conventionally, it has been difficult to read by needle inspection using image recognition, and a technique for improving the reading accuracy has been desired. Therefore, an object is to provide a program, an information processing apparatus, and an information processing method that improve the accuracy of numerical reading in needle inspection of analog meters.

Means for Solving the Problem

[0005] According to the embodiment, the program causes a computer to function as a storage unit, an inclination information acquisition unit, a correction unit, a digit recognition unit, a pointer detection unit, and a measurement value detection unit. The storage unit stores an inclination recognition model generated from learning data regarding the inclination of digits in digit image data related to images of digits of an analog meter. The inclination information acquisition unit acquires the inclination information of each digit image data from the inclination recognition model for a plurality of digit image data extracted from the image data obtained by photographing the analog meter. The correction unit performs rotational correction on the digit image data based on the inclination information. The digit recognition unit recognizes the value of the digit from the rotationally corrected digit image data. The pointer detection unit detects the position of the tip of the pointer of the analog meter. The measurement value detection unit detects the measurement value indicated by the pointer from the positions of a plurality of digits represented on the display board of the analog meter, the values indicated by the digits, and the position of the tip of the pointer.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

[0007] Hereinafter, an embodiment will be described with reference to the drawings. (Configuration) FIG. 1 is a diagram showing an example of a cloud system that provides an equipment inspection support service according to an embodiment. Hereinafter, application examples to the needle inspection work of meters used in factories, office equipment, or various measuring instruments will be described.

[0008] In FIG. 1, the cloud system has a server 400 connected to a network 500 as the core. A client terminal 300 can access the server 400 via the network 500. Further, a mobile terminal 100 can access the server 400 via a base station 200 of the network 500.

[0009] The mobile terminal 100 is a smartphone, tablet, notebook computer, etc., and is carried by a meter reader who performs the meter reading operation. The mobile terminal 100 is used in a state where the form creation support app according to the embodiment is installed. The mobile terminal 100 optically reads the measured values of the meters MT1 to MTn installed at the site, creates form data by the form creation support app, and transmits it to the server 400.

[0010] The base station 200 communicably connects the mobile terminal 100 and the network 500. If the network 500 is a LAN (Local Area Network), the base station 200 is an access point that implements a wireless LAN (IEEE802.11 series). If the network 500 is a public or local mobile communication network, the base station 200 is a wireless base station such as 3G, LTE (registered trademark), 4G, 5G, etc.

[0011] The client terminal 300 is an information processing device such as a desktop or laptop personal computer. The client terminal 300 is installed, for example, in the management department of a meter reading service vendor and is operated by an operator in that department. The client terminal 300 has functions such as acquiring, referring to, browsing, editing, aggregating, data converting / file creating, and issuing forms for the data files uploaded from the mobile terminal 100 to the server 400.

[0012] The server 400 provides services such as storing, managing, data processing of the data uploaded from the mobile terminal 100 and the client terminal 300, or data sharing among users.

[0013] The meters MT1 to MTn are related to various facilities and measuring instruments, and display, for example, measured values such as current, voltage, pressure and flow rate of gas or liquid, and values indicating the operating state of facilities and equipment. There are two types of meters MT1 to MTn: numerical meters and analog meters, and both types can be mixed as the target of meter reading by the meter reader.

[0014] The numerical meter is of a rotary type or a digital type, and visually displays measured values and the like. The digital numerical meter uses, for example, a plurality of 7-segment displays on which numbers from 0 to 9 are displayed to indicate a numerical value. The rotary numerical meter includes, for example, a mechanism in which a plurality of number wheels with arithmetic numbers from 0 to 9 are arranged in series, and the number wheels are rotated to indicate a numerical value.

[0015] The analog meter indicates a numerical value by the position of a pointer that moves on a display board on which arithmetic numbers and scales are marked. That is, the numerical value such as the measured value is analogically indicated by the relative positional relationship between the numbers, scales, and the pointer.

[0016] FIG. 2 is an external view showing an example of a rectangular meter. For example, in many AC ammeters, the scale values on the display board are drawn concentrically. In this type of meter, the scale values on the scale plate are often marked concentrically. In this case, the numerical values on the dial plate not only include a mixture of horizontal and non-horizontal ones, but also the inclination of each numerical value is different, making it difficult to read by machine. In addition, there are also cases where the width of the scale is interrupted or uneven. In the embodiment, a technology capable of coping with such a situation will be described. Hereinafter, the numbers written on the display board as the reading values of the meter will be referred to as written numbers for explanation.

[0017] FIG. 3 is a functional block diagram showing an example of the mobile terminal 100. The mobile terminal 100 as an example of an information processing apparatus includes a communication unit 101, an input unit 102, a display unit 103, a camera 104, a GNSS (Global Navigation Satellite System) reception unit 105, a storage unit 106, and a processor 110. That is, the mobile terminal is a computer including a processor and a memory.

[0018] The communication unit 101 is a wireless communication interface, establishes a wireless communication link with the base station 200, and communicates with the server 400 via the network 500.

[0019] The input unit 102 includes input devices such as the touch panel of the display unit 103 and the key switches provided on the exterior of the mobile terminal 100, and receives input operations and instructions from the needle inspector. The display unit 103 includes a display device such as a touch panel, and provides visual information to the needle inspector. The touch panel receives information input by a stylus or a finger. There are various types of touch panels, such as capacitive, resistive film, and projection infrared. Examples of display devices include liquid crystal panels, organic EL (Electro Luminescence) panels, and electronic paper.

[0020] The display unit 103 displays, for example, information input fields, soft keys, and various images (photos, CG (Computer Graphics) images) on the display device. The display related to the information input fields and soft keys in the display unit 103 is controlled by the processor 110 so as to correspond to the operations on the input unit 102.

[0021] The camera 104 is a digital camera that generates digital image data. The camera 104, for example, photographs the meters MT1 to MTn (FIG. 1) and acquires image data including data such as the image of the indicated numbers. That is, the camera 104 includes an optical system including a lens, an imaging unit including an image sensor such as CMOS (Complementary MOS), and a signal processing unit that generates image data in a predetermined format (for example, JPEG (Joint Photographic Experts Group)) from the imaging signal from the imaging unit. The processor 110 may add metadata (for example, Exif (Exchangeable image file format) data) indicating the shooting date and time, location, and shooting conditions (shutter speed, aperture value, angle of view, latitude and longitude, etc.) to the image data.

[0022] The GNSS receiver unit 105 receives positioning signals transmitted from a plurality of GNSS satellites typified by GPS (Global Positioning System) satellites and ground reference stations, etc., and measures the position of the mobile terminal 100. The positioning signal also includes time information, and the current time can also be known using this.

[0023] The storage unit 106 is a recording device such as a flash memory like RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), or HDD (Hard Disk Drive). Different types of recording devices may be provided in combination according to the characteristics of the data to be handled.

[0024] The storage unit 106 stores, for example, data generated along with the operation of the processor 110, various parameters, data input from the meter reader, image data acquired by the camera 104 (for example, image data obtained by photographing the measured values of the meters MT1 to MTn), data acquired (downloaded) from the server 400, or temporary data for information processing, etc. In the embodiment, the storage unit 106 stores the form data 106a, the setting information data 106b, the image data 106c, the learned model 600, and the program 700.

[0025] The form data 106a is created by aggregating the pointer values of each meter for each predetermined meter reading period. As soon as new form data 106a is created, it is immediately uploaded to the server 400. The form data 106a is, for example, data created for each meter reading site and includes information such as "meter information", "meter image", "current meter reading value", "difference from the previous time", "previous meter reading value", "difference obtained previously". When a plurality of meters are installed at the meter reading site, the form data 106a includes the above information for each meter.

[0026] Here, the "meter information" includes a master code for identifying the meter, identification information indicating the object measured by the meter (such as a tenant, a parking lot, a shared space, etc.), and a type (such as gas, electricity, water, etc.). The "meter image" is an image that serves as the basis for the meter reading value, that is, an image taken during the meter reading.

[0027] The "current meter reading value" is the value read in the current meter reading. The "difference from the previous time" is the difference between the current meter reading value and the previous meter reading value. The "previous meter reading value" is the value read in the previous meter reading. The "difference obtained in the previous time" is the difference between the previous meter reading value and the meter reading value of the time before the previous time.

[0028] The setting information data 106b is setting information necessary for aggregating the meter reading values of each meter. The setting information data 106b is created for each meter at each meter reading site, similar to the form data 106a.

[0029] The image data 106c is image data generated by photographing the meter indicating the meter reading value with the camera 104, and is recorded in the form data 106a as an evidentiary meter image. At that time, only the necessary part including the numerical value may be selected and cut out from the image immediately after shooting.

[0030] The learned model 600 is generated by machine learning a neural network such as a DNN (Deep Neural Network) or a CNN (Convolutional Neural Network). In the embodiment, learning data including digital image data including an image of a numerical value and the slope of the numerical value shown in the digital image data is repeatedly input to the neural network, and the learned model 600 is generated by machine learning that minimizes the error between the output slope and the correct value by the gradient descent method or the like.

[0031] Here, the digital image data is image data obtained by cutting out the minimum necessary area from the image data. Although unprocessed image data may be used as the digital image data, it is considered that the learning efficiency will not increase.

[0032] FIG. 4 is a diagram for explaining the learned model 600. The learned model 600 as a tilt recognition model is generated, for example, in the server 400. Then, when the form creation support application installed in the mobile terminal 100 is launched for the first time, the learned model 600 is downloaded from the server 400 and stored in the storage unit 106.

[0033] As shown in FIG. 4(a), the learned model 600 in FIG. 4(b) is generated by repeatedly providing teacher data, which is a set of digital image data including numerical values and the tilt of the numerical values, to an unlearned neural network and stacking the procedure of feeding back the error. For example, teacher data with an image including the numerical value "3878" and a tilt of 4 degrees, or teacher data with an image including the numerical value "10" and a tilt of 18 degrees, etc. are prepared in advance and repeatedly provided to a CNN, DNN, etc. for learning. When image data including numerical values is separately provided to the learned model 600 obtained by such machine learning, the tilt (tilt information) of the image numerical values, rather than the numerical values themselves, is output. Here, the teacher data may be created by an existing framework in, for example, the server 400 or another computer, or image data collected by the checker on-site may be used as the teacher data.

[0034] Returning to FIG. 3 to continue the explanation. The program 700 in the storage unit 106 includes an OS (Operating System), firmware, application software, etc., and causes the mobile terminal 100 to execute various functions according to the embodiment.

[0035] The processor 110 includes processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), a chipset, a RAM, a ROM, etc., and controls each part of the mobile terminal 100. The ROM stores firmware, set values (various parameters), etc. The processor 110 reads the program 700 in the storage unit 106 into the RAM, and also uses the RAM as a work area (working area) to realize various functions.

[0036] The processor 110 executes the instructions included in the program 700 to realize at least the following functions. That is, the processor 110 functions as a communication control unit 111, an input control unit 112, a display control unit 113, an image processing unit 114, a position information processing unit 115, and a stitch data aggregation processing unit 116. Here, the program 700 includes instructions for causing the mobile terminal 100 as a computer to function as the storage unit 106, the communication control unit 111, the input control unit 112, the display control unit 113, the image processing unit 114, the position information processing unit 115, and the stitch data aggregation processing unit 116.

[0037] The communication control unit 111 controls the communication between the mobile terminal 100 and the base station 200, the network 500, and the server 400 according to a predetermined communication protocol. The communication control unit 111 performs, for example, downloading of files and data from the server 400 and uploading of data to the server 400.

[0038] The input control unit 112 interprets the requirements of the stitcher from the operations on the input unit 102, and accepts the input of information (such as a character string), the capture of data, the change of the association between data, etc. In addition, the input control unit 112 acquires the image data generated by the camera 104, transfers it to the storage unit 106, and stores it as the image data 106c.

[0039] The display control unit 113 displays various types of information including characters and images, various icons, a GUI (Graphical User Interface) such as an information input field and soft keys, etc. on the display unit 103. Further, the display control unit 113 performs a guide display indicating the image capture position.

[0040] The image processing unit 114 performs image processing on the image data captured by the camera 104. The image processing unit 114 has, for example, a function as a recognition engine and a function as an analysis engine.

[0041] The recognition engine recognizes various types of information from an image based on the image data acquired by the camera 104, for example, by pattern recognition applying AI (Artificial Intelligence). The recognition engine has, for example, a function of detecting a QR code (registered trademark), distinguishing (determining) whether a meter shown in the image is a numerical meter or an analog meter, detecting a portion on the display panel of the meter where the measured value is displayed, and also detecting an object (a numerical sequence of the measured value of the numerical meter, a pointer or a notation number of the analog meter, etc.) and its arrangement (coordinates) within the display panel.

[0042] The analysis engine performs a process of analyzing the information detected by the recognition engine. The analysis engine, for example, for an image of the meter portion, decodes a QR code (registered trademark), converts the measured value of the numerical meter into text by character recognition processing (OCR: Optical Character Recognition) such as optical character recognition, or detects the value indicated by the pointer from the positional relationship of the objects described in the meter portion of the analog meter.

[0043] Regarding the character recognition process, multiple types of analysis engines are prepared in advance. Each analysis engine corresponds to, for example, each type of meter, and uses an analysis algorithm suitable for the notation numbers of the corresponding meter to convert the displayed measurement value into text. That is, since the fonts of arithmetic numbers vary depending on the meter, and the combinations of notation numbers and background colors are different, analysis engines with analysis algorithms suitable for the display of each meter are prepared.

[0044] The position information processing unit 115 is equipped with an acceleration sensor, a gyro sensor, etc. Based on the detection results of these, the positioning results by the GNSS receiving unit 105, and / or the positioning information obtained from the base station 200 by the communication control unit 111, the position of the mobile terminal 100 is measured. The needle counting data aggregation processing unit 116 comprehensively controls various processes related to the aggregation of needle counting data.

[0045] FIG. 5 is a functional block diagram for explaining the functions provided in the image processing unit 114. The image processing unit 114 includes a classification unit 114a, a correction unit 114b, and a notation number detection unit 114c. That is, the program 700 includes instructions for causing the mobile terminal 100 to function as the classification unit 114a, the correction unit 114b, and the notation number detection unit 114c.

[0046] The classification unit 114a provides the image data 106c including the notation number data to the learned model 600 to obtain the inclination information of the image data 106c. The correction unit 114b performs rotation correction on the image data 106c based on the inclination information. The notation number detection unit 114c detects the value and position of the notation number based on the rotation-corrected image data 106c.

[0047] FIG. 6 is a diagram for explaining the operations of the classification unit 114a, the correction unit 114b, and the notation digit detection unit 114c. The classification unit 114a provides the image data 106c to the learned model 600 and obtains the inclination information for each image data. For example, the inclination of the image data including the notation digit "0.04" is 0 degrees, the inclination of the image data including "45" is 52 degrees, and the inclination of the image data including "0" is -45 degrees, and such inclination information is obtained. This inclination information is input to the correction unit 114b in a set with each image data.

[0048] The correction unit 114b rotationally corrects each image data 106c so as to cancel the inclination information and make it upright, that is, so that the angle becomes 0. The rotational correction of the image can be performed by an existing technique using, for example, open source software or libraries.

[0049] The notation digit detection unit 114c detects the value and position of the notation digit based on the rotationally corrected image data 106c. In FIG. 6, the values "0.04", "45", and "0" are obtained respectively.

[0050] FIG. 7 is a functional block diagram for explaining the functions provided in the needling data totaling processing unit 116. The needling data totaling processing unit 116 includes a form creation unit 116a, a numerical meter needling unit 116b, an analog meter needling unit 116c, and a correction processing unit 116d.

[0051] The form creation unit 116a creates form data based on the needling values read by the numerical meter needling unit 116b or the analog meter needling unit 116c, or the pointer values corrected by the correction processing unit 116d.

[0052] The numerical meter needling unit 116b reads, as pointer values, the numerical strings of the measured values displayed in the numerical data by using the recognition engine and the analysis engine of the image processing unit 114 as needed, based on the image of the numerical data taken.

[0053] The analog meter needle detection unit 116c uses the recognition engine and analysis engine of the image processing unit 114 as needed, and reads the value indicated by the needle on the analog meter as the needle value based on the image of the analog meter taken.

[0054] The correction processing unit 116d receives an instruction to correct the needle value in response to the operation of the needle inspector via the input unit 102, and corrects the needle value read by the numerical value meter needle detection unit 116b or the analog meter needle detection unit 116c.

[0055] FIG. 8 is a functional block diagram for explaining the functions provided in the analog meter needle detection unit 116c. The analog meter needle detection unit 116c includes a needle detection unit 6a, a circle detection unit 6b, an intersection detection unit 6c, an intersection number detection unit 6d, and a measured value detection unit 6e. That is, the program 700 includes instructions for causing the mobile terminal 100 to function as the needle detection unit 6a, the circle detection unit 6b, the intersection detection unit 6c, the intersection number detection unit 6d, and the measured value detection unit 6e.

[0056] The needle detection unit 6a detects the position of the tip of the needle based on the image data 106c obtained by photographing the analog meter. The circle detection unit 6b detects a circle passing through the position of the notation number detected by the notation number detection unit 114c and the position of the center of this circle.

[0057] The intersection detection unit 6c obtains the position of the intersection of the line segment connecting the tip of the needle detected by the needle detection unit 6a and the center of the circle detected by the circle detection unit 6b and the circumference of the circle detected by the circle detection unit 6b. The intersection number detection unit 6d detects a first notation number located on one circumference of the intersection and a second notation number located on the other circumference of the intersection among the notation numbers detected by the notation number detection unit 114c, respectively. The measured value detection unit 6e detects the measured value indicated by the needle based on the position of the first notation number, the position of the second notation number, the position of the center of the circle, the position of the intersection, the value of the first notation number, and the value of the second notation number.

[0058] (Operation) Next, the operation of the cloud system with the above configuration will be described. FIG. 9 and FIG. 10 are flowcharts showing an example of the processing procedure of the mobile terminal 100 according to the embodiment. With reference to FIGS. 9 and 10, the analog meter needling process for reading the needling value from the analog meter will be described.

[0059] In step S301 of FIG. 9, the analog meter needling unit 116c of the mobile terminal 100 reads the setting information data 106b from the storage unit 106 and proceeds to step S302. The read setting information data 106b is used as the parameter (initial setting value) of the initial setting in the subsequent processing.

[0060] As an example of the setting information data 106b, for various analog meters, information such as the minimum value and the maximum value of the numerical value displayed on the display panel associated with the identification information of the meter, or learning data obtained by AI (Artificial Intelligence) learning for identifying the meter is included.

[0061] The learning data is generated by machine learning based on information such as the center point of the meter, the start point and the end point of the movable range of the pointer, the bottom point at the bottom of the display panel, the range of the pointer (minimum value, maximum value), and the threshold value in the image of the analog meter taken.

[0062] In step S302, the analog meter needling unit 116c activates the camera 104, images the analog meter MT (for example, any one of the meters MT1 to MTn), acquires the image data, and the processing procedure proceeds to step S303.

[0063] Specifically, in step S302, the camera 104 is activated, and the video captured by the camera 104 is displayed on the display unit 103. The needle inspector checks this display, adjusts the shooting composition, and performs a shutter operation in a state where the entire analog meter MT is captured. As a result, image data in which the entire analog meter MT is captured is obtained, and the captured analog meter MT is displayed on the display unit 103.

[0064] In step S303, based on the image data obtained in step S302, the analog meter needle inspection unit 116c displays the captured analog meter MT on the display unit 103 and detects the notation numbers and pointers on the display panel of the analog meter MT captured in the image data. Note that the recognition engine of the image processing unit 114 is used for this detection.

[0065] Also in step S303, the analog meter needle inspection unit 116c surrounds the detected notation numbers and pointers with rectangular frame lines (numerical value frames) on the displayed analog meter MT and proceeds to step S304.

[0066] FIG. 11 is a diagram showing an example of the frame lines surrounding the notation numbers. These displays are performed through the display control unit 113, and the same applies to the displays described below. Note that the detection of the notation numbers and pointers on the analog meter MT can be realized, for example, by using AI.

[0067] That is, based on the learning data and image data included in the setting information data 106b, the analog meter needle inspection unit 116c identifies the type of the analog meter MT and further detects the notation numbers and pointers on the display panel from the learning data for the identified analog meter MT.

[0068] In step S304, the analog meter needle detection unit 116c detects the position (coordinates) of the center of the frame line surrounding the notation number displayed in step S303, and proceeds to step S305. The coordinates detected here are referred to as notation number coordinates in the following description. The notation number coordinates are based on the two-dimensional coordinate axes set on the displayed analog meter MT, and the coordinates in the following description are also based on the above coordinate axes.

[0069] In step S305, the analog meter needle detection unit 116c obtains an approximate circle (pseudo circle) passing through the center of the frame line based on the notation number coordinates detected in step S304. That is, the analog meter needle detection unit 116c obtains a pseudo circle in which the notation number coordinates exist on the circumference. Further, the analog meter needle detection unit 116c calculates the coordinates of the center point O of this circle and its radius R, and proceeds to step S306.

[0070] FIG. 12 is a diagram showing an example of a pseudo circle and the center point O. In step S306, the analog meter needle detection unit 116c detects the coordinates Ac of the tip A of the pointer based on the frame line surrounding the pointer and the center point O of the pseudo circle, and the processing procedure proceeds to step S307.

[0071] FIG. 13 is a diagram showing an example of the frame line surrounding the pointer and the center point O of the circle. Further, FIG. 14 shows an example of an image including a dial inclined backward. In the example of FIG. 14, among the four vertices of the rectangular frame line surrounding the pointer, the vertex forming a diagonal with the vertex close to the center point O of the circle is detected as the coordinates Ac of the tip A of the pointer.

[0072] However, depending on the angle of the pointer, as shown in FIG. 15, instead of the vertex b1 which is a point close to the center O of the circle, it may be possible to detect an inappropriate vertex b2. Therefore, the vertex b1 may be detected in consideration of the distances between the midpoints c1 to c4 of each side of the frame line and the center O, and the coordinates of the vertex a corresponding to the tip A of the pointer, which forms a diagonal with this vertex b1, may be detected.

[0073] In step S307, the analog meter needle detection unit 116c arranges the notation digital coordinates detected in step S304 in order, and the processing procedure proceeds to step S308.

[0074] In step S307, specifically, as indicated by the arrow in FIG. 16, starting from the starting point S on the circumference below the center point O of the circle, the analog meter needle detection unit 116c arranges the notation digital coordinates on the circumference in order clockwise. This is in consideration of the fact that the reading value (increases) of a general analog meter changes clockwise.

[0075] Next, in step S501, the image processing unit 114 cuts out the numerical image of the frame line (numerical frame), and in step S502, recognizes the rotation angle of the numerical value in the cut-out numerical image. The rotation angle is output as tilt information from the learned model 600. Using this tilt information, in step S503, the image processing unit 114 performs horizontal correction processing on the cut-out numerical image to generate image data of notation digits without tilt. As a result, as shown in FIG. 17, image data in which all the numerical values on the dial are horizontal is generated.

[0076] In step S308, as shown in FIG. 17, the analog meter needle detection unit 116c reads the numerical value of the notation digit within the frame line surrounded in step S303, and the processing procedure proceeds to step S309.

[0077] The reading (detection) of the numerical value of the notation digit can be realized, for example, by using AI. That is, based on the learning data and image data included in the setting information data 106b, the analog meter needle detection unit 116c can identify the type of the analog meter MT and the like. Furthermore, the numerical value of the notation digit on the display panel can be detected from the learning data for this identified analog meter MT.

[0078] In step S309, the analog meter needle detection unit 116c determines whether the value read in step S308 contains "0". Here, if the value read in step S308 contains "0", the processing procedure proceeds to step S310; if not, the processing procedure proceeds to step S312.

[0079] In step S310, based on the arrangement performed in step S307 and the value read in step S308, the analog meter needle detection unit 116c determines whether there is a value to the left of the numeral representing the value "0" when viewed from the center point O. Here, if there is a value to the left, the processing procedure proceeds to step S311; if not, the processing procedure proceeds to step S314.

[0080] In step S311, for the values to the left of the value "0" among the values read in step S308, the analog meter needle detection unit 116c converts them to negative values of the same magnitude and proceeds to step S314. For the values on the right side, they remain positive values.

[0081] In the case of an analog meter as shown in FIG. 18, the values "0.05" and "0.1" exist to the left of the value "0". Therefore, the analog meter needle detection unit 116c converts these values to "-0.05" and "-0.1" respectively.

[0082] In step S312, based on the arrangement performed in step S307 and the value read in step S308, the analog meter needle detection unit 116c compares two adjacent values and determines whether there is a value that is greater than or equal to the right - hand value. That is, it determines whether there is a value such that the left - hand value is greater than or equal to the right - hand value when viewed from the center point O. If there is a value greater than or equal to the right - hand value, the processing procedure proceeds to step S311; if not, the processing procedure proceeds to step S314.

[0083] In step S313, the analog meter needle inspection unit 116c reverses the sign of the numerical values among the numerical values read in step S308 that are equal to or greater than the numerical value on the right, and the processing procedure proceeds to step S314.

[0084] In the case of an analog meter as shown in FIG. 18, the numerical value "0.05" exists on the right side of the leftmost numerical value "0.1". Therefore, the analog meter needle inspection unit 116c converts this numerical value "0.1" to "-0.1". Similarly, since the numerical value "0" exists on the right side of the left numerical value "0.05", the analog meter needle inspection unit 116c converts this numerical value "0.05" to "-0.05".

[0085] In step S314 of FIG. 9, the analog meter needle inspection unit 116c detects the coordinates Xc of the intersection point X between the line segment connecting the coordinates Ac of the tip A of the pointer detected in step S306 and the coordinates of the center point O of the circle detected in step S305, and the circumference of the circle O obtained in step S305, as shown in FIG. 19, for example. Then, regardless of the presence or absence of the intersection point X, the processing procedure proceeds to step S401 of FIG. 10.

[0086] In step S401 of FIG. 10, the analog meter needle inspection unit 116c determines whether the coordinates Xc of the intersection point X were detected in step S314. If the coordinates Xc of the intersection point X were detected (i.e., there is an intersection point X), the processing procedure proceeds to step S403. If the coordinates Xc of the intersection point X cannot be detected, such as when there is no intersection point X, the processing procedure proceeds to step S402.

[0087] In step S402, the analog meter needle inspection unit 116c extends the pointer to create the intersection point X between the extended pointer and the circumference, and detects the coordinates Xc of the intersection point X. Then, the processing procedure proceeds to step S402. The detection of the coordinates Xc of the intersection point X can be realized, for example, by extending in the direction from the center point O to the coordinates Ac based on the coordinates Ac of the tip A of the pointer detected in step S306 and the center point O.

[0088] In step S403, the analog meter needle detection unit 116c searches for the notation numbers on the circumferences on the left and right of the coordinate Xc of the intersection point X detected in step S314 or step S402, and determines whether or not there are such notation numbers.

[0089] In the example of FIG. 20, “20” exists as the notation number NL on the circumference to the left of the coordinate Xc of the intersection point X, and “40” exists as the notation number NR on the circumference to the right.

[0090] Here, when notation numbers can be detected on both the left and right, the processing procedure proceeds to step S404. When there is a notation number on only one of the left or right, as the pointer value, the minimum value (when only NR is detected) or the maximum value (when only NL is detected) is detected, and the processing procedure proceeds to step S409.

[0091] In FIG. 21(a), similar to FIG. 20, “20” exists as the notation number NL on the circumference to the left of the coordinate Xc of the intersection point X, and “40” exists as the notation number NR on the circumference to the right. Therefore, the processing procedure proceeds to step S404.

[0092] On the other hand, as shown in FIG. 21(b), when there is no notation number on the left side of the intersection point X (up to the starting point of the circumference) as seen from the center point O, it is determined as the detected minimum value based on the initial setting value as the pointer value, and the processing procedure proceeds to step S409.

[0093] Note that when there is no notation number on the right side of the intersection point X (up to the ending point of the circumference), it is determined as the detected maximum value based on the initial setting value as the pointer value, and the processing procedure proceeds to step S409.

[0094] Also, as shown in Fig. 21(c), when the pointer overlaps with the notation number "40" and the notation number "40" is not detected in step S303 or step S304, this notation number "40" will not be detected. Therefore, "40" will not be used in the determination of step S403. However, since the other left and right notation numbers "20" and "60" are detected as NL and NR respectively in step S303 and step S304, after the presence of these notation numbers is detected in step S403, the processing procedure proceeds to step S404.

[0095] Furthermore, as shown in Fig. 21(d), when the pointer overlaps with the notation number "40" and for some reason the notation number "60" is not detected, the notation numbers "40" and "60" will not be used in the determination of step S403. However, since the other left and right notation numbers "20" and "80" are detected in step S303 and step S304, after these notation numbers are detected as NL and NR respectively in step S403, the processing procedure proceeds to step S404.

[0096] As shown in Fig. 22, in step S404, the analog meter needle detection unit 116c first detects the angle S1 formed by the line segment connecting the coordinates of the notation number NL on the left side of the intersection point X and the coordinates of the center point O and the line segment connecting the coordinates of the intersection point X and the coordinates of the center point O among the two notation numbers NL and NR detected in step S403.

[0097] Next, the analog meter needle detection unit 116c detects the angle S2 formed by the line segment connecting the coordinates of the notation number NR on the right side of the intersection point X and the coordinates of the center point O and the line segment connecting the coordinates of the intersection point X and the coordinates of the center point O among the two notation numbers NL and NR detected in step S403, and proceeds to step S405.

[0098] As shown in Fig. 23, in step S405, the analog meter needle detection unit 116c calculates the pointer value VD corresponding to the intersection point X based on the angles S1, S2, the radius R, the numerical value VL of the left notation number NL, and the numerical value VR of the right notation number NR. After that, the processing procedure proceeds to step S406.

[0099] Here, the pointer value VD can be calculated, for example, as VD = (S1 * VR + S2 * VL) / (S1 + S2).

[0100] In step S406, the analog meter needle detection unit 116c detects the maximum number of digits among the digits after the decimal point based on the numerical value of the notation digits read in step S308, and the processing procedure proceeds to step S402.

[0101] For an analog meter as shown in FIG. 24, since the read numerical value is an integer value, the maximum number of digits after the decimal point is detected as 0. For an analog meter as shown in FIG. 25, the read numerical value includes "0.1" and "0.05", and based on "0.05", the maximum number of digits is detected as 2.

[0102] In step S407, the analog meter needle detection unit 116c obtains the value obtained by adding 1 to the maximum number of digits detected in step S406 as the number of significant digits after the decimal point. Then, the analog meter needle detection unit 116c determines the value up to the number of significant digits after the decimal point among the calculation results of step S405 as the pointer value, and the processing procedure proceeds to step S408.

[0103] In the analog meter shown in FIG. 24, in step S405, when the pointer value "30.04567..." is calculated, since the number of significant digits after the decimal point is 1, the pointer value is determined as "30.0" up to the first decimal place. Note that the value of the first decimal place is determined by rounding the value of the second decimal place.

[0104] In step S408, based on the image data obtained in step S302, the analog meter needle detection unit 116c generates data of a trimmed image (hereinafter referred to as trimmed image data) that trims the range including the pointer, the notation numerical values in its vicinity, and the intersection point X in the image of the photographed analog meter MT, and proceeds to step S409.

[0105] In step S409, the correction processing unit 116d displays the trimming image on the display unit 103 based on the trimming image data generated in step S408, instead of the currently displayed image, and also displays on the display unit 103 the pointer value determined in step S403 or the pointer value determined in step S407. Then, the processing procedure proceeds to step S410.

[0106] FIG. 26 is a diagram showing an example of an interface for correcting the detected pointer value. That is, a trimming image in which a range including the intersection point X in the image data is enlarged is displayed, and a drum roll and a slide bar are displayed on the display unit 103 as GUIs functioning as soft keys.

[0107] The drum roll display is an input interface for correcting the pointer value, and virtually displays graphically drum rolls (number wheels) corresponding to the values of each digit of the pointer value. The drum roll can be operated through a touch panel (input unit 102) placed on the display unit 103.

[0108] For example, when the input unit 102 detects a flick input for rotating each drum roll up and down, the image of the drum roll is rotated and displayed in response, and the pointer value can be changed to an arbitrary value displayed accordingly.

[0109] The slide bar display is an input interface for setting a decimal point between the drum rolls, corresponding vertically to the positions of the drum rolls, and by sliding the pointer P left and right, a decimal point can be set between the digits (drum rolls).

[0110] FIG. 27 is a diagram showing an example of a GUI for correcting the position of the decimal point of the detected pointer value. FIGS. 27(a) to (e) show how the position of the decimal point of the pointer value changes according to the position of the pointer P. In FIG. 27, the position of the decimal point of the pointer value and the position of the pointer P are set to coincide in the vertical direction.

[0111] Note that the position of the pointer P and the position of the decimal point in the numerical sequence represented by the drum roll do not necessarily have to coincide vertically. For example, for the display range of the numerical sequence of the pointer value, a slider may be displayed in a wider range than that, and the displayable range on the display unit 103 may be used to the maximum extent for decimal point input. In that case, the position of the decimal point is varied according to the relative position of the pointer P within the movable range of the pointer P.

[0112] Returning to FIG. 10 and continuing the explanation, in step S410, the correction processing unit 116d requests the inspector to input confirmation for the pointer value displayed on the display unit 103. Here, when an OK input operation is performed on the input unit 102, the displayed pointer value is determined as the measured value, and the processing procedure proceeds to step S412. On the other hand, when an NG input operation is performed on the input unit 102, the processing procedure proceeds to step S411.

[0113] In step S411, the correction processing unit 116d detects operations on the drum roll display and the slider display through the input unit 102. The correction processing unit 116d receives a correction instruction for the pointer value from the inspector, determines the corrected (displayed) pointer value as the measured value, and the processing procedure proceeds to step S412.

[0114] In step S412, the form creation unit 116a stores, as form data 106a in the storage unit 106, the image data obtained in step S302, the trimming image data generated in step S408, and the measured value determined in step S410 or step S411, in association with the identification information of the analog meter to be measured. After this, the processing procedure proceeds to step S413.

[0115] In step S413, the analog meter needle detection unit 116c determines whether an instruction to end the work has been given from the needle inspector through the input unit 102. Here, if an instruction to end the work is detected, the series of processing procedures ends. If there is an instruction to continue the work, the processing procedure shifts to step S302, and the needle inspection of the analog meter MT is started again.

[0116] (Effect) As described above, according to the embodiment, a learned model 600 that classifies image data by "slope" instead of "numerical value" shown in the image is created by machine learning and stored in the mobile terminal 100. The mobile terminal 100 provides the image data 106c including the written numerals to the learned model 600 and acquires the slope information.

[0117] As shown in FIG. 28, the mobile terminal 100 cuts out the image data with the recognized numerical value frame to generate digital image data, and inputs the digital image data to the AI-based learned model 600. Then, the rotation angle of each numerical value is recognized by the AI. Then, with the center point of the numerical value frame as the axis, the digital image data is rotated by the angle obtained by the rotation angle recognition, and horizontal correction processing is performed.

[0118] Here, since the numerical value frame in FIG. 28 is a quadrilateral, for example, the intersection of the diagonals of the four vertices can be extracted as the center point of the numerical value frame. In the embodiment, a pseudo circle passing through the center point of the numerical value frame is detected, and this pseudo circle is used for the horizontal correction processing. The pseudo circle is obtained along the arc drawn by the center point of the digital image data.

[0119] FIG. 29 is a diagram for supplementarily explaining a numerical value frame and a pseudo circle. As shown in FIG. 29(a), based on the recognition results of the numerical value frame and the pointer, the coordinates of the center point of the numerical value frame and the coordinates of the center (center point) of the pseudo circle are detected. Further, a line segment connecting the center of the pseudo circle and the center point of the numerical value frame is detected. Then, the angle between this line segment and the side (for example, the bottom side of the numerical value frame) of the four sides of the numerical value frame that intersects the line segment is detected, and by performing rotational correction so as to cancel this angle, as shown in FIG. 29(b), image data in a state where the numerical value is horizontal can be obtained. Then, from the division ratio of the pseudo circle by the pointer of the meter in the pseudo circle, the measured value indicated by the pointer can be calculated.

[0120] Note that geometrically, it can be said that three points are necessary and sufficient to set a pseudo circle. If there are three center points of the numerical value frame, a pseudo circle passing through these three points is uniquely determined. On the other hand, when there are four or more center points of the numerical value frame, for example, the least squares method can be applied using the coordinates of each center point to calculate a unique pseudo circle.

[0121] As described above, according to the embodiment, the obliquely written numbers in the captured image data are corrected to be horizontal. Therefore, since it is only necessary to read the numerical value in the horizontal state, the accuracy of numerical value reading can be dramatically improved compared to directly applying the image data including the numerical value in the oblique state to the numerical value recognition process.

[0122] That is, in the embodiment, instead of directly classifying the image data into numerical values, the image data is first classified into tilt information and then the image data is corrected to be horizontal. Then, image recognition processing is performed using the horizontal image data to read the numerical value. Therefore, according to the embodiment, it is possible to provide a program, an information processing apparatus, and an information processing method that improve the accuracy of numerical value reading in the needle inspection of an analog meter. As a result, the human load related to the needle inspection work can be reduced.

[0123] Note that the present invention is not limited to the above-described embodiments. For example, the application examples are not limited to factories, office facilities, or various measuring instruments. For example, the meters MT1 to MTn may indicate not only measured values such as current, voltage, pressure, and flow rate of gas or liquid, and the operating states of facilities and devices, but also other physical quantities and states.

[0124] Also, the air interface between the mobile terminal 100 and the base station 200 in FIG. 1 is not limited to wireless LAN, 4G, 5G, etc. In short, any wireless communication method may be applied as long as the mobile terminal 100 can be connected to the network 500 via a wireless link.

[0125] In addition, part or all of each function included in the processor 110 in FIG. 2 may be integrated with other functions, or each function may be divided into a plurality of functional blocks from another perspective and described in a different expression. Also, the neural network in FIG. 4 is not limited to CNN or DNN, and neural networks with different formats such as SVM (Support Vector Machine) can be appropriately used.

[0126] Also, the inspector may instruct from the input unit 102 whether to inspect numerical data or analog data. Alternatively, the camera 104 may capture the meter to be inspected, and the recognition engine may distinguish (determine) whether the type of the meter shown in the image is numerical data or analog data.

[0127] In the above embodiment, the case where the mobile terminal 100 performs the shooting of the analog meter, the detection of the measured value of the analog meter, and the creation of the form data 106a recording the detected value has been described as an example, but the present invention is not limited thereto. For example, only the shooting of the analog meter may be performed by the mobile terminal 100, and the detection of the measured value of the analog meter and the creation of the form data 106a recording the detected value may be realized by information processing in the client terminal 300 or the server 400.

[0128] That is, all or some of the functions of the mobile terminal 100 may be implemented on the client terminal 300 or the server 400. For example, an image acquired by the mobile terminal 100 may be transferred to the client terminal 300, and processes such as image processing, angle recognition, rotation correction, and numerical recognition may be performed on the client terminal 300. It is possible to flexibly determine how much processing is shared between the edge (mobile terminal 100) and the cloud (client terminal 300, server 400) according to the system requirements.

[0129] Furthermore, the services provided by the server 400 may be provided in a subscription manner to the mobile terminal 100 or the client terminal 300 owned (or leased) by the client receiving the service.

[0130] Furthermore, as described with reference to FIG. 29, it is also possible to generate image data in a horizontal state by detecting the angle formed by the center of the pseudo circle and the numerical frame through image processing and performing rotation correction to cancel this angle. According to such a procedure, it is possible to perform horizontal correction processing without relying on the learned model 600.

[0131] Although the embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0132] 6a…Pointer detection unit, 6b…Circle detection unit, 6c…Intersection detection unit, 6d…Intersection number detection unit, 6e…Measured value detection unit, 100…Mobile terminal, 101…Communication unit, 102…Input unit, 103…Display unit, 104…Camera, 105…GNSS receiver, 106…Memory unit, 106a…Form data, 106b…Setting information data, 106c…Image data, 110…Processor, 111…Communication control unit, 112…Input control unit, 113…Display control unit, 114…Image processing unit, 114a…Classification unit, 114b…Correction unit, 114c…Notation number detection unit, 115…Position information processing unit, 116…Needle inspection data aggregation processing unit, 116a…Form creation unit, 116b…Numerical data needle inspection unit, 116c…Analog meter needle inspection unit, 116d…Correction processing unit, 200…Base station, 300…Client terminal, 400…Server, 500…Network, 600…Trained model, 700…Program, MT1~MTn…Meters.

Claims

1. A computer, a storage unit that stores a tilt recognition model generated from learning data regarding the tilt of the numerals in digital image data related to digital images of the numerals of an analog meter; a tilt information acquisition unit that acquires the tilt information of each of the digital image data from the tilt recognition model for a plurality of the digital image data extracted from the image data obtained by photographing the analog meter; a correction unit that extracts the center point of the digital image data based on the tilt information, rotates the digital image data about the center point to correct the tilt of the digital image data, and performs rotational correction; a numeral recognition unit that recognizes the value of the numeral from the digitally image data after rotational correction; a pointer detection unit that detects the position of the tip of the pointer of the analog meter; a measured value detection unit that detects the measured value indicated by the pointer from the positions of a plurality of numerals represented on the display board of the analog meter, the values indicated by the numerals, and the position of the tip of the pointer; a circle detection unit that detects a virtual circle passing through the position of the written numeral of the digital image data and the position of the center of the virtual circle; A program for causing the computer to function as such.

2. The circle detection unit detects the center point of rotation related to the pointer of the analog meter and the center points of the plurality of digital image data, obtains a virtual circle centered on the center point of rotation related to the pointer and having the center points of the digital image data on an arc, The measured value detection unit calculates the measured value indicated by the pointer from the division ratio of the virtual circle by the pointer in the virtual circle. The program according to claim 1.

3. An information processing apparatus having a storage unit and a processor, wherein the storage unit stores a tilt recognition model generated from learning data regarding the tilt of the numerals in digital image data related to digital images of the numerals of an analog meter, and the processor is a tilt information acquisition unit that acquires the tilt information of each of the digital image data from the tilt recognition model for a plurality of the digital image data extracted from the image data obtained by photographing the analog meter; is a correction unit that extracts the center point of the digital image data based on the tilt information, rotates the digital image data about the center point to correct the tilt of the digital image data, and performs rotational correction; is a numeral recognition unit that recognizes the value of the numeral from the digitally image data after rotational correction; is a pointer detection unit that detects the position of the tip of the pointer of the analog meter; A measurement value detection unit that detects a measurement value indicated by the pointer from the positions of a plurality of numbers represented on the display board of the analog meter, the values indicated by the numbers, and the position of the tip of the pointer; A circle detection unit that detects a virtual circle passing through the position of the written number in the digital image data and the position of the center of the virtual circle; An information processing apparatus comprising the above. **Claim 4**: The circle detection unit detects the rotation center point related to the pointer of the analog meter and the center points of the plurality of digital image data; obtains a virtual circle centered on the rotation center point related to the pointer and having the center points of the digital image data on an arc; The measurement value detection unit calculates a measurement value indicated by the pointer from the division ratio of the virtual circle by the pointer in the virtual circle. The information processing apparatus according to claim 3. **Claim 5**: A computer For a plurality of digital image data related to an image of a number extracted from image data obtained by photographing an analog meter, a classification step of obtaining inclination information of each of the digital image data from an inclination recognition model generated from learning data regarding the inclination of the number in the digital image data; A correction step of extracting the center point of the digital image data based on the inclination information, rotating the digital image data around the center point to correct the inclination of the digital image data, and performing rotation correction; A number recognition step of recognizing the value of the number from the digitally image data after rotation correction; A pointer detection step of detecting the position of the tip of the pointer of the analog meter; A measurement value detection step of detecting a measurement value indicated by the pointer from the positions of a plurality of numbers represented on the display board of the analog meter, the values indicated by the numbers, and the position of the tip of the pointer; A circle detection step of detecting a virtual circle passing through the position of the written number in the digital image data and the position of the center of the virtual circle; An information processing method comprising the above. **Claim 6**: The circle detection step A computer detects the rotation center point related to the pointer of the analog meter and the center points of the plurality of digital image data; is a step of obtaining a virtual circle centered on the rotation center point related to the pointer and having the center points of the digital image data on an arc; The measurement value detection step is a step in which a computer calculates a measurement value indicated by the pointer from the division ratio of the virtual circle by the pointer in the virtual circle. The information processing method according to claim 5.

Images