Indication value reading program, indication value reading device, and indication value reading method

The indication value reading program and device automate analog meter reading using machine learning, enabling rapid detection and response to factory abnormalities.

JP7589911B2Active Publication Date: 2024-11-26THE PUBLIC UNIV THE UNIV OF AIZU
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2020123114
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-07-17
Publication Date
2024-11-26
Estimated Expiration
2040-07-17

AI Technical Summary

Technical Problem

Manual calculations of analog meter readings in factories are time-consuming and delay the detection of abnormalities, preventing quick response to equipment failures.

Method used

An indication value reading program and device that extracts and processes image data from analog meters using machine learning to automatically determine and output meter values, enabling rapid detection of abnormalities.

Benefits of technology

Facilitates quick detection of factory abnormalities, reducing human error and response time to equipment issues.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589911000002
    Figure 0007589911000002
  • Figure 0007589911000003
    Figure 0007589911000003
  • Figure 0007589911000004
    Figure 0007589911000004
Patent Text Reader

Abstract

To provide an indicated value reading program, an indicated value reading device, and an indicated value reading method that enable quick detection of abnormalities that occur in factories.SOLUTION: An indicated value reading device extracts partial image data on an indicator from image data to be inspected including the indicator with a pointer, identifies a first direction indicated by the pointer included in the partial image data based on the extracted partial image data, identifies a value indicated by the pointer included in the partial image data based on the identified first direction and a range of values which the pointer can indicate in the indicator, and outputs the identified value indicated by the pointer.SELECTED DRAWING: Figure 12
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an indication value reading program, an indication value reading device, and an indication value reading method. [Background technology]

[0002] For example, in a factory where semiconductors are manufactured, many analog meters are generally installed for measuring the temperature, pressure, etc. of each device.

[0003] In such a factory, workers, for example, patrol the factory at regular intervals and visually check the values ​​indicated by each analog meter. Then, the workers, for example, perform necessary calculations using the values ​​of each analog meter and determine whether or not an abnormality (e.g., equipment failure, etc.) has occurred in the factory. As a result, if it is determined that an abnormality has occurred in the factory, the workers, for example, make arrangements to replace the equipment in which the abnormality has occurred (see Patent Documents 1 and 2 and Non-Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2004-133560 A [Patent Document 2] JP 2017-126187 A [Non-patent literature]

[0005] [Non-Patent Document 1] Yixiao Fang, Yan Dai, Guoli He and Donglian Qi, "A Mask RCNN based Automatic Reading Method for Pointer Meter", 38th Chinese Control Conference, 8466-8471. [Non-Patent Document 2] HaoWen Lai, Qi Kang, Le pan and Can Cui, "A Novel Scale Recognition Method for Pointer Meters Adapted to Different Types and Shape", 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), 374-379. Summary of the Invention [Problem to be solved by the invention]

[0006] Here, the calculations using the values ​​of the analog meters as described above are performed manually by an operator, and may take a certain amount of time. Furthermore, such calculations may need to be performed after moving to a location other than the factory (such as the operator's office).

[0007] As a result, workers may not be able to quickly detect abnormalities occurring in the factory, and it may take a long time to resolve the abnormalities.

[0008] SUMMARY OF THE PRESENT EMBODIMENTS An object of the present invention is to provide an indication value reading program, an indication value reading device, and an indication value reading method that enable rapid detection of abnormalities occurring in a factory. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the indication value reading program of the present invention is characterized in that it causes a computer to execute the following processes: extracting partial image data for a display having a pointer from inspection object image data which includes the display, identifying a first orientation indicated by the pointer contained in the partial image data based on the extracted partial image data, identifying a value indicated by the pointer contained in the partial image data based on the identified first orientation and the range of values ​​which the pointer can indicate on the display, and outputting the identified value indicated by the pointer.

[0010] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it generates multiple learning data by adding position information of a display having a pointer contained in each learning image data to each of multiple learning image data, generates a learning model by performing machine learning using the multiple generated learning data, obtains a value output from the learning model as the inspection object image data is input, and extracts the partial image data corresponding to the obtained value from the inspection object image data.

[0011] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it generates multiple learning data by adding information indicating the orientation of the pointer contained in each learning image data to each of multiple learning image data for a display device having a pointer, generates a learning model by performing machine learning using the multiple generated learning data, obtains a value output from the learning model as the partial image data is input, and identifies the orientation of the pointer from the obtained value.

[0012] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the length of the pointer from the value, and if the calculated length is within a predetermined range, performs a process of identifying the orientation of the pointer from the value.

[0013] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that, if the length is not within the specified range, the process of extracting the partial image data and the process of obtaining the value are performed again before performing the process of identifying the orientation of the needle from the value.

[0014] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it determines the value indicated by the pointer from the minimum value included in the range of values ​​that the pointer can indicate, the maximum value included in the range of values ​​that the pointer can indicate, the first orientation, the second orientation indicated by the pointer when the pointer indicates the minimum value, and the third orientation indicated by the pointer when the pointer indicates the maximum value.

[0015] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates a first value by subtracting an angle corresponding to the second orientation from an angle corresponding to the first orientation, calculates a second value by subtracting an angle corresponding to the second orientation from an angle corresponding to the third orientation, calculates a third value by subtracting the minimum value from the maximum value, calculates a fourth value by dividing the first value by the second value, calculates a fifth value by multiplying the fourth value and the third value, and calculates the value indicated by the pointer by adding the minimum value to the fifth value.

[0016] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it acquires the inspection object image data of the display device imaged by an imaging device, performs a homography transformation on the acquired inspection object image data, and extracts the partial image data from the inspection object image data after the homography transformation.

[0017] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it identifies one or more landmarks on the display device, and performs a homography transformation on the inspection object image data so that the identified one or more landmarks move to a pre-specified position.

[0018] In order to achieve the above object, in one aspect, the indication value reading program of the present invention is characterized in that it calculates an average value of the values ​​indicated by the pointer and outputs the calculated average value.

[0019] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the average value if the calculated standard deviation is greater than or equal to a predetermined threshold value.

[0020] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the standard deviation in addition to the average value.

[0021] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the standard deviation of the value indicated by the pointer identified in the process of identifying the value indicated by the pointer, and if the calculated standard deviation is equal to or greater than a predetermined threshold value, outputs information indicating that the pointer is vibrating in addition to the average value.

[0022] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the difference between the maximum and minimum values ​​indicated by the pointer identified in the process of identifying the value indicated by the pointer, and outputs the average value if the calculated difference is greater than or equal to a predetermined threshold value.

[0023] In addition, in one aspect, the indication value reading program of the present invention for achieving the above-mentioned object is characterized in that it calculates the difference between the maximum and minimum values ​​indicated by the pointer identified in the process of identifying the value indicated by the pointer, and if the calculated difference is greater than or equal to a predetermined threshold, outputs information indicating that the pointer is vibrating in addition to the average value.

[0024] In addition, in order to achieve the above-mentioned object, the indication value reading device of the present invention is characterized in that it has a partial image extraction unit that extracts partial image data for a display having a pointer from image data of an object to be inspected, the image data including the display having a pointer; a direction identification unit that identifies a first orientation indicated by the pointer contained in the partial image data based on the extracted partial image data; a value identification unit that identifies a value indicated by the pointer contained in the partial image data based on the identified first orientation and the range of values ​​that the pointer can indicate on the display; and a value output unit that outputs the identified value indicated by the pointer.

[0025] In addition, the method of reading an indication value of the present invention for achieving the above-mentioned object is characterized in that it has a computer execute a process of extracting partial image data for a display having a pointer from image data of an object to be inspected, the partial image data including the display including the pointer being inspected, identifying a first orientation indicated by the pointer contained in the partial image data based on the extracted partial image data, identifying a value indicated by the pointer contained in the partial image data based on the identified first orientation and the range of values ​​that the pointer can indicate on the display, and outputting the identified value indicated by the pointer. Effect of the Invention

[0026] According to the indication value reading program, indication value reading device, and indication value reading method of the present invention, it is possible to quickly detect abnormalities occurring in a factory. [Brief description of the drawings]

[0027] [Figure 1]FIG. 1 is a diagram showing an example of a configuration of an information processing device 1 according to the first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an outline of the indication value reading process in the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an outline of the indication value reading process in the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating details of the indication value reading process in the first embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating details of the indication value reading process in the first embodiment. [Figure 6] FIG. 6 is a flow chart illustrating details of the indication value reading process in the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating details of the indication value reading process in the first embodiment. [Figure 8] FIG. 8 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 9] FIG. 9 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 13] FIG. 13 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 14] FIG. 14 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 15] FIG. 15 is a diagram for explaining details of the indication value reading process in the first embodiment. [Figure 16] FIG. 16 is a diagram for explaining details of the indication value reading process in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the technical scope of the present invention is not limited to these embodiments.

[0029] First, a configuration example of an information processing device 1 (hereinafter also referred to as an indication value reading device 1) in the first embodiment will be described. Fig. 1 is a diagram showing a configuration example of the information processing device 1 in the first embodiment.

[0030] The information processing device 1 is a computer device, for example, a general-purpose PC (Personal Computer). The information processing device 1 performs a process (hereinafter also referred to as an indicated value reading process) of reading a value indicated by a needle of an analog meter (hereinafter also referred to as a display device) not shown.

[0031] 1, the information processing device 1 has a hardware configuration of a general-purpose computer device, and includes, for example, a CPU 101 which is a processor, a memory 102, a communication interface 103, and a storage medium 104. Each unit is connected to each other via a bus 105.

[0032] The storage medium 104 has, for example, a program storage area (not shown) for storing a program (not shown) for performing the indication value reading process.

[0033] The storage medium 104 also includes a storage unit 110 (hereinafter also referred to as a storage area 110) that stores information used when performing the indication value reading process, for example. The storage medium 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD).

[0034] The CPU 101 executes a program loaded from the storage medium 104 to the memory 102 to perform an indication value reading process.

[0035] The communication interface 103 communicates with an operation terminal 2 such as a smartphone via a network NW such as the Internet.

[0036] [Outline of the first embodiment] Fig. 2 and Fig. 3 are diagrams for explaining an outline of the indication value reading process in the first embodiment. Specifically, Fig. 2 is a diagram for explaining the learning stage process of the indication value reading process in the first embodiment. Also, Fig. 3 is a diagram for explaining the inference stage process of the indication value reading process in the first embodiment.

[0037] First, the learning stage of the indication value reading process in the first embodiment will be described.

[0038] As shown in FIG. 2, the first data generation unit 111 of the information processing device 1 generates learning data (hereinafter also referred to as first learning data) by adding information about the position and size of the analog meter in the first learning image data (hereinafter collectively referred to as position information) to learning image data that partially shows an analog meter.

[0039] Then, the first model generation unit 112 of the information processing device 1 generates a learning model (hereinafter also referred to as a first learning model) by performing machine learning using the multiple first learning data generated by the first data generation unit 111. After that, the first model generation unit 112 stores the generated first learning model in the memory area 110.

[0040] That is, the first model generation unit 112 uses a plurality of first learning data to generate a first learning model used when extracting partial image data of an analog meter from inspection object image data.

[0041] In addition, the second data generation unit 113 of the information processing device 1 generates learning data (hereinafter also referred to as second learning data) by adding information indicating the direction of the needle of the analog meter in partial image data extracted by the first learning model generated by the first model generation unit 112 to learning image data (hereinafter also referred to as second learning image data) about an analog meter, for example.

[0042] Then, the second model generation unit 114 of the information processing device 1 generates a learning model (hereinafter also referred to as a second learning model) by performing machine learning using the multiple second learning data generated by the second data generation unit 113. After that, the second model generation unit 114 stores the generated second learning model in the memory area 110.

[0043] That is, the second model generation unit 114 generates a second learning model used when identifying (estimating) the direction indicated by the needle of the analog meter, by using a plurality of second learning data.

[0044] Next, the inference stage of the indication value reading process in the first embodiment will be described.

[0045] The partial image extraction unit 121 of the information processing device 1 extracts partial image data regarding the analog meter from the verification target image data that partially includes the analog meter.

[0046] Specifically, the partial image extraction unit 121 identifies position information indicated by a value output in response to input of the verification target image data to the first learning model stored in the memory area 110. Then, the partial image extraction unit 121 extracts the partial image data corresponding to the identified position information from the verification target image data.

[0047] Then, based on the partial image data extracted by the partial image extraction unit 121, the direction determination unit 122 of the information processing device 1 identifies the direction indicated by the needle of the analog meter shown in the partial image data (hereinafter also referred to as a first direction).

[0048] Specifically, the direction determining unit 122 identifies a first direction indicated by a value output in response to an input of partial image data to the second learning model stored in the memory area 110.

[0049] Next, the value determination unit 123 of the information processing device 1 determines the value indicated by the needle of the analog meter contained in the partial image data extracted by the partial image extraction unit 121 based on the first orientation determined by the direction determination unit 122 and the range of values ​​that the needle of the analog meter can indicate.

[0050] Thereafter, the value output unit 124 of the information processing device 1 outputs the value identified by the value identifying unit 123 to the operation terminal 2, for example.

[0051] That is, the information processing device 1 in this embodiment uses image data showing an analog meter placed in a factory as an input to a learning model to automatically identify the direction in which the needle of the analog meter is pointing. Then, the information processing device 1 identifies the value indicated by the needle of the analog meter based on the identified direction.

[0052] After that, the information processing device 1 calculates (estimates) the probability of occurrence of an abnormal value in the equipment in the factory from the current value and past value indicated by the needle of each analog meter, for example. Then, when the calculated probability of occurrence of an abnormal value exceeds a predetermined threshold, the information processing device 1 determines that there is a high possibility that an abnormality has occurred in the factory.

[0053] This enables the information processing device 1 to automatically and quickly determine whether or not there is a high possibility that an abnormality has occurred in the factory. Then, for example, when the information processing device 1 determines that there is a high possibility that an abnormality has occurred in the factory, it becomes possible to notify the worker to perform a necessary task (operation).

[0054] In this regard, the occurrence of an abnormality in the factory may lead to a major accident in the factory, so workers need to take prompt action against the abnormality that occurs in the factory in order to prevent the occurrence of a major accident in the factory.

[0055] Therefore, for example, when the information processing device 1 determines that there is a high possibility that an abnormality will occur in the factory, it instructs the worker to take appropriate measures. Also, for example, when the information processing device 1 determines that there is an extremely high possibility that an abnormality will occur in the factory (when it determines that the situation is such that it is possible to conclude that an abnormality has occurred), it instructs the worker to take emergency measures.

[0056] Furthermore, by automatically reading the values ​​indicated by the needles of each analog meter, the information processing device 1 can prevent human errors (such as reading errors or value input errors) that may occur when an operator reads the values.

[0057] [Details of the first embodiment] Next, the details of the indication value reading process in the first embodiment will be described. Figures 4 to 7 are flow charts for explaining the details of the indication value reading process in the first embodiment. Figures 8 to 16 are diagrams for explaining the details of the indication value reading process in the first embodiment.

[0058] [Processing during the learning phase] First, a description will be given of the details of the processing in the learning stage among the details of the indication value reading processing in the first embodiment. Fig. 4 is a diagram for explaining the details of the processing in the learning stage.

[0059] 4, the first data generation unit 111 waits until the learning timing comes (NO in S11). The learning timing may be, for example, the timing when the worker inputs the first learning image data via the operation terminal 2. The learning timing may also be, for example, the timing when the worker inputs information to the effect that each learning model will be generated.

[0060] Then, when the learning timing arrives (YES in S11), the first data generation unit 111 generates first learning data by adding position information of the analog meter contained in each image data to each of the first learning image data (S12).

[0061] Specifically, the first data generation unit 111 adds position information of the analog meter to the first learning image data DT11, for example, by using a bounding box BB11 (bounding box BB11 specified by the worker) indicating the position and size of the analog meter, as shown in FIG. 7.

[0062] More specifically, the worker prepares, for example, image data corresponding to each frame included in video data captured by an imaging device (not shown) as first learning image data. Then, the worker adds position information of an analog meter to each piece of first learning image data by using a bounding box. After that, the first data generating unit 111 generates the first learning data by, for example, associating the first learning image data with the coordinates (X coordinate and Y coordinate) of the upper left corner of the bounding box and the vertical and horizontal lengths of the bounding box.

[0063] Thereafter, the first model generation unit 112 generates a first learning model by performing machine learning using the first learning data generated in the process of S12 (S13).

[0064] Next, the second data generation unit 113 generates second learning data by adding information indicating the orientation of the needle in each piece of image data to each piece of second learning image data (S14).

[0065] Specifically, as shown in FIG. 8, the second data generation unit 113 adds needle position information to the second learning image data DT12, for example, by using a bounding box BB12 (bounding box BB12 specified by the operator) indicating the position of the needle of an analog meter.

[0066] More specifically, the second data generation unit 113 acquires, for example, image data output in association with input of the first learning image data to the first learning model generated in the process of S13 as the second learning image data. Then, the worker adds position information of the needle to each piece of the second learning image data by using a bounding box. After that, the second data generation unit 113 generates the second learning data by, for example, associating the second learning image data with the coordinates (X coordinate and Y coordinate) of the top left corner of the bounding box and the vertical and horizontal lengths of the bounding box.

[0067] Here, because of the need to generate a second learning model with high estimation accuracy, the second data generation unit 113 needs to generate, for example, hundreds to thousands of second learning image data showing an analog meter in a state where a needle is pointing to each scale for each scale of the analog meter. Furthermore, for example, when multiple analog meters are arranged in a factory, the second data generation unit 113 needs to generate, for example, the second learning image data for each analog meter.

[0068] In this regard, when performing the indicated value reading process in this embodiment, the worker may prepare only the first learning image data corresponding to a 1 / 4 region of the scale of the analog meter (for example, the region between "0.1" and "0.3" in the analog meter shown in FIG. 8) and input these to the first learning model to generate the second learning image data. Then, after generating the second learning data from the generated second learning image data, the worker may further generate new second learning data corresponding to other regions (for example, regions other than the region between "0.1" and "0.3" in the analog meter shown in FIG. 8) by flipping the generated second learning image data horizontally or vertically (flipping horizontally and vertically), thereby generating second learning data corresponding to all regions of the scale of the analog meter.

[0069] Specifically, for example, when the value indicated by the needle of an analog meter displayed in the second learning image data DT12 shown in FIG. 8 is "0.178", the value indicated by the needle of an analog meter displayed in image data obtained by flipping the second learning image data DT12 horizontally is "0.422", the value indicated by the needle of an analog meter displayed in image data obtained by flipping the second learning image data DT12 vertically is "0.022", and the value indicated by the needle of an analog meter displayed in image data obtained by flipping the second learning image data DT12 horizontally and vertically is "0.578".

[0070] Therefore, in this case, the worker generates new second learning data by adding position information of the pointer when pointing to "0.422" to image data obtained by flipping the second learning image data DT12 horizontally, new second learning data by adding position information of the pointer when pointing to "0.022" to image data obtained by flipping the second learning image data DT12 vertically, and new second learning data by adding position information of the pointer when pointing to "0.578" to image data obtained by flipping the second learning image data DT12 horizontally and vertically.

[0071] That is, the second learning model in this embodiment is not a learning model that recognizes verification target image data showing an analog meter, but a learning model that detects the direction (angle) indicated by the needle of an analog meter. Therefore, when generating the second learning model, the worker can also use image data in which the characters indicating the scale (such as "0.1") are inverted. Therefore, the worker can generate only second learning data corresponding to a partial area of ​​the scale of the analog meter, and further generate new second learning data by inverting various parts of the second learning image data included in the generated second learning data.

[0072] This allows the operator to reduce the burden of generating the second learning data. In addition, in this case, the operator can generate a second learning model that can identify (estimate) the orientation of the pointer of another analog meter that has a different type of pointer, number of scales, etc.

[0073] Thereafter, the second model generation unit 114 generates a second learning model by performing machine learning using the second learning data generated in the process of S14 (S15).

[0074] Note that the operator may be configured to input new first learning image data at any time to the information processing device 1. Then, the information processing device 1 may be configured to generate new first learning data including the new first learning image data at any time (S12).

[0075] Thereafter, for example, if the proportion of the newly added first learning image data exceeds a predetermined proportion of all the first learning image data, the information processing device 1 may generate the first learning model and the second learning model again (S13 to S15).

[0076] This enables the worker to continuously improve the judgment accuracy of the first learning model and the second learning model.

[0077] [Processing in the inference stage] Next, a description will be given of the details of the processing in the inference stage among the details of the indication value reading processing in the first embodiment. Figures 5 and 6 are diagrams for explaining the details of the processing in the inference stage.

[0078] 5, the partial image extraction unit 121 waits until the judgment timing arrives (NO in S21). The judgment timing may be, for example, the timing when the operator inputs the verification target image data via the operation terminal 2. The judgment timing may also be, for example, the timing when the operator inputs information to the effect that the verification target learning data will be judged.

[0079] Then, when the determination timing arrives (YES in S21), the partial image extraction unit 121 acquires inspection object image data of the analog meter captured by an imaging device (not shown) (S22).

[0080] Specifically, as shown in Fig. 9, the partial image extraction unit 121 receives inspection target image data DT21 of an analog meter transmitted from an imaging device that an operator uses to capture an image. The imaging device may be, for example, a non-fixed camera. The imaging device may also be, for example, one built into the operation terminal 2.

[0081] Next, the partial image extraction unit 121 acquires a value output from the first learning model in response to input of the inspection target image data acquired in the process of S22 (S23).

[0082] Then, the partial image extraction unit 121 extracts the partial image data corresponding to the value acquired in the process of S23 from the inspection object image data acquired in the process of S22 (S24).

[0083] Specifically, as shown in FIG. 10, the partial image extraction unit 121 refers to the position information (coordinates) corresponding to the value acquired in the processing of S23, and acquires the portion of the inspection object image data acquired in the processing of S21 that shows the analog meter as partial image data DT22.

[0084] Next, as shown in FIG. 6, the direction determination unit 122 acquires a value output from the second learning model in response to input of the partial image data extracted in the process of S24 (S31).

[0085] In this case, the direction determination unit 112 may calculate the length from the axis of rotation of the needle of the analog meter to the tip (the length of the diagonal of the bounding box corresponding to the needle's position) by using the value acquired in the process of S31. If the calculated length of the diagonal is not within a predetermined allowable range (for example, within a range of ±5(%)), the direction determination unit 112 may determine that an error (partial image data reading error) has occurred in the process of S24, and may perform the processes from S24 onwards again.

[0086] This enables the direction determining unit 112 to improve the reading accuracy of the partial image data in the process of S24.

[0087] Then, the direction determination unit 122 identifies a first direction in which the needle of the analog meter is facing, from the value acquired in the process of S31 (S32).

[0088] Specifically, as shown in FIG. 11, the direction determination unit 122 specifies a direction DRp as a first direction in which the needle of an analog meter shown in the partial image data DT22 extracted in the process of S24 faces, for example.

[0089] Next, the value identification unit 123 identifies the value indicated by the needle included in the partial image data extracted in the process of S24, based on the first orientation identified in the process of S32 and the range of values ​​indicated by the needle on the analog meter (S33).

[0090] Specifically, value identification unit 123 identifies the value indicated by the pointer included in the partial image data extracted in the process of S24, for example, from the minimum value in the range of values ​​that the pointer can indicate (hereinafter also simply referred to as the minimum value), the maximum value in the range of values ​​that the pointer can indicate (hereinafter also simply referred to as the maximum value), the first direction identified in the process of S32, the direction indicated by the pointer when it indicates the minimum value (hereinafter also referred to as the second direction), and the direction indicated by the pointer when it indicates the maximum value (hereinafter also referred to as the third direction). A specific example of the process of S33 will be described below.

[0091] [Example of S33 processing] The value specification unit 123 specifies the value indicated by the pointer included in the partial image data extracted in the process of S24, for example, according to the following formula 1. Note that, hereinafter, as shown in Fig. 12, the second direction is also referred to as direction DRm, the third direction is also referred to as direction DRM, and the reference direction (X-axis direction) when calculating each angle is also referred to as direction DRX.

[0092]

number

[0093] In the above formula 1, θ p indicates the angle when moving clockwise from the direction DRX to the direction DRp, and θ m indicates the angle when moving clockwise from the direction DRX to the direction DRm, and θ M indicates the angle when moving clockwise from the direction DRX to the direction DRM. Also, in the above formula 1, v p indicates a value corresponding to the direction DRp within the range of values ​​that the pointer can indicate, and v m indicates the minimum value that the pointer can indicate (the value corresponding to the direction DRm), and v M indicates the maximum value (value corresponding to the direction DRM) among the values ​​that the pointer can indicate. p and θ M is θ mIf it is smaller than θ, add 360 degrees to the angle from the direction DRX to the direction DRM. M Let us assume that.

[0094] Specifically, as shown in FIG. m is 135 degrees, and θ M is 405 degrees, and θ p is 215 degrees, the value specifying unit 123 p This gives us approximately 0.178.

[0095] Returning to FIG. 6, the value output unit 124 outputs the value indicated by the indicator identified in the process of S33 (S34).

[0096] Specifically, the value output unit 124 outputs to the operation terminal 2, for example, the value indicated by the indicator identified in the process of S33.

[0097] The value output unit 124 may perform a predetermined calculation using the value indicated by the indicator identified in the process of S33, and output the calculation result to the operation terminal 2. That is, the value output unit 124 may perform a calculation for determining whether or not an abnormality has occurred in the factory, by using the value indicated by the indicator identified in the process of S33, and output the calculation result to the operation terminal 2.

[0098] In this manner, the information processing device 1 in the present embodiment extracts partial image data of an analog meter having a pointer from the inspection object image data showing the analog meter. Then, based on the extracted partial image data, the information processing device 1 identifies the first direction indicated by the pointer included in the partial image data.

[0099] Furthermore, the information processing device 1 identifies a value indicated by the needle included in the partial image data based on the identified first orientation and the range of values ​​that the needle can indicate on the analog meter. Then, the information processing device 1 outputs the identified value indicated by the needle.

[0100] That is, the information processing device 1 in this embodiment receives image data of each analog meter in a factory as an input, and automatically identifies (reads) the value indicated by each analog meter.

[0101] This allows the information processing device 1 to calculate (estimate) the probability that an abnormality has occurred in the factory, for example, by using the values ​​indicated by each analog meter. Then, when the information processing device 1 determines that an abnormality has occurred in the factory based on the calculated probability, for example, it becomes possible to notify workers to perform the necessary work (operation).

[0102] In addition, by automatically reading the values ​​indicated by the needles of each analog meter, the information processing device 1 can prevent human errors (such as reading errors or value input errors) that may occur when an operator reads the values.

[0103] Here, the information processing device 1 may acquire video data of an analog meter captured by an imaging device (not shown) in the process of S22. The information processing device 1 may perform the processes of S23 to S34 for each of a plurality of pieces of inspection object image data (e.g., a plurality of pieces of inspection object image data corresponding to 10 consecutive frames) included in the video data.

[0104] Specifically, in this case, the information processing device 1 calculates, for example, the standard deviation of the values ​​calculated in the process of S33 (the values ​​calculated for each of the multiple pieces of inspection target image data). If it is determined that the standard deviation exceeds a threshold value, the information processing device 1 determines that the needle of the analog meter shown in the video data acquired in the process of S22 is vibrating, and outputs the average value of the values ​​calculated in the process of S33.

[0105] This allows the information processing device 1 to determine whether the needle of the analog meter is vibrating or not. Furthermore, even if the needle of the analog meter is vibrating, the information processing device 1 can output the value indicated by the needle with high accuracy.

[0106] In this case, the information processing device 1 may output, for example, in addition to the average value of the values ​​calculated in the process of S33, the standard deviation of the values ​​calculated in the process of S33, or information indicating that the needle of the analog meter shown in the video data acquired in the process of S22 is vibrating. Moreover, instead of determining whether or not the standard deviation of the values ​​calculated in the process of S33 exceeds a threshold value, the information processing device 1 may determine whether or not the difference between the maximum value and the minimum value of the values ​​calculated in the process of S33 exceeds a threshold value.

[0107] Furthermore, even if it is determined that the standard deviation does not exceed the threshold value, the information processing device 1 may output the average value of the values ​​calculated in the process of S33. This enables the information processing device 1 to improve the accuracy of estimating the value indicated by the needle of the analog meter.

[0108] [Homography Transformation] Next, homography transformation of partial image data will be described.

[0109] The information processing device 1 may perform homography transformation on the partial image data extracted in the process of S24 before performing the process of S31, for example. Specifically, the information processing device 1 may perform homography transformation so that one or more marks on an analog meter shown in the partial image data extracted in the process of S24 move to a pre-specified position. Then, the information processing device 1 may input the partial image data subjected to homography transformation to the second learning model in the process of S31.

[0110] This enables the information processing device to convert, for example, detection object image data showing a tilted analog meter into detection object image data showing the analog meter in the correct orientation (the same orientation as when photographed from directly in front). Therefore, even when the information processing device 1 acquires detection object image data showing a tilted analog meter, it is possible to suppress a decrease in the estimation accuracy of the value indicated by the needle. A specific example of homography transformation will be described below.

[0111] [Example of homography transformation] 13 and 14 are diagrams for explaining a specific example of homography transformation.

[0112] For example, as shown in FIG. 13, an operator attaches four markers MR31, MR32, MR33, and MR34, which are stickers in colors not used in analog meters, to the outer frame FL of an analog meter that is to be imaged by an imaging device (not shown).

[0113] Specifically, as shown in Fig. 13, the worker attaches the marks MR31 and MR33 to the analog meter to be inspected so that the line connecting the marks MR31 and MR33 (hereinafter also referred to as the first line) is perpendicular to the ground and the first line passes through the base end of the pointer (the rotation axis of the pointer). The worker also attaches the marks MR32 and MR34 to the analog meter to be inspected so that the line connecting the marks MR32 and MR34 (hereinafter also referred to as the second line) is horizontal to the ground and the second line passes through the base end of the pointer.

[0114] 14(A), for example, when partial image data DT31 showing an analog meter tilted to the left is extracted in the process of S24, the information processing device 1 performs homography transformation so that the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data, as shown in Fig. 14(B), to generate partial image data DT31a. After that, the information processing device 1 inputs the partial image data DT31a to the second learning model in the process of S31.

[0115] Similarly, as shown in Fig. 14(C), for example, in the process of S24, when partial image data DT32 showing an analog meter tilted to the right is extracted, the information processing device 1 performs homography transformation so that the first straight line is parallel to the horizontal side of the partial image data and the second straight line is parallel to the vertical side of the partial image data, and generates partial image data DT32a, as shown in Fig. 14(D). After that, the information processing device 1 inputs the partial image data DT32a to the second learning model in the process of S31.

[0116] [Estimation results of the second learning model when homography transformation is performed] Next, the estimation result of the second learning model when homography conversion is performed will be described. FIG. 15 and FIG. 16 are diagrams for explaining the estimation result of the second learning model when homography conversion is performed. Specifically, FIG. 15(A) is a scatter plot showing the estimation result of the second learning model when homography conversion is not performed, and FIG. 15(B) is a histogram showing the estimation result of the second learning model when homography conversion is not performed. Also, FIG. 16(A) is a scatter plot showing the estimation result of the second learning model when homography conversion is performed, and FIG. 16(B) is a histogram showing the estimation result of the second learning model when homography conversion is performed.

[0117] The horizontal axis of the scatter diagrams shown in Figures 15(A) and 16(A) indicates the value actually indicated by the analog meter, and the vertical axis indicates the error between the value specified in the process of S33 and the value actually indicated by the analog meter. The horizontal axis of the histograms shown in Figures 15(B) and 16(B) indicates the error between the value specified in the process of S33 and the value actually indicated by the analog meter, and the vertical axis indicates the number of partial image data input to the second learning model.

[0118] Specifically, the diagrams in Fig. 16 show, for example, that there is an increase in the number of partial image data in which the error between the value specified in the process of S33 and the value actually indicated by the analog meter is 0, when compared with the diagrams in Fig. 15. Also, the diagrams in Fig. 16 show that, as a whole, the errors corresponding to the respective partial image data are smaller, when compared with the diagrams in Fig. 15.

[0119] That is, the examples shown in FIGS. 15 and 16 show that performing homography transformation on the partial image data extracted in the process of S24 improves the estimation accuracy of the second learning model. [Explanation of symbols]

[0120] 1: Information processing device 2: Operation terminal 101:CPU 102: Memory 103: Communication interface 104:Storage medium 105: Bus

Claims

1. A method for generating a plurality of learning data by adding information indicating the orientation of a pointer contained in each of a plurality of learning image data for a display device having a pointer; generating a learning model by performing machine learning using the generated plurality of learning data; Extracting partial image data of a display having a pointer from inspection object image data including the display; Obtaining a value output from the learning model in response to inputting the extracted partial image data; Calculating the length of the pointer included in the partial image data from the acquired value; If the calculated length is within a predetermined range, a first direction indicated by the pointer included in the partial image data is identified from the acquired value; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values ​​that the pointer can indicate on the display included in the partial image data; outputting a value indicated by the identified indicator; An indication reading program for causing a computer to execute a process.

2. In claim 1, further comprising: generating a plurality of learning data by adding position information of a display having a pointer, the position information being included in each of the plurality of learning image data; generating a learning model by performing machine learning using the generated plurality of learning data; The process is executed by a computer, In the process of extracting the partial image data, Obtaining a value output from the learning model in response to input of the inspection object image data; extracting the partial image data corresponding to the acquired value from the inspection object image data; An indication value reading program comprising:

3. In claim 1, In the process of identifying the first orientation, if the length is not within the predetermined range, the process of extracting the partial image data and the process of obtaining the value are performed again before the process of identifying the orientation of the pointer from the value is performed. An indication value reading program comprising:

4. In claim 1, In the process of identifying the value indicated by the pointer, the value indicated by the pointer is identified from a minimum value included in a range of values ​​that the pointer can indicate, a maximum value included in a range of values ​​that the pointer can indicate, the first orientation, a second orientation indicated by the pointer when the pointer indicates the minimum value, and a third orientation indicated by the pointer when the pointer indicates the maximum value. An indication value reading program comprising:

5. In claim 4, In the process of identifying the value indicated by the guideline, calculating a first value by subtracting an angle corresponding to the second orientation from an angle corresponding to the first orientation; calculating a second value by subtracting an angle corresponding to the second orientation from an angle corresponding to the third orientation; calculating a third value by subtracting the minimum value from the maximum value; calculating a fourth value by dividing the first value by the second value; calculating a fifth value by multiplying the fourth value and the third value; calculating a value indicated by the pointer by adding the minimum value to the fifth value; An indication value reading program comprising:

6. In claim 1, further comprising: acquiring the inspection object image data of the display unit captured by an imaging device; performing homography transformation on the acquired inspection object image data; The process is executed by a computer, In the process of extracting the partial image data, the partial image data is extracted from the inspection object image data that has been subjected to homography transformation. An indication value reading program comprising:

7. In claim 6, In the process of performing the homography transformation, Identifying one or more indicia on the indicator; performing a homography transformation on the inspection object image data so that the one or more identified landmarks are moved to a pre-specified position; An indication value reading program comprising:

8. In claim 1, further comprising: performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data; calculating an average value of the values ​​indicated by the indicator identified in the process of identifying the values ​​indicated by the indicator; The process is executed by a computer, In the process of outputting the value indicated by the pointer, the calculated average value is output. An indication value reading program comprising:

9. In claim 8, further comprising: Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator. The process is executed by a computer, In the process of outputting the value indicated by the indicator, if the calculated standard deviation is equal to or greater than a predetermined threshold value, In some cases, the average value is output. An indication value reading program comprising:

10. In claim 8, further comprising: Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator. The process is executed by a computer, In the process of outputting the value indicated by the pointer, the standard deviation is output in addition to the average value. An indication value reading program comprising:

11. In claim 8, further comprising: Calculating a standard deviation of the value indicated by the indicator identified in the process of identifying the value indicated by the indicator. The process is executed by a computer, In the process of outputting the value indicated by the indicator, When the calculated standard deviation is equal to or greater than a predetermined threshold, in addition to the average value, information indicating that the pointer is vibrating is output. An indication value reading program comprising:

12. In claim 8, further comprising: calculating a difference between a maximum value and a minimum value among the values ​​indicated by the indicator identified in the process of identifying the value indicated by the indicator; The process is executed by a computer, In the process of outputting the value indicated by the indicator, When the calculated difference is equal to or greater than a predetermined threshold, the average value is output. An indication value reading program comprising:

13. In claim 8, further comprising: calculating a difference between a maximum value and a minimum value among the values ​​indicated by the indicator identified in the process of identifying the value indicated by the indicator; The process is executed by a computer, In the process of outputting the value indicated by the indicator, When the calculated difference is equal to or greater than a predetermined threshold, in addition to the average value, information indicating that the pointer is vibrating is output. An indication value reading program comprising:

14. Extracting partial image data of a display having a pointer from inspection object image data including the display; Identifying a first direction indicated by the pointer included in the partial image data based on the extracted partial image data; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values ​​that the pointer can indicate on the display device; performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data; Calculating the average value of the values ​​indicated by the identified guidelines; Calculate the standard deviation of the values ​​indicated by the identified guidelines; If the calculated standard deviation is equal to or greater than a predetermined threshold, output the calculated average value. An indication reading program for causing a computer to execute a process.

15. A data generating unit that generates a plurality of learning data by adding information indicating the orientation of a pointer contained in each of a plurality of learning image data for a display device having a pointer; a model generation unit that generates a learning model by performing machine learning using the generated plurality of learning data; a partial image extraction unit that extracts partial image data of a display having a pointer from inspection object image data including the display; a direction specification unit that obtains a value output from the learning model in response to input of the extracted partial image data, calculates a length of the pointer included in the partial image data from the obtained value, and specifies a first direction indicated by the pointer included in the partial image data from the obtained value if the calculated length is within a predetermined range; a value specifying unit that specifies a value indicated by the pointer included in the partial image data based on the specified first orientation and a range of values ​​that the pointer can indicate on the display; A value output unit that outputs a value indicated by the specified pointer. An indication value reading device.

16. A partial image extracting unit for extracting partial image data of a display having a pointer from each of a plurality of inspection object image data including the display; a direction specifying unit that specifies a first direction indicated by the pointer included in the partial image data based on the extracted partial image data for each of the plurality of inspection object image data; a value specifying unit that specifies, for each of a plurality of pieces of inspection object image data, a value indicated by the pointer included in the partial image data based on the specified first orientation and a range of values ​​that the pointer can indicate on the display; and a value output unit that calculates an average value of the values ​​indicated by the identified indicator, calculates a standard deviation of the values ​​indicated by the identified indicator, and outputs the calculated average value if the calculated standard deviation is equal to or greater than a predetermined threshold value. An indication value reading device.

17. A method for generating a plurality of learning data by adding information indicating the orientation of a pointer contained in each of a plurality of learning image data for a display device having a pointer, generating a learning model by performing machine learning using the generated plurality of learning data; Extracting partial image data of a display having a pointer from inspection object image data including the display; Obtaining a value output from the learning model in response to inputting the extracted partial image data; Calculating the length of the pointer included in the partial image data from the acquired value; If the calculated length is within a predetermined range, a first direction indicated by the pointer included in the partial image data is identified from the acquired value; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values ​​that the pointer can indicate on the display included in the partial image data; outputting a value indicated by the identified indicator; A method for reading an indication value, the processing of which is executed by a computer.

18. Extracting partial image data of a display having a pointer from inspection object image data including the display, Identifying a first direction indicated by the pointer included in the partial image data based on the extracted partial image data; identifying a value indicated by the pointer included in the partial image data based on the identified first orientation and a range of values ​​that the pointer can indicate on the display device; performing the process of extracting the partial image data, the process of identifying the first orientation, and the process of identifying the value indicated by the pointer for a plurality of the inspection object image data; Calculating the average value of the values ​​indicated by the identified guidelines; Calculate the standard deviation of the values ​​indicated by the identified guidelines; If the calculated standard deviation is equal to or greater than a predetermined threshold, output the calculated average value. A method for reading an indication value, the processing of which is executed by a computer.

Citation Information

Patent Citations

  • Data logger

    JP1983137042A

  • Instrument indication reader

    JP2003223693A

  • Automatic reading method and automatic reader for analog meter

    JP2004133560A

  • Indicated value measurement system

    JP2012146154A

  • Measuring device for electric characteristics of semiconductor, measuring method for electric characteristics of semiconductor, control device for measuring device for electric characteristics of semiconductor, and computer program

    JP2015108548A