Monitoring device of an analysis device

Abstract

The present application provides a monitoring device of an analysis device, which enables a user to easily grasp the state of the analysis device. The monitoring device (10) includes: an acquisition section (111) that acquires a captured image of a display panel (41) of a control device (40) that controls the analysis device; an image storage section (12) that stores the captured image; and a state determination section (113) that determines the state of the analysis device based on the captured image.

Description

Technical Field

[0001] This invention relates to a monitoring device for an analytical apparatus. Background Technology

[0002] Previously, the operating status of analytical devices was managed by the user. For example, in the fatigue testing machine described in Patent Document 1, the control device for controlling the test includes a display that shows the test conditions, etc. The user can monitor and manage the operating status of the fatigue testing machine by visually observing the display.

[0003] [Existing technical documents]

[0004] [Patent Literature]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2006-292400 Summary of the Invention

[0006] [The problem the invention aims to solve]

[0007] In the conventional structure, users had to travel to the location of the control device to confirm the operating status of the fatigue testing machine. The images or videos from the control device's display panel could be sent to a remote location or stored in a storage device, allowing for monitoring of the machine's status changes remotely or after fatigue testing. However, this method requires users to visually judge the machine's status or pinpoint changes from the large image or video displayed on the panel, which is not easy.

[0008] Furthermore, the number of users with multiple analytical devices, not limited to material testing machines such as fatigue testing machines, has been increasing in recent years. These multiple analytical devices sometimes include those with control devices that cannot be connected to communication networks. Moreover, there are also cases where analytical devices from different manufacturers are mixed together. In these situations, even if a communication network is available, it is sometimes impossible to collect and analyze the analysis results using the network. For example, if the control application used to control the analytical device is limited to a single control application, even with a communication network, it is impossible to collect and analyze the analysis results.

[0009] This invention was made in view of this situation, with the aim of enabling users to easily monitor the status of the analysis device.

[0010] [Technical means to solve the problem]

[0011] One aspect of the monitoring device of the present invention includes: an acquisition unit for acquiring images captured by the display panel of a control device that controls the analysis device; an image storage unit for storing the captured images; and a status determination unit for determining the status of the analysis device based on the captured images.

[0012] [The effects of the invention]

[0013] According to the present invention, the state of the analysis device is determined based on images captured by the display panel of the control device that controls the analysis device. Therefore, the user can easily grasp the state of the analysis device. Furthermore, since the state of the analysis device is determined based on images captured by the display panel of the control device, the control device itself does not need to be connected to a communication network or have communication functionality. Moreover, even when the state determination involves control devices from different manufacturers, by using the state determination unit to determine the information displayed on the display panel, a common application can be used to extract or analyze control or analysis information. Attached Figure Description

[0014] Figure 1 This figure shows an example of the utilization mode of a monitoring device for monitoring the state of a material testing machine according to an embodiment of the present invention.

[0015] Figure 2 This is a diagram illustrating an example of the structure of the display panel of a control device used to control a material testing machine.

[0016] Figure 3A This diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0017] Figure 3B This diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0018] Figure 3C This diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0019] Figure 3D This diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0020] Figure 3E This diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0021] Figure 3FThis diagram illustrates an example of the relationship between the illumination patterns of the four indicator lights on the power unit operation keys of the control device and the hydraulic source status of the material testing machine.

[0022] Figure 4A This diagram illustrates an example of the relationship between the illumination state of the two indicator lights on the test operation keys of the control device and the test status of the material testing machine.

[0023] Figure 4B This diagram illustrates an example of the relationship between the illumination state of the two indicator lights on the test operation keys of the control device and the test status of the material testing machine.

[0024] Figure 5 This is a diagram illustrating an example of the structure of a monitoring device.

[0025] Figure 6 This diagram illustrates the operation of an image recognition unit that acquires a local image of a display area from an image of interest.

[0026] Figure 7 This is a diagram showing an example of a data display screen output from the output unit to the display unit.

[0027] Figure 8 This is a flowchart illustrating an example of the processing performed by the monitoring device.

[0028] Figure 9 It means Figure 8 A flowchart of an example of the image extraction and processing flow.

[0029] Figure 10 It means Figure 8 The flowchart is an example of the process for determining the state of the hydraulic source.

[0030] Figure 11 It means Figure 8 The flowchart is an example of the test status determination process.

[0031] Figure 12 It means Figure 8 A flowchart illustrating an example of the numerical data generation and processing workflow.

[0032] Figure 13 It means Figure 8 The flowchart is an example of the process for updating the first loop number in the process.

[0033] Figure 14 It means Figure 8 The flowchart is an example of the second loop number update process.

[0034] Figure 15 This is another example of a data display screen output from the output unit to the display unit.

[0035] Figure 16 This is another example of a display panel for a monitoring device.

[0036] Figure 17 This is a diagram illustrating an example of the structure of the monitoring device according to the second embodiment.

[0037] Figure 18 This is a flowchart illustrating an example of masking image generation and processing.

[0038] Figure 19 This is a schematic diagram illustrating an example of masking image generation and processing.

[0039] Figure 20 This is a flowchart illustrating an example of image processing before a decision is made.

[0040] Figure 21 This is a flowchart illustrating an example of the masking image generation process in the third embodiment.

[0041] Figure 22 This is a flowchart illustrating an example of the operation of the state determination unit in the fourth embodiment.

[0042] [Explanation of Symbols]

[0043] 10, 10a: Monitoring devices

[0044] 11: Processor

[0045] 111: Acquisition Department

[0046] 112: Image Recognition Department

[0047] 113: Status Determination Unit

[0048] 114: Update Department

[0049] 115: Output Department

[0050] 12: Memory

[0051] 121: Monitoring Procedures

[0052] 122: Image Data

[0053] 123a: Hydraulic Condition Recognition Model

[0054] 123b: Test State Recognition Model

[0055] 124: Judgment Result Data

[0056] 125: Numerical data

[0057] 211: Image Processing Department

[0058] 221: Masking image data

[0059] 13: Input Section

[0060] 14: Display Section

[0061] 141: Data display screen

[0062] 142: Focus on the image display section

[0063] 143, 144, 145: Local image display section

[0064] 146: Judgment Result Display Section

[0065] 147: Numerical Display Section

[0066] 15: Machine connection interface

[0067] 16: Communication Interface

[0068] 20: Material testing machine (analytical device)

[0069] 21: Abutment

[0070] 22a, 22b: Supports

[0071] 23: Forklift

[0072] 24: Raising the roof beam

[0073] 25: Hydraulic actuator

[0074] 25a: Piston rod

[0075] 26a: Lower clamp

[0076] 26b: Upper clamp

[0077] 27: Force sensor

[0078] 28: Servo Valve

[0079] 29: Differential Transformer

[0080] 30: Hydraulic source

[0081] 30a: Hydraulic pump

[0082] 30b: manifold

[0083] 30c: Piping valve

[0084] 30d: Load valve

[0085] 40: Control device

[0086] 41: Display panel

[0087] 410: Power switch

[0088] 411: Function Key

[0089] 412: Dial

[0090] 413: Setting button

[0091] 414: Numeric keypad

[0092] 415: Emergency Stop Switch

[0093] 50: Display screen

[0094] 501: Cycle Number Display Section

[0095] 502: Specific symbol display section

[0096] 503: Test Status Display Section

[0097] 51: Power Unit Operation Key

[0098] 51a, 51b, 51c, 51d: Indicator lights

[0099] 511: Start button

[0100] 512: Stop button

[0101] 513: Manifold Button

[0102] 514: Pressurization button

[0103] 52: Test Operation Key

[0104] 52a, 52b: Indicator lights

[0105] 521: Start button

[0106] 522: Stop button

[0107] 53, 54, 55: Display area

[0108] 531, 532, 533, 534, 541, 542, 551, 552: Markers

[0109] 60: Data display screen

[0110] 61: Judgment Result Display Section

[0111] 62: Numerical display unit

[0112] 63: Focus on the image display section

[0113] 64: Overall Image Display Unit

[0114] 65: Test piece image display section

[0115] 70: Display panel

[0116] 71, 72: Indicator lights

[0117] 73: Reset button

[0118] 74: Emergency Stop Switch

[0119] 751, 752, 753, 754: Markers

[0120] 80: Camera

[0121] 81: Tripod

[0122] SP: Test Film Detailed Implementation

[0123] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0124] [1. First Implementation Method]

[0125] [1.1. Utilization Forms of Monitoring Devices]

[0126] The following example, a material testing machine, is cited as an example of an analytical apparatus. However, this invention is not limited to a material testing machine, but is generally applicable to inventions of analytical apparatuses. Examples of analytical apparatuses include chromatographic apparatuses, mass analysis apparatuses, optical analysis apparatuses, electronic balances, etc., but other apparatuses may also be included.

[0127] Figure 1 This diagram illustrates an example of how the monitoring device 10 is used. The monitoring device 10 is connected to a camera 80. The camera 80 captures images of the display panel 41 of the control device 40. The control device 40 controls the material testing machine 20.

[0128] The material testing machine 20, for example, is a fatigue testing machine, which is constructed by forming a load frame on a base 21 through a pair of support columns 22a and 22b and a cross head 23, and fixing an upper beam (cross head) 24 on the support columns 22a and 22b.

[0129] A hydraulic actuator 25 is mounted on the base 21. A lower clamp 26a for fixing the lower end of the test piece SP is mounted on the piston rod 25a of the hydraulic actuator 25. Furthermore, an upper clamp 26b for fixing the upper end of the test piece SP is mounted on the upper beam 24 via a force sensor 27. Both the lower clamp 26a and the upper clamp 26b include chuck mechanisms for gripping the test piece SP.

[0130] The force sensor 27 detects the test force applied to the test piece SP.

[0131] The hydraulic actuator 25 controls the direction and quantity of pressurized oil via a servo valve 28, thereby extending and retracting the piston rod 25a. As a result, a test force is applied to the test piece SP, which is fixed between the upper clamp 26b and the lower clamp 26a. The stroke of the hydraulic actuator 25, i.e., the displacement of the test piece SP, is detected by a differential transformer 29 mounted on the hydraulic actuator 25.

[0132] A hydraulic source 30 is provided in the material testing machine 20. The hydraulic source 30 supplies hydraulic pressure to the hydraulic actuator 25 to drive the hydraulic actuator 25. That is, the hydraulic actuator 25 is driven by the hydraulic pressure supplied from the hydraulic source 30, and the hydraulic pressure is adjusted by the servo valve 28, thereby causing the piston rod 25a to extend and retract. The hydraulic source 30 includes a hydraulic pump 30a, a manifold 30b, a piping valve 30c, and a load valve 30d. In addition to supplying the hydraulic pressure generated by the hydraulic pump 30a to the material testing machine 20, the manifold 30b can also supply hydraulic pressure to other devices (e.g., other material testing machines).

[0133] Piping valve 30c and load valve 30d open and close according to instructions from control device 40. When piping valve 30c is opened, hydraulic pressure generated by hydraulic pump 30a is introduced into manifold 30b. Furthermore, when load valve 30d is opened, hydraulic pressure within manifold 30b is introduced into hydraulic actuator 25. With both piping valve 30c and load valve 30d open, hydraulic pressure generated by hydraulic pump 30a can be introduced into hydraulic actuator 25 via servo valve 28.

[0134] The control device 40 acquires the test force signal FS output from the force sensor 27, and performs an analog-to-digital (A / D) conversion on the test force signal FS to generate test force information. The control device 40 acquires the displacement signal DS output from the differential transformer 29, and performs an A / D conversion on the displacement signal DS to generate displacement information. The control device 40 generates command information based on the test force information and the displacement information. The control device 40 generates a command signal CS by performing a digital-to-analog (D / A) conversion on the command information, and outputs the generated command signal CS to the servo valve 28.

[0135] The servo valve 28 controls the direction and quantity of hydraulic oil pressure on the hydraulic actuator 25 according to the command signal CS output from the control device 40. Alternatively, an amplifier can be installed between the material testing machine 20 and the control device 40 to amplify the test force signal FS, displacement signal DS, and command signal CS respectively.

[0136] The material testing machine 20 is controlled by the control device 40, for example, to perform fatigue testing on a test piece SP. During the fatigue test, the material testing machine 20 repeatedly applies a tensile stress σ to the test piece SP. The tensile stress σ and the upper limit of the number of repetitions are preset. The upper limit of the number of repetitions is, for example, 10. 2 10 times 8 Second-rate.

[0137] The camera 80 is, for example, fixed to a tripod 81 and positioned to capture the entire display panel 41 of the control device 40. The monitoring device 10 acquires images of the display panel 41 of the control device 40 at predetermined time intervals using the camera 80. Based on the acquired images, the monitoring device 10 digitizes the display on the monitor provided on the display panel 41 of the control device 40.

[0138] Regarding "datafication", refer to... Figure 5 Let me explain further.

[0139] [1.2. Display panel structure of the control device]

[0140] Figure 2 This is a diagram showing an example of the structure of the display panel 41 of the control device 40.

[0141] The display panel 41 includes a power switch 410, function keys 411, and a dial 412. The power switch 410 turns the power to the control device 40 on and off. The function keys 411 instruct the control device 40 to perform specific functions. The dial 412 is used for operations such as changing settings.

[0142] Furthermore, the display panel 41 is equipped with a setting key 413, a numeric keypad 414, and an emergency stop switch 415. The setting key 413 sets the operation of the control device 40. The numeric keypad 414 is used when inputting values. The emergency stop switch 415 causes the material testing machine 20 to stop immediately.

[0143] Furthermore, the display panel 41 is equipped with a display screen 50, a power unit operation key 51, and a test operation key 52. ​​The power unit operation key 51 is used when operating the hydraulic source 30 of the material testing machine 20. The test operation key 52 indicates the start and stop of the test in the material testing machine 20.

[0144] The display screen 50 is, for example, a display screen including a liquid crystal display (LCD) and a touch panel with a touch sensor disposed on the display screen.

[0145] The display screen 50 displays various information about the operation of the material testing machine 20, such as the test force, piston displacement, and remaining time until the end of the fatigue test during the fatigue test performed by the material testing machine 20. In the first embodiment and the embodiments described below, the display screen 50 is specially provided with a cycle count display unit 501 and a specific symbol display unit 502. The cycle count display unit 501 displays the number of stress cycles applied during the fatigue test. The specific symbol display unit 502 displays pre-defined specific symbols (e.g., icons).

[0146] Furthermore, a test status display unit 503 is provided on the display screen 50. The test status display unit 503 indicates whether a test is being performed in the material testing machine 20. On the test status display unit 503, for example, the text "Testing" or "Stopped" is displayed over a predetermined background color. In the first embodiment and the embodiments described below, for example, during the execution of the test, the text "Testing" is displayed over a green background, and during the test being stopped, the text "Stopped" is displayed over a red background.

[0147] The power unit operation keys 51 include a start button 511, a stop button 512, a manifold button 513, and a pressurization button 514. The start button 511 and stop button 512 are used to start and stop the hydraulic pump 30a, respectively. The manifold button 513 is used to open / close the piping valve 30c. Furthermore, the pressurization button 514 is used to open / close the load valve 30d. The start button 511, stop button 512, manifold button 513, and pressurization button 514 each have indicator lights 51a, 51b, 51c, and 51d, respectively.

[0148] The test operation keys 52 include a start key 521 and a stop key 522. The start key 521 and stop key 522 each have an indicator light 52a and an indicator light 52b, respectively. When the user presses the start key 521, the material testing machine 20 begins fatigue testing, and the text "Testing" is displayed on the test status display unit 503 of the display screen 50. Furthermore, when the user presses the stop key 522, the material testing machine 20 stops the ongoing test operation, and the text "Stopped" is displayed on the test status display unit 503 of the display screen 50.

[0149] Here, the indicator lights 51a, 51b, 51c, and 51d included in the display screen 50 and the power unit operation keys 51, and the indicator lights 52a and 52b included in the test operation keys 52, each correspond to an example of a display provided on the display panel 41. Furthermore, the display screen 50 corresponds to an example of a numerical display that displays specific symbols and values.

[0150] On the display panel 41, at least two corners of each display area are marked with barcodes, the barcodes representing identification codes for these display areas. At least one display area is rectangular. The marks are positioned at the four corners of the at least one rectangular display area. As described later, image distortion in images captured by the display panel 41 is corrected by means of the marks positioned at the four corners of the rectangular display areas.

[0151] Figure 2 In the example shown, four markers, 531, 532, 533, and 534, are respectively arranged at the four corners of the rectangular display area 53, which has a numerical display, i.e., a display screen 50. Furthermore, markers 541 and 542 are arranged at the two corners of the rectangular display area 54, which has a power unit operation key 51. Additionally, markers 551 and 552 are arranged at the two corners of the rectangular display area 55, which has a test operation key 52. ​​These markers can be labels affixed to the display panel 41 before the camera 80 takes a picture, or they can be pre-printed on the display panel 41.

[0152] Each mark is accompanied by a barcode that identifies the corresponding display area. The barcode contains information indicating the location of the mark bearing the barcode within the corresponding display area, such as "top left," "top right," "bottom right," or "bottom left" of the display area.

[0153] For example, the barcodes affixed to labels 531, 532, 533, and 534 respectively contain an identification code for the display area 53 where the display screen 50 is located, and information indicating the location of each position within the display area 53. Similarly, the barcodes affixed to labels 541 and 542 contain an identification code for the display area 54 where the power unit operation key 51 is located, and information indicating the location of each position within the display area 54. Likewise, the barcodes affixed to labels 551 and 552 contain an identification code for the display area 55 where the test operation key 52 is located, and information indicating the location of each position within the display area 54.

[0154] Indicator lights 51a, 51b, 51c, 51d, 52a, and 52b are, for example, light-emitting diodes (LEDs). These indicator lights can be either indicator lights with one (monochrome) illumination color or indicator lights capable of selectively illuminating two or more illumination colors.

[0155] Indicator lights 51a, 51b, 51c, and 51d, through combinations of their illumination patterns, indicate the status of the hydraulic source 30 in the material testing machine 20. Furthermore, indicator lights 52a and 52b, through combinations of their illumination patterns, indicate the execution status of the test operation in the material testing machine 20, that is, whether the test is in progress or stopped.

[0156] Here, the term "lighting state" includes the on or off state of each lamp, and the lighting color or brightness in the on state. In the first embodiment and the embodiments described below, display lamps 51a, 51b, 51c, 51d, 52a, and 52b are monochrome LEDs, and the term "lighting state" refers to the combination of the on and off states of each display lamp.

[0157] Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E as well as Figure 3F This is a diagram illustrating an example of the relationship between the combination of the illumination patterns of the indicator lights 51a, 51b, 51c, and 51d included in the power unit operation key 51 and the state of the hydraulic source of the material testing machine 20. Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E as well as Figure 3F In the diagram, indicator lights 51a, 51b, 51c, and 51d are respectively represented by black to indicate the lit state and white to indicate the off state.

[0158] Figure 3A In the process, the indicator light 51a is lit while the other indicator lights 51b, 51c, and 51d are off, indicating that the hydraulic source of the material testing machine 20 is in the RUN state, that is, the hydraulic pump 30a has been started.

[0159] Figure 3B In the process, the indicator light 51b is lit while the other indicator lights 51a, 51c, and 51d are off, indicating that the hydraulic source of the material testing machine 20 is in a stopped state, that is, the hydraulic pump 30a has stopped.

[0160] Figure 3C In the process, the indicator lights 51a, 51c, and 51d are lit while the indicator light 51b is off, indicating that the hydraulic power source of the material testing machine 20 is in the manifold load (LOAD_MANI) state, that is, the piping valve 30c is open.

[0161] Figure 3D In the process, the indicator lights 51b and 51c are lit while the indicator lights 51a and 51d are off, indicating that the hydraulic power source of the material testing machine 20 is in the manifold stop (STOP_MANI) state, that is, the piping valve 30c is closed.

[0162] Figure 3E In the process, the indicator lights 51a and 51d are lit while the indicator lights 51b and 51c are off, indicating that the hydraulic source of the material testing machine 20 is in a loaded state, that is, the piping valve 30c and the load valve 30d are open.

[0163] and, Figure 3F In the process, the fact that all indicator lights 51a, 51b, 51c, and 51d are off indicates that the hydraulic power source of the material testing machine 20 is in the POWER OFF state, that is, the power supply for the operation of the hydraulic pump 30a, piping valve 30c, and load valve 30d is disconnected.

[0164] Figure 4A as well as Figure 4B This is a diagram illustrating an example of the relationship between the combination of the illumination states of the indicator lights 52a and 52b of the test operation key 52 and the test states in the material testing machine 20. Figure 4A In this process, the indicator light 52a is lit while the indicator light 52b is off to indicate that the material testing machine 20 is in the start state, that is, in the state of performing fatigue testing.

[0165] and, Figure 4B In this process, the indicator light 52a is lit while the indicator light 52b is off, indicating that the material testing machine 20 is in a stopped state, that is, the fatigue test has been stopped or suspended.

[0166] [1.3. Structure of the monitoring device]

[0167] Figure 5 This is a diagram illustrating an example of the structure of the monitoring device 10. The monitoring device 10 may include, for example, a personal computer, but is not limited thereto, and may also include one or more suitable electronic circuits. Such electronic circuits may include programmed hardware such as a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or a System-on-a-Chip (SoC) FPGA.

[0168] The monitoring device 10 includes a processor 11, a memory 12, an input unit 13, a display unit 14, a machine connection interface (machine connection I / F) 15, and a communication interface (communication I / F) 16.

[0169] Processor 11 includes a central processing unit (CPU) or a micro-processor (MPU), etc.

[0170] The memory 12 includes read-only memory (ROM) or random access memory (RAM), etc. The memory 12 may also include storage devices such as hard disk drives (HDDs) or solid-state drives (SSDs). The memory 12 stores the monitoring program 121 executed by the processor 11, image data 122, hydraulic state recognition model 123a, test state recognition model 123b, judgment result data 124, and numerical data 125. The memory 12 corresponds to an example of an image storage unit.

[0171] Input unit 13 is an input device used by the user to input data or commands to monitoring device 10, such as a keyboard, switch, and mouse. Display unit 14 is a display screen that displays text, graphics, images, etc., such as an LCD. Input unit 13 and display unit 14 may also be configured as a touch panel with a touch sensor disposed on a display screen such as an LCD.

[0172] Machine connection I / F15 is an interface for connecting the monitoring device 10 to peripheral machines, such as a USB transceiver for USB (registered trademark) communication. In the first embodiment, the monitoring device 10 is connected to the camera 80 via the machine connection I / F15.

[0173] The Communication I / F16 is a communication device used by the monitoring device 10 to communicate with other devices (such as other computer devices or server devices) via communication networks such as the Internet or intranet, and to perform wired or wireless communication.

[0174] The processor 11 includes an acquisition unit 111, an image recognition unit 112, a status determination unit 113, an update unit 114, and an output unit 115, serving as functional elements or functional units. These functional elements included in the processor 11 are implemented, for example, by the processor 11 executing a monitoring program 121 stored in the memory 12.

[0175] The acquisition unit 111 acquires images captured by the camera 80 of the display panel 41 of the control device 40 at predetermined time intervals via the machine connection I / F 15. In the first embodiment, the images acquired by the acquisition unit 111 are images captured by the display panel 41 when the material testing machine 20 performs or stops fatigue testing. Furthermore, in the first embodiment, the acquisition unit 111 directly acquires images captured by the display panel 41 from the camera 80, but the target of the acquired images is not limited to the camera 80. For example, the acquisition unit 111 may also acquire images by sequentially reading images captured in advance at predetermined time intervals and stored in the memory 12 or other devices (e.g., a server device on a communication network). Hereinafter, images acquired by the acquisition unit 111 at a specific time will be referred to as "images of interest," distinguishing them from images acquired before that time. In the first embodiment, the acquisition unit 111 saves the acquired images as part of the image data 122 in the memory 12.

[0176] The image recognition unit 112 performs image recognition processing on the image of interest acquired by the acquisition unit 111. Through image recognition processing, the image recognition unit 112 acquires the identification codes shown by the barcodes attached to the marks 531, 532, 533, 534, 541, 542, 551, and 552 on the display panel 41 contained in the image of interest. Furthermore, based on the position of these marks and the identification codes, the image recognition unit 112 determines the image area on the image of interest corresponding to each of the display areas 53, 54, and 55 equipped with the display.

[0177] Therefore, even if the position of the camera 80 deviates during the fatigue test of the material testing machine 20, the image area of ​​each display area on the image of interest can be properly determined. Such deviation of the camera 80 position may occur, for example, if the user moves the position of the camera 80 to operate the control device 40.

[0178] Furthermore, the image recognition unit 112 corrects image distortion in the determined image of interest corresponding to the display areas 53, 54, and 55 based on the positions of the markers 531, 532, 533, and 534 arranged at the four corners of the rectangular display area 53 on the image of interest. Thus, even if the shape of image distortion changes during fatigue testing of the material testing machine 20, the image distortion can be appropriately assessed based on the marker arrangement, allowing for rapid image correction. Such changes in image distortion may occur, for example, when the user moves the optical axis of the camera 80 to operate the control device 40.

[0179] Figure 6 This diagram illustrates the correction of image distortion performed by the image recognition unit 112. Figure 6 The upper part of the figure shows an example of an image of interest acquired by the acquisition unit 111 from the camera 80. In the example shown, the image of interest is distorted into a trapezoidal shape. This image distortion occurs because the normal direction of the display panel 41 is tilted relative to the optical axis of the camera 80.

[0180] Image recognition unit 112 from Figure 6 In the image of interest above, groups of markers bearing the same identification code are identified. Specifically, the image recognition unit 112 identifies a group of four markers 531, 532, 533, and 534 bearing the identification code of display area 53, a group of two markers 541 and 542 bearing the identification code of display area 54, and a group of two markers 551 and 552 bearing the identification code of display area 55. Thus, the image recognition unit 112... Figure 6 As shown in the left figure in the middle section, the display area 53 is defined as the image area of ​​interest, which is a trapezoid with the positions of the four marks 531, 532, 533, and 534 as the four corners.

[0181] Moreover, the image recognition unit 112, such as Figure 6 As shown in the central diagram of the middle section, the display area 54 is defined as a rectangular image region in the image of interest, with the positions of the two markers 541 and 542 as the two diagonal corners. Similarly, the image recognition unit 112... Figure 6 As shown in the right figure in the middle section, the display area 55 is defined as the image area of ​​interest, which is a rectangle with the positions of the two markers 551 and 552 as the two opposite corners.

[0182] Next, the image recognition unit 112, based on the determined trapezoidal image area of ​​the display area 53, ( Figure 6 (Left image in the middle section) A correction function for correcting image distortion is calculated. In the first embodiment, the correction function is defined as a projective transformation matrix used to correct the trapezoidal image region of the determined display area 53 into a rectangular image.

[0183] The image recognition unit 112 uses the projective transformation matrix of the predetermined correction function to correct the image. Figure 6 The images in the middle section are used to obtain... Figure 6The following section shows corrected partial images of display areas 53, 54, and 55. The image recognition unit 112 sends the corrected partial images of display areas 53, 54, and 55 to the status determination unit 113. Furthermore, the image recognition unit 112 saves the corrected partial images of display areas 53, 54, and 55 as part of image data 122, associated with the image of interest.

[0184] in addition, Figure 6 The example shown is from Figure 6 The image in the upper section that is identified as the focus is Figure 6 Image correction is performed on the respective image areas of display areas 53, 54, and 55 shown in the middle section, thereby obtaining... Figure 6 The image below shows a corrected local image, but the image correction process is not limited to this. For example, the image recognition unit 112 can also use a correction function calculated as described above to correct the local image. Figure 6 The image of interest shown in the previous paragraph is subjected to overall image correction, and the image is directly obtained from the image of interest that has been corrected. Figure 6 The following section shows the corrected partial images of display areas 53, 54, and 55.

[0185] Furthermore, the image recognition unit 112 generates numerical data representing the values ​​displayed on the display screen 50 based on the image of interest acquired by the acquisition unit 111.

[0186] Specifically, the image recognition unit 112 calculates the size of a specific symbol displayed on the specific symbol display unit 502 on the display screen 50 within the image of interest based on a corrected partial image of the display area 53 on which the display screen 50 is equipped. Furthermore, based on the calculated size of the specific symbol, the image recognition unit 112 infers the size of the text representing the value displayed on the display screen 50 within the image of interest. The image recognition unit 112 generates numerical data representing the value displayed on the display screen 50 using image recognition processing based on the inferred text size. Therefore, by using appropriate text sizes for text recognition, the possibility of misrecognizing the displayed value and generating incorrect numerical data can be reduced.

[0187] The numerical data may include, in addition to the cycle count data displayed on the cycle count display unit 501 of the display screen 50, the numerical data of the cycle count. Figure 2The numerical data shown, such as "50.1215" and "16.6601", pertains to test force, piston displacement, etc. The image recognition unit 112 sends the generated cycle number data to the update unit 114. Furthermore, the image recognition unit 112 stores the generated numerical data, excluding the cycle number, in association with the image of interest as part of the numerical data 125 in the memory 12.

[0188] The state determination unit 113 determines the state of the material testing machine 20 based on the image of interest acquired by the acquisition unit 111. Specifically, based on the partial images of the display area 54 and the display area 55 extracted by the image recognition unit 112, it determines the hydraulic source state of the material testing machine 20 represented by the combination of the lighting patterns of the indicator lights 51a, 51b, 51c, and 51d, and the test state of the material testing machine 20 represented by the combination of the lighting patterns of the indicator lights 52a and 52b, respectively. Therefore, the user does not need to determine the state of the material testing machine represented by the combination of lighting patterns, thus improving user convenience. Furthermore, the partial images of the display area 54 equipped with indicator lights 51a, 51b, 51c, and 51d, and the partial images of the display area 55 equipped with indicator lights 52a and 52b, respectively correspond to an example of the "first captured image".

[0189] The state determination unit 113 uses artificial intelligence, for example, to infer the state of the material testing machine 20 based on the first captured image. The memory 12 stores a hydraulic state recognition model 123a, which learns through machine learning the relationship between images captured of various combinations of illumination patterns of indicator lights 51a, 51b, 51c, and 51d and the hydraulic source state of the material testing machine 20 corresponding to those combinations. Furthermore, the memory 12 also stores a test state recognition model 123b, which learns through machine learning the relationship between images captured of various combinations of illumination patterns of indicator lights 52a and 52b and the test state of the material testing machine 20 corresponding to those combinations. The hydraulic state recognition model 123a and the test state recognition model 123b are equivalent to state recognition models.

[0190] The state determination unit 113 infers the hydraulic source state of the material testing machine 20 by inputting a partial image of the display area 54 extracted from the image of interest, i.e., the first captured image, into the hydraulic state recognition model 123a. Furthermore, the state determination unit 113 infers the test state of the material testing machine 20 by inputting a partial image of the display area 55 extracted from the image of interest, i.e., the first captured image, into the test state recognition model 123b. Therefore, the state determination unit 113 does not need to individually identify the lighting pattern of each indicator light, and can quickly determine the state of the material testing machine 20. Additionally, the partial image of the display area 54 input to the hydraulic state recognition model 123a, i.e., the first captured image, and the partial image of the display area 55 input to the test state recognition model 123b, i.e., the first captured image, can each be a partial image after image distortion correction in the image recognition unit 112.

[0191] The outputs of the hydraulic state identification model 123a and the test state identification model 123b respectively include the prediction results and accuracy of the predictions regarding the hydraulic source state, and the prediction results and accuracy of the predictions regarding the test state. The accuracy of the predictions is expressed as a percentage value representing the accuracy of the corresponding prediction.

[0192] The state determination unit 113 determines the hydraulic source state of the material testing machine 20 based on the prediction result using the hydraulic state recognition model 123a, and uses the accuracy of the prediction output by the hydraulic state recognition model 123a as the determination accuracy. The state determination unit 113 attaches a warning message corresponding to the determination accuracy to the data representing the determination result of the hydraulic source state and outputs it to the output unit 115. The warning message is, for example, set as "GOOD" indicating that the determination accuracy is above a specified value, or "WARNING" indicating that the determination accuracy is below a specified value. Thus, the user can easily grasp the reliability of the determination result based on the warning message attached to the determination result. The specific process for determining the hydraulic source state will be referred to... Figure 10 To be continued later.

[0193] Similarly, the state determination unit 113 determines the test state of the material testing machine 20 based on the prediction result using the test state recognition model 123b, and uses the accuracy of the prediction output by the test state recognition model 123b as the determination accuracy. The state determination unit 113 attaches a warning message corresponding to the determination accuracy to the data representing the test state determination result and outputs it to the output unit 115. The warning message is set to "GOOD" or "WARNING" for example, depending on whether the determination accuracy is above or below a specified value. The specific process for determining the test state will be referred to... Figure 11To be continued later.

[0194] The updating unit 114 determines the loop count value, i.e., the loop count determination value, that should be associated with the captured images based on the loop count numerical data generated by the image recognition unit 112 from a series of captured images during the fatigue test. Specifically, the updating unit 114 updates or maintains the loop count determination value associated with the image of interest relative to the loop count determination value associated with the previous image (previous value) based on the loop count numerical data generated from the series of captured images. Here, the previous image refers to the captured image acquired by the acquisition unit 111 before the image of interest. Hereinafter, the value of the loop count numerical data generated by the image recognition unit 112 based on the captured images will be referred to as the loop count generation value.

[0195] More specifically, the update unit 114 calculates a regression line of the loop number generation value generated based on at least three captured images taken before the image of interest, relative to time. Furthermore, when the difference between the loop number generation value (the value of interest) generated based on the image of interest and the regression line is within a predetermined amount, the update unit 114 updates the loop number determination value using the value of interest. Thus, the update unit 114 can associate a highly reliable loop number determination value, taking into account the temporal variation trend of the loop number represented by the regression line, with the captured images. The update unit 114 sends the updated loop number determination value to the output unit 115 and saves it in the memory 12 as part of the numerical data 125, associated with the image of interest.

[0196] On the other hand, if the difference is not within a predetermined amount, the update unit 114 determines whether the predetermined number of cycle number generation values ​​generated based on a predetermined number of images taken before the image of interest are continuously the same fixed value. Furthermore, when the predetermined number of cycle number generation values ​​are continuously the same fixed value, the update unit 114 updates the cycle number determination value associated with the image of interest using this fixed value. Moreover, when the predetermined number of cycle number generation values ​​are continuously the same fixed value, the update unit 114 updates the regression line. That is, the update unit 114 discards the current regression line and recalculates the regression line using three or more cycle number generation values ​​generated based on subsequently taken images. Therefore, even if the stress application period of the fatigue test is changed and the temporal variation trend of the cycle number changes, the update unit 114 can avoid associating incorrect cycle number determination values ​​with the captured images. Furthermore, after updating the regression line, the update unit 114 appropriately grasps the new temporal variation trend of the cycle number using the updated regression line, thereby enabling the association of highly reliable cycle number determination values ​​with the captured images.

[0197] On the other hand, when at least one of the generated loop count values ​​is different from the others, the updating unit 114 does not update the loop count determination value associated with the image of interest, but maintains it at the same value as the previous value. Therefore, even if the numerical data of the loop count changes erroneously during a period when there is no change in the loop count, the updating unit 114 maintains the loop count determination value at the previous value, thereby keeping the error of the loop count determination value small. When the updating unit 114 does not update the loop count determination value, it sends a warning message to the output unit 115 for the loop count determination value that remains the same as the previous value. The warning message can be, for example, set to "WARNING". Thus, the user can easily understand the possibility that the reliability of the loop count determination value may be low based on the warning message attached to the loop count determination value. Alternatively, when the updating unit 114 updates the loop count determination value, it can also attach a warning message such as "GOOD" to the loop count determination value. The update unit 114 saves the cycle number determination value with warning information and the image of interest as part of the numerical data 125 to the memory 12.

[0198] Whenever the acquisition unit 111 acquires a captured image, the output unit 115 outputs a data display screen to the display unit 14. The data display screen includes the acquired captured image (the image of interest), a display of the hydraulic source status and test status determined by the status determination unit 113 regarding the image of interest, and a cycle count determination value determined by the update unit 114. Alternatively, the output unit 115 may also print out the display of the hydraulic source status and test status, along with the cycle count determination value, via a printer device (not shown).

[0199] Figure 7 This diagram shows an example of a data display screen output from the output unit 115 to the display unit 14. Figure 7 The data display screen 141 shown includes a focus image display unit 142, a partial image display unit 143, a partial image display unit 144, a partial image display unit 145, a determination result display unit 146, and a numerical display unit 147.

[0200] The image of interest is displayed on the image of interest display unit 142. Local image display units 143, 144, and 145 respectively display local images of display areas 53, 54, and 55, which have been extracted from the image of interest and had their image distortion corrected. The determination result display unit 146 displays the determination results of the hydraulic source state and test state of the material testing machine 20, based on the image of interest. Furthermore, the numerical display unit 147 displays the cycle number determination value related to the image of interest.

[0201] Figure 7In the determination result display unit 146, "Hydraulic source accuracy" and "Test status accuracy" respectively represent warning messages attached to the determination results of the hydraulic source status and test status received from the status determination unit 113. Furthermore, Figure 7 The “Cycle Count Accuracy” displayed on the numerical display unit 147 is based on the warning message attached to the cycle count display value received by the output unit 115 from the update unit 114.

[0202] The monitoring device 10, having the aforementioned structure, generates numerical data and data indicating the state determination results of the material testing machine 20 based on the display screen 50 (i.e., the display panel 41 of the control device 40) and the images captured by the indicator lights 51a, 51b, 51c, 51d, 52a, and 52b. Furthermore, the generated numerical data and determination result data are output to the display unit 14 and stored in the memory 12 in association with the captured images.

[0203] Therefore, users can easily grasp the operating status of the material testing machine 20 based on the generated numerical data and judgment result data. Thus, even if the number of images taken at specified time intervals during the fatigue test performed by the material testing machine 20 is large, users can easily grasp the points of change (or timing of change) in the state of the material testing machine 20 by viewing the output of numerical data and judgment result data generated based on each image.

[0204] Furthermore, while this embodiment uses a material testing machine as an example, graphical information, such as waveforms displayed on the screens of other analytical devices, can be acquired. Known waveform analysis algorithms or graphical analysis algorithms are applied to the acquired graphical information to extract information from the captured images. Moreover, the method is not limited to graphical information; even simply extracting warning displays can achieve the same effect as described in this application.

[0205] [1.4. Processing in the monitoring device]

[0206] [1.4.1 Overall Process]

[0207] Next, refer to Figure 8 The flowchart shown illustrates the operation of the monitoring device 10. Figure 8 The process shown begins, for example, when the user inputs an execution instruction via input unit 13. When inputting the execution instruction, the user can input the end conditions for capturing images, such as the number of times the target image is captured or the target acquisition time.

[0208] When processing begins, the acquisition unit 111 acquires a captured image from the camera 80 (S11). The acquired captured image is the image of interest. Next, the image recognition unit 112 performs image extraction processing on the image of interest (S12). In the image extraction processing, the image recognition unit 112 extracts a partial image of the display area 53 equipped with the display screen 50, a partial image of the display area 54 equipped with display lamps 51a, 51b, 51c, and 51d, and a partial image of the display area 55 equipped with display lamps 52a and 52b from the image of interest. Details of the image extraction processing will be referred to later. Figure 9 To illustrate.

[0209] Next, the state determination unit 113 performs state determination processing. This state determination processing includes hydraulic source state determination processing (S13) and test state determination processing (S14). In the hydraulic source state determination processing, the state determination unit 113 inputs a partial image of the display area 54 extracted from the image of interest to the hydraulic state recognition model 123a to infer the hydraulic source state, and determines the hydraulic source state of the material testing machine 20 based on the inference result and its accuracy. Furthermore, in the test state determination processing, the state determination unit 113 inputs a partial image of the display area 55 extracted from the image of interest to the test state recognition model 123b to infer the test state, and determines the test state of the material testing machine 20 based on the inference result and its accuracy. Details regarding the hydraulic source state determination processing and the test state determination processing will be provided later. Figure 10 , Figure 11 To illustrate.

[0210] Next, the image recognition unit 112 performs numerical data generation processing (S15). In the numerical data generation processing, the image recognition unit 112 generates numerical data based on a local image of the display area 53 extracted from the image of interest, according to the numerical display shown on the display screen 50. The numerical data generation processing will be described later. Figure 12 To illustrate.

[0211] Subsequently, the update unit 114 performs loop count update processing. In the loop count update processing, the update unit 114 first determines whether the test state determined by the state determination unit 113 in the test state determination processing (S14) is "Start" (i.e., test start state) (S16). And, when the test state is "Start" (S16; Yes), the update unit 114 performs the first loop count update processing (S17). On the other hand, when the test state is not "Start" (i.e., "Stop" (S16; No), the update unit 114 performs the second loop count update processing (S18).

[0212] In the first loop count update process, based on the increase in loop count during the fatigue test, the update unit 114 determines the loop count determination value to be associated with the image of interest based on the change in the loop count generation value generated from a series of captured images. Furthermore, in the second loop count update process, based on the case where the loop count does not change upon the cessation of the fatigue test, the update unit 114 determines the loop count determination value to be associated with the image of interest based on the loop count generation value generated from a series of captured images. The first and second loop count update processes will be referred to later. Figure 13 as well as Figure 14 To illustrate.

[0213] Next, the output unit 115 outputs data representing the results of the state determination process and the loop number update process to the display unit 14 (S19). Specifically, the output unit 115 outputs, for example... Figure 7 The data display screen showing the image of interest is displayed on the display unit 14. Then, the acquisition unit 111, based on the user's input... Figure 8 When the processing begins, the acquisition end condition is input to the input unit 13 to determine whether the acquisition of the captured image has ended (S20).

[0214] Furthermore, when the acquisition of the captured image is completed (S20; Yes), the acquisition unit 111 terminates this process. On the other hand, when the acquisition of the captured image has not yet been completed (S20; No), the acquisition unit 111 returns to step S11 to acquire a new captured image.

[0215] [1.4.2 Actions in Image Extraction Processing]

[0216] Figure 9 It means Figure 8 The flowchart shown is an example of an image extraction process in the processing described.

[0217] The so-called "image extraction processing" refers to the process of extracting partial images from the image of interest, including partial images of the display area 53 equipped with the display screen 50, partial images of the display area 54 equipped with display lamps 51a, 51b, 51c, and 51d, and partial images of the display area 55 equipped with display lamps 52a and 52b.

[0218] In the image extraction process, firstly, the image recognition unit 112 detects a mark projected onto the image of interest (S121). For each mark, a barcode is attached to identify the area of ​​the display equipped with the mark. For each group of marks with the same identification code, the image recognition unit 112 determines the image area on the image of interest corresponding to the area equipped with the display, based on the position of each mark and the identification code (S122). In the first embodiment and the embodiments described later, the area equipped with the display refers to the display area 53 equipped with a numerical display, i.e., a display screen 50; the display area 54 equipped with display lamps 51a, 51b, 51c, and 51d; and the display area 55 equipped with display lamps 52a and 52b.

[0219] The image recognition unit 112 calculates a correction function for correcting image distortion in the captured image based on the positions of four markers with the same identification code on the captured image (S123). As described above, the correction function is defined, for example, by a projective transformation matrix. The image recognition unit 112 uses the calculated correction function to correct the image distortion in the image region determined in step S122 (S124). The image recognition unit 112 extracts each corrected image region as a local image (S125). The image recognition unit 112 sends the extracted local images of each region to the state determination unit 113 and saves them as part of the image data 122 in association with the captured image to the memory 12.

[0220] [1.4.3 Actions in the hydraulic source status determination process]

[0221] Figure 10 It means Figure 8 The flowchart is an example of the process for determining the state of the hydraulic source.

[0222] The so-called "hydraulic source status determination processing" refers to the processing of determining the hydraulic source status of the material testing machine 20 based on a local image of the display area 54 extracted from the image of interest.

[0223] In the hydraulic source state determination process, firstly, the state determination unit 113 inputs a partial image of the display area 54 equipped with indicator lights 51a, 51b, 51c, and 51d, received from the image recognition unit 112, to the hydraulic state recognition model 123a to infer the hydraulic source state of the material testing machine 20 (S131). Next, the state determination unit 113 determines whether the accuracy of the inference output by the hydraulic state recognition model 123a is greater than or equal to a predetermined value (e.g., 70%) (S132).

[0224] Furthermore, when the estimation accuracy is above a predetermined value (S132; Yes), the state determination unit 113 determines the estimation result obtained by the hydraulic state recognition model 123a as the determination result of the hydraulic source state regarding the image of interest (S133). On the other hand, when the estimation accuracy is less than a predetermined value (S132; No), the state determination unit 113 sets the determination result regarding the image of interest to be the same as the determination result of the hydraulic source state already determined for the previous image (S134).

[0225] Next, the state determination unit 113 takes the predicted accuracy output by the hydraulic state recognition model 123a as the determination accuracy, and attaches a warning message corresponding to the determination accuracy to the determination result determined in step S133 or step S134 (S135). The state determination unit 113 sends the determination result of the hydraulic source state with the warning message to the output unit 115, and saves it to the memory 12 as part of the determination result data 124 in association with the image of interest.

[0226] [1.4.4 Processing in Test Status Determination]

[0227] Figure 11 It means Figure 8 The flowchart is an example of the test status determination process.

[0228] The so-called "test status determination processing" refers to the processing of determining the test status of the material testing machine 20 based on a local image of the display area 55 extracted from the image of interest.

[0229] In the test state determination process, firstly, the state determination unit 113 inputs a partial image of the display area 55 equipped with indicator lights 52a and 52b, received from the image recognition unit 112, to the test state recognition model 123b to infer the test state of the material testing machine 20 (S1401). This inference is set as the first inference. During this inference, the test state recognition model 123b outputs the inference accuracy regarding this inference.

[0230] Next, the status determination unit 113, based on the partial image of the display area 53 received from the image recognition unit 112, infers the test status of the material testing machine 20 according to the display of the test status display unit 503 on the display screen 50 (S1402). This inference is designated as a second inference. Specifically, the status determination unit 113 counts, for example, the number of green pixels and the number of red pixels on the test status display unit 503, and calculates, for example, the ratio of the number of green pixels to the total number of counts and the ratio of the number of red pixels to the total number of counts as percentages. Furthermore, when the ratio of the number of green pixels is 20% or more greater than the ratio of the number of red pixels, the status determination unit 113 infers that the display of the test status display unit 503 is "testing" and the test status is "start," and uses the ratio of the number of green pixels as the accuracy of the inference.

[0231] Furthermore, when the ratio of red pixels is more than 20% greater than the ratio of green pixels, the status determination unit 113 assumes that the test status display unit 503 is displaying "Stopped" and the test status is "STOP," and uses the ratio of red pixels as the accuracy of the assumption. On the other hand, when the difference between the ratio of green pixels and the ratio of red pixels is less than 20%, the status determination unit 113 considers the display of the test status display unit 503 and the assumption of the test status to be uncertain.

[0232] Next, the state determination unit 113 determines whether the result of the first prediction in step S141 is the same as the result of the second prediction in step S1402 (S1403). And, when the result of the first prediction is the same as the result of the second prediction (S1403; Yes), the state determination unit 113 takes the result of the first prediction or the result of the second prediction as the determination result regarding the test state (S1404).

[0233] On the other hand, in step S1403, when the results of the first prediction and the second prediction are different (S1403; No), the state determination unit 113 determines whether the result of the second prediction is uncertain (S1405). Furthermore, when the result of the second prediction is not uncertain (S1405; No), the state determination unit 113 takes the result of the second prediction as the determination result regarding the test state (S1406).

[0234] On the other hand, in step S1405, when the result of the second prediction is indeterminate (S1405; Yes), the state determination unit 113 determines whether the prediction accuracy of the first prediction is greater than or equal to a predetermined value (e.g., 90%) (S1407). Furthermore, when the prediction accuracy of the first prediction is greater than or equal to the predetermined value (S1407; Yes), the state determination unit 113 uses the result of the first prediction as the determination result regarding the test state (S1408).

[0235] Furthermore, on the other hand, in step S1407, when the accuracy of the first prediction is less than a predetermined value (S1407; No), the state determination unit 113 sets the determination result of the current test state (i.e., regarding the image of interest) to be the same as the determination result of the test state that has been determined regarding the previous image (S1409).

[0236] Next, the status determination unit 113 attaches a warning message (S1410) corresponding to the determination accuracy of the test status determined in steps S1404, S1406, S1408, or S1409. Regarding the determination accuracy, for example, when the determination result is determined in step S1404, it can be set to 95%. Furthermore, when the determination result is determined in step S1406, the determination accuracy of the determination result can be set to the same value as the prediction accuracy in the second prediction.

[0237] Furthermore, when a determination result is determined in step S1408 or step S1409, the determination accuracy of these determination results can be set to the same value as the prediction accuracy in the first prediction. When the determination accuracy is a predetermined value (e.g., 90%) or higher, the status determination unit 113 can attach "GOOD" as a warning message to the determination result; if the determination accuracy is less than the predetermined value, it can attach "WARNING" as a warning message.

[0238] The status determination unit 113 sends the determination result of the test status with warning information to the output unit 115, and stores it in the memory 12 as part of the determination result data 124 in association with the image of interest.

[0239] [1.4.5 Processing in Numerical Data Generation and Processing]

[0240] Figure 12 It means Figure 8 A flowchart illustrating an example of the numerical data generation and processing workflow.

[0241] The term "numerical data generation processing" refers to the process of generating numerical data based on a local image of the display area 53 extracted from the image of interest, according to the numerical display shown on the screen 50.

[0242] First, the image recognition unit 112 detects a specific symbol from a partial image of the display area 53 extracted in the image extraction process, and calculates the size of the specific symbol in the partial image (S151). In the first embodiment, the specific symbol is a specific icon displayed on the specific symbol display unit 502 of the display screen 50. Next, the image recognition unit 112 infers the size of the numerical text displayed in the partial image of the display area 53 based on the calculated size of the specific symbol (S152). Then, based on the inferred text size, the image recognition unit 112 recognizes the numerical numbers displayed on the display screen 50 in the partial image of the display area 53 through text recognition processing, and generates numerical data based on the numerical display (S153). The image recognition unit 112 sends the numerical data of the loop count in the generated numerical data to the update unit 114. Furthermore, the image recognition unit 112 saves the generated numerical data, excluding the numerical data of the loop count, as part of the numerical data 125 associated with the image of interest to the memory 12.

[0243] [1.4.6 Actions in the first loop number update process]

[0244] Figure 13 It means Figure 8 The flowchart is an example of the process for updating the first loop number in the process.

[0245] The so-called "first cycle number update processing" refers to the process of determining the cycle number judgment value that should be associated with the image of interest when the material testing machine 20 is in the test start state.

[0246] As described above, in the first cycle count update process, based on the increase in cycle count during the fatigue test, the update unit 114 determines whether to update the cycle count determination value associated with the image of interest relative to the previous value (the cycle count determination value associated with the previous image) based on the change in the cycle count generation value generated from a series of captured images. As described above, the value represented by the numerical data of the cycle count generated by the image recognition unit 112 based on the captured images is called the cycle count generation value, and the cycle count generation value for the image of interest is called the attention value. Furthermore, the captured image taken before the image of interest is called the previous image, and the cycle count determination value determined by the update unit 114 for the previous image is called the previous value.

[0247] In the first cycle number update process, the update unit 114 first determines whether the value of interest is greater than the previous value (S1601). When the value of interest is greater than the previous value (S1601; Yes), the update unit 114 calculates a regression line of the cycle number generation value with respect to time based on the cycle number generation values ​​of at least three captured images taken before the image of interest (S1602). In the first embodiment, the regression line is calculated using the cycle number generation values ​​obtained from the most recent captured images, which are three to one hundred images included in the previous images taken before the image of interest. For example, the update unit 114 may temporarily store the cycle number generation values ​​of the most recent captured images used for calculating the regression line in the memory 12 as regression line data, and calculate the regression line based on these temporarily stored cycle number generation values.

[0248] The update unit 114 determines whether the difference between the value of interest and the regression line is within a predetermined amount (e.g., ±5%) (S1603). If the difference is within the predetermined amount (S1603; Yes), the update unit 114 updates the cycle number determination value based on the value of interest (S1604). The update unit 114 sends the updated cycle number determination value to the output unit 115 and saves it to the memory 12 as part of the numerical data 125, associated with the image of interest.

[0249] On the other hand, in step S1603, when the difference exceeds a predetermined amount (S1603; No), the update unit 114 determines whether the most recent first predetermined number of cycle count generation values ​​remains a fixed value (S1605). Here, the "most recent first predetermined number of cycle count generation values" refers to the cycle count generation value generated based on the most recent first predetermined number of captured images, including the image of interest and multiple captured images taken before the image of interest. In the first embodiment, the first predetermined number is, for example, 10. The update unit 114 may, for example, temporarily store the most recent cycle count generation value generated based on the most recent first predetermined number of captured images in the memory 12, thereby performing the determination in step S1603.

[0250] In step S1605, when the most recent first predetermined number of cycle count generation value remains at a fixed value (S1605; Yes), the update unit 114 updates the cycle count determination value using the fixed value (S1606). The update unit 114 sends the updated cycle count determination value to the output unit 115 and saves it to the memory 12 as part of the numerical data 125 in association with the image of interest.

[0251] Furthermore, the update unit 114 excludes the cycle number generation value, which is obtained based on the image of interest and the image taken before the image of interest, from the calculation objects of the regression line in step S1602, thereby initializing and updating the regression line (S1607). Specifically, for example, the update unit 114 deletes the excluded cycle number generation value from the regression line data temporarily stored in the memory 12.

[0252] On the other hand, in step S1605, if the most recent first predetermined cycle count generation value does not remain at a fixed value (S1605; No), the update unit 114 does not update the cycle count determination value (S1609). That is, the update unit 114 determines the previous value (i.e., the value of the cycle count determination value in the previous image) as the cycle count determination value for the image of interest. Moreover, the update unit 114 attaches a warning message to the determined cycle count determination value (S1610). The warning message is, for example, "WARNING". The update unit 114 sends the cycle count determination value with the warning message to the output unit 115 and saves it in the memory 12 as part of the numerical data 125 in association with the image of interest.

[0253] Furthermore, on the other hand, in step S1601, when the value of interest is lower than the previous value (S1601; No), the update unit 114 determines whether the most recent second predetermined number of cycle counts generated remains a fixed value (S1608). In the first embodiment, the second predetermined number is, for example, 50. The update unit 114 may, for example, temporarily store the most recent second predetermined number of cycle counts generated in the memory 12, thereby performing the determination in step S1608.

[0254] Furthermore, in step S1608, when the most recent second predetermined number of cycle count generation value remains at a fixed value (S1608; Yes), the update unit 114 moves the processing to step S1606 and updates the cycle count determination value using the fixed value.

[0255] On the other hand, in step S1608, if the most recent second predetermined cycle number generation value does not remain at a fixed value (S1608; No), the update unit 114 moves the processing to step S1609 and does not update the cycle number determination value. That is, the update unit 114 determines the previous value as the cycle number determination value for the image of interest.

[0256] [1.4.7 Actions in the second loop number update process]

[0257] Figure 14 It means Figure 8 The flowchart is an example of the second loop number update process.

[0258] The so-called "second cycle number update processing" refers to the process of determining the cycle number judgment value that should be associated with the image of interest when the material testing machine 20 is in the test stop state.

[0259] As described above, in the second cycle number update process, based on the fact that the cycle number does not change during the cessation of the fatigue test, the update unit 114 determines the cycle number determination value that should be associated with the image of interest based on the cycle number generation value generated by a series of captured images.

[0260] First, the update unit 114 determines whether the most recent third-predetermined number of loop counts generated remains a fixed value (S171). Here, the "most recent third-predetermined number of loop counts generated" refers to the loop count generated based on the most recent third-predetermined number of captured images, which includes the image of interest and multiple captured images taken before the image of interest. In the first embodiment, the third predetermined number is, for example, 10. The update unit 114 may temporarily store the most recent loop count generated based on the most recent third-predetermined number of captured images in the memory 12, thereby performing the determination in step S171.

[0261] In step S171, when the most recent third predetermined cycle number generation value remains at a fixed value (S171; Yes), the update unit 114 updates the cycle number determination value using the fixed value (S172). The update unit 114 sends the updated cycle number determination value to the output unit 115 and saves the numerical data 125 in association with the image of interest to the memory 12.

[0262] On the other hand, when the most recent third predetermined cycle number generation value does not remain a fixed value (S171; No), the update unit 114 does not update the cycle number determination value (S173). That is, the update unit 114 determines the previous value as the cycle number determination value for the image of interest. Moreover, the update unit 114 attaches a warning message to the determined cycle number determination value (S174). The warning message is, for example, "WARNING". The update unit 114 sends the cycle number determination value with the warning message to the output unit 115 and saves it in the memory 12 as part of the numerical data 125 in association with the image of interest.

[0263] [1.5. Variations]

[0264] [1.5.1 Variation Example 1]

[0265] In the described embodiment, a camera 80 is connected to the monitoring device 10 to capture images of the display panel 41 of the control device 40, but the number of cameras connected to the monitoring device 10 is not limited to one. For example, in addition to the camera 80 capturing images of the display panel 41, an additional camera capturing images of the entire material testing machine 20 may be connected to the monitoring device 10. In this case, the acquisition unit 111 acquires, for example, the image of the display panel 41 captured by the camera 80 as the image of interest, and acquires an overall image of the material testing machine 20 from the additional camera at the same time as the image of interest.

[0266] Moreover, at this time, the output unit 115 can Figure 15 The data display screen 60 shown is output to the display unit 14. The data display screen 60 includes, for example, a judgment result display unit 61, a numerical display unit 62, a focus image display unit 63, an overall image display unit 64, and a test piece image display unit 65. On the judgment result display unit 61 and the numerical display unit 62, and... Figure 7 Similarly, the judgment result display unit 146 and the numerical display unit 147 on the data display screen show the judgment results of the hydraulic source state and the test state, as well as the cycle count judgment value, respectively. Furthermore, on the image display unit 63, and... Figure 7 Similarly, the image of interest display unit 142 displays the image of interest (i.e., the image captured by the display panel 41 acquired by the acquisition unit 111 this time). Furthermore, the overall image display unit 64 displays an overall image of the material testing machine 20 captured simultaneously with the image of interest, and the test piece image display unit 65 displays an image of the vicinity of the test piece SP in the overall image that is magnified.

[0267] [1.5.2 Variation Example 2]

[0268] In the described embodiment, as an example of a control device for controlling the material testing machine 20, a control device 40 is shown that includes a display panel 41, which has a display screen 50 and six indicator lights 51a, 51b, 51c, 51d, 52a, and 52b. However, the display panel of the control device is not limited to a structure like display panel 41. The display panel of the control device may also not include a display screen and may function as a display, or it may include two or more display screens. Moreover, the display panel may include any number of indicator lights other than six to function as a display.

[0269] For example, the control device may include, Figure 16 The simplified display panel 70 shown is an example of a display panel with a simple structure. This simplified display panel 70 could be, for example, a display panel for an operation box attached to the control device 40. As an example, in... Figure 16The display panel 70 shown includes an indicator light 71 for displaying the power status of the hydraulic source 30 and an indicator light 72 for displaying warnings of abnormal conditions. The indicator lights 71 and 72 can express specific states of the material testing machine 20 through combinations of their illumination patterns.

[0270] Furthermore, the display panel 70 is equipped with a reset button 73 to clear warnings of abnormal states, and a button related to... Figure 2 The same emergency stop switch 74 as the emergency stop switch 415 shown is also present. Furthermore, at the four corners of the rectangular display panel 70, there are... Figure 2 The same barcodes 751, 752, 753, and 754 are shown, including mark 531. The barcodes on these marks contain information indicating that the area enclosed by these marks is an area equipped with display lamps 71 and 72 as displays.

[0271] At this time, the monitoring device 10 can also acquire the image captured by the display panel 70 through the same action as described above, determine the specific state of the material testing machine 20 represented by the combination of the lighting patterns of the indicator lights 71 and 72, and output the data representing the result of the determination to the display unit 14. For example, the memory 12 of the monitoring device 10 stores a learned state recognition model that has learned the relationship between various lighting patterns of the indicator lights 71 and 72 and the specific state of the material testing machine 20, and the state determination unit 113 can use the state recognition model to determine the specific state.

[0272] [2. Second Implementation]

[0273] Figure 17 This figure shows an example of the structure of the monitoring device 10a according to the second embodiment. The monitoring device 10a, like the monitoring device 10 described in the first embodiment, can be used in combination with the material testing machine 20. The usage configuration of the monitoring device 10a is the same as in the first embodiment, therefore, description is omitted. Furthermore, in the second embodiment, the same symbols are used for structures common to the first embodiment, and descriptions are omitted.

[0274] like Figure 17 As shown, the monitoring device 10a includes a processor 11a. The processor 11a is configured in the same way as the processor 11. That is, the processor 11a includes the acquisition unit 111, the image recognition unit 112, the status determination unit 113, the update unit 114, and the output unit 115. Furthermore, the processor 11a includes an image processing unit 211.

[0275] The monitoring device 10a has a memory 12a. The memory 12a is configured similarly to the memory 12. That is, the memory 12a stores the monitoring program 121, image data 122, hydraulic state recognition model 123a, test state recognition model 123b, judgment result data 124, and numerical data 125. Furthermore, the memory 12a stores shielding image data 221. Shielding image data 221 is image data of a shielding image. Shielding image data 221 includes, for example, images of shielding images. Figure 2 The data of the masking image corresponding to the partial image of the display area 54 shown, and the .... Figure 2 The data of the masking image corresponding to the partial image of the display area 55 shown.

[0276] The image processing unit 211 performs image processing on the partial images of display area 54 and display area 55 extracted by the image recognition unit 112 using masking image data 221, thereby generating a local image for determination. The local image for determination refers to an image processed to easily determine the lighting state of display lamps 51a, 51b, 51c, and 51d, or the lighting state of display lamps 52a and 52b. The local image for determination corresponds to an example of an image for determination.

[0277] The masking image is an image that obscures a portion of the local image. That is, when the masking image is superimposed on the local image extracted by the image recognition unit 112, a portion of the local image is obscured while the remaining portion is not. Here, the portion of the local image obscured by the masking image is called the masked region, and the portion of the local image not obscured by the masking image is called the unmasked region. The image processing unit 211 generates a local image for determination by extracting the unmasked region from the local image. The unmasked region is an area where the lighting state of indicator lights 51a, 51b, 51c, 51d, or indicator lights 52a and 52b can be easily determined. The masked region corresponds to an example of the first region, and the unmasked region corresponds to an example of the second region.

[0278] As an example, let's consider the case where the state determination unit 113 determines the lighting state of the indicator lamp 51a. In this case, the state determination unit 113 determines whether the indicator lamp 51a is on or off, or the lighting color of the indicator lamp 51a. The shielding image corresponding to the display area 54 is used in the local image extracted by the image recognition unit 112 to shield areas that do not change with the change in the lighting state of the indicator lamp 51a, or areas that change only slightly. The change in lighting state includes switching between on and off of the indicator lamp 51a, and changes in the lighting color of the indicator lamp 51a. Therefore, with the change in the lighting state of the indicator lamp 51a, the unshielded area of ​​the local image changes significantly. Therefore, by using the local image for determination generated by the image processing unit 211, compared with the case where no shielding image is used, the state determination unit 113 can determine the lighting state of the indicator lamp 51a with higher accuracy. The same applies to indicator lights 51b, 51c, 51d, 52a, 52b, and display area 55.

[0279] The shielding image corresponding to the display area 54 can also be an image that completely overlaps with a partial image obtained by photographing the display area 54. Furthermore, the shielding image corresponding to the display area 54 can also be the image corresponding to each of the start button 511, stop button 512, manifold button 513, and pressurization button 514. That is, the shielding image corresponding to the display area 54 can also include four shielding images.

[0280] Similarly, the shielding image corresponding to the display area 55 can also be an image that completely overlaps with a partial image obtained by capturing the display area 55. Furthermore, the shielding image corresponding to the display area 55 can also be the image corresponding to the start button 521 and the stop button 522 respectively. That is, the shielding image corresponding to the display area 55 can also include two shielding images.

[0281] Image processing unit 211, in the state determination unit 113 shown in Modified Example 2, performs the following steps: Figure 16 When determining the illumination state of the indicator lights 71 and 72, image processing of a local area can also be performed. The indicator lights 71 and 72 are, for example, a simple structure that allows switching between on and off states. The state determination unit 113 determines whether indicator light 71 is on or off, and whether indicator light 72 is on or off, based on a local image of the display panel 70.

[0282] At this time, the shielding image corresponding to the display panel 70 shields the portion of the display lamp 71 with small brightness changes when it is lit and when it is turned off, as well as the portion of the display lamp 72 with small brightness changes when it is lit and when it is turned off.

[0283] The shielded image data 221 can be pre-generated and stored in memory 12 by a device different from the monitoring device 10a, or it can be generated by the monitoring device 10a.

[0284] Here, the operation of the monitoring device 10a when generating shielded image data 221 is shown. Figure 18 .

[0285] Figure 18 This is a flowchart illustrating an example of the masking image generation process performed by the image processing unit 211. Figure 19 This is a schematic diagram illustrating an example of shielding image generation processing, showing an example of processing an image captured by the display lamp 71. "Shielding image generation processing" refers to the process of generating a shielding image.

[0286] Figure 18 The actions are performed, for example, by the image processing unit 211.

[0287] Image processing unit 211 acquires an ON image (S201) of the object portion that is the target of the masking image generation. An ON image is an image captured when the light on the object portion is on. Image processing unit 211 acquires an OFF image (S202) of the object portion. An OFF image is an image captured when the light on the object portion is off. The OFF image acquired in step S202 is an image obtained by capturing the same object portion as the ON image acquired in step S201.

[0288] Figure 19 The images 301 and 302 represent the on and off states when the object is the indicator light 71. The on image 301 is an image captured when the indicator light 71 is on; specifically, it is an image captured when the indicator light 71 is lit. The off image 302 is an image captured when the indicator light 71 is off; specifically, it is an image captured when the indicator light 71 is off. The on and off images 301 and 302 are, for example, cropped from the image of interest. The on and off images 301 and 302 can be images without distortion correction or images after distortion correction.

[0289] return Figure 18 The image processing unit 211 calculates the brightness difference between the open and closed images for each position in both the open and closed images (S203). For example, the image processing unit 211 correlates the coordinates in the open image with the coordinates in the closed image, calculates the brightness of a first position in the open image and the brightness of a second position in the closed image corresponding to the first position, and calculates the difference in brightness. In calculating the brightness, the image processing unit 211 can calculate either the average brightness of a region with a predetermined area or the brightness of a single pixel.

[0290] The image processing unit 211 generates a masked image (S204) that masks the regions where the brightness difference calculated in step S203 is below a threshold. The brightness value is, for example, 0 to 255. At this time, the threshold for the brightness difference can be set to a value between 0 and 255.

[0291] Figure 19 The shielded image 311 shown includes a shielded region 312 and an unshielded region 313. The shielded region 312 is the area where the brightness difference between the on image 301 and the off image 302 is below a threshold value. The unshielded region 313 is the area in the shielded image 311 other than the shielded region 312. The brightness difference between the on image 301 and the off image 302 in the unshielded region 313 exceeds the threshold value and is greater than that in the shielded region 312.

[0292] In other words, the local image for determination generated by extracting the unmasked area is an image obtained by extracting the region whose brightness changes significantly according to the lighting pattern of the object part. Therefore, by using the local image for determination, the lighting pattern of the object part can be determined more easily and with higher accuracy compared to the case where the local image for determination is not used.

[0293] Figure 18 The operation is performed for each object part. As a result, a masking image is generated for each object part. The data of the generated masking image is included in the masking image data 221. The object part is the object that the state determination unit 113 determines to be in display mode or illuminated mode, and includes display lamps 51a-51d, 52a, 52b, 71, 72, etc. The masking image data 221 may include data of the masking images corresponding to all object parts that the state determination unit 113 has set as determination objects.

[0294] Figure 20 This is a flowchart illustrating an example of pre-determination image processing performed by the image processing unit 211. Pre-determination image processing is the process by which the image processing unit 211 generates a local image for determination before the state determination unit 113 performs the determination. For example, pre-determination image processing is performed in… Figure 8 This is performed after step S12 and before step S14.

[0295] In the pre-determination image processing, the image processing unit 211 acquires the local image extracted by the image recognition unit 112 (S211). The image processing unit 211 selects and acquires a shielding image corresponding to the processing performed by the state determination unit 113 (S212). For example, when the state determination unit 113 performs processing to determine the lighting state of the display lamp 71, in step S212, the image processing unit 211 acquires data of the shielding image corresponding to the local image of the display lamp 71 from the shielding image data 221. The size of the shielding image selected in step S212 may not be exactly the same as the local image acquired in step S211. For example, the image processing unit 211 may also acquire a shielding image in step S212 that overlaps with a part of the local image acquired in step S211.

[0296] The image processing unit 211 uses the shielded image acquired in step S212 to extract the unshielded area from the local image acquired in step S211, thereby generating a local image for determination (S213). The local image for determination generated in step S213 is an image of the object region processed by the state determination unit 113. The image processing unit 211 outputs the generated local image for determination in a state that the state determination unit 113 can process (S214).

[0297] exist Figure 20 The local image generated during the processing is not limited to an image containing only one object region. For example, the image processing unit 211 may apply a masking image to a portion of a local image containing multiple object regions. In this case, in step S213, the unmasked region is extracted from the image of the object region to which the masking image is applied, and images of other object regions are extracted in an unprocessed state. Thus, a portion of the multiple object regions reflected in the local image is processed by the image processing unit 211.

[0298] In step S214, the image processing unit 211 may also apply multiple masking images to a local image acquired in step S211. For example, the image processing unit 211 may also acquire multiple masking images in step S212 and apply the multiple masking images to a local image to generate a local image for determination.

[0299] Furthermore, a masked image can also contain images of multiple object parts.

[0300] In the second embodiment, the state determination unit 113 performs test state determination processing (step S14) using the local image generated by the image processing unit 211. As a result, the test state of the monitoring device 10a can be determined with higher accuracy compared to the case where the local image is not used.

[0301] The shielded image is generated using open and closed images of the object being photographed. These open and closed images are, for example, images captured by the camera 80 when the material testing machine 20 and monitoring device 10a are set up. In this case, the shielded image generated through the shielded image generation process shields areas affected by the camera 80's shooting environment, including ambient light. By utilizing the shielded image, the image processing unit 211 can extract areas in the camera 80's captured image that are prone to image changes corresponding to the lighting pattern of the object being photographed as unshielded areas. Therefore, the influence of the shooting environment related to the determination of the state determination unit 113 can be suppressed, making the determination made by the state determination unit 113 less susceptible to changes in ambient light, thereby achieving highly robust determinations. Thus, it is expected that the determination accuracy of the state determination unit 113 can be further improved.

[0302] The monitoring device 10a can be used for image processing of the shielded image in display area 53, display area 54 and display area 55.

[0303] Moreover, it can also be passed through Figure 18 The local image for determination generated by the pre-determination image processing is used in the hydraulic source state determination process of step S12. At this time, the image processing unit 211 performs image processing on the local image of the display area 53 using a masking image, thereby generating a local image for determination of the display area 53. The state determination unit 113 uses the local image for determination related to the display area 53 generated by the image processing unit 211 to perform the hydraulic source state determination process. Therefore, compared to the case where the local image is not used, the state of the hydraulic source of the material testing machine 20 can be determined with higher accuracy.

[0304] [3. Third Implementation Method]

[0305] In the second embodiment, a process for generating a shielding image based on the brightness difference between the open and closed images is described. Besides brightness, the monitoring device 10a can also perform processing based on chroma and hue to generate the shielding image. This example will be described as a third embodiment. The structures of the monitoring device 10a and the material testing machine 20 in the third embodiment are the same as in the second embodiment.

[0306] Figure 21 This is a flowchart illustrating an example of the masking image generation process in the third embodiment. Figure 21 The masking image generation process is performed, for example, by the image processing unit 211 using images captured by the camera 80.

[0307] Actions and references for steps S201 and S202 Figure 18The described actions are common. Following step S202, the image processing unit 211 calculates the tonal difference between the open and closed images for each position in both the open and closed images (S211). For example, the image processing unit 211 correlates the coordinates in the open image with the coordinates in the closed image, calculates the tonal value of a first position in the open image and the tonal value of a second position in the closed image corresponding to the first position, and calculates the difference between the calculated values. In calculating the tonal value, the image processing unit 211 can calculate either the average tonal value of a region with a predetermined area or the tonal value of a single pixel.

[0308] Subsequently, the image processing unit 211 generates a masked image (S222) that masks the regions where the hue difference calculated in step S221 is below a threshold. The hue value is, for example, 0 to 255. At this time, the threshold for the hue difference can be set to a value between 0 and 255.

[0309] Furthermore, the image processing unit 211 calculates the chroma difference between the open and closed images for each position in both the open and closed images (S223). For example, the image processing unit 211 correlates the coordinates in the open image with the coordinates in the closed image, calculates the chroma value of a first position in the open image and the chroma value of a second position in the closed image corresponding to the first position, and calculates the difference between the calculated values. In calculating the chroma value, the image processing unit 211 can calculate either the average chroma value of a region with a predetermined area or the chroma value of a single pixel. Subsequently, the image processing unit 211 generates a masked image that masks the region where the chroma difference calculated in step S221 is below a threshold (S224). The chroma value is, for example, 0 to 255. At this time, the threshold for the chroma difference can be set to a value between 0 and 255.

[0310] The thresholds for brightness, chroma, and hue are determined, for example, based on images captured experimentally in a test environment simulating the actual setup of the monitoring device 10a.

[0311] Furthermore, the image processing unit 211 calculates the brightness difference between the open and closed images for each position in both the open and closed images (S225). For example, the image processing unit 211 correlates the coordinates in the open image with the coordinates in the closed image, calculates the brightness value of a first position in the open image and the brightness value of a second position in the closed image corresponding to the first position, and calculates the difference between the calculated values. In the process of calculating the brightness value, the image processing unit 211 can calculate either the average brightness value of a region with a predetermined area or the brightness value of a single pixel. Subsequently, the image processing unit 211 generates a masking image that masks the region where the brightness difference calculated in step S221 is below a threshold (S226).

[0312] Here, the processing of steps S221, S223, and S225 will be explained.

[0313] Hue values ​​can be calculated, for example, through processes 1 to 3 below.

[0314] Step 1. Obtain the R, G, and B values ​​of the first position of the opened image. Here, the R, G, and B values ​​refer to pixel values. The following example illustrates this: the opened image is a 24-bit color image, with R values ​​ranging from 0 to 255, G values ​​ranging from 0 to 255, and B values ​​ranging from 0 to 255.

[0315] Step 2. Determine the maximum and minimum values ​​among R, G, and B, and set the maximum value as NMAX and the minimum value as NMIN.

[0316] Step 3. Calculate the hue value H. Specifically, when R is at its maximum, the hue H value is calculated using equation (1) below. When G is at its maximum, the hue H value is calculated using equation (2) below. When B is at its maximum, the hue H value is calculated using equation (3) below. When the values ​​of R, G, and B are all the same, set H = 0.

[0317] H=60×((GB)÷(NMAX-NMIN))…(1)

[0318] H=60×((BR)÷(NMAX-NMIN))+120…(2)

[0319] H=60×((RG)÷(NMAX-NMIN))+240…(3)

[0320] The chroma value S can be calculated, for example, using NMAX and NMIN obtained in processes 1 to 2, by the following formula (4).

[0321] S=(NMAX-NMIN)÷NMAX…(4)

[0322] The brightness value V can be calculated, for example, using NMAX and NMIN obtained in processes 1 to 2, by the following formula (5).

[0323] V = NMAX…(5)

[0324] This method can also be applied to the process of calculating the brightness of the image in the second embodiment.

[0325] The image processing unit 211 calculates the hue, chroma, and brightness of the turned-on image through the aforementioned process, and calculates the hue, chroma, and brightness of the turned-off image through the same process.

[0326] Through the above actions, a hue masking image, a chroma masking image, and a masking image are generated. As described in the second embodiment, the hue masking image, chroma masking image, and masking image are included in the masking image data 221. The hue masking image refers to the masking image generated in step S222, and the chroma masking image refers to the masking image generated in step S224. In other words, the hue masking image is a masking image that masks areas where the hue difference is below a threshold, and the chroma masking image is a masking image that masks areas where the chroma difference is below a threshold. These masking images can be used in the image recognition processing performed by the image recognition unit 112, and in the hydraulic source state determination processing (step S13) and test state determination processing (step S14) performed by the state determination unit 113. Using these masking images improves the accuracy of the processing for recognizing or determining the display mode of the display screen 50, or the illumination mode of the display area 53, display area 54, and display area 55.

[0327] The object recognized by the image recognition unit 112 and the object determined by the status determination unit 113 are defined as the object location. The object location may be, for example, the entire display of the display screen 50, the cycle count display 501 displayed on the display screen 50, the specific symbol display 502, or the test status display 503. Furthermore, the object location may be, for example, the start button 511, the stop button 512, the manifold button 513, the pressurization button 514, the start button 521, the stop button 522, the indicator light 71, or the indicator light 72.

[0328] In the third embodiment, the recognition performed by the image recognition unit 112 and the determination performed by the state determination unit 113 can be performed based on one or more of hue, chroma, and brightness. When the display mode or illumination mode of the object changes, causing a color change in the image captured by the camera 80, it is preferable to perform recognition or determination based on hue or chroma. For example, the display lamps 51a to 51d, 52a, and 52b can be configured to switch between two or more illumination colors. When the display lamp 51a switches between a first illumination color and a second illumination color, it is easy to identify the first illumination color and the second illumination color based on the hue difference or chroma difference. Furthermore, this is also suitable, for example, for situations where the color of the item displayed on the display screen 50 changes.

[0329] Furthermore, when the color tone changes little due to variations in the display or illumination pattern of the object, using chroma difference allows for more accurate identification and judgment compared to not using chroma difference. Specifically, this can be illustrated by cases where the first and second illuminated colors are in the same color family or are similar colors. Moreover, chroma difference is also suitable for identification or judgment when the object's structure involves switching between the illumination and extinguishing of a light source.

[0330] Furthermore, when there are significant variations in hue due to changes in the display or illumination patterns of the object's parts, utilizing hue differences allows for more accurate identification and judgment compared to cases where hue differences are not used. Specifically, examples include cases where the first and second highlight colors are not homonyms or are complementary colors.

[0331] Furthermore, when the brightness varies significantly depending on the display or illumination pattern of the object part, utilizing the brightness difference allows for more accurate identification and judgment compared to cases where the brightness difference is not utilized. Specifically, this can be illustrated by cases where the object part is a structure that switches between lighting and extinguishing the light source.

[0332] Image processing unit 211 performs processing using at least one of tone-masking image, chroma-masking image, and luminance-masking image. Figure 20 The image processing before determination is shown. At this time, the image processing unit 211 can also select any one of the tone masking image, chroma masking image, and luminance masking image according to the object of determination, i.e., the object region, performed by the state determination unit 113. At this time, it is also possible to pre-set which of the three masking images—tone masking image, chroma masking image, and luminance masking image—is applicable for each object region.

[0333] For example, the image processing unit 211 calculates the average hue, average chroma, and average brightness of the unmasked area for both the open and closed images of the target region. The image processing unit 211 also calculates the differences in the average hue, chroma, and brightness between the open and closed images of the target region. Furthermore, the image processing unit 211 determines the largest difference among these three average differences. When the difference in the average hue is the largest, the image processing unit 211 associates a hue-masked image with the target region. Similarly, when the difference in the average chroma is the largest, it associates a chroma-masked image with the target region. Finally, when the difference in the average brightness is the largest, it associates a brightness-masked image with the target region.

[0334] Furthermore, the image processing unit 211 can also perform pre-determination image processing using a hue-masking image, pre-determination image processing using a chroma-masking image, and pre-determination image processing using a luminance-masking image, respectively. In this case, the state determination unit 113 selects any one of the local image for determination generated using a hue-masking image, a local image for determination generated using a chroma-masking image, and a local image for determination generated using a luminance-masking image. The state determination unit 113 uses the selected local image for determination to perform the hydraulic source state determination processing in step S13 or the test state determination processing in step S14.

[0335] When generating the local image for determination used by the image recognition unit 112, the image processing unit 211 can also perform pre-determination image processing using a hue-masking image, pre-determination image processing using a chroma-masking image, and pre-determination image processing using a luminance-masking image, respectively. In this case, the image recognition unit 112 selects to use any one of the local images for determination generated using a hue-masking image, a chroma-masking image, or a luminance-masking image.

[0336] Thus, according to the structure of the third embodiment, the monitoring device 10a has shielded image data 221 including a hue-masking image, a chroma-masking image, and a luminance-masking image, and the image processing unit 211 performs image processing using the shielded images. Therefore, when the state determination unit 113 determines the display mode of the display screen 50 or the illumination mode of the display area 53, display area 54, or display area 55, the determination accuracy can be further improved, making the determination easier. Specifically, for object areas lacking any of the hue difference, chroma difference, or luminance difference accompanying changes in display mode or illumination mode, high-precision determination can be performed. The processing performed by the image recognition unit 112 is also improved by having the image processing unit 211 perform image processing using the hue-masking image, chroma-masking image, and luminance-masking image.

[0337] Furthermore, since the display or illumination mode can be determined based on the hue, saturation, and brightness of the captured image, the influence of the camera's shooting environment can be suppressed, achieving a highly robust determination that is not easily affected by changes in ambient light.

[0338] Furthermore, in addition to brightness changes caused by variations in display or illumination patterns, changes in hue or chroma can also be used for determination. Therefore, even when there are as many as three or four state changes in the display or illumination patterns of the object, the display or illumination pattern can be accurately determined.

[0339] [4. Fourth Implementation Method]

[0340] In the second and third embodiments, the following example is described: the image processing unit 211 generates a local image for determination, and the state determination unit 113 performs hydraulic source state determination processing (step S13) and / or test state determination processing (step S14) using the local image for determination.

[0341] As an example of the present invention, the state determination unit 113 may also utilize the masked image in the hydraulic source state determination process (step S13) and / or the test state determination process (step S14). Regarding this example, see [reference needed]. Figure 22 Let me explain.

[0342] Figure 22 This is a flowchart illustrating an example of the operation of the state determination unit 113 in the fourth embodiment. Figure 22 The action shown is to replace Figure 11 The step S1402 is executed.

[0343] The material testing machine 20 and monitoring device 10a in the fourth embodiment have the same structure as in the third embodiment. The monitoring device 10a stores the masked image data 221, which includes a hue masking image, a chroma masking image, and a luminance masking image, generated by the image processing unit 211 or other devices, into the memory 12.

[0344] The state determination unit 113 acquires the local image corrected by the image recognition unit 112 (S231). The state determination unit 113 overlays the tonal masking image contained in the masking image data 221 onto the local image to extract the unmasked region (S232). The state determination unit 113 calculates the average tonal value of the unmasked region (S233).

[0345] The state determination unit 113 overlays the chroma masking image contained in the masking image data 221 onto the local image acquired in step S231 to extract the unmasked region (S234). The state determination unit 113 calculates the average chroma of the unmasked region (S235).

[0346] The state determination unit 113 overlays the brightness shielding image contained in the shielding image data 221 onto the local image acquired in step S231 to extract the unshielded area (S236). The state determination unit 113 calculates the average brightness of the unshielded area (S237).

[0347] The state determination unit 113 determines the lighting state based on the average hue calculated in step S233, the average chroma calculated in step S235, and the average brightness calculated in step S237 (S238). The determination objects in step S238 are display lamps 51a to 51d, display lamp 52a, and display lamp 52b, etc. The state determination unit 113 infers the test state of the material testing machine 20 based on the determination result of step S238 (S239). The inference result of step S239 corresponds to a second inference.

[0348] In step S239, the state determination unit 113 determines, for example, which of the open and closed images the average hue value calculated in step S233 is closest to. That is, the state determination unit 113 determines which of the two hue values ​​the average hue value of the local image is closest to: the average hue value of the unmasked area in the open image or the average hue value of the unmasked area in the closed image. Similarly, the state determination unit 113 determines which of the two images the average chroma value calculated in step S235 is closest to. Likewise, the state determination unit 113 determines which of the two images the average luminance value calculated in step S237 is closest to.

[0349] The state determination unit 113 determines the lighting state based on whether the average value of the hue, the average value of the chroma, and the average value of the brightness of the local image are closer to an on or off image. For example, if two or more of the average values ​​of the hue, chroma, and brightness are closer to an on image, the state determination unit 113 determines that the object area to be determined is in an on state. Furthermore, if two or more of the average values ​​of the hue, chroma, and brightness are closer to an off image, the state determination unit 113 determines that the object area to be determined is in a off state.

[0350] In this determination, the state determination unit 113 may also pre-set weights for hue, chroma and brightness respectively.

[0351] Specifically, the hue determination result is set to HD. If the average hue value is close to that of an on image, it is considered "HD=1", and if the average hue value is close to that of an off image, it is considered "HD=-1". Similarly, the chroma determination result is set to SD. If the average chroma value is close to that of an on image, it is considered "SD=1", and if the average chroma value is close to that of an off image, it is considered "SD=-1". Likewise, the luminance determination result is set to VD. If the average luminance value is close to that of an on image, it is considered "VD=1", and if the average luminance value is close to that of an off image, it is considered "VD=-1".

[0352] Here, the weighted value corresponding to hue is set as WH, the weighted value corresponding to chroma is set as WS, and the weighted value corresponding to brightness is set as WV. The state determination unit 113 calculates the determination value DD using the following formula (6).

[0353] DD=WH×HD+WS×SD+WV×VD…(6)

[0354] The state determination unit 113 determines that the lighting mode is on when the calculated DD value is 0 or above, and determines that the lighting mode is off when the DD value is negative.

[0355] As described in the fourth embodiment, the state determination unit 113 uses a shielded image for determination, thereby enabling a more accurate determination of the test state of the material testing machine 20 compared to the case where a shielded image is not used.

[0356] Figure 22 Taking the action as an example, the state determination unit 113 can also be used in the hydraulic source state determination process (for example). Figure 10 In step S131, a determination using the masking image is performed. Furthermore, Figure 22 The actions shown can also be performed by the state determination unit 113 alone, or at least a portion of the processing can be performed by the masking image data 221.

[0357] [5. Form and Effect]

[0358] Those skilled in the art will understand that the first to fourth embodiments are specific examples of the following forms.

[0359] (First item)

[0360] A monitoring device for an analysis apparatus according to one embodiment includes: an acquisition unit for acquiring images captured by the display panel of a control device that controls the analysis apparatus; an image storage unit for storing the captured images; and a status determination unit for determining the status of the analysis apparatus based on the captured images.

[0361] According to the monitoring device described in the first item, the status determination unit determines the status of the analysis device based on the image captured by the display panel of the control device that controls the analysis device.

[0362] Therefore, users can easily grasp the status of the analysis device based on the data representing the results of the determination.

[0363] (Second item)

[0364] In the monitoring device described in the first item, the display panel includes a plurality of display lights as a display, and the state determination unit determines the state of the analysis device represented by the combination of the lighting patterns of the plurality of display lights based on a first captured image included in the captured image and corresponding to the plurality of display lights.

[0365] According to the monitoring device described in the second item, the status determination unit determines the status of the analysis device represented by the combination of lighting patterns of the display lamps included in the display panel based on the images of the display lamps.

[0366] Therefore, users do not need to determine the state of the analysis device represented by the combination of lighting patterns themselves. This improves user convenience.

[0367] (Third item)

[0368] The monitoring device described in the second item includes: an image processing unit that determines a first region in the first captured image and a second region whose image change is greater than that of the first region when the illumination mode of the indicator light changes, and generates a determination image containing the second region; the state determination unit acquires the determination image as the first captured image.

[0369] According to the monitoring device described in the third item, by utilizing the image for determining the area of ​​large image change when the illumination pattern of the indicator light changes, it is possible to make judgments with higher accuracy.

[0370] (Item 4)

[0371] In the monitoring device described in the third item, the image processing unit applies the shielding image that shields the first area to the first captured image, thereby generating the determination image.

[0372] According to the monitoring device described in the fourth item, it is possible to easily generate an image for determining areas where the image changes significantly when the illumination pattern of the indicator light changes.

[0373] (Item 5)

[0374] The monitoring device described in the second or third item includes: a state recognition model, which learns the relationship between the first captured image of various lighting patterns of the plurality of display lights and the state of the analysis device through machine learning; the state determination unit inputs the first captured image into the state recognition model to determine the state of the analysis device.

[0375] According to the monitoring device described in item 5, the state determination unit uses a state recognition model that has learned the relationship between the combination of various lighting patterns of multiple indicator lights and the state of the analysis device through machine learning to determine the state of the analysis device.

[0376] Therefore, it is not necessary to individually identify the lighting pattern of each indicator light, thus the status of the analysis device can be quickly determined.

[0377] (Item 6)

[0378] In the monitoring device described in the fifth item, when the state determination unit uses the state recognition model to determine the state of the analysis device, it attaches a warning message to the state determination result corresponding to the accuracy of the determination.

[0379] According to the monitoring device described in item 6, when the state determination unit uses the state recognition model to determine the state of the analysis device, it attaches a warning message to the determination result that corresponds to the accuracy of the determination.

[0380] Therefore, users can easily grasp the reliability of the judgment result based on the warning information attached to the judgment result.

[0381] (Seventh item)

[0382] The monitoring device according to any one of the second to sixth items includes: an image recognition unit that performs image recognition processing on the captured image; on the display panel, at least two corners of each of the display areas are provided with a barcode, the barcode representing an identification code of the display area; the image recognition unit obtains the identification code from the captured image containing the barcode, and determines the image area in the captured image corresponding to each of the display areas based on the position of the barcode and the identification code.

[0383] According to the monitoring device described in the seventh item, the image recognition unit obtains an identification code from an image of a mark disposed on a display panel, and determines an image area disposed on the display based on the position of the mark on the captured image and the identification code.

[0384] Therefore, even if the camera position is moved during the test, the image portion equipped with the display can be easily determined on the captured image based on the marked position, thereby enabling rapid status determination.

[0385] (Item 8)

[0386] In the monitoring device described in the seventh item, at least one of the display areas is rectangular, and the mark is arranged at the four corners of the at least one rectangular display area on the display panel. The image recognition unit corrects the image distortion of the image area based on the position of the mark arranged at the four corners of the rectangular display area.

[0387] According to the monitoring device described in item 8, the image recognition unit corrects image distortion based on the positions of markers arranged at the four corners of a rectangular display area equipped with a display.

[0388] Therefore, even if the camera position is moved during the test and the shape of the image distortion changes, the image distortion can be properly controlled according to the configuration of the markers, so that image correction can be performed quickly.

[0389] (Item 9)

[0390] In the monitoring device described in the seventh or eighth item, at least one of the displays is a numerical display that displays specific symbols and values, and the image recognition unit infers the text size of the value displayed on the numerical display based on the size of the symbol displayed on the numerical display, and generates numerical data displayed on the numerical display by using image recognition processing based on the inferred text size.

[0391] According to the monitoring device described in item 9, the image recognition unit infers the size of the text displayed in the numerical value based on the size of a specific symbol displayed on the numerical display, and uses the inferred text size to recognize the displayed numerical value and generate numerical data.

[0392] Therefore, it can reduce the possibility of generating incorrect numerical data due to misrecognition and display of numerical values ​​caused by using inappropriate text sizes.

[0393] (Item 10)

[0394] In the monitoring device described in the ninth item, the analysis device performs a fatigue test, the numerical display shows a value including the number of cycles of the fatigue test, the acquisition unit acquires a plurality of images taken at predetermined time intervals during the fatigue test, the image recognition unit generates numerical data representing the value of the number of cycles based on each of the plurality of images, the monitoring device includes an update unit, the update unit updates the value of the number of cycles associated with the images, i.e., the number of cycles determination value, based on the number of cycles value data generated by the image recognition unit, the update unit calculates a regression line of the number of cycles value data generated based on each of at least three images taken before the image of interest, which is one of the plurality of images, relative to time, and if the difference between the value of the number of cycles value data generated based on the image of interest, i.e., the attention value, and the regression line is within a predetermined amount, the number of cycles determination value is updated using the attention value.

[0395] According to the monitoring device described in item 10, the updating unit updates the cycle number determination value associated with the image of interest by means of the attention value, provided that the difference between the regression line of the cycle number generated based on the image of interest taken before the image of interest and the value of the cycle number generated based on the image of interest (attention value) is within a predetermined amount.

[0396] Therefore, a highly reliable cycle number determination value that takes into account the time variation trend of the cycle number represented by the regression line can be associated with the captured image.

[0397] (Item 11)

[0398] In the monitoring device described in the tenth item, when the difference between the value of interest and the regression line is not within a predetermined amount, the updating unit updates the cycle number determination value and the regression line by the same value when all the numerical data of the cycle number generated based on a predetermined number of images taken before the image of interest are the same.

[0399] According to the monitoring device described in item eleven, when the difference between the value of interest and the regression line is not within a specified amount, and all the numerical data of the cycle number generated based on the previously specified number of captured images are the same value, the updating unit updates the cycle number determination value using the same value, and updates the regression line.

[0400] Therefore, even if the stress application period of the fatigue test is changed and the time variation trend of the cycle number changes, it is possible to avoid erroneous cycle number determination values ​​being associated with the captured images. Furthermore, the new time variation trend of the cycle number can be grasped through the updated regression line, thereby associating highly reliable cycle number determination values ​​with the captured images.

[0401] (Item 12)

[0402] In the monitoring device described in the tenth or eleventh item, the updating unit does not update the cycle number determination value when the difference between the value of interest and the regression line is not within a predetermined amount, and when at least one of the numerical data of the cycle number generated based on a predetermined number of images taken before the image of interest is a different value from the others.

[0403] According to the monitoring device described in item 12, the updating unit does not update the cycle number determination value when the difference between the value of interest and the regression line is not within a specified amount, and at least one of the cycle number values ​​generated based on the previously specified number of captured images is a different value from the others.

[0404] Therefore, if the numerical data of the loop number changes erroneously during a period when there is no change in the loop number, the loop number determination value is maintained at the previous value, thereby keeping the error of the loop number determination value small.

[0405] (Item 13)

[0406] In the monitoring device described in the twelfth item, the updating unit attaches a warning message to the cycle number determination value when it does not update the cycle number determination value.

[0407] According to the monitoring device described in item thirteen, when the update unit does not update the cycle number determination value, it attaches a warning message to the cycle number determination value.

[0408] Therefore, users can easily understand the possibility that the reliability of the loop count determination value may be low based on the warning information attached to the loop count determination value.

[0409] [6. Other Implementation Methods]

[0410] Furthermore, the monitoring device 10, monitoring device 10a, and material testing machine 20 described in the first to fourth embodiments are merely examples of the form of the material testing machine of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

[0411] For example, in the various embodiments described, the material testing machine 20 may be a fatigue testing machine, but it is not limited to that. The material testing machine 20 can be any testing machine that applies a testing force to the test piece SP, causing it to deform and thus performing material testing. For example, the material testing machine 20 can also be a tensile testing machine, a compression testing machine, a bending testing machine, or a torsion testing machine.

[0412] and, Figure 5 The functional units shown represent functional structures, and their specific installation configurations are not particularly limited. That is, it is not necessarily required to install hardware corresponding to each functional unit; of course, a structure in which multiple functional units' functions are implemented by a single processor executing a program can also be used. Furthermore, in the described embodiment, a portion of the functionality implemented using software can also be implemented using hardware, or vice versa. Regarding... Figure 17 The same applies.

[0413] Furthermore, to facilitate understanding of the processing in monitoring device 10, Figures 8 to 14 , Figure 18 , Figure 20 , Figure 21 , Figure 22 The processing units in the flowchart shown are divided according to the main processing content. They are not subject to... Figures 8 to 14 , Figure 18, Figure 20 , Figure 21 , Figure 22 The flowchart is not limited to the segmentation method or name of the processing units shown. It can also be divided into more processing units based on the processing content, or a single processing unit can contain more processing steps. Furthermore, the processing order of the flowchart is not limited to the example shown.

[0414] Furthermore, the monitoring program 121 executed by the processor 11 of the monitoring device 10 and monitoring device 10a can also be pre-recorded on a recording medium that can be read and recorded by a computer. As the recording medium, magnetic, optical, or semiconductor memory devices can be used. Specifically, examples include portable or stationary recording media such as floppy disks, HDDs, compact disk-only memory (CD-ROM), digital versatile discs (DVDs), Blu-ray discs, optical disks, flash memory, and card-type recording media. Moreover, the recording medium can also be a non-volatile storage device, such as RAM, ROM, or HDD, included in the monitoring device 10. Furthermore, the monitoring program 121 can be pre-stored in a server device or the like, and downloaded from the server device to the memory 12 of the monitoring device 10.

Claims

1. A monitoring device for an analytical apparatus, characterized in that, include: The acquisition unit acquires images captured by the display panel of the control device that controls the analysis device; Image storage unit, for storing the captured images; The status determination unit determines the status of the analysis device based on the captured image; as well as The image recognition unit performs image recognition processing on the captured image. The analytical device performs fatigue testing. The display panel includes a display that shows numerical values. The value displayed on the monitor includes the number of cycles of the fatigue test. The acquisition unit acquires a plurality of images taken at predetermined time intervals during the fatigue test. The image recognition unit generates numerical data representing the cycle number based on each of the plurality of captured images. The monitoring device includes an update unit that updates the value of the loop number associated with the captured image, i.e., the loop number determination value, based on the numerical data of the loop number generated by the image recognition unit. The update unit calculates a regression line of the numerical data of the loop number generated based on each of at least three captured images taken before the image of interest, which is one of the plurality of captured images, relative to time. If the difference between the numerical value of the cycle number generated based on the focus image (i.e., the focus value) and the regression line is within a specified amount, then the cycle number determination value is updated using the focus value.

2. The monitoring device of the analytical apparatus according to claim 1, wherein, The display panel includes multiple indicator lights to serve as a display. The state determination unit determines the state of the analysis device based on the combination of the lighting patterns of the plurality of display lights, which is included in the captured image and corresponds to the plurality of display lights.

3. The monitoring device of the analytical apparatus according to claim 2, wherein, include: The image processing unit determines a first region in the first captured image and a second region whose image change is greater than that of the first region when the illumination pattern of the indicator light changes, and generates a determination image that includes the second region. The state determination unit acquires the determination image as the first captured image.

4. The monitoring device of the analytical apparatus according to claim 3, wherein, The image processing unit applies the shielding image that masks the first region to the first captured image, thereby generating the determination image.

5. The monitoring device of the analytical apparatus according to claim 2 or 3, wherein, include: The state recognition model uses machine learning to learn the relationship between the first captured image of various lighting patterns of the multiple display lights and the state of the analysis device. The state determination unit inputs the first captured image into the state recognition model to determine the state of the analysis device.

6. The monitoring device of the analytical apparatus according to claim 5, wherein, When the state determination unit uses the state recognition model to determine the state of the analysis device, it attaches a warning message to the state determination result that corresponds to the accuracy of the determination.

7. The monitoring device of the analytical apparatus according to claim 2 or 3, wherein, include: On the display panel, at least two corners of each display area are provided with a mark bearing a barcode, the barcode representing an identification code for the display area. The image recognition unit obtains the identification code from the captured image containing the marked image. The image regions in the captured image corresponding to the display area are determined based on the location of the marker and the identification code.

8. The monitoring device of the analytical apparatus according to claim 7, wherein, At least one of the display areas is rectangular, and the markings are arranged at the four corners of at least one rectangular display area on the display panel. The image recognition unit corrects image distortion in the image area based on the positions of the marks disposed at the four corners of the rectangular display area.

9. The monitoring device of the analytical apparatus according to claim 1, wherein, At least one of the displays is a numerical display that shows specific symbols and values. The image recognition unit infers the text size of the numerical value displayed on the numerical display based on the size of the symbol displayed on the numerical display. Numerical data is generated and displayed on the numerical display by using image recognition processing based on the inferred text size.

10. The monitoring device of the analytical apparatus according to claim 1, wherein, When the difference between the focus value and the regression line is not within a specified amount, and when all the numerical data of the cycle number generated based on a specified number of images taken before the focus image are the same, the updating unit updates the cycle number determination value and the regression line using the same value.

11. The monitoring device of the analytical apparatus according to claim 1, wherein, If the difference between the focus value and the regression line is not within a specified amount, and if at least one of the numerical data of the cycle number generated based on a specified number of images taken before the focus image is a different value from the others, the updating unit does not update the cycle number determination value.

12. The monitoring device of the analytical apparatus according to claim 11, wherein, When the updating unit does not update the loop count determination value, it attaches a warning message to the loop count determination value.

Images