Monitoring device and monitoring system

Abstract

To enable detection of a non-light-emitting marker provided on a display panel of a display device from a photographed image of a camera even when exposure of the camera is low.SOLUTION: Disclosed is a monitoring device for a material testing machine equipped with a display device having a non-light-emitting marker on a display panel, which includes: an acquisition part for acquiring an image taken by a camera which photographs the display panel; a first generation part for generating a first processed image by gray-scaling the photographed image; a second generation part for generating a second processed image obtained by blurring the first processed image; a third generation part for generating a third processed image obtained by binarizing the first processed image based on the second processed image; and a first detection part for detecting the marker from the third processed image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a monitoring device and a monitoring system.

Background Art

[0002] Conventionally, a technique for managing the operating state and test data of a material testing machine by a computer has been proposed. For example, Patent Document 1 discloses a remote platform that acquires data regarding the status of a test device and test data by communicating with the test device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, it is conceivable to photograph the display provided in a material testing machine with a camera and remotely monitor the display panel of the display by the photographed image of the camera. Further, in this monitoring, it is also conceivable to detect a marker that the display panel has from the photographed image of the camera for a predetermined use. However, when the marker that the display panel has is a non-emitting marker, if the exposure of the camera is set low, the marker will appear dark in the photographed image of the camera, and there is a possibility that the marker cannot be detected from the photographed image of the camera. Here, it is conceivable to set the exposure of the camera high, but if the exposure of the camera is increased, the influence of ambient light may occur in the photographed image, and the photographed image may be inappropriate as the photographed image used for monitoring the display panel.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to enable detection of a non-emitting marker that the display panel of a display has from the photographed image of a camera even when the exposure of the camera is low. [Means for solving the problem]

[0006] A first aspect of the present invention relates to a monitoring device for a material testing machine equipped with a display unit having a non-emitting marker on a display panel, comprising: an acquisition unit that acquires an image captured by a camera that photographs the display panel; a first generation unit that generates a first processed image obtained by converting the captured image to grayscale; a second generation unit that generates a second processed image obtained by blurring the first processed image; a third generation unit that generates a third processed image obtained by binarizing the first processed image based on the second processed image; and a first detection unit that detects the marker from the third processed image.

[0007] A second aspect of the present invention relates to a monitoring system comprising a monitoring device for a material testing machine equipped with a display unit having a non-emitting marker on a display panel, and a camera for photographing the display panel, wherein the monitoring device comprises an acquisition unit for acquiring an image captured by the camera for photographing the display panel, a first generation unit for generating a first processed image obtained by converting the captured image to grayscale, a second generation unit for generating a second processed image obtained by blurring the first processed image, a third generation unit for generating a third processed image obtained by binarizing the first processed image based on the second processed image, and a first detection unit for detecting the marker from the third processed image. [Effects of the Invention]

[0008] According to a first aspect of the present invention, the first detection unit can highlight and capture non-emitting markers in the image used for detection. Therefore, even when the camera exposure is low, non-emitting markers on the display panel of the display unit can be detected from the image captured by the camera.

[0009] According to a second aspect of the present invention, the same effects as those of the first aspect of the present invention are achieved. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing an example of a monitoring system configuration. [Figure 2] This is a diagram showing an example of the configuration of a fatigue testing machine. [Figure 3] This is a diagram showing an example of the configuration of a display panel. [Figure 4] This is a diagram showing an example of a marker. [Figure 5] This is a diagram showing an example of the configuration of a server device. [Figure 6] This is a diagram showing an example of a grayscale image. [Figure 7] This is a diagram showing an example of a blurred processed image. [Figure 8] This is a diagram showing an example of a binary image. [Figure 9] This is a diagram showing an example of the processing of a specific part and an image processing part. [Figure 10] This is a diagram showing an example of a first screen. [Figure 11] This is a diagram showing an example of a second screen. [Figure 12] This is a flowchart showing the operation of a server device. [Figure 13] This is a diagram showing an example of a binary image. [Figure 14] This is a flowchart showing the operation of a server device. [Figure 15] This is a diagram showing an example of the configuration of a server device. [Figure 16] This is a flowchart showing the operation of a server device. [Embodiments for Carrying Out the Invention]

[0011] [1. First Embodiment] First, the first embodiment will be described. [1-1. Configuration of Monitoring System] FIG. 1 is a diagram showing an example of the configuration of a monitoring system 100 according to this embodiment. The monitoring system 100 is a system that monitors the fatigue testing machine 1. In FIG. 1, a configuration in which the monitoring system 100 monitors one fatigue testing machine 1 is illustrated. In the present embodiment, the case where the monitoring system 100 monitors one fatigue testing machine 1 will be described, but the number of fatigue testing machines 1 to be monitored is not limited to one. The number of fatigue testing machines 1 to be monitored by the monitoring system 100 may be plural. The fatigue testing machine 1 corresponds to an example of a "material testing machine".

[0012] The monitoring system 100 includes the fatigue testing machine 1. The fatigue testing machine 1 includes a testing machine main body 10 and a control device 20. The fatigue testing machine 1 performs a fatigue test on the test piece SP by controlling the testing machine main body 10 with the control device 20. For example, the fatigue testing machine 1 repeatedly applies a tensile stress to the test piece SP. The tensile stress and the number of repetitions are preset. The number of repetitions is, for example, 10 2 times to 10 8 times. The control device 20 corresponds to an example of a "display".

[0013] The monitoring system 100 includes a first camera 2, a second camera 3, a third camera 4, and a server device 5. The third camera 4 corresponds to an example of a "camera". The first camera 2 and the second camera 3 photograph the testing machine main body 10. The third camera 4 photographs the control device 20. In the present embodiment, the case where the monitoring system 100 includes three cameras for one fatigue testing machine 1 to be monitored will be described, but it is not limited thereto. The monitoring system 100 may include one, two, or four or more cameras for one fatigue testing machine 1 to be monitored. In the following description, when the first camera 2, the second camera 3, and the third camera 4 are not distinguished, they are denoted as "camera �".

[0014] Camera 6 is a digital camera that captures a predetermined shooting range. Camera 6 may be a digital still camera that captures still images or a digital video camera that captures moving images. Camera 6 has a communication function and performs data communication via the network NW1. When a picture is taken, Camera 6 transmits the image data of the captured image to the server device 5 via the network NW1.

[0015] In this embodiment, the case in which camera 6 has a communication function is described, but it is not limited to this. At least one of the cameras 6 may be configured without a communication function. In this case, at least one of the cameras 6 is connected to a computer (not shown). This computer has a communication function that can perform data communication via the network NW1, and a function to control the connected camera 6. This computer may be, for example, a personal computer, a tablet computer, or a smartphone. In this case, the camera 6 connected to the computer performs shooting according to the computer's control, and this computer transmits the image data of the captured image to the server device 5.

[0016] Network NW1 connects camera 6 and server device 5 to network NW2 in a communicative manner. Camera 6 is connected to network NW1, and network NW1 is connected to network NW2 via a router, gateway, or other communication network (not shown). Network NW1 is connected to network NW2. Server device 5 is connected to network NW2. Server device 5 corresponds to an example of a "monitoring device".

[0017] Network NW1 is, for example, a LAN (Local Area Network). Network NW1 includes, for example, Ethernet® standard cables, routers (not shown), gateways, etc. Network NW1 may also be a wireless communication line such as Wi-Fi (registered trademark). Network NW2 connects the camera 6 and the server device 5 to enable communication via Network NW1. Network NW2 is, for example, a global network such as the Internet. Network NW2 may also be a WAN (Wide Area Network).

[0018] Server device 5 is a server computer with communication capabilities. Server device 5 may be composed of multiple server computers, and may, for example, be a cloud server. Server device 5 is a so-called application server. Server device 5 is located, for example, in a monitoring room. In the monitoring room, for example, a user monitors the fatigue testing machine 1 by monitoring the screen displayed on the display unit 56 of server device 5.

[0019] [1-2. Configuration of the fatigue testing machine] Figure 2 shows an example of the configuration of the fatigue testing machine 1. The testing machine body 10 performs fatigue testing on the test specimen SP according to instructions from the control device 20. The control device 20 controls the operation of the testing machine body 10.

[0020] As shown in Figure 2, the test machine body 10 has a load frame formed on a base 11 by a pair of support columns 12A and 12B and a yoke 13, and crossheads 14 are movably attached to the support columns 12A and 12B.

[0021] A hydraulic actuator 15 is positioned on the base 11, and a lower fixture 16A for fixing the lower end of the test piece SP is attached to the piston rod 15A of the hydraulic actuator 15. An upper fixture 16B for fixing the upper end of the test piece SP is attached to the crosshead 14 via a load cell 17. Both the lower fixture 16A and the upper fixture 16B are equipped with a chuck mechanism for gripping the test piece SP. The load cell 17 detects the test force acting on the test specimen SP.

[0022] The hydraulic actuator 15's piston rod 15A extends and retracts, controlled by a servo valve 18, which regulates the direction and volume of the pressurized oil. As a result, a test force is applied to the test piece SP fixed between the upper fixture 16B and the lower fixture 16A. The stroke of the hydraulic actuator 15, i.e., the displacement of the test piece SP, is detected by a differential transformer 19 attached to the hydraulic actuator 15.

[0023] The testing machine body 10 is equipped with a power source and a hydraulic power source (not shown). The power source supplies power to various parts of the testing machine body 10. The power source supplies power to various motors, for example, and drives them. The hydraulic power source supplies hydraulic pressure to the hydraulic system that constitutes the testing machine body 10. The hydraulic power source supplies hydraulic pressure to a hydraulic actuator 15, for example, and drives the hydraulic actuator 15. That is, the hydraulic actuator 15 is driven by the hydraulic pressure supplied from the hydraulic power source, causing the piston rod 15A to extend and retract. The hydraulic power source includes, for example, a hydraulic pump and a hydraulic control valve, and generates hydraulic pressure by driving the hydraulic pump. The hydraulic control valve adjusts the hydraulic pressure output from the hydraulic power source. Power is supplied to the hydraulic pump and hydraulic control valve from the power source.

[0024] The control device 20 acquires the test force signal FS output from the load cell 17 and generates test force information by A / D conversion of the test force signal FS. The control device 20 acquires the displacement signal DS output from the differential transformer 19 and generates displacement information by A / D conversion of the displacement signal DS. The control device 20 generates command information based on the test force information and the displacement information. The control device 20 generates a command signal CS by D / A conversion of the command information and outputs the generated command signal CS to the servo valve 18.

[0025] The servo valve 18 controls the direction and amount of pressurized oil to the hydraulic actuator 15 according to the command signal CS output from the control device 20.

[0026] The control device 20 has a display 30. The display 30 is made up of an LCD (Liquid Crystal Display) or the like. The control device 20 displays the test force information, displacement information acquired from the testing machine body 10, and the command information output to the testing machine body 10 on the display 30.

[0027] In Figure 2, amplifiers that amplify the test force signal FS, the displacement signal DS, and the command signal CS may be placed between the test machine body 10 and the control device 20.

[0028] The first camera 2 captures the shooting range PA1 of the test machine body 10. The shooting range PA1 includes almost the entire test machine body 10. The second camera 3 captures the shooting range PA2 of the main body of the testing machine 10. The shooting range PA2 includes the test specimen SP mounted on the main body of the testing machine 10. Specifically, the shooting range PA2 includes the upper fixture 16B, the lower fixture 16A, and the test specimen SP. The third camera 4 captures the shooting range PA3 of the control device 20. The shooting range PA3 includes the entire display panel 21 on the housing of the control device 20, where the display 30 is located. In the following description, images captured by the first camera 2 will be referred to as "test machine body images" and denoted with the symbol "P1". Images captured by the second camera 3 will be referred to as "test piece images" and denoted with the symbol "P2". Images captured by the third camera 4 will be referred to as "panel images" and denoted with the symbol "P3".

[0029] [1-3. Display Surface Configuration] Figure 3 shows an example of the configuration of the display panel 21. The display panel 21 accepts user input to the control device 20 and displays various information indicating the operation or status of the fatigue testing machine 1.

[0030] As shown in Figure 3, the display panel 21 includes a power switch 210, function keys 211, and a dial 212. The power switch 210 receives power on and off operations for the control device 20. The function keys 211 receive operations to instruct the control device 20 to perform a specific function. The dial 212 receives operations such as changing setting values.

[0031] The display panel 21 further includes a setting key 213, a numeric keypad 214, and an emergency stop switch 215. The setting key 213 accepts operations to set the operation of the control device 20. The numeric keypad 214 accepts operations to input numerical values. The emergency stop switch 215 accepts operations to emergency stop the fatigue testing machine 1.

[0032] The display panel 21 further comprises a display 30, a power unit operation key 31, and a test operation key 32. The power unit operation key 31 receives commands for the hydraulic power source of the fatigue testing machine 1. The test operation key 32 receives commands to start and stop material testing in the fatigue testing machine 1. The display 30 is configured as a touch panel comprising a display, for example, an LCD, and a touch sensor positioned on the display surface of the display.

[0033] The display 30 shows various information such as the test force, piston displacement, and remaining time until the end of the fatigue test being performed by the fatigue testing machine 1. The display screen of the display 30 includes a cycle count display unit 301, a specific symbol display unit 302, and a test status display unit 303. The cycle count display unit 301 displays the number of cycles in the fatigue test. The specific symbol display unit 302 displays a predetermined specific symbol (for example, an icon). The test status display unit 303 indicates whether or not a fatigue test is being performed in the fatigue testing machine 1. For example, the test status display unit 303 displays the words "Testing in progress" or "Stopped" overlaid on a predetermined background color.

[0034] The power unit operation key 31 includes a start button 311, a stop button 312, a manifold button 313, and a pressurizing button 314. The start button 311 accepts the operation to start the hydraulic pump. The stop button 312 accepts the operation to stop the hydraulic pump. The manifold button 313 accepts operations to open and close the piping valve. The piping valve is located in the hydraulic piping between the hydraulic pump and the load valve. The pressurizing button 314 accepts operations to open and close the load valve. The load valve is located in the hydraulic piping between the piping valve and the servo valve 18.

[0035] The start button 311 has an indicator lamp 311A, the stop button 312 has an indicator lamp 312B, the manifold button 313 has an indicator lamp 313C, and the pressurizing button 314 has an indicator lamp 314D. Each of the indicator lamps 311A, 312B, 313C, and 314D is composed of an LED (Light Emitting Diode) or the like. Each of the indicator lamps 311A, 312B, 313C, and 314D may be equipped with multiple LEDs and capable of selectively illuminating two or more colors.

[0036] When the hydraulic pump is running, indicator lamp 311A ​​lights up; when the hydraulic pump is stopped, indicator lamp 311A ​​turns off. When the hydraulic pump is running, indicator lamp 312B is off; when the hydraulic pump is stopped, indicator lamp 312B is lit. When the piping valve is open, indicator lamp 313C lights up; when the piping valve is closed, indicator lamp 313C turns off. When the load valve is open, indicator lamp 314D lights up; when the load valve is closed, indicator lamp 314D turns off.

[0037] Indicator lamps 311A, 312B, 313C, and 314D indicate the hydraulic power source status in the fatigue testing machine 1, i.e., the operating status of the hydraulic power source, through a combination of their lighting patterns.

[0038] The test operation key 32 has a start key 321 and a stop key 322. The start key 321 has an indicator lamp 321A, and the stop key 322 has an indicator lamp 322B. Each of the indicator lamps 321A and 322B is composed of an LED or the like. Each of the indicator lamps 321A and 322B may have multiple LEDs and be capable of selectively illuminating two or more colors. When fatigue testing machine 1 is performing a fatigue test, indicator lamp 321A lights up; when fatigue testing machine 1 is not performing a fatigue test, indicator lamp 321A turns off. When fatigue testing machine 1 is performing a fatigue test, indicator lamp 322B is off, and when fatigue testing machine 1 is not performing a fatigue test, indicator lamp 322B is lit.

[0039] Indicator lamps 321A and 322B indicate the test status of the fatigue test in the fatigue testing machine 1, i.e., whether the test is in progress or stopped, through a combination of their lighting patterns.

[0040] When the user presses the start key 321, the fatigue testing machine 1 starts the fatigue test, and the words "Testing" are displayed on the test status display section 303 of the display 30. When the user presses the stop key 322, the fatigue testing machine 1 stops the ongoing test operation, and the words "Stopped" are displayed on the test status display section 303 of the display 30.

[0041] The display 30 is positioned in a rectangular first display area 33. The first display area 33 is the area on the display panel 21 in which the display 30 is positioned. The first display area 33 shows an example of a "predetermined area".

[0042] In Figure 3, markers MK are provided at each of the four corners of the first display area 33. Specifically, marker MK1 is placed in the upper left corner of the first display area 33. Marker MK2 is placed in the upper right corner of the first display area 33. Marker MK3 is placed in the lower right corner of the first display area 33. Marker MK4 is placed in the lower left corner of the first display area 33.

[0043] The power unit operation keys 31 are located in a rectangular second display area 34. The second display area 34 is the area on the display panel 21 where the power unit operation keys 31 are located. The second display area 34 shows an example of a "predetermined area".

[0044] In Figure 3, markers MK are provided at the two corners of the second display area 34. Specifically, marker MK5 is placed in the upper right corner of the second display area 34. Also, marker MK6 is placed in the lower left corner of the second display area 34.

[0045] The test operation key 32 is located in a rectangular third display area 35. The third display area 35 is the area on the display panel 21 where the test operation key 32 is located. The third display area 35 shows an example of a "predetermined area".

[0046] In Figure 3, markers MK are provided at the two corners of the third display area 35. Specifically, marker MK7 is placed in the upper right corner of the third display area 35. Also, marker MK8 is placed in the lower left corner of the third display area 35.

[0047] Each of the markers MK provided on the display panel 21 is a non-luminescent marker. That is, each of the markers MK does not emit light. The markers MK in this embodiment are rectangular, but their shape is not limited to a rectangle. Each of the markers MK is, for example, a label that is pre-attached to the display panel 21. Alternatively, each of the markers MK may be pre-printed on the display panel 21.

[0048] Figure 4 shows an example of a marker MK. Marker MK has a white area and a black area. The outer edge of Marker MK is composed of the white area. A code is formed inside the outer edge of Marker MK. In this embodiment, the code is composed of a white area and a black area. The code formed on Marker MK indicates the identification code of the display area (first display area 33, second display area 34, and third display area 35) corresponding to Marker MK. The code formed on Marker MK also includes information indicating the position of the corresponding Marker MK in the corresponding display area, such as "upper left," "upper right," "lower right," or "lower left" of that display area.

[0049] Each of the codes formed on markers MK1, MK2, MK3, and MK4 includes an identification code for the first display area 33 and information indicating their respective positions within the first display area 33. For example, the code formed on marker MK1 includes an identification code for the first display area 33 and information indicating the "top left" position within the first display area 33. Each of the codes formed on markers MK5 and MK6 includes an identification code for the second display area 34 and information indicating their respective positions within the second display area 34. Each of the codes formed on markers MK7 and MK8 includes an identification code for the third display area 35 and information indicating their respective positions within the third display area 35.

[0050] [1-4. Server Device Configuration] Figure 5 shows an example of the configuration of server device 5. Server device 5 is connected to the first camera 2, second camera 3, and third camera 4 via networks NW1 and NW2, enabling communication between them. The server device 5 includes a processor 50, memory 52, input unit 54, display unit 56, and communication interface 58.

[0051] The processor 50 consists of a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), etc. The memory 52 consists of a ROM (Read Only Memory), RAM (Random Access Memory), an HDD, an SSD, etc. The memory 52 stores the control program 521, marker image data 522, the first trained model 523, and the second trained model 524. The marker image data 522, the first trained model 523, and the second trained model 524 will be described later.

[0052] The input unit 54 is equipped with various switches and keys for use by the user. The input unit 54 may be connected to an input device used by the user and may accept user input by detecting operations on the input device. Examples of input devices include pointing devices such as a mouse or trackpad, and a keyboard.

[0053] The display unit 56 includes a display such as an LCD. The display unit 56 displays various screens, including the first screen 800 and the second screen 900 described later, in accordance with the control of the processor 50. The input unit 54 may include a touch sensor superimposed on the display surface of the display unit 56.

[0054] The communication interface 58 is connected to network NW2 and communicates with camera 6 via network NW2 and network NW1. The communication interface 58 includes, for example, a connector to which a communication cable can be connected and a communication circuit. The communication interface 58 may also be configured to include a wireless communication interface and perform wireless communication.

[0055] The processor 50 executes the control program 521 stored in the memory 52, so that the server device 5 functions as an acquisition unit 501, a first image generation unit 502, a second image generation unit 503, a third image generation unit 504, a first detection unit 505, a second detection unit 506, a specific unit 507, an image processing unit 508, a test value generation unit 509, a determination unit 510, and a display control unit 511. Furthermore, the memory 52 functions as a marker position storage unit 525 and an image storage unit 526 when the processor 50 executes a control program 521. The first image generation unit 502 is an example of a "first generation unit". The second image generation unit 503 is an example of a "second generation unit". The third image generation unit 504 is an example of a "third generation unit".

[0056] The marker position storage unit 525 stores the position of each marker MK in the panel image P3 as information. Specifically, the marker position storage unit 525 stores the coordinates of each marker MK in the coordinate system defined in the panel image P3 as the position of each marker MK.

[0057] The image storage unit 526 stores the captured images acquired by the acquisition unit 501. The captured images stored in the image storage unit 526 include the test machine body image P1, the test piece image P2, and the panel image P3.

[0058] The acquisition unit 501 acquires captured images from the camera 6. Specifically, the acquisition unit 501 receives image data of captured images from the camera 6 via the communication interface 358. The acquisition unit 501 may also acquire captured images from the camera 6 each time the camera 6 generates an image.

[0059] The first image generation unit 502 generates a grayscale image P4. Figure 6 shows an example of a grayscale image P4. The grayscale image P4 is a grayscale image of the panel image P3 acquired by the acquisition unit 501. In this embodiment, the display panel 21 as seen in the panel image P3 is distorted into a trapezoidal shape. This is because the normal direction of the display panel 21 is tilted with respect to the optical axis of the third camera 4. Therefore, as shown in Figure 6, the display panel 21 as seen in the grayscale image P4 is distorted into a trapezoidal shape. Grayscale image P4 is an example of a "first processed image".

[0060] The second image generation unit 503 generates a blurred image P5. Figure 7 shows an example of a blurred image P5. The blurred image P5 is an image obtained by blurring the grayscale image P4 generated by the first image generation unit 502. Blurring is a process that blurs an image. Blurring includes blurring that equalizes the pixels of the image and filtering using a median filter. In this embodiment, the second image generation unit 503 blurs the grayscale image P4 using a Gaussian filter and generates a blurred image P5 in which each pixel of the grayscale image P4 is smoothed.

[0061] The second image generation unit 503 changes the values ​​of the filter parameters used to generate the blurred image P5. In this embodiment, the filter size is used as an example of a filter parameter. The method for changing the filter size value will be described later.

[0062] The third image generation unit 504 generates a binarized image P6. Figure 8 shows an example of a binarized image P6. The binarized image P6 is an image obtained by binarizing the grayscale image P4 generated by the first image generation unit 502 based on the blurred image P5 generated by the second image generation unit 503. The third image generation unit 504 generates the binarized image P6 by treating pixels in the grayscale image P4 with lower brightness than pixels in the blurred image P5 as black, and pixels in the grayscale image P4 with higher brightness than pixels in the blurred image P5 as white, for pixels with the same coordinates. The binarized image P6 corresponds to an example of a "third processed image".

[0063] The first detection unit 505 detects marker MK from the binarized image P6. The first detection unit 505 reads marker image data 522 from the memory 52. ​​Marker image data 522 stores image data of each marker MK that the display panel 21 has. Marker image data 522 stores a number of marker MK image data corresponding to the number of marker MKs that the display panel 21 has. The first detection unit 505 uses the images of marker MK indicated by the marker image data 522 to detect marker MK from the binarized image P6 by pattern matching. Specifically, the first detection unit 505 determines whether there are any images in the binarized image P6 that have a degree of agreement with the image indicated by the marker image data 522 that is equal to or greater than a predetermined value (for example, 90 percent). The first detection unit 505 then detects images that have a degree of agreement with the image indicated by the marker image data 522 that is equal to or greater than a predetermined value as marker MK.

[0064] The second detection unit 506 detects the position of marker MK detected by the first detection unit 505 in the panel image P3. The second detection unit 506 detects the position of marker MK in the binarized image P6, which was detected by the first detection unit 505, as the position of marker MK in the panel image P3. The second detection unit 506 detects the coordinates in the coordinate system defined in the binarized image P6 as the position of marker MK. When the second detection unit 506 detects the position of marker MK, it stores information indicating the detected position of marker MK in the marker position storage unit 525.

[0065] The specific unit 507 and the image processing unit 508 will be described with reference to Figure 9. Figure 9 is a diagram showing an example of the processing performed by the specific unit 507 and the image processing unit 508.

[0066] The identification unit 507 identifies the areas corresponding to the first display area 33, the second display area 34, and the third display area 35 from the panel image P3 acquired by the acquisition unit 501. The identification unit 507 identifies the marker MK in the panel image P3 using the position of the marker MK stored in the marker position storage unit 525. Specifically, the identification unit 507 identifies the marker MK in the panel image P3 by assuming that the marker MK is at the position of the marker MK stored in the marker position storage unit 525. Next, the identification unit 507 reads the code formed on the identified marker MK to determine which position the marker MK is located at, and identifies the areas corresponding to the first display area 33, the second display area 34, and the third display area 35. As shown in the middle left of Figure 9, when the identification unit 507 identifies the positions of markers MK1, MK2, MK3, and MK4 in the panel image P3, it identifies a rectangular area with the identified positions of markers MK1, MK2, MK3, and MK4 as the four corners as the area corresponding to the first display area 33 in the panel image P3. Also, as shown in the middle center of Figure 9, when the identification unit 507 identifies the positions of markers MK5 and MK6 in the panel image P3, it identifies a rectangular area with the identified positions of markers MK5 and MK6 as the diagonals as the area corresponding to the second display area 34 in the panel image P3. Furthermore, as shown in the middle right of Figure 9, when the identification unit 507 identifies the positions of markers MK7 and MK8 in the panel image P3, it identifies a rectangular area with the identified positions of markers MK7 and MK8 as the diagonals as the area corresponding to the third display area 35 in the panel image P3.

[0067] The image processing unit 508 extracts an image of the region corresponding to the first display area 33 identified by the identification unit 507 (middle left figure in Figure 9) from the panel captured image P3, and calculates a correction function to correct the distortion of the image based on the extracted image. In this embodiment, the correction function is defined by a projection transformation line example for correcting the image of the region corresponding to the first display area 33 identified by the identification unit 507 into a rectangular image. The image processing unit 508 uses the calculated projection transformation line example to generate a display image P7, a first lamp image P8, and a second lamp image P9 converted into rectangular images. Display image P7 is the image obtained after correcting the image of the area corresponding to the first display area 33 into a rectangular image. First lamp image P8 is the image obtained after correcting the image of the area corresponding to the second display area 34 into a rectangular image. Second lamp image P9 is the image obtained after correcting the image of the area corresponding to the third display area 35 into a rectangular image.

[0068] Now, returning to the description of the server device 5's configuration, the test value generation unit 509 generates test values ​​that are included in the display image P7 and displayed on the display panel 21. For example, the test value generation unit 509 estimates the character size indicating the cycle number based on the size of the specific symbol displayed in the specific symbol display unit 302 included in the display image P7. Then, the test value generation unit 509 generates the cycle number value through image recognition processing (in this case, character recognition processing) using the estimated character size. Here, we will explain the case in which the test value generation unit 509 generates the cycle number value, but the test value generation unit 509 also generates test values ​​such as test force and displacement using the same processing as the cycle number value. Furthermore, the test value generation unit 509 generates a cycle count value through character recognition processing and outputs the accuracy of the cycle count value in the character recognition processing. The accuracy of the cycle count value is a value that expresses the certainty of the cycle count value as a percentage.

[0069] The determination unit 510 determines the hydraulic power source state based on the first lamp image P8. The determination unit 510 also determines the test state of the fatigue test based on the second lamp image P9.

[0070] First, let's explain the process for determining the hydraulic power source state. The determination unit 510 determines the hydraulic power source status based on the combination of the lighting patterns of indicator lamps 311A, 312B, 313C, and 314D. In this embodiment, "lighting pattern" refers to whether the lamp is lit or off.

[0071] If indicator lamp 311A ​​lights up and indicator lamps 312B, 313C, and 314D are off, the determination unit 510 determines that the hydraulic power source state is "RUN state," that is, the hydraulic pump is started. If indicator lamp 312B lights up and indicator lamps 311A, 313C, and 314D are off, the determination unit 510 determines that the hydraulic power source state is in the "STOP state," that is, the hydraulic pump is stopped.

[0072] If indicator lamps 311A, 313C, and 314D are lit and indicator lamp 312B is off, the determination unit 510 determines that the hydraulic power source state is "LOAD_MANI state," that is, that the piping valve and load valve are in the open state. If indicator lamps 312B and 313C are lit and indicator lamps 311A ​​and 314D are off, the determination unit 510 determines that the hydraulic power source state is "STOP_MANI state," that is, the piping valve is open.

[0073] If indicator lamps 311A ​​and 314D are lit and indicator lamps 312B and 313C are off, the determination unit 510 determines that the hydraulic power source state is "LOAD state," that is, the load valve is in the open state. If all indicator lamps 311A, 312B, 313C, and 314D are off, the determination unit 510 determines that the hydraulic power source state is "POWER OFF state," that is, the power supply for the operation of the hydraulic pump, piping valves, and load valves is turned off.

[0074] Furthermore, the determination unit 510 determines the hydraulic power source state using the first trained model 523. Specifically, the determination unit 510 determines the hydraulic power source state by inputting the first lamp image P8 into the first trained model 523. The first trained model 523 is a trained model that has been trained by machine learning (so-called "supervised learning") to learn the relationship between the first lamp image P8, which is a photograph taken of various combinations of lighting patterns of the indicator lamps 311A ​​to 314D, and the hydraulic power source state corresponding to the combination of lighting patterns of the indicator lamps 311A ​​to 314D. The determination unit 510 determines the hydraulic power source state by inputting the first lamp image P8 into the first trained model 523.

[0075] Next, we will explain the process for determining the test state in the fatigue test. The determination unit 510 determines the test status of the fatigue test based on the combination of the lighting patterns of multiple indicator lamps. Specifically, the determination unit 510 determines the test status of the fatigue test based on the combination of the lighting patterns of indicator lamp 321A and indicator lamp 322B.

[0076] If indicator lamp 321A lights up and indicator lamp 322B turns off, the determination unit 510 determines that the fatigue test state is in the "START state," that is, the fatigue test is being performed. If indicator lamp 322B lights up and indicator lamp 321A turns off, the determination unit 510 determines that the fatigue test is in a "STOP state," that is, the fatigue test has been stopped.

[0077] Furthermore, the determination unit 510 determines the test state of the fatigue test using the second trained model 524. Specifically, the determination unit 510 determines the test state of the fatigue test by inputting the second lamp image P9 into the second trained model 524. The second trained model 524 is a trained model that has been trained by machine learning (so-called "supervised learning") to learn the relationship between the second lamp image P9, which is a photograph taken of various combinations of lighting patterns of the indicator lamps 321A and 322B, and the test state of the fatigue test corresponding to the combination of lighting patterns of the indicator lamps 321A and 322B. The determination unit 510 determines the test state of the fatigue test by inputting the second lamp image P9 into the second trained model 524.

[0078] The output of the first trained model 523 includes the accuracy of the judgment regarding the hydraulic source state. The output of the second trained model 524 includes the accuracy of the judgment regarding the test state of the fatigue test. The accuracy of the judgment is expressed as a percentage of the likelihood of the judgment being correct. In other words, the determination unit 510 determines the hydraulic power source state and outputs the accuracy of that determination. The determination unit 510 also determines the test state of the fatigue test and outputs the accuracy of that determination.

[0079] The display control unit 511 generates a first screen 800 and displays the generated first screen 800 on the display unit 56. The display control unit 511 also generates a second screen 900 and displays the generated second screen 900 on the display unit 56.

[0080] Figure 10 shows an example of the first screen 800. The first screen 800 includes a cycle count display unit 801, a test force display unit 802, a test status display unit 803, a main unit image display button 804, a test piece image display button 805, and a panel image display button 806.

[0081] The cycle count display unit 801 displays the cycle count value of the fatigue testing machine 1. The cycle count value is generated by the test value generation unit 509. The test force display unit 802 displays the test force value of the fatigue testing machine 1. The test force value is generated by the test value generation unit 509. The test status display unit 803 displays the test status of the fatigue testing machine 1. The test status of the fatigue testing machine 1 is determined by the determination unit 510 based on the second lamp image P10.

[0082] The main unit image display button 804 is clicked by the user to display the image P1 of the fatigue testing machine 1. When the main unit image display button 804 is clicked, the display control unit 511 reads the image P1 of the testing machine from the image storage unit 526 and displays the read image P1 of the testing machine in the display area AR. The specimen image display button 805 is clicked by the user to display the specimen image P2. When the specimen image display button 805 is clicked, the display control unit 511 reads the specimen image P2 from the image storage unit 526 and displays the read specimen image P2 in the display area AR. The panel image display button 806 is clicked by the user to display the panel image P3. When the panel image display button 806 is clicked, the display control unit 511 reads the panel image P3 from the image storage unit 526 and displays the read panel image P3 in the display area AR.

[0083] Figure 11 shows an example of the second screen 900. The second screen 900 includes a judgment result display unit 901, a numerical display unit 902, a panel image display unit 903, a main unit image display unit 904, and a test piece image display unit 905.

[0084] The judgment result display unit 901 displays the judgment result from the judgment unit 510, namely the hydraulic power source status and the test status of the fatigue test. The judgment result display unit 901 will, for example, display that the hydraulic power source status is "LOAD status". The judgment result display unit 901 will also display, for example, "WARNING" indicating that the accuracy of the hydraulic power source status determination is less than the first threshold. The first threshold is, for example, 80%. Furthermore, the judgment result display unit 901 will display, for example, that the test state of the fatigue test is in the "START state". Also, the judgment result display unit 901 will display, for example, "GOOD", indicating that the accuracy of the judgment of the test state of the fatigue test is above the second threshold. The second threshold is, for example, 80%.

[0085] The numerical display unit 902 displays the test value generated by the test value generation unit 509. The numerical display unit 902 displays, for example, the value of the cycle count. The numerical display unit 902 also displays, for example, "GOOD," indicating that the accuracy of the cycle count value is above the third threshold. The third threshold is, for example, 80%.

[0086] The panel image display unit 903 displays the panel image P3. The main unit image display unit 904 displays the test machine main unit image P1. The test piece image display unit 905 displays the test piece image P2.

[0087] [1-5. Server Device Operation] Next, we will explain the operation of server device 5. First, let's explain the operation related to remembering the position of marker MK. Figure 12 is a flowchart showing the operation of the server device 5.

[0088] When a predetermined start trigger occurs, the server device 5 performs the operation shown in Figure 12. Examples of predetermined start triggers include receiving a command from the user to update the position of marker MK in the panel image P3, or receiving a command from the user to set a new position for marker MK in the panel image P3.

[0089] The acquisition unit 501 acquires the panel image P3 (step SA1). Next, the second image generation unit 503 sets a default value for the blurring filter size (step SA2).

[0090] Next, the second detection unit 506 determines whether or not it has been able to detect the positions of all the markers MK to be detected (step SA3). If the second detection unit 506 has been able to detect the positions of each different marker MK, corresponding to the number of image data stored in the marker image data 522 stored in the memory 52, it makes a positive determination in step SA3; otherwise, it makes a negative determination in step SA3.

[0091] If the second detection unit 506 determines that it has not been able to detect the positions of all the markers MK to be detected (step SA3: NO), the first image generation unit 502 generates a grayscale image P4 by converting the panel image P3 acquired in step SA1 into grayscale (step SA4).

[0092] Next, the second image generation unit 503 generates a blurred image P5 by blurring the grayscale image P4 generated in step SA3 (step SA5). If the process in step SA5 is the first process since starting the flowchart in Figure 12, the panel captured image P3 will be blurred with a default value set to the filter size. If the process in step SA5 is the second or later process since starting the flowchart in Figure 12, the panel captured image P3 will be blurred with a value set to the filter size as changed in step SA10.

[0093] Next, the third image generation unit 504 generates a binarized image P6 using the grayscale image P4 generated in step SA4 and the blurred image P5 generated in step SA5 (step SA6).

[0094] Next, the first detection unit 505 performs marker detection processing (step SA7). Marker detection processing is the process of detecting marker MK from the binarized image P6 generated in step SA6.

[0095] Next, the second detection unit 506 performs a marker position detection process (step SA8). The marker position detection process is the process of detecting the position of marker MK detected in the marker detection process of step SA7 in the panel image P3. If marker MK is not detected in the marker detection process, the processor 50 skips the process of step SA8.

[0096] Next, the second detection unit 506 stores the position of marker MK detected in the marker position detection process in the marker position storage unit 525 (step SA9).

[0097] Next, the second image generation unit 503 changes the filter size value (step SA10).

[0098] In step SA10, the second image generation unit 503 may change the filter size value within the range of lower and upper limits determined from the size of the marker MK. The second image generation unit 503 determines the lower and upper limits using the size of the marker MK. For example, the second image generation unit 503 sets the upper limit of the filter size to 1.5 times the maximum size of the marker MK estimated in the panel image P3, and sets the lower limit of the filter size to 0.5 times the minimum size of the marker MK estimated in the panel image P3. The estimated maximum and minimum sizes of the marker MK are stored as data in the memory 52 in advance. In this embodiment, the smallest unit of the filter size corresponds to 1 pixel.

[0099] In step SA10, the second image generation unit 503 may increase or decrease the filter size value by one or more steps from the currently set filter size value. For example, if the currently set filter size is 10×10, the second image generation unit 503 may change the filter size to 11×11. Note that the steps in which the filter size is increased are not limited to one step, but may be multiple steps. When making this stepwise change, the second image generation unit 503 may change it within the range of the upper and lower limits of the filter size as described above.

[0100] In step SA10, the second image generation unit 503 may change the filter size value using gradient descent. Stochastic gradient descent or steepest descent methods may be used as gradient descent methods. For example, the second image generation unit 503 calculates the difference between the highest match rate in the previous mark detection process and the highest match rate in the mark detection process before that, and changes the filter size value according to the calculated difference. For example, the second image generation unit 503 changes the filter size value such that the larger the calculated difference, the smaller the degree of change in the filter size value. When making this change using gradient descent, the second image generation unit 503 may change it within the range of the upper and lower limits of the filter size described above.

[0101] When the filter size value is changed, the second detection unit 506 performs the determination in step SA3 again. The processor 50 repeats steps SA3 to SA10 until it has detected the positions of all the markers MK to be detected.

[0102] As described above, the server device 5 generates a binarized image P6 and detects the marker MK from the generated binarized image P6. The third camera 4 in this embodiment takes images in a low-exposure state. A low-exposure state refers to a state where the exposure is low, specifically a state where the exposure is below a predetermined value. This predetermined value is determined through prior tests and simulations, based on the consideration that the first display area 33, the second display area 34, and the third display area 35 are properly captured in the panel image P3. As a result, the non-emitting marker MK appears dark in the panel image P3. However, as shown in Figure 8, the marker MK is emphasized in the binarized image P6. Therefore, the server device 5 can detect the marker MK from the image captured by the third camera 4 even when the third camera 4 is in a low-exposure state.

[0103] Furthermore, as explained using Figure 12, the server device 5 changes the filter size until all markers MK to be detected are detected. Depending on the relative position of the third camera 4 with respect to the control device 20 and the field of view of the third camera 4, the size of the markers MK that appear in the panel image P3 will differ. However, because the filter size is changed, an appropriate filter size can be set for each marker MK. Therefore, the server device 5 can detect markers MK from the image captured by the third camera 4, regardless of the relative position of the third camera 4 with respect to the control device 20 and the field of view of the third camera 4.

[0104] It should be noted that a larger filter size is not necessarily better, nor is a smaller filter size necessarily better. The larger the filter size, the more elements other than the marker MK are included in the filter size when blurring the marker MK area. Therefore, the larger the filter size, the more elements other than the marker MK are included in the leveling process when blurring the marker MK area, which may prevent the marker MK from being properly leveled. Also, when blurring the marker MK area, the smaller the filter size, the more bias there will be in the proportion of black and white included in the filter size. Therefore, if the filter size is too small, the marker MK area may not be properly leveled. This can lead to the marker MK being emphasized and not being captured in the binarized image P6. Figure 13 is an example of a binarized image P6 where the filter size was too small and the marker MK was not captured properly. As explained above, it is necessary to determine an appropriate filter size, but the server device 5 can change the filter size value and set it to an appropriate value.

[0105] Next, we will explain the operation of the server device 5 using the position of marker MK stored in the flowchart of Figure 14. Figure 14 is a flowchart showing the operation of server device 5.

[0106] The acquisition unit 501 acquires the panel image P3 (step SB1). Next, the identification unit 507 identifies the areas corresponding to the first display area 33, the second display area 34, and the third display area 35 in the panel image P3 acquired in step SB1 (step SB2).

[0107] Next, the image processing unit 508 generates a display image P7, a first lamp image P8, and a second lamp image P9 from the region identified in step SB2 (step SB3).

[0108] Next, the determination unit 510 performs a state determination process (step SB4). The state determination process includes a process for determining the hydraulic power source state and a process for determining the test state of the fatigue test.

[0109] Next, the test value generation unit 509 generates test values ​​(step SB5). Next, the display control unit 511 determines whether to display the first screen 800 or the second screen 900 based on the user's operation (step SB6).

[0110] If the display control unit 511 determines that the first screen 800 should be displayed (step SB5: first screen), it causes the display unit 56 to display the first screen 800 (step SB6).

[0111] Next, if the display control unit 511 determines that the second screen 900 should be displayed (step SB5: second screen), it causes the display unit 56 to display the second screen 900 (step SB7).

[0112] [2. Second Embodiment] Next, a second embodiment will be described. Regarding the configuration of each part of the monitoring system 100 in the second embodiment, components that are the same as those in the first embodiment are denoted by the same reference numerals and detailed descriptions are omitted.

[0113] [2-1. Server Device Configuration] Figure 15 shows an example of the configuration of the server device 5 according to the second embodiment. As is clear from comparing Figure 15 and Figure 5, the memory 52 of the second embodiment stores the third trained model 528. The third pre-trained model, 528, corresponds to an example of a "pre-trained model".

[0114] The third pre-trained model 528 is a pre-trained model that has been trained using machine learning (so-called "supervised learning") to learn the relationship between the filter parameter values, the position of marker MK in the panel image P3, and the probability of marker MK existing at that position. The probability of existence is expressed, for example, as a percentage. When the filter parameter values ​​are input to the third pre-trained model 528, it outputs a combination of the position of marker MK in the panel image P3 and the probability of marker MK existing at that position. The third pre-trained model 528 outputs multiple such combinations.

[0115] In the second embodiment, the second image generation unit 503 modifies the values ​​of the filter parameters using the third trained model 528. In the second embodiment, the filter size is also used as an example of a filter parameter. The modification of the filter size value in the second embodiment will be described later.

[0116] [2-2. Server Device Operation] Referring to Figure 16, the operation of the server device 5 related to the storage of the marker MK's position will be explained. Note that the operation of the first screen 800 and the second screen 900 using the marker MK's position is the same as that of the server device 5 in the second embodiment. Figure 16 is a flowchart showing the operation of the server device 5.

[0117] When a predetermined start trigger occurs, the server device 5 performs the operation shown in Figure 16. Examples of predetermined start triggers include receiving an instruction from the user to update the position of marker MK in the panel image P3, or receiving an instruction from the user to set the position of marker MK in the panel image P3.

[0118] The acquisition unit 501 acquires the panel image P3 (step SC1). Next, the second image generation unit 503 determines a value to set for the filter size of the blurring process and sets the determined value (step SC2).

[0119] In step SC2, the second image generation unit 503 increases or decreases the candidate values ​​for the filter size in one or more steps until at least one of the existence probabilities output by the third trained model 528 is equal to or greater than a predetermined probability. The second image generation unit 503 inputs the changed filter size to the third trained model 528 each time the candidate values ​​for the filter size are changed. The number of steps for changing the filter size is not limited to one, but may be multiple. When at least one of the existence probabilities output by the third trained model 528 is equal to or greater than a predetermined probability, the second image generation unit 503 determines the candidate value for the filter size most recently input to the third trained model 528 as the value to be set for the filter size. The second image generation unit 503 may change the filter size value within the range of lower and upper limits determined from the size of the marker MK, as in the first embodiment.

[0120] In step SC2, the second image generation unit 503 may determine candidate filter size values ​​using gradient descent with respect to the existence probability output by the third trained model 528. The second image generation unit 503 inputs the candidate filter size values ​​into the third trained model 528 and obtains the existence probability from the third trained model 528. If the output existence probability is less than or equal to a predetermined probability, the second image generation unit 503 changes the candidate filter size value, inputs the changed candidate filter size value into the third trained model 528, and obtains the existence probability from the third trained model 528 again. Next, if the existence probability obtained this time is less than or equal to a predetermined probability, the second image generation unit 503 calculates the difference between the existence probability obtained this time and the existence probability obtained last time, and changes the candidate filter size value according to the calculated difference. For example, the larger the calculated difference, the smaller the degree of change in the candidate filter size value, and the second image generation unit 503 changes the candidate filter size value. The second image generation unit 503 continues to change candidate filter size values ​​until the existence probability output by the third trained model 528 exceeds a predetermined probability. Then, when at least one of the existence probabilities output by the third trained model 528 exceeds the predetermined probability, the second image generation unit 503 determines the most recently input filter size candidate to the third trained model 528 as the value to set for the filter size. In addition, the second image generation unit 503 may change the filter size value within the range of lower and upper limits determined from the size of the marker MK, similar to the first embodiment.

[0121] Next, the second detection unit 506 determines whether or not it has been able to detect the positions of all the markers MK to be detected (step SC3). The determination in step SC3 is made using the same method as in step SA3.

[0122] If the second detection unit 506 determines that it has not been able to detect the positions of all the markers MK to be detected (step SC3: NO), the first image generation unit 502 generates a grayscale image P4 by converting the panel image P3 acquired in step SC1 into grayscale (step SC4).

[0123] Next, the second image generation unit 503 generates a blurred image P5 by blurring the grayscale image P4 generated in step SC3 (step SC5). If this step SC5 is the first time since starting the flowchart in Figure 16, the panel captured image P3 will be blurred with the filter size set to the value determined in step SC2. If this step SC5 is the second or later since starting the flowchart in Figure 16, the panel captured image P3 will be blurred with the filter size set to the value changed in step SC10.

[0124] Next, the third image generation unit 504 generates a binarized image P6 using the grayscale image P4 generated in step SC4 and the blurred image P5 generated in step SA5 (step SC6).

[0125] Next, the first detection unit 505 performs marker detection processing (step SC7). The marker detection processing in step SC7 is the process of detecting markers MK from the binarized image P6 generated in step SC6. If the processing in step SC6 is the first processing since starting the flowchart in Figure 16, the first detection unit 505 performs pattern matching on the positions in the binarized image P6 where the probability of existence in the processing in step SC2 was determined to be greater than or equal to a predetermined probability. If the processing in step SC6 is the second or later processing since starting the flowchart in Figure 16, the first detection unit 505 performs pattern matching on the positions in the binarized image P6 where the probability of existence in the processing in step SC10 was determined to be greater than or equal to a predetermined probability.

[0126] Next, the second detection unit 506 performs a marker position detection process (step SC8). The marker position detection process in step SC8 is the process of detecting the position of marker MK detected in the marker detection process in step SC7 in the panel image P3. If marker MK is not detected in the marker detection process, the processor 50 skips the process in step SC8.

[0127] Next, the second detection unit 506 stores the position of marker MK detected in the marker position detection process in the marker position storage unit 525 (step SC9).

[0128] Next, the second image generation unit 503 changes the filter size value (step SC10). The change in step SC10 is performed in the same way as the value determination in step SC2.

[0129] When the filter size value is changed, the second detection unit 506 performs the determination in step SC3 again. The processor 50 repeats steps SC3 to SC10 until it has detected the positions of all the markers MK to be detected.

[0130] According to the second embodiment, the same effects as the first embodiment can be achieved.

[0131] [3. Appearance] Those skilled in the art will understand that each of the embodiments described above is a specific example of the following embodiments.

[0132] (Section 1) A monitoring device according to one embodiment is a monitoring device for a material testing machine equipped with a display unit having a non-emitting marker on a display panel, comprising: an acquisition unit that acquires an image captured by a camera that photographs the display panel; a first generation unit that generates a first processed image obtained by converting the captured image to grayscale; a second generation unit that generates a second processed image obtained by blurring the first processed image; a third generation unit that generates a third processed image obtained by binarizing the first processed image based on the second processed image; and a first detection unit that detects the marker from the third processed image.

[0133] According to the monitoring device described in paragraph 1, the first detection unit can highlight and capture non-emitting markers in the image used for detection. Therefore, even when the camera exposure is low, non-emitting markers on the display panel of the display unit can be detected from the image captured by the camera.

[0134] (Section 2) The monitoring device described in paragraph 1 includes a second detection unit that detects the position of the marker detected by the first detection unit in the captured image, the second generation unit changes the value of the filter parameter for the blurring process, and if the second detection unit has not detected all the positions of the marker to be detected, it generates the second processed image that has been blurred using the changed filter parameter, the third generation unit generates the third processed image again based on the second processed image that has been blurred using the changed filter parameter and the first processed image, and the first detection unit detects the marker from the third processed image that has been regenerated by the third generation unit.

[0135] According to the monitoring device described in paragraph 2, if not all positions of all detected markers have been detected, the marker detection is performed again by changing the value of the filter parameter. Therefore, even when the camera exposure is low, all non-emitting markers on the display panel of the display unit can be detected from the camera's captured image.

[0136] (Section 3) The monitoring device described in paragraph 2 includes a trained model that has been trained by machine learning to determine the relationship between the value of the filter parameter, the position of the marker in the captured image, and the probability of the marker being present at that position, and the second generation unit changes the value of the filter parameter based on the probability of existence output by the trained model in response to the input of the filter parameter.

[0137] According to the monitoring device described in Section 3, the filter parameter values ​​are changed using the probability of existence output by the trained model, so the filter parameter values ​​can be appropriately changed to filter parameter values ​​that have a high probability of detecting markers. Therefore, even when the camera exposure is low, non-emitting markers on the display panel of the display unit can be accurately detected from the camera's captured image.

[0138] (Section 4) In the monitoring device described in paragraph 3, the second generation unit changes the value of the filter parameter by gradient descent using the existence probability.

[0139] According to the monitoring device described in Section 4, the filter parameter values ​​can be changed to appropriate values ​​without having to change them step by step. Therefore, since the filter parameter values ​​can be changed efficiently, non-emitting markers can be detected quickly.

[0140] (Section 5) A monitoring device according to any one of configurations 1 to 4, comprising: a second detection unit that detects the position of the marker detected by the first detection unit in the captured image; and a identification unit that identifies a region corresponding to a predetermined region of the display panel from the captured image based on the position of the marker detected by the second detection unit.

[0141] According to the monitoring device described in paragraph 5, even when the camera's exposure is low, a predetermined area of ​​the display panel can be identified from the image captured by the camera using a non-emitting marker.

[0142] (Section 6) Another embodiment of the monitoring system is a monitoring device for a material testing machine equipped with a display unit having a non-luminescent marker on a display panel, and a camera for photographing the display panel, wherein the monitoring device includes an acquisition unit for acquiring an image captured by the camera for photographing the display panel, a first generation unit for generating a first processed image obtained by grayscaleizing the captured image, a second generation unit for generating a second processed image obtained by blurring the first processed image, a third generation unit for generating a third processed image obtained by binarizing the first processed image based on the second processed image, and a first detection unit for detecting the marker from the third processed image.

[0143] The monitoring system described in paragraph 6 produces the same effect as the monitoring device described in paragraph 1.

[0144] [4. Other Embodiments] It should be noted that the fatigue testing machine 1 in each embodiment is merely an example of the material testing machine according to the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

[0145] For example, in the embodiments described above, the case where the material testing machine is a fatigue testing machine 1 is explained, but the invention is not limited to this. The material testing machine may be, for example, a tensile testing machine, a compression testing machine, a bending testing machine, a torsion testing machine, etc.

[0146] For example, in each of the embodiments described above, the filter size was used as an example of a filter parameter. However, if the blurring process is a Gaussian filter, the filter parameter may also be the standard deviation or the mean.

[0147] For example, in the embodiments described above, a Gaussian filter was used as an example of blurring, but the blurring process is not limited to a Gaussian filter and may be other filters such as an averaging filter. For example, if the blurring process is an averaging filter, the filter parameter may be the filter size.

[0148] For example, in each of the embodiments described above, the display unit is not limited to the control device 20. The display unit may be separate from the control device 20, as long as it has the same configuration as the display panel 21.

[0149] Furthermore, the color of the marker MK is not limited to the color of the embodiment described above. The marker MK can be any non-luminescent marker MK.

[0150] Furthermore, the functional units shown in Figures 5 and 15 represent functional configurations, and the specific implementation form is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each functional unit be implemented individually, and it is certainly possible to have a configuration in which a single processor executes a program to realize the functions of multiple functional units. Also, in the above embodiment, some of the functions realized by software may be realized by hardware, or conversely, some of the functions realized by hardware may be realized by software.

[0151] Furthermore, the processing units in the flowcharts shown in Figures 12, 14, and 16 are divided according to their main processing content to facilitate understanding of the processing performed by the server device 5. The division and naming of these processing units in the flowcharts are not restrictive; they can be further divided into more processing units depending on the processing content, or each processing unit can be divided to include even more processing. Also, the processing order in the flowcharts is not limited to the examples shown.

[0152] The control program 521 to be executed by the processor 50 can also be recorded on a recording medium that is readable by a computer. Magnetic, optical, or semiconductor memory devices can be used as recording media. Specifically, examples include portable or fixed recording media such as flexible disks, HDDs, CD-ROMs (Compact Disk Read Only Memory), DVDs, Blu-ray® Discs, magneto-optical disks, flash memory, and card-type recording media. Furthermore, the recording medium may be a non-volatile storage device such as RAM, ROM, or HDD, which is an internal storage device of the server device 5. Alternatively, the control program 521 may be stored in another server device different from the server device 5, and the server device 5 may download the control program 521 from that other server device. [Explanation of symbols]

[0153] 1. Fatigue testing machine (material testing machine) 4. Third camera (camera) 5. Server equipment (monitoring equipment) 20 Control device (display) 33 1st display area (predetermined area) 34 2nd display area (predetermined area) 35 Third display area (predetermined area) 100 monitoring systems 501 Acquisition Department 502 First image generation unit (first generation unit) 503 Second image generation unit (second generation unit) 504 Third image generation unit (third generation unit) 505 First detection unit 506 Second detection unit 507 Specific section 521 Control Program 528 Third pre-trained model (pre-trained model) MK Marker P3 Panel image (image taken by camera) P4 Grayscale image (first processed image) P5 blurred image (second processed image) P6 binarized image (third processed image)

Claims

1. A monitoring device for a material testing machine, which is equipped with a display unit having a non-luminescent marker on the display panel, An acquisition unit that acquires an image captured by a camera that photographs the display panel, A first generation unit generates a first processed image by converting the captured image into grayscale, A second generation unit generates a second processed image obtained by blurring the aforementioned captured image, A third generation unit generates a third processed image obtained by binarizing the first processed image based on the second processed image, The system comprises a first detection unit that detects the marker from the third processed image, monitoring equipment.

2. The system includes a second detection unit that detects the position of the marker detected by the first detection unit in the captured image, The second generation unit changes the value of the filter parameter for the blurring process, and if the second detection unit has not detected all the positions of the markers to be detected, it generates the second processed image that has been blurred using the changed filter parameter. The third generation unit generates the third processed image again based on the second processed image, which has been blurred by the filter parameters whose values ​​have been changed, and the first processed image. The first detection unit detects the marker from the third processed image that is again generated by the third generation unit. The monitoring device according to claim 1.

3. The system includes a trained model that has been trained by machine learning to determine the relationship between the value of the filter parameter, the position of the marker in the captured image, and the probability of the marker being present at that position. The second generation unit modifies the values ​​of the filter parameters based on the existence probability output by the trained model in response to the input of the filter parameters. The monitoring device according to claim 2.

4. The second generation unit modifies the value of the filter parameter by gradient descent using the existence probability. The monitoring device according to claim 3.

5. A second detection unit detects the position of the marker detected by the first detection unit in the captured image, The system includes a specification unit that identifies a predetermined area of ​​the display panel based on the position of the marker detected by the second detection unit, A monitoring device according to any one of claims 1 to 4.

6. A monitoring device for a material testing machine equipped with a display unit having a non-emitting marker on the display panel, and a monitoring system comprising a camera for photographing the display panel, The aforementioned monitoring device is An acquisition unit that acquires an image captured by a camera that photographs the display panel, A first generation unit generates a first processed image by converting the captured image into grayscale, A second generation unit generates a second processed image obtained by blurring the aforementioned captured image, A third generation unit generates a third processed image obtained by binarizing the first processed image based on the second processed image, The system comprises a first detection unit that detects the marker from the third processed image, Monitoring system.

Images