Material testing systems, methods, and programs

Abstract

To provide technology that more reliably ensures the safety of operators in material testing machines. [Solution] In a material testing system, the computer performs the following steps: acquires an image of the material testing machine; uses a machine learning model for object detection to detect human body parts in the image; detects a dangerous area in the image; and determines safety based on the detection of the human body parts and the dangerous area in the image.

Description

Technical Field

[0001] The present invention relates to the safety of a material testing system including a material testing machine.

Background Art

[0002] A material testing machine is used to evaluate the physical properties of materials. Conventionally, various safety measures have been proposed for material testing machines.

[0003] As one of the tests performed on a material testing machine, a tensile test on a test piece is carried out. In the tensile test, the test piece may suddenly break and a part of the test piece may scatter. Japanese Unexamined Patent Application Publication No. 2019-100751 (Patent Document 1) discloses a scattering prevention cover for preventing the scattering of the test piece outside the apparatus and ensuring the safety of the operator.

[0004] Attempts have been made to monitor the surrounding information of a material testing machine with a photographing device such as a camera. However, in photographing from the outside environment of the testing machine, since the operator occupies most of the photographing range, the testing machine is reflected in the shadow of the photographer. Therefore, it has been difficult to make a safety determination by camera determination. In order to solve this problem, the development of a testing machine provided with a camera in the testing machine housing has been promoted.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In material testing machines, technologies to more reliably ensure operator safety are still needed. By using a camera built into the testing machine housing, the relationship between the operator and the testing machine can be captured, making it possible to perform safety assessments using image processing technology. Therefore, the inventor conducted research on a safety assessment system using a classification model, which is a common AI technology. A classification model is a model that uses labeled data as training data to statistically identify which label unknown data is closest to when it is input. At least "dangerous state" and "safe state" are required as labels, and a classification model can be generated by using a dataset in which these labels are linked to image data as training data.

[0007] Classification models are known to be effective for simple problems, and high effectiveness was expected in environments where a portion of the testing machine is continuously photographed from a predetermined position. However, even when using the generated classification model, high performance in safety judgment was not demonstrated. Upon investigating the cause, it was found that the model was learning incorrect features as the basis for judging "dangerous conditions" and "safe conditions." One detailed cause was that the working space of the testing machine (around the test piece gripper) does not receive much external light, making it difficult for light and dark contrast to be reflected in the image information. Another detailed cause was that even in such a working space, the operator's hands occupied a large portion of the shooting range, resulting in phenomena such as the gripper being difficult to see in the image and the test piece being difficult to see in the image, making it impossible to determine whether the information shown in the image was of a dangerous condition or a safe condition. This invention was conceived in view of these circumstances, and its purpose is to provide a technology that more reliably ensures the safety of operators in material testing machines. [Means for solving the problem]

[0008] A material testing system according to certain aspects of the present disclosure comprises a material testing machine, an imaging device, and a computer capable of communicating with the material testing machine and the imaging device, wherein the computer includes a processor and a storage device for storing a program, and the processor performs a method by executing a program, the method comprising the steps of: acquiring an image of the material testing machine; detecting human body parts in the image using a machine learning model for object detection; detecting a hazardous area in the image; and determining safety based on the detection of human body parts and a hazardous area in the image.

[0009] A material testing system according to certain aspects of the present disclosure comprises a material testing machine, an imaging device, and a computer capable of communicating with the material testing machine and the imaging device, wherein the computer includes a processor and a storage device for storing a program, and the processor performs a method by executing a program, the method comprising the steps of: acquiring an image of the material testing machine; detecting a human body part in the image using a machine learning model for object detection and determining whether the location of the human body part is dangerous; and outputting a warning notification if it is determined that the location of the human body part in the image is dangerous, wherein the machine learning model for object detection has undergone training using a plurality of images with correct labels as training data, each of the plurality of images includes a human body part and a dangerous area of ​​the material testing machine, the correct label indicates whether the location of the human body part is dangerous, and the machine learning model for object detection is configured, through training, to output a classification result of whether the location of the human body part is dangerous in the input image, and the determination step includes determining whether the location of the human body part is dangerous based on the classification result output from the machine learning model for object detection in response to the input image of the image.

[0010] A method according to certain aspects of this disclosure is a method for determining the safety of a material testing system including a material testing machine, comprising the steps of: acquiring an image of the material testing machine; detecting human body parts in the image using a machine learning model for object detection; detecting a hazardous area in the image; and determining safety based on the detection of human body parts and a hazardous area in the image.

[0011] A method following a certain aspect of the present disclosure is a method for determining the safety of a material testing system including a material testing machine, comprising the steps of: acquiring an image of the material testing machine; using a machine learning model for object detection to detect human body parts in the image and determining whether the location of the human body parts is dangerous; and, if it is determined that the location of the human body parts in the image is dangerous, outputting a warning notification, wherein the machine learning model for object detection has been trained using a plurality of images with correct labels as training data, each of the plurality of images including a human body part and a dangerous area of ​​the material testing machine, the correct label indicates whether the location of the human body part is dangerous, and the machine learning model for object detection is configured, through training, to output a classification result of whether the location of the human body part is dangerous in the input image, and the step of determining whether the location of the human body part is dangerous based on the classification result output from the machine learning model for object detection in response to the input image.

[0012] A program that conforms to certain aspects of this disclosure, when executed by a computer, causes the computer to carry out the methods described above. [Effects of the Invention]

[0013] In accordance with certain aspects of this disclosure, a technology is provided for more reliably ensuring the safety of operators in material testing machines. [Brief explanation of the drawing]

[0014] [Figure 1]It is a diagram showing a configuration example of the stress measurement device 100 in Embodiment 1. [Figure 2] It is a schematic diagram showing an example of the hardware configuration of the control device 14. [Figure 3] It is a flowchart of the process performed to notify a warning in the control device 14 of Embodiment 1. [Figure 4] It is a diagram showing an example of the display screen of the "DANGER" notification in Embodiment 1. [Figure 5] It is a diagram showing another example of the display screen of the "DANGER" notification in Embodiment 1. [Figure 6] It is a diagram showing a configuration example of the stress measurement device 100 in Embodiment 2. [Figure 7] It is a diagram schematically showing the training phase of the machine learning model. [Figure 8] It is a diagram showing an example of a set of learning data with an "OK" label. [Figure 9] It is a diagram showing an example of a set of learning data with a "DANGER" label. [Figure 10] It is a diagram schematically showing the usage phase. [Figure 11] It is a flowchart of the process performed to notify a warning in the control device 14 of Embodiment 2. [Figure 12] It is a diagram showing an example of the display screen of the "DANGER" notification in Embodiment 2. [Figure 13] It is a diagram showing another example of the display screen of the "DANGER" notification in Embodiment 2. [Figure 14] It is a diagram showing an example of the display screen of the "OK" notification in Embodiment 2.

Mode for Carrying Out the Invention

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the figures, the same or corresponding parts are denoted by the same reference numerals and their description will not be repeated.

[0016] [Embodiment 1] (Configuration of Stress Measuring Device 100) FIG. 1 is a diagram showing a configuration example of a stress measuring device 100 according to Embodiment 1. The stress measuring device 100 measures mechanical properties (for example, tensile strength, yield point, elongation, reduction of area, etc.) of the specimen 2 by applying a tensile force to the specimen 2. The specimen 2 is, for example, a test piece defined in Japanese Industrial Standard (JIS) Z-2201 "Tensile Test Piece for Metallic Materials".

[0017] As shown in FIG. 1, the stress measuring device 100 includes a tensile testing machine 4, a control device 6, an imaging device 8, a driving device 12, a control device 14, and a storage device 16. As the tensile testing machine 4, for example, a precision universal testing machine (product name: Autograph AG-Xplus, manufactured by Shimadzu Corporation) is used. The tensile testing machine 4 corresponds to an example of a "material testing machine". The control device 14 corresponds to an example of a "computer".

[0018] The stress measuring device 100 determines whether a part of an operator (also referred to as a "user" in the present disclosure) is located in a dangerous position with respect to the tensile testing machine 4, and outputs a warning notification when it is determined that the part is located in a dangerous position. In this sense, the stress measuring device 100 corresponds to an example of a warning notification system.

[0019] The tensile testing machine 4 includes a table 40, a crosshead 42, a pair of screw rods 44, 46, an upper gripper 48, a lower gripper 50, and a load cell 52. The pair of screw rods 44, 46 are erected on the table 40 so as to be rotatable in a vertical direction. The pair of screw rods 44, 46 are made of ball screws.

[0020] The crosshead 42 is connected to each screw rod 44, 46 via nuts (not shown). The crosshead 42 is configured to be movable in the vertical direction along the pair of screw rods 44, 46. A load mechanism (not shown) for raising and lowering the crosshead 42 is mounted inside the table 40.

[0021] The upper grip 48 is connected to the crosshead 42 and grips the upper end of the specimen 2. The lower grip 50 is connected to the table 40 and grips the lower end of the specimen 2. An example of the distance L1 between the upper grip 48 and the lower grip 50 is 120 mm. During a tensile test, the tensile testing machine 4 applies a tensile force to the specimen 2 by raising the crosshead 42 in accordance with the control of the control device 14, while gripping both ends of the specimen 2 with the upper grip 48 and the lower grip 50.

[0022] The load cell 52 is a sensor for detecting the test force, which is the tensile force applied to the test specimen 2. The load cell 52 outputs a signal indicating the detected test force to the control device 14.

[0023] The control device 6 communicates with the tensile testing machine 4 and controls the tensile operation performed by the tensile testing machine 4. The control device 6 receives user operations such as setting various parameters, including the test conditions for the tensile test, and issuing execution instructions, and controls the loading mechanism according to the received user operations. Furthermore, the control device 6 receives various signals from the tensile testing machine 4, including the output signal from the load cell 52 and a signal indicating the displacement of the crosshead 42, and analyzes data such as the detected value of the test force.

[0024] The control device 6 includes a processor, memory, an interface circuit for connecting peripheral devices, and a display unit 62. The processor is, for example, a CPU (Central Processing Unit). The memory is, for example, ROM (Read Only Memory) and / or RAM (Random Access Memory). The various functions described above are realized by the processor executing the tensile test program stored in the memory.

[0025] The display unit 62 displays various information based on signals input to the control device 6. For example, during the execution of a tensile test, the display unit 62 displays the test force detected by the load cell 52. The display unit also displays the displacement amount indicating the displacement (stroke) of the crosshead 42.

[0026] The imaging device 8 is positioned so as to include at least the hazardous area in the tensile testing machine 4 within its field of view. The stress measuring device 100 may further have a light source for illuminating the field of view.

[0027] In one implementation example, the tensile testing machine 4 may have a housing, and the upper grips 48 and lower grips 50 may be located inside the housing. The housing may have a cover that is opened and closed to insert and remove the test specimen 2. An example of the above-mentioned hazard zone may be the outer edge of the cover (the entry point for the test specimen 2 in the tensile testing machine 4). Another part, such as the gripping teeth of the fixture, may be designated as the hazard zone.

[0028] The imaging device 8 includes an optical system such as a lens and an image sensor. The image sensor is implemented by, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor generates an image by converting light incident from the field of view into an electrical signal via the optical system.

[0029] The control device 14 is connected to the imaging device 8. The control device 14 is also connected to the control device 6 of the tensile testing machine 4 by a communication line 15. The control device 14 can comprehensively control the tensile testing machine 4 and the imaging device 8 by exchanging data with the control device 6 via the communication line 15. Communication between the control device 14 and the control device 6 may be achieved by wireless communication.

[0030] In this embodiment, the control device 14 and the control device 6 are provided as separate components, but the control device 14 and the control device 6 may be integrated into a single unit.

[0031] (Hardware configuration of control device 14) Figure 2 is a schematic diagram showing an example of the hardware configuration of the control device 14. As shown in Figure 2, the control device 14 includes a processor 140, memory 141, a communication interface 146, an input / output interface 145, a display interface 143, and a non-volatile storage device 16. These components are connected to each other in a communicative manner via an internal bus 147. The processor 140 is, for example, a CPU. The memory 141 is, for example, ROM and / or RAM. The storage device 16 includes a program area 160 for storing programs and a data area 162 for storing data used to execute programs.

[0032] The communication interface 146 exchanges data with other devices. These other devices include the control unit 6 and external devices (not shown). The communication interface 146 may be configured to download various types of data from these external devices.

[0033] The input / output interface 145 is connected to the control unit 144 and receives signals indicating user operation from the control unit 144. The control unit 144 typically consists of a keyboard, mouse, touch panel, touchpad, etc., and accepts user input. The control unit 144 may be configured integrally with the control device 14 or separately from the control device 14.

[0034] The display interface 143 is connected to the display unit 142 and outputs image signals to the display unit 142 for displaying images, according to commands from the processor 140 or the like. The display unit 142 is composed of an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays various information to the user.

[0035] Figure 2 shows an example configuration in which the processor 140 provides the necessary functions by executing a program. However, some or all of the provided functions may be implemented using dedicated hardware circuits (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)).

[0036] (Machine learning model) In this embodiment, the control device 14 uses the image captured by the imaging device 8 to determine whether a worker's body part (for example, a hand) is in a dangerous position relative to the tensile testing machine 4. In this determination, the control device 14 has a machine learning model for object detection extract the worker's body part from the captured image. As the machine learning model, for example, a model trained according to the R-CNN (Regions with Convolutional Neural Networks), Fast R-CNN, Faster R-CNN, or YOLO (You Only Look Once) algorithm may be used.

[0037] The training data used for the machine learning model consists of images captured while an operator is actually performing tasks on the testing machine (e.g., setting up test specimens). Since the training data and inference data must be equivalent, it is preferable to use a camera installed inside the testing machine for the training data.

[0038] If there is no camera inside the test machine, it is desirable to install a camera at the camera position expected in the actual test and capture training data. In addition, it is necessary to attach information called a correct answer label to the training data. In this example, body parts such as hands in the image are enclosed in a rectangle, and information indicating that it is a body part is added. The same applies to the hazard zone. When preparing the training data, it is advisable to prepare images that reflect the expected variations in the operating environment. For example, test machine operators often wear gloves. Since the presence or absence of gloves and the type of gloves affect the judgment, it is preferable to prepare images of operation with gloves of different types and colors as training data.

[0039] The data for realizing the machine learning model may be stored in the storage device 18, or it may be stored in the storage device of an external server (for example, an API (Application Programming Interface) server) from the control device 14. In the latter case, the processor 140 provides the captured image to the external server and obtains detection results (whether the worker's parts were detected, the areas identified as the worker's parts in the captured image, etc.) from the external server.

[0040] (Process flow) Figure 3 is a flowchart of the process performed in the control device 14 of Embodiment 1 to notify a warning. In one implementation example, the process is performed in the control device 14 by the processor 140 executing a given program. In one implementation example, the process in Figure 3 is started when an instruction to start the above process is input to the operation unit 144.

[0041] In step S10, the control device 14 acquires a predetermined "hazardous area" in the captured image. More specifically, the stress measurement device 100 identifies the area in the tensile testing machine 4 that is considered dangerous for workers to approach. In the image captured by the imaging device 8, the area corresponding to this area is identified as the "hazardous area". The "hazardous area" is set in advance in the control device 14. The "hazardous area" will be described later as hazardous area 501 with reference to Figures 4 and 5. In step S10, the control device 14 reads information identifying the "hazardous area" from the storage device 16. In step S10, the control device 14 may also accept the setting of the "hazardous area". In this embodiment, the "hazardous area" is determined by user operation, but the "hazardous area" may also be extracted using a machine learning model.

[0042] In step S12, the control device 14 acquires the captured image from the imaging device 8. In step S14, the control device 14 inputs the captured image acquired in step S12 into the machine learning model.

[0043] In step S16, the control device 14 obtains detection results from the machine learning model. In step S16, it may be determined that there is a danger if the worker and the hazardous area are both visible in the same image. In this case, a hazardous situation that occurs during testing can be detected. However, this determination cannot be used to perform a deterrent determination while preparing test specimen sets, etc. Therefore, step S18 is performed for such cases.

[0044] In step S18, if the detection result obtained in step S16 includes information identifying an area identified as a part of the worker, the control device 14 determines whether the shortest distance between that area and the hazardous area is less than a predetermined threshold. More specifically, by comparing the area and the hazardous area, the control device 14 determines the shortest distance between them and determines whether that shortest distance is less than the threshold. If the shortest distance is less than the threshold (YES in step S18), the control device 14 proceeds to step S20; otherwise (NO in step S18), it proceeds to step S22. The shortest distance may be calculated by determining the distance between the center coordinates of the area identified as a part of the worker and the hazardous area. The present invention is not limited to this calculation method, and a general shortest distance calculation algorithm may be used.

[0045] In step S20, the control device 14 outputs a "DANGER" notification. After that, the control device 14 returns control to step S12.

[0046] The "DANGER" notification corresponds to an example of a warning notification. The output of the "DANGER" notification is realized, for example, by display and / or audio output on the display unit 142.

[0047] In step S20, the control device 14 may further output a signal to the tensile testing machine 4 instructing it to stop operation (or to transition to a predetermined safe state). Upon receiving this signal from the control device 14, the tensile testing machine 4 will stop operation. Alternatively, the tensile testing machine 4 may stop operation immediately when it is determined to be dangerous without displaying a warning.

[0048] Figure 4 shows an example of the "DANGER" notification display screen in Embodiment 1. Figure 5 shows another example of the "DANGER" notification display screen in Embodiment 1.

[0049] In the example shown in Figure 4, screen 500 primarily displays an example of the captured image acquired in step S12. Screen 500 includes the danger zone 501 and the bounding box 502.

[0050] In the example shown in Figure 5, screen 510 primarily displays other examples of the captured images acquired in step S12. Screen 510 includes the danger zone 501 and the bounding box 502.

[0051] In Figures 4 and 5, the bounding box 502 points to a rectangle containing the area detected as the worker's "hand". In each example in Figures 4 and 5, the bounding box 502 is displayed in a specific color (for example, red). The display of the bounding box 502 in a specific color enables the output of a "DANGER" notification.

[0052] Returning to Figure 3, in step S22, the control device 14 outputs an "OK" notification. After that, the control device 14 returns control to step S12.

[0053] The "OK" notification corresponds to an example of a safety notification. The output of the "OK" notification is realized, for example, by a display and / or audio output on the display unit 142. In one implementation example, the output of the "OK" notification is realized by the bounding box being displayed in a color other than the "specific color" mentioned above (for example, blue).

[0054] In the process described with reference to Figure 3, the control device 14 continuously acquires captured images, inputs the captured images into a machine learning model, and outputs a "DANGEER" notification or an "OK" notification regarding the captured images.

[0055] As described above, in Embodiment 1, a machine learning model for object detection detects human body parts in the captured image of the material testing machine, and a warning is issued if the distance between the detected human body part and the danger zone of the material testing machine is less than a predetermined threshold. As a result, by using a machine learning model for object detection to detect human body parts, the occurrence of an event in which a human body part is located within the danger zone is immediately detected. Therefore, the occurrence of the above event is immediately detected, and a warning is issued.

[0056] [Embodiment 2] (Configuration of stress measuring device 100) Figure 6 shows an example of the configuration of the stress measurement device 100 in Embodiment 2. In Embodiment 2, one imaging device 8 may be placed in the location indicated by frame 101, or in the location indicated by frame 102.

[0057] (Machine learning model) In this embodiment, the control device 14 uses the image captured by the imaging device 8 to determine whether a worker's body part (for example, a hand) is in a dangerous position relative to the tensile testing machine 4. In this determination, the control device 14 has a machine learning model for object detection determine whether a worker's body part is in a dangerous position in the captured image. As the machine learning model, for example, a model trained according to the R-CNN, Fast R-CNN, Faster R-CNN, or YOLO algorithm may be used.

[0058] Figure 7 schematically illustrates the training phase of a machine learning model. In Embodiment 2, the machine learning model 300 undergoes a training process using the training data 310. This trains the machine learning model 300. The training data 310 includes multiple sets of training data, in which each set of training data has images labeled "OK" or "DANGER". Based on these labels, the machine learning model 300 can be trained to determine safe distances and / or dangerous distances.

[0059] Figure 8 shows an example of a set of training data labeled "OK". In the example in Figure 8, image 600 is labeled "OK", as shown in label 602. Image 600 includes a bounding box 601. The bounding box 601 represents the position of the worker's body part (hand) detected in image 600.

[0060] Figure 9 shows an example of a set of training data labeled "DANGER". In the example in Figure 9, image 610 is labeled "DANGER", as shown in label 612. Image 610 includes a bounding box 611. The bounding box 611 represents the position of the worker's body part (hand) detected in image 610.

[0061] Image 610 includes a case where the distance between the worker's part and the hazard zone of the tensile testing machine 4 is below the threshold.

[0062] In one implementation example, the labels "OK" or "DANGER" are added to the image by the user.

[0063] In one implementation example, a "DANGER" label is applied if the shortest distance in real space between a worker's body part and the hazardous area is less than a predetermined threshold, and an "OK" label is applied if the shortest distance is equal to or greater than the threshold. The "DANGER" label indicates that the location of the body part is hazardous. The "OK" label indicates that the location of the body part is not hazardous.

[0064] In Embodiment 2, as described with reference to frames 101 and 102 in Figure 6, the position of the imaging device 8 relative to the tensile testing machine 4 is changeable. That is, in the training data 310, the images in the first set may be images taken from a different location than the images in the second set.

[0065] Even if the actual spatial distance between the worker's body part and the hazard zone is the same, the distance (number of pixels) in the captured image may differ depending on the shooting location. For example, the distance in the captured image between the worker's body part and the hazard zone may increase as the imaging device 8 that captured the image is located closer to the tensile testing machine 4, even if the positional relationship between the worker's body part and the hazard zone remains constant.

[0066] If the type of label is determined according to the shortest distance in real space between the worker's body part and the hazard zone, then even if the spacing in the images is the same, the type of label may differ. That is, for images with the same spacing between the worker's body part and the hazard zone, an "OK" label may be applied if the image is taken from a first location, while a "DANGER" label may be applied if the image is taken from a location closer to the tensile testing machine 4 than the first location.

[0067] Figure 10 is a schematic diagram illustrating the usage phase. As shown in Figure 10, the machine learning model 300 is trained to output a classification result of "OK" or "DANGER" for input images through a learning process using the training data 310.

[0068] As described above, the position of the imaging device 8 relative to the tensile testing machine 4 is changeable. Therefore, the machine learning model 300 is expected to output classification results based not only on the number of pixels between the worker's part and the danger zone of the tensile testing machine 4 in the input image, but also on the size of the worker's part and / or the danger zone of the tensile testing machine 4 in the input image. In other words, the machine learning model 300 is expected to be trained to output classification results based not on the number of pixels between the worker's part and the danger zone in the input image, but on the ratio of the number of pixels to the size of the part and / or the danger zone.

[0069] Furthermore, by changing the position of the imaging device 8 relative to the tensile testing machine 4, not only the distance between the imaging device 8 and the tensile testing machine 4, but also the angle of the imaging device 8 relative to the tensile testing machine 4 may change. Therefore, if the training data 310 includes multiple images taken with the imaging device 8 in different configurations, it is expected that the machine learning model 300 will be trained to output classification results according to the relative positions of human body parts and danger zones in the input images with good accuracy, even if the distance, angle, and / or orientation of the imaging device 8 relative to the tensile testing machine 4 that takes the input images is changed.

[0070] Based on the above, it is expected that the machine learning model 300 will be trained to output a classification result based on the shortest distance in real space between the worker's part and the tensile testing machine 4, which is assumed from the classification results. It is expected that the machine learning model 300 will be trained to output a "DANGER" classification result if the shortest distance is less than the threshold, and an "OK" classification result otherwise.

[0071] (Process flow) Figure 11 is a flowchart of the process performed in the control device 14 of Embodiment 2 to notify a warning. In the process shown in Figure 11, a machine learning model trained by the learning process described with reference to Figures 8 to 9 is used. In one implementation, the process is performed in the control device 14 by the processor 140 executing a given program. In one implementation, the process shown in Figure 11 is started when an instruction to start the process is input to the operation unit 144.

[0072] In step S30, the control device 14 acquires an image from the imaging device 8. In step S32, the control device 14 inputs the captured image acquired in step S30 into the machine learning model.

[0073] In step S34, the control device 14 obtains the classification result from the machine learning model. In step S36, the control device 14 determines whether the classification result obtained in step S34 is "DANGER". If the classification result is "DANGER" (YES in step S36), the control device 14 proceeds to step S38; otherwise (NO in step S36), it proceeds to step S40.

[0074] In step S38, the control device 14 outputs a "DANGER" notification, similar to step S20 in Figure 3, and then returns control to step S30.

[0075] In step S38, the control device 14 may further output a signal to the tensile testing machine 4 instructing it to stop operation (or to transition to a predetermined safety state). Upon receiving this signal from the control device 14, the tensile testing machine 4 stops operation.

[0076] In step S40, the control device 14 outputs an "OK" notification, similar to step S22 in Figure 3, and then returns control to step S30.

[0077] Figure 12 shows an example of the "DANGER" notification display screen in Embodiment 2. Figure 13 shows another example of the "DANGER" notification display screen in Embodiment 2.

[0078] In the example shown in Figure 12, screen 700 primarily displays an example of an image acquired in step S30. Screen 700 includes a bounding box 701.

[0079] In the example shown in Figure 13, screen 710 primarily displays other examples of the captured images acquired in step S30. Screen 710 includes a bounding box 701.

[0080] In Figures 12 and 13, the bounding box 701 points to a rectangle containing the area detected as the worker's "hand". In each example in Figures 12 and 13, the bounding box 701 is displayed in a specific color (for example, red). The bounding box 701 is also labeled with the string "Danger". The display of the bounding box 701 in a specific color and / or the labeling of the string "Danger" to the bounding box 701 results in the output of a "DANGER" notification.

[0081] Figure 14 shows an example of the display screen for the "OK" notification in Embodiment 2. In the example shown in Figure 14, screen 720 primarily displays other examples of the captured images acquired in step S30. Screen 720 includes a bounding box 721.

[0082] The bounding box 721 points to a rectangle that includes the area detected as the worker's "hand". In the example in Figure 14, the bounding box 721 is displayed in a different color (for example, blue) than the specific color mentioned above. The bounding box 721 is also labeled with the string "OK". The display of the bounding box 721 in the above color (for example, blue) and / or the labeling of the bounding box 701 with the string "OK" results in the output of the "OK" notification.

[0083] Based on the above, in Embodiment 2, a machine learning model for object detection is used to classify whether the location of a human body part in an image captured from a material testing machine is dangerous or not. If the classification result indicates that the location of a human body part is dangerous, a warning is issued. This allows for immediate detection of the occurrence of an event where the location of a human body part is dangerous, and a warning can be issued. In Embodiment 2, the type of notification to be output ("OK" or "DANGER") is determined according to the classification result from the machine learning model. In other words, it is not necessary to compare the area of ​​the detected human body part with the area of ​​a previously identified danger zone in order to determine the type of notification to be output. This avoids the situation where the amount of computation required in the control device 14 from the input of the captured image to the output of the notification increases due to this comparison, and thus avoids the situation where the time required from the input of the captured image to the output of the notification is prolonged due to this comparison.

[0084] According to this disclosure, safety judgments can be made based on the fact that the location of human body parts and the danger zone are captured, thereby improving the accuracy of the judgment. Furthermore, because this disclosure uses an object detection algorithm rather than a classification model, it is possible to separately extract and judge human body parts and danger zones from a single image. In addition, because this disclosure performs safety judgments based on the distance between human body parts and danger zones, the accuracy of safety judgments is improved, and the basis for the judgment can be easily confirmed afterward.

[0085] [Pattern] Those skilled in the art will understand that the above-described exemplary embodiments are specific examples of the following embodiments.

[0086] (Section 1) A material testing system according to one embodiment comprises a material testing machine, an imaging device, and a computer capable of communicating with the material testing machine and the imaging device, wherein the computer includes a processor and a storage device for storing a program, and the processor performs a method by executing the program, the method may include the steps of: acquiring an image of the material testing machine; detecting a human body part in the image using a machine learning model for object detection; detecting a hazardous area in the image; and determining safety based on the detection of the human body part and the hazardous area in the image.

[0087] According to the material testing system described in paragraph 1, technology is provided to more reliably ensure the safety of operators in material testing machines.

[0088] (Clause 2) In the material testing system described in paragraph 1, the method may further include the steps of: determining whether the distance between the human body part and the danger range in the captured image is less than a predetermined threshold by comparing the area of ​​the human body part with a predetermined danger range of the material testing machine in the captured image; and determining that a danger condition has occurred if it is determined that the distance between the human body part and the danger range in the captured image is less than a predetermined threshold.

[0089] According to the material testing system described in paragraph 2, the occurrence of a hazardous condition can be determined based on the position of human body parts in the captured image.

[0090] (3) In the material testing system described in paragraph 2, the processor may be configured to stop the operation of the material testing machine based on the determination that the dangerous condition has occurred in the determination step.

[0091] According to the material testing system described in paragraph 3, when a dangerous condition is determined to have occurred, the deterioration of the situation due to the continued operation of the material testing machine can be avoided.

[0092] (Clause 4) In the material testing system described in paragraph 2, the method may further include a step of outputting a warning notification based on the determination that the hazardous condition has occurred in the determination step.

[0093] According to the material testing system described in Section 4, it is possible to make the user aware that a hazardous condition has occurred.

[0094] (Clause 5) In the material testing system described in paragraph 2, the method may further include the step of outputting a safety notification when it is determined that the distance between the human body part and the hazardous area in the captured image is not less than the threshold.

[0095] According to the material testing system described in Section 5, the user may be notified that no hazardous situation has occurred.

[0096] (Clause 6) A material testing system according to one embodiment comprises a material testing machine, an imaging device, and a computer capable of communicating with the material testing machine and the imaging device, wherein the computer includes a processor and a storage device for storing a program, and the processor performs a method by executing the program, the method including the steps of: acquiring an image of the material testing machine; detecting human body parts in the image using a machine learning model for object detection and determining whether the location of the human body parts is dangerous; and outputting a warning notification if it is determined that the location of the human body parts in the image is dangerous, The object detection machine learning model is trained using multiple images with correct labels as training data, each of which includes a human body part and the hazardous area of ​​the material testing machine, the correct label indicates whether the location of the human body part is hazardous or not, and the object detection machine learning model is configured to output a classification result of whether the location of the human body part is hazardous or not in the input image after undergoing the training process, and the determination step may include determining whether the location of the human body part is hazardous or not based on the classification result output from the object detection machine learning model in response to the input image.

[0097] The material testing system described in Section 6 provides technology to more reliably ensure the safety of operators in material testing machines.

[0098] (Clause 7) In the material testing system described in Clause 6, the correct label may specify that the position of the human body part is dangerous if the distance in real space between the human body part and the inlet of the test specimen in the material testing machine is less than a predetermined threshold in each of the plurality of images, and that the position of the human body part is not dangerous if the distance in real space between the human body part and the inlet of the test specimen in the material testing machine is not less than a predetermined threshold.

[0099] According to the material testing system described in Section 7, a determination can be made using a machine learning model for object detection to determine whether the position of a human body part is dangerous or not, based on the distance between the human body part and the inlet of the test specimen.

[0100] (Clause 8) In the material testing system described in paragraph 6 or 7, the plurality of images may include images taken from a first position relative to the material testing machine and images taken from a second position different from the first position relative to the material testing machine.

[0101] According to the material testing system described in paragraph 8, the arrangement of the imaging device for capturing images can be appropriately configured.

[0102] (Clause 9) In a material testing system described in any one of paragraphs 6 to 8, the method may further include the step of outputting a safety notification if it is determined that the position of the human body part in the captured image is not dangerous.

[0103] According to the material testing system described in Section 9, the user may be notified that no hazardous situation has occurred.

[0104] (Clause 10) In the material testing system described in paragraph 4 and any one of paragraphs 6 to 9, the step of outputting the warning notification may include instructing the material testing machine to stop operating.

[0105] According to the material testing system described in paragraph 10, when a hazardous situation occurs, not only notification but also appropriate control of the material testing machine can be implemented.

[0106] (Clause 11) In the material testing system described in any one of paragraphs 1 to 7, the hazardous area may include the inlet of the test specimen in the material testing machine.

[0107] According to the material testing system described in paragraph 11, a technology is provided that can promptly notify a warning if a worker's body part is located in a hazardous area.

[0108] (Clause 12) In the material testing system described in any one of paragraphs 1 to 11, the human body part may include a human hand.

[0109] According to the material testing system described in paragraph 12, notifications may be issued regarding parts that are likely to become dangerous to the human body.

[0110] (Clause 13) A method according to one embodiment is a method for determining the safety of a material testing system including a material testing machine, which may include the steps of: acquiring an image of the material testing machine; detecting a human body part in the image using a machine learning model for object detection; detecting a hazardous area in the image; and determining safety based on the detection of the human body part and the hazardous area in the image.

[0111] The method described in paragraph 13 provides a technique for more reliably ensuring worker safety in material testing machines.

[0112] (Clause 14) A method according to one embodiment is a method for determining the safety of a material testing system including a material testing machine, comprising the steps of: acquiring an image of the material testing machine; detecting a human body part in the image using a machine learning model for object detection and determining whether the location of the human body part is dangerous; and outputting a warning notification if it is determined that the location of the human body part in the image is dangerous, wherein the machine learning model for object detection has undergone a learning process using a plurality of images with correct labels as training data, each of the plurality of images includes a human body part and the dangerous range of the material testing machine, the correct label indicates whether the location of the human body part is dangerous, the machine learning model for object detection is configured to output a classification result of whether the location of the human body part is dangerous in the input image after undergoing the learning process, and the step of determining whether the location of the human body part is dangerous based on the classification result output from the machine learning model for object detection in response to the input of the image.

[0113] The method described in paragraph 14 provides a technique for more reliably ensuring worker safety in material testing machines.

[0114] (Clause 15) A program according to one embodiment may be executed by a computer, causing the computer to perform the method described in paragraph 13 or 14.

[0115] According to the program described in Section 15, technologies are provided to more reliably ensure the safety of operators in material testing machines.

[0116] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims rather than by the description of the embodiments above, and all modifications within the meaning and scope equivalent to the claims are intended to be included. Furthermore, each technology in the embodiments is intended to be practiced individually or, as far as possible, in combination with other technologies in the embodiments. [Explanation of Symbols]

[0117] 2 specimens, 4 tensile testing machines, 6, 14 control devices, 8 imaging devices, 16, 18 memory devices, 62, 142 display units, 100 stress measurement devices, 101, 102 frames, 140 processors, 141 memory, 144 operation units, 300 machine learning models, 310 training data, 500, 510, 700, 710, 720 screens, 501 hazard range, 502, 601, 611, 701, 721 bounding boxes, 600, 610 images, 602, 612 labels.

Claims

1. Material testing machine, Imaging device and The system comprises a material testing machine and a computer capable of communicating with the imaging device, The computer includes a processor and a storage device for storing programs, The processor performs the method by executing the program. The aforementioned method, The steps include acquiring an image captured by the aforementioned material testing machine, The steps include: detecting human body parts in the captured image using a machine learning model for object detection; The steps include detecting the danger zone in the captured image, A material testing system comprising the step of determining safety based on the detection of the human body part and the hazardous area in the captured image.

2. The aforementioned method, The steps include: comparing the area of ​​the human body part in the captured image with a predetermined danger range of the material testing machine to determine whether the distance between the human body part and the danger range in the captured image is less than a predetermined threshold; The material testing system according to claim 1, further comprising the step of determining that a dangerous condition has occurred when it is determined that the distance between the human body part and the dangerous area in the captured image is less than a predetermined threshold.

3. The material testing system according to claim 2, wherein the processor is configured to stop the operation of the material testing machine based on the determination that the dangerous condition has occurred in the determination step.

4. The material testing system according to claim 2, wherein the method further includes a step of outputting a warning notification based on the determination that the dangerous condition has occurred in the determination step.

5. The material testing system according to claim 2, further comprising the step of outputting a safety notification when it is determined that the distance between the human body part and the danger range in the captured image is not less than the threshold.

6. Material testing machine, Imaging device and The system comprises a material testing machine and a computer capable of communicating with the imaging device, The computer includes a processor and a storage device for storing programs, The processor performs the method by executing the program. The aforementioned method, The steps include acquiring an image captured by the aforementioned material testing machine, The steps include: using a machine learning model for object detection to detect human body parts in the captured image and determining whether the location of the human body parts is dangerous; The step of outputting a warning notification if the position of the human body part in the captured image is determined to be dangerous, The aforementioned machine learning model for object detection has undergone training using multiple images with correct labels as training data. Each of the aforementioned images includes a human body part and the hazard zone of the material testing machine, The aforementioned correct label indicates whether the position of the human body part is dangerous or not. The machine learning model for object detection is configured to output a classification result indicating whether the position of a human body part in the input image is dangerous or not, after undergoing the training process. The material testing system includes a determination step which involves determining whether the location of the human body part is dangerous based on the classification result output from the object detection machine learning model in response to the input of the captured image.

7. The correct label is, in each of the multiple images, The position of the human body part is defined as dangerous if the distance in real space between the human body part and the inlet of the test specimen in the material testing machine is less than a predetermined threshold. The material testing system according to claim 6, wherein the position of the human body part is not dangerous if the distance in real space between the human body part and the inlet of the test specimen in the material testing machine is not less than a predetermined threshold.

8. The material testing system according to claim 6, wherein the plurality of images include an image taken from a first position relative to the material testing machine and an image taken from a second position different from the first position relative to the material testing machine.

9. The material testing system according to claim 6, further comprising the step of outputting a safety notification when it is determined that the position of the human body part in the captured image is not dangerous.

10. The material testing system according to any one of claims 4 and 6 to 9, wherein the step of outputting the warning notification includes instructing the material testing machine to stop operating.

11. The material testing system according to any one of claims 1 to 9, wherein the hazardous area includes the inlet for the test specimen in the material testing machine.

12. The material testing system according to any one of claims 1 to 9, wherein the aforementioned human body part includes a human hand.

13. A method for determining the safety of a materials testing system, including a materials testing machine, The steps include acquiring an image captured by the aforementioned material testing machine, The steps include: detecting human body parts in the captured image using a machine learning model for object detection; The steps include detecting the danger zone in the captured image, A method comprising the step of determining safety based on the detection of the human body part and the hazardous area in the captured image.

14. A method for determining the safety of a materials testing system, including a materials testing machine, The steps include acquiring an image captured by the aforementioned material testing machine, The steps include: using a machine learning model for object detection to detect human body parts in the captured image and determining whether the location of the human body parts is dangerous; The step of outputting a warning notification if the position of the human body part in the captured image is determined to be dangerous, The aforementioned machine learning model for object detection has undergone training using multiple images with correct labels as training data. Each of the aforementioned images includes a human body part and the hazard zone of the material testing machine, The aforementioned correct label indicates whether the position of the human body part is dangerous or not. The machine learning model for object detection is configured to output a classification result indicating whether the position of a human body part in the input image is dangerous or not, after undergoing the training process. The method includes determining whether the position of the human body part is dangerous based on the classification result output from the machine learning model for object detection in response to the input of the captured image.

15. A program that, when executed by a computer, causes the computer to perform the method described in claim 13 or claim 14.

Images