State determination device, state determination method, and program

Abstract

A state determination device according to the present invention acquires, as fixed images, each of time series images obtained by an imaging device capturing successive images of a rotating object via an optical device for rotating and emitting incident light. Using each of the acquired fixed images, the displacements of a plurality specific points on the object are calculated, and with any one of the displacements of the specific points as a reference specific point, the difference between the displacement of the reference specific point and the displacement of a specific point other than the reference specific point is calculated. Using the difference, displacement of the surface of the object on an imaging plane of the imaging device in accordance with movement of the object in the normal direction of the imaging plane is calculated as an out-of-plane displacement. Using the difference and the out-of-plane displacement, displacement of the surface of the object on the imaging plane in accordance with movement of the surface of the object in the direction along the surface is calculated as an in-plane displacement. An abnormality in the object is identified on the basis of the distribution of the in-plane displacement, or a change in the in-plane displacement over time.

Description

[Technical Field]

[0001] This invention relates to a state determination device and a state determination method for determining the state of an object, and further, to a program for realizing these. Mu To relate to. [Background technology]

[0002] When defects such as surface cracks, delamination, or internal cavities occur in objects that move or rotate, such as wind turbine propellers, turbine blades, or the wheels of a moving vehicle, it can adversely affect the integrity of the structure containing the object. Therefore, it is necessary to detect these defects as abnormalities as quickly and accurately as possible.

[0003] One method for inspecting objects involves stopping the object's movement and having an inspector perform a visual inspection or tapping test to detect any abnormalities. However, this method incurs significant human resource costs and has the drawback of preventing the use of the structure containing the object during the inspection.

[0004] Therefore, methods have been developed to determine the state of an object based on an image of the object without stopping its motion. For example, in the techniques disclosed in Non-Patent Documents 1 and 2, an object is captured in video using a stereo camera, and the movement of multiple points on the object is determined from the time-series frame images that make up the video, thereby detecting anomalies in the moving object.

[0005] Furthermore, the technologies disclosed in Patent Documents 1 and 2 detect the amount of displacement along the surface of an object (hereinafter also referred to as "in-plane displacement") and the amount of displacement along the optical axis of the camera (hereinafter also referred to as "out-of-plane displacement") from video footage taken with a single camera, rather than a stereo camera. In addition, the technologies disclosed in Patent Documents 1 and 2 detect defects (abnormalities) in the object, such as cracks, delamination, or internal cavities, based on the detected out-of-plane and in-plane displacements. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2016 / 152075 [Patent Document 2] International Publication No. 2016 / 152076 [Non-patent literature]

[0007] [Non-Patent Document 1] Lee, Hyoseong, et al. "Measurement of 3-D vibration by dynamic photogrammetry using least-square image matching for sub-pixel targeting." SENSORS, 2015 IEEE. IEEE, 2015. [Non-Patent Document 2] Winstroth, Jan, and JorgSeume. "Wind turbine rotor blade monitoring using digital image correlation: Assessment on a scaled model." 32nd ASME Wind Energy Symposium. 2014. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] The technologies disclosed in Non-Patent Documents 1 and 2 require the use of a stereo camera to capture video, which presents the problem of increased device size due to the size of the stereo camera. On the other hand, the technologies disclosed in Patent Documents 1 and 2 use only one camera, which can reduce the size compared to a stereo camera, thus preventing the problem of increased device size caused by the stereo camera.

[0009] However, in the technologies disclosed in Patent Documents 1 and 2, even when attempting to measure out-of-plane and in-plane displacements of a moving object, the displacement due to vibrations (noise) associated with the object's motion is added to both the out-of-plane and in-plane displacements, making it difficult to obtain accurate measurements. As a result, it becomes difficult to capture the characteristics that appear in the out-of-plane and in-plane displacements caused by anomalies in the object from the measured values, and consequently, the accuracy of detecting anomalies in the object decreases.

[0010] Furthermore, when an object is photographed, motion blur occurs in the resulting image. Therefore, in the technologies disclosed in Patent Documents 1 and 2, the accuracy of displacement measurement is reduced due to motion blur, which in turn reduces the accuracy of detecting anomalies in the object.

[0011] This disclosure, in one aspect, provides a state determination device, a state determination method, and a device that accurately detect abnormalities in rotating objects while suppressing the increase in size of the device. program The purpose is to provide. [Means for solving the problem]

[0012] To achieve the above objective, the state determination device in one aspect is: A fixed image acquisition unit acquires each of the time-series images obtained by continuously photographing a rotating object with an imaging device via a first optical device that rotates and emits incident light, as a fixed image. A difference calculation unit calculates the displacement of multiple specific points of the object using each of the acquired fixed images, and using one of the multiple specific points as a reference specific point, calculates the difference between the displacement of the reference specific point and the displacement of the other specific points. An out-of-plane displacement calculation unit calculates the displacement of the object's surface on the imaging surface as it moves in the direction normal to the imaging surface of the imaging device, using the difference described above, An in-plane displacement calculation unit that calculates, as in-plane displacement, the displacement of the surface of the object on the imaging surface as the surface of the object moves in a direction along the surface, using the difference and the out-of-plane displacement; An abnormality determination unit that identifies an abnormality of the object based on the distribution of the in-plane displacement or the temporal change of the out-of-plane displacement; characterized by comprising the above.

[0013] Also, in order to achieve the above object, a state determination method on one side A step of acquiring, as fixed images, each of the time-series images obtained by continuously photographing a rotating object with an imaging device via a first optical device that rotates incident light and emits it; A step of calculating the displacement amounts of a plurality of specific points of the object using each of the acquired fixed images, and calculating a difference between the displacement amount of a reference specific point and the displacement amounts of the specific points other than the reference specific point, with any one of the plurality of specific points as the reference specific point; A step of calculating, as out-of-plane displacement, the displacement of the surface of the object on the imaging surface as the object moves in the normal direction of the imaging surface of the imaging device, using the difference; A step of calculating, as in-plane displacement, the displacement of the surface of the object on the imaging surface as the surface of the object moves in a direction along the surface, using the difference and the out-of-plane displacement; A step of identifying an abnormality of the object based on the distribution of the in-plane displacement or the temporal change of the out-of-plane displacement; characterized by comprising the above.

[0014] Furthermore, in order to achieve the above object, on one side program is A computer A step of acquiring, as fixed images, each of the time-series images obtained by continuously photographing a rotating object with an imaging device via a first optical device that rotates incident light and emits it; Using each of the acquired fixed images, the displacement amounts of multiple specific points of the object are calculated, and one of the multiple specific points is designated as a reference point. The difference between the displacement amount of the reference point and the displacement amounts of the other specific points is then calculated. Using the difference, the step of calculating the displacement of the object's surface on the imaging surface as an out-of-plane displacement due to the object moving in the direction normal to the imaging surface of the imaging device, Using the difference and the out-of-plane displacement, the step of calculating the displacement of the object's surface on the imaging surface as an in-plane displacement due to the object's surface moving in a direction along the surface, A step of identifying an abnormality in the object based on the distribution of the in-plane displacement or the temporal change in the out-of-plane displacement, Let's execute it ru, It is characterized by the following: [Effects of the Invention]

[0015] One aspect of this disclosure is that it is possible to accurately detect anomalies in objects while suppressing the increase in the size of the device. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 is a schematic diagram illustrating the general configuration of the state determination device in Embodiment 1. [Figure 2] Figure 2 is a diagram illustrating an example of the specific configuration of the state determination device of Embodiment 1. [Figure 3] Figure 3 is a configuration diagram showing an example of an optical device used in Embodiment 1. [Figure 4A] Figure 4A shows the relationship between the image rotation speed and the summation image rotation speed of each dove prism when the rotation speed of the object is constant. [Figure 4B] Figure 4B shows the relationship between the image rotation speed of each dove prism and the summation image rotation speed when the rotation speed of the object is variable. [Figure 5] Figure 5 shows another example of an optical device. [Figure 6A]Figure 6A is a diagram illustrating the process by which the fixed image acquisition unit determines the rotation speed and direction of an object, and shows a specific frame of the object's time-series image. [Figure 6B] Figure 6B is a diagram illustrating the process by which the fixed image acquisition unit determines the rotation speed and direction of an object, and shows the frame following the frame shown in Figure 6A. [Figure 7] Figure 7 shows an image captured by the imaging device through a rotating dove prism. [Figure 8] Figure 8 shows an example of a time-series fixed image when no extraneous vibrations are occurring in the object. [Figure 9] Figure 9 shows an example of a time-series fixed image when an object is experiencing extraneous vibrations. [Figure 10A] Figure 10A shows the change in position over time at each specific point on the object shown in Figures 8 and 9. [Figure 10B] Figure 10B shows the change in position over time at each of the specific points on the object shown in Figures 8 and 9, with one of the specific points being used as a reference point. [Figure 11] Figure 11 shows the optical arrangement of an object during imaging. [Figure 12] Figure 12 is a diagram illustrating the calculation process of depth movement by the out-of-plane displacement calculation unit. [Figure 13A] Figure 13A shows the out-of-plane displacement on the imaging plane when an object moves uniformly in the direction of the optical axis of the imaging device. [Figure 13B] Figure 13B shows the relationship between out-of-plane displacement and distance from the imaging center when an object moves uniformly in the direction of the optical axis of the imaging device. [Figure 13C] Figure 13C shows the relationship between out-of-plane displacement and distance from the imaging center when an object does not move uniformly in the optical axis direction of the imaging device. [Figure 14A] Figure 14A shows the relationship between out-of-plane displacement and in-plane displacement on the imaging plane when an object moves uniformly in the direction of the optical axis of the imaging device. [Figure 14B]Figure 14B shows the relationship between the sum of out-of-plane and in-plane displacements and the distance from the imaging center when an object moves uniformly in the direction of the optical axis of the imaging device. [Figure 14C] Figure 14C shows the relationship between the sum of out-of-plane and in-plane displacements and the distance from the imaging center when an object does not move uniformly in the optical axis direction of the imaging device. [Figure 15A] Figure 15A shows the frequency characteristics of the displacement at the specific point A shown in Figure 8. [Figure 15B] Figure 15B shows the frequency characteristics of the difference [AD] between the displacement of a specific point A and the displacement of a specific point D, as shown in Figure 8. [Figure 16A] Figure 16A is a schematic diagram showing a portion of an object whose state is to be determined. [Figure 16B] Figure 16B shows the relationship between the out-of-plane displacement of a portion of the object shown in Figure 16A and the vibration frequency. [Figure 17A] Figure 17A schematically shows a static state of a portion of an object in which a crack has formed. [Figure 17B] Figure 17B schematically shows a state in which a part of the object shown in Figure 17A is undergoing rotational motion. [Figure 17C] Figure 17C shows the time evolution of the in-plane displacement of a portion of the object shown in Figure 17A. [Figure 17D] Figure 17D shows the relationship between the in-plane displacement and position of a portion of the object shown in Figure 17A. [Figure 18] Figure 18 is a flowchart showing the operation of the state determination device in Embodiment 1. [Figure 19] Figure 19 is a block diagram showing the configuration of the state determination device in Embodiment 2. [Figure 20] Figure 20 shows the configuration of the optical device. [Figure 21] Figure 21 is a schematic diagram illustrating an object undergoing translational motion. [Figure 22] Figure 22 is a block diagram showing an example of a computer that implements the state determination device in Embodiments 1 and 2. [Modes for carrying out the invention]

[0017] (Embodiment 1) The state determination device in Embodiment 1 will be described below with reference to the drawings.

[0018] [Device configuration] First, the schematic configuration of the state determination device in Embodiment 1 will be described using Figure 1. Figure 1 is a configuration diagram illustrating the schematic configuration of the state determination device in Embodiment 1.

[0019] The state determination device 100 is a device that detects the displacement of the surface of an object from a time-series image obtained by imaging the object, and determines whether the object is abnormal. The state determination device 100 includes a fixed image acquisition unit 1, a difference calculation unit 2, an out-of-plane displacement calculation unit 3, an in-plane displacement calculation unit 4, and an abnormality determination unit 5.

[0020] The fixed image acquisition unit 1 acquires each of the time-series images obtained by continuously photographing a rotating object with an imaging device via an optical device that rotates and emits incident light, as a fixed image.

[0021] The difference calculation unit 2 uses each of the acquired fixed images to calculate the displacement amount of multiple specific points on the object, and then... point One of the specified points is designated as the reference point, and the difference between the displacement of the reference point and the displacement of the other specified points is calculated.

[0022] The out-of-plane displacement calculation unit 3 uses the difference to calculate the displacement of the object's surface on the imaging surface as an out-of-plane displacement, which occurs when the object moves in the direction normal to the imaging surface of the imaging device.

[0023] The in-plane displacement calculation unit 4 uses the difference and out-of-plane displacement to calculate the displacement of the object's surface on the imaging surface as an in-plane displacement, which occurs as the object's surface moves in a direction along its surface.

[0024] The abnormality detection unit 5 identifies an abnormality in the object based on the distribution of in-plane displacement or the temporal change in out-of-plane displacement.

[0025] According to the above-described state determination device 100, unlike the technologies disclosed in Non-Patent Documents 1 and 2, a stereo camera is not required, and abnormalities in an object can be determined using only a single imaging device, thus suppressing the need for a larger device. Furthermore, by suppressing errors caused by extraneous vibrations (noise) generated in a rotating object, abnormalities in a rotating object can be detected with high accuracy.

[0026] Next, the configuration and function of the state determination device in Embodiment 1 will be described in detail. Figure 2 is a configuration diagram illustrating an example of the specific configuration of the state determination device 100 in Embodiment 1.

[0027] The state determination device 100 is connected to an imaging device 101 for photographing the object 20 that is the subject of state determination. The imaging device 101 is equipped with a solid-state image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor). In Embodiment 1, the object 20 is a moving object that performs rotational motion, such as a wind power generation propeller or turbine. The state determination device 100 performs state determination particularly on the blades of the propeller and turbine. In the example in Figure 2, the object 20 is a propeller. The object 20 consists of a blade 21 and a spinner 22.

[0028] In Embodiment 1, the imaging device 101 has a function to output image data continuously at set intervals, and generates an image (hereinafter referred to as "time-series image") composed of consecutive frames in a time series. In Embodiment 1, the frame rate of the time-series image is appropriately set considering the natural frequency of the object 20, and is set to, for example, 400 fps (frames per second). The imaging device 101 outputs the image data of the time-series image to the state determination device 100 on a frame-by-frame basis.

[0029] Furthermore, as shown in Figure 2, in Embodiment 1, an optical device 102 is positioned between the imaging device 101 and the object 20 to be imaged. The optical device 102 includes rotating optical systems 32 and 33 to rotate and emit incident light.

[0030] The configuration of the optical device 102 will be specifically explained using Figure 3. Figure 3 is a configuration diagram showing an example of the optical device used in Embodiment 1.

[0031] As shown in Figure 3, the rotating optical system 32 comprises a hollow motor (not shown), a dove prism 35, and a rotation drive unit 37. The dove prism 35 is held inside the hollow motor. The rotation drive unit 37 drives the hollow motor. In the rotating optical system 32, when the rotation drive unit 37 drives the hollow motor, the dove prism 35 rotates.

[0032] Similarly, as shown in Figure 3, the rotating optical system 33 also includes a hollow motor (not shown), a dove prism 36, and a rotation drive unit 38. The dove prism 36 is also held inside the hollow motor. In the rotating optical system 33 as well, when the rotation drive unit 38 drives the hollow motor, the dove prism 36 rotates.

[0033] Furthermore, as shown in Figure 3, the rotating optical systems 32 and 33 are arranged in series along the optical axis (axis passing through the center of the imaging plane) 34 of the imaging device 101. Specifically, the rotating optical systems 32 and 33 are arranged such that their respective dove prisms 35 and 36 are located on the optical axis 34. The dove prisms 35 and 36 rotate around the optical axis 34.

[0034] Since the dove prisms 35 and 36 each rotate around the optical axis 34, they rotate the incident image around the optical axis 34. Therefore, the imaging device 101 images the object 20 through the dove prisms 35 and 36, which rotate at a rotational speed corresponding to the rotational speed of the object 20, and as a result, it is possible to capture an image in which the rotation of the object 20 is canceled out.

[0035] Furthermore, the dove prisms 35 and 36 can rotate an optically incident image at twice their rotational speed (number of rotations). Therefore, if dove prisms 35 and 36 are rotated in opposite directions, the rotational speed of the image that enters from dove prism 35 and exits from dove prism 36 will be the sum of twice the rotational speeds of dove prisms 35 and 36. For example, if dove prisms 35 and 36 are rotated at the same rotational speed but in opposite directions, the rotational speed of the image will be four times the rotational speeds of dove prisms 35 and 36.

[0036] For example, suppose the cross-sectional size of dove prism 35 (the cross-section perpendicular to the optical axis 34) is 10 mm x 10 mm, and the cross-sectional size of dove prism 36 is 20 mm x 20 mm. Also, suppose the rotation speed of dove prism 35 is controlled within the range of 0 rpm to 4800 rpm, and the rotation speed of dove prism 36 is controlled between 0 rpm and 2400 rpm. If dove prism 35 is rotated at 4800 rpm and dove prism 36 is rotated at 2400 rpm, the image that has passed through the two dove prisms 35 and 36 will rotate at 14400 rpm (= 4800 rpm x 2 + 2400 rpm x 2), which is the sum of twice the rotation speed of each dove prism. In other words, in this case, the rotation speed of the image that has passed through dove prisms 35 and 36 is controlled between 0 rpm and 14400 rpm.

[0037] In Embodiment 1, the sizes of the dove prisms 35 and 36 are determined according to the field of view of the imaging lens of the imaging device 101. For example, as the distance between the dove prisms 35 and 36 and the imaging device 101 increases, the field of view of the imaging lens increases, so the sizes of the dove prisms 35 and 36 are set to be larger.

[0038] Note that more than three dove prisms may be used. For example, if a prism with a cross-sectional size of 5 mm x 5 mm is rotated at a speed of 7200 rpm, a prism with a cross-sectional size of 10 mm x 10 mm is rotated at a speed of 4800 rpm, and a prism with a cross-sectional size of 20 mm x 20 mm is rotated at a speed of 1800 rpm, the image that has passed through the three dove prisms will rotate at a speed of 27600 rpm (= 7200 rpm x 2 + 4800 rpm x 2 + 1800 rpm x 2). In this case, the rotation speed of the image is controlled between 0 rpm and 27600 rpm.

[0039] Thus, when multiple dove prisms are used, a larger image rotation speed can be obtained with less driving force compared to the case of a single dove prism, enabling appropriate response control according to the movement (rotation) of the object 20. In Embodiment 1, the rotation speed of each dove prism is determined by the fixed image acquisition unit 1, as will be described later.

[0040] Figure 4A shows the relationship between the image rotation speed of each dove prism 35 and 36 and the summation image rotation speed when the rotation speed of the object is constant. Figure 4B shows the relationship between the image rotation speed of each dove prism 35 and 36 and the summation image rotation speed when the rotation speed of the object varies. The image rotation speed of each dove prism 35 and 36 is the rotation speed of the image rotated by each dove prism 35 and 36. The summation image rotation speed is the rotation speed of the image rotated by the two dove prisms 35 and 36. When the rotation speed of object 20 is constant, the relationship between the image rotation speed of the dove prism 36 rotating at low speed, the image rotation speed of the dove prism 35 rotating at high speed, and the summation image rotation speed by dove prisms 35 and 36 remains constant over time, as shown in Figure 4A. On the other hand, when the rotation speed of object 20 varies, the above relationship changes over time, as shown in Figure 4B.

[0041] In Embodiment 1, the optical device 102 is not limited to a configuration that includes dove prisms 35 and 36. For example, the optical device 102 may be an image rotator composed of three mirrors 102A, 102B, and 102C that rotate around axis 102D, as shown in Figure 5. Figure 5 shows another example of the optical device 102.

[0042] Next, we will describe the various parts of the state determination device 100 of Embodiment 1, as shown in Figure 2.

[0043] In Embodiment 1, the fixed image acquisition unit 1 can cause the imaging device 101 to photograph the object 20 without rotating the dove prism of the optical device 102. In this case, the fixed image acquisition unit 1 acquires image data of the time-series images captured by the imaging device 101, and determines the rotation speed and direction of the rotating object 20 based on the differences between frames of the acquired time-series images.

[0044] Figure 6A is a diagram illustrating the process by which the fixed image acquisition unit 1 determines the rotation speed and direction of an object, and shows a specific frame of the time-series image of object 20. Figure 6B is a diagram illustrating the process by which the fixed image acquisition unit 1 determines the rotation speed and direction of an object, and shows the frame following the frame shown in Figure 6A.

[0045] Object 20 is a propeller that rotates around a rotation axis 20A and has two blades 21 and a spinner 22. The number of blades on object 20 is not particularly limited. The fixed image acquisition unit 1 first performs a known object detection process (e.g., pattern matching) using frame images of the time-series images of object 20 shown in Figure 6A. As a result, the fixed image acquisition unit 1 detects the blades 21 of object 20 and sets a rectangular selection region 23 surrounding the detected blades 21. At this time, the coordinates of the upper left corner of the selection region 23 are set to (X1, Y1). Note that detection of multiple blades 21 may be performed simultaneously. In this case, a selection region 23 is set for each blade 21.

[0046] The dashed line in Figure 6A shows the selected region 23 in Figure 6A rotated by an angle θ around the rotation axis 20A. The fixed image acquisition unit 1 rotates the selected region 23 around the rotation axis 20A by an angle θ of, for example, 1° increments from 0° to 360°, generating multiple images with different rotation angles. Then, for each image of the rotated selected region 23, the fixed image acquisition unit 1 performs an image correlation calculation with the next frame image shown in Figure 6B, and based on the calculation results for each image, it identifies the angle θ' that maximizes the correlation value (image similarity).

[0047] Here, affine transformation can be used as the image processing method for rotating the image. In the example above, the angle rotated in one rotation is set to 1°, but this is not particularly limited in Embodiment 1. The angle rotated in one rotation can be set appropriately according to the required accuracy or computational cost. In addition, various image processing methods other than image correlation calculation may be used to evaluate the similarity of the images. Furthermore, various image interpolation processes may be performed during image rotation. Moreover, various methods may be selected for the processing method used to evaluate the similarity of the images and the image processing method used for image rotation, depending on the computational cost.

[0048] The fixed image acquisition unit 1, upon determining the angle θ', calculates the rotational speed (number of rotations) of the blades 21 of the object 20 using the angle θ' and the frame rate. The fixed image acquisition unit 1 determines the rotational speeds of the dove prisms 35 and 36 respectively so that the sum of the rotational speeds matches the calculated rotational speed of the object 20. The fixed image acquisition unit 1 then instructs the rotation drive units 37 and 38 to rotate the dove prisms 35 and 36 in opposite directions at the determined rotational speeds. As a result, the imaging device 101 images the object 20 through the rotating dove prisms 35 and 36. Consequently, as shown in Figure 7, the imaging device 101 captures an image in which the rotational blur (motion blur) of each blade 21 of the object 20 is canceled out. Figure 7 shows an image captured by the imaging device 101 through the rotating dove prisms 35 and 36.

[0049] The fixed image acquisition unit 1 detects the blades 21 of the object 20 in the captured image shown in Figure 7 and sets a rectangular selection region 24 surrounding the detected blades 21. Also, similar to the selection region 23 described above, the coordinates of the upper left corner of the selection region 24 are set to (X1, Y1). In this case as well, detection of multiple blades 21 may be performed simultaneously, in which case a selection region 24 is set for each blade 21. In Embodiment 1, the image shown in Figure 7, in which this rotational blur has been canceled out, is referred to as the "fixed image". The fixed image acquisition unit 1 repeats the process described above for each frame image to acquire a time-series of fixed images in which rotational blur has been suppressed, and sets a selection region 24 for each fixed image.

[0050] As described above, the difference calculation unit 2 uses the time-series fixed images generated by the fixed image acquisition unit 1 to calculate the displacement amounts of multiple specific points on the object 20. The displacement calculation process in the difference calculation unit 2 will be explained using Figures 8 and 9. Figure 8 is a diagram showing an example of a time-series fixed image when no extra vibration is occurring in the object. Figure 9 is a diagram showing an example of a time-series fixed image when extra vibration is occurring in the object.

[0051] In Figures 8 and 9, the solid line indicates the outer edge of the object in a particular frame, and the dashed line indicates the outer edge of object 20 in the next frame. As shown in Figures 8 and 9, specific points A, B, C, and D on the blade 21 of object 20 are displaced by the rotation of object 20 and move to specific points A', B', C', and D', respectively. If no extra vibration is occurring in object 20, the positions of specific points A', B', C', and D' are the same in both Figure 8 and Figure 9. However, in Figure 9, extra vibration is occurring in object 20. Therefore, the positions of specific points A', B', C', and D' are different in Figures 8 and 9.

[0052] Here, we will use Figures 10A and 10B to explain the time change in the position of the specific points shown in Figures 8 and 9. Figure 10A is a diagram showing the time change in the position of each specific point on the object 20 shown in Figures 8 and 9. Figure 10B is a diagram showing the time change in the position of each specific point on the object 20 shown in Figures 8 and 9, with one of the specific points as a reference point.

[0053] As shown in Figure 10A, if object 20 is subjected to the extra vibrations shown in Figure 9, in the fixed image, specific points A, B, C, and D will have displacements due to the extra vibrations shown by the dashed lines, in addition to the displacement due to the rotation of object 20 shown by the solid lines (displacement shown in Figure 8).

[0054] Therefore, the difference calculation unit 2 uses one of the specific points A to D as the reference specific point (in this case, specific point D), and calculates the difference between the displacement of specific point D and the displacements of specific points A, B, and C, respectively. As a result, as shown in Figure 10B, the displacement due to extraneous vibrations is removed from the displacement of each specific point in the fixed image shown in Figure 9. In Figure 10B, the displacement of each specific point is shown with specific point D as the reference. In Figure 10B, "AD" shows the difference between the displacement of specific point D and the displacement of specific point A, "BD" shows the difference between the displacement of specific point D and the displacement of specific point B, and further, "CD" shows the difference between the displacement of specific point D and the displacement of specific point C. In addition, the displacement of each specific point shown in Figure 10B has characteristics similar to the displacement of specific points A, B, and C when there is no extraneous vibration as shown in Figure 8.

[0055] Furthermore, in Embodiment 1, the out-of-plane displacement calculation unit 3 calculates the out-of-plane displacement of the object as described above. In Embodiment 1, the in-plane displacement calculation unit 4 calculates the in-plane displacement of the object as described above. Here, the calculation process of out-of-plane displacement by the out-of-plane displacement calculation unit 3 and the calculation process of in-plane displacement by the in-plane displacement calculation unit 4 will be specifically explained using Figures 11 to 14.

[0056] Figure 11 shows the optical arrangement of object 20 during imaging.

[0057] In FIG. 11, the surface of the object 20 (particularly the blade 21) photographed in a specific frame is indicated by reference numeral 25, and the surface of the object 20 photographed in the next frame is indicated by reference numeral 26. In FIG. 11, the z-axis coincides with the optical axis direction of the imaging device 101 (the normal direction of the imaging surface), and the x-axis and y-axis are two axes orthogonal to the optical axis direction, and coincide with the vertical direction and the horizontal direction in the solid-state imaging device.

[0058] As shown in FIG. 11, it is assumed that the object 20 moves and its surface 25 moves by Δz in the z direction. Also, it is assumed that an image at coordinates (x, y) on the surface 25 is projected onto the coordinates (x i , y i ) on the imaging surface. In this case, in the field of view of the imaging surface of the imaging device 101, the coordinate x i of the image at coordinates (x, y) on the surface 25 is displaced by δx i in the x direction. Similarly, the coordinate y i of the image at coordinates (x, y) on the surface 25 is displaced by δy i in the y direction. While Δz is called the depth movement amount, the displacements δx i , δy i on the imaging surface are hereinafter referred to as out-of-plane displacements.

[0059] Also, as shown in FIG. 11, when the surface 25 of the blade 21 of the object 20 moves by the depth movement amount Δz in the z direction, it is distorted, and thereby surface displacements Δx and Δy occur on the surface 26. The displacements Δx i , Δy i projected onto the imaging surface at this time are hereinafter referred to as in-plane displacements.

[0060] The out-of-plane displacement and the in-plane displacement are represented by Equation 1 and Equation 2, respectively, when the focal length of the lens of the imaging device 101 is represented by f and the imaging distance is represented by L.

[0061]

Equation

[0062]

Equation

[0063] In Embodiment 1, the out-of-plane displacement calculation unit 3 calculates the out-of-plane displacement δx from the depth movement amount Δz described above. i δy i The calculation is performed. Figure 12 is a diagram illustrating the calculation process of the depth movement Δz by the out-of-plane displacement calculation unit 3.

[0064] As shown in Figure 12, assume that a point located at coordinate x1 and a point located at coordinate x2 on the surface 25 of the object move uniformly by Δz in the optical axis direction of the imaging device 101. In this case, the out-of-plane displacement at the point located at coordinate x1 is δx1, and the out-of-plane displacement at the point located at coordinate x2 is δx2. Here, using the difference δd between the out-of-plane displacements δx1 and δx2 shown in Equation 3, Δz can be obtained from Equation 4.

[0065] Also, the coordinates of the point located at coordinate x1 when projected onto the imaging surface are x i1 The coordinates of the point located at coordinate x2 when projected onto the imaging plane are x i2 Let α be the distance between coordinates x1 and x2, and the coordinate x projected onto the imaging plane. i1 and coordinate x i2 If the distance between α and β is denoted as β, then the relationship between distance α and distance β is given by equation 5. Furthermore, the difference δd has the same characteristics as the difference in displacement amounts [AD], [BD], or [CD] shown in Figure 10B, and can therefore be calculated by the difference calculation unit 2. Accordingly, the out-of-plane displacement calculation unit 3 calculates the depth movement amount Δz using equations 3 and 4 below.

[0066]

number

[0067]

number

[0068]

number

[0069] Out-of-plane displacement will be explained in detail using Figures 13A to 13C. Figure 13A shows the out-of-plane displacement on the imaging plane when the object 20 moves uniformly in the optical axis direction of the imaging device 101. Figure 13B shows the relationship between the out-of-plane displacement and the distance from the imaging center when the object 20 moves uniformly in the optical axis direction of the imaging device 101. Figure 13C shows the relationship between the out-of-plane displacement and the distance from the imaging center when the object 20 does not move uniformly in the optical axis direction of the imaging device 101.

[0070] For example, as shown in Figure 12, suppose that object 20 moves uniformly along the optical axis of the imaging device 101. In this case, as shown in Figure 13A, the out-of-plane displacement on the imaging plane becomes a group of radial vectors centered on the optical axis of the imaging device 101. In other words, the out-of-plane displacement vector increases in proportion to the distance from the imaging center (from equation 1 above). At this time, for example, the magnitude of the out-of-plane displacement δx in the x-axis direction is a straight line, as shown in Figure 13B. The slope of this straight line corresponds to the depth movement Δz.

[0071] Furthermore, if the movement of the object 20 in the optical axis direction of the imaging device 101 is not uniform, as shown in Figure 13C, the slope of the time change of the out-of-plane displacement will differ in each interval (xi3, xi4), interval (xi4, xi1), and interval (xi1, xi2). In this case, the depth movement amount Δz will differ in each interval. Therefore, the out-of-plane displacement calculation unit 3 calculates the out-of-plane displacement based on the depth movement amount Δz corresponding to each interval, using Equation 1 above.

[0072] The relationship between out-of-plane displacement vectors and in-plane displacement vectors is explained using Figures 14A to 14C. Figure 14A shows the relationship between out-of-plane displacement and in-plane displacement on the imaging plane when an object moves uniformly in the optical axis direction of the imaging device. Figure 14B shows the relationship between the sum of out-of-plane and in-plane displacements and the distance from the imaging center when an object moves uniformly in the optical axis direction of the imaging device. Figure 14C shows the relationship between the sum of out-of-plane and in-plane displacements and the distance from the imaging center when an object does not move uniformly in the optical axis direction of the imaging device.

[0073] In Figure 14A, the dotted arrow indicates the displacement amount of a specific point in the time-series fixed image calculated by the difference calculation unit 2 as explained in Figure 10B (hereinafter referred to as the measurement vector V(Vx)). i ,Vy i This represents the measurement vector V(Vx). i ,Vy i ) is the out-of-plane displacement vector δ(δx i ,δy i ) and the in-plane displacement vector Δ(Δx i ,Δy i This is the composite vector of ) and ).

[0074] The solid lines in Figures 14B and 14C represent the out-of-plane displacement vector (x component) shown in Figures 13B and 13C. The dashed lines in Figures 14B and 14C represent the displacement calculated by the difference calculation unit 2, which is the measured vector (x component) obtained by adding the in-plane displacement vector (x component) to the out-of-plane displacement vector. When the magnitude of the in-plane displacement is sufficiently smaller than the magnitude of the out-of-plane displacement, Δz obtained in Equation 4 can be approximated as the depth movement. Therefore, in the out-of-plane displacement calculation unit 3, the out-of-plane displacement in each section can be obtained by substituting Δz into Equation 1.

[0075] Furthermore, the in-plane displacement vector (x component) can be separated by subtracting the out-of-plane displacement vector (x component) from the measurement vector (x component) at each point within each interval calculated by the difference calculation unit 2. The separation of the out-of-plane displacement vector (x component) is performed by the in-plane displacement calculation unit 4.

[0076] Here, we have described a method for calculating out-of-plane displacement from out-of-plane displacement obtained by projection in the x-direction. However, the out-of-plane displacement in the y-direction, which is perpendicular to the optical axis direction (z-direction) and the x-direction of the lens of the imaging device 101, can also be calculated in the same way as in the x-direction.

[0077] In Embodiment 1, the abnormality detection unit 5 estimates an abnormality in the blade 21 of the object 20. The processing in the abnormality detection unit 5 will be explained using Figures 15 to 17.

[0078] In Embodiment 1, the abnormality detection unit 5 analyzes the out-of-plane displacement distribution of the rotating object 20 at each time point. The abnormality detection unit 5 also analyzes the in-plane displacement distribution of the rotating object. The abnormality detection unit 5 then uses the analysis results of the out-of-plane displacement distribution and the analysis results of the in-plane displacement distribution to determine if the object 20 is abnormal. This will be explained in detail below.

[0079] First, let's explain the analysis of out-of-plane displacement. Figures 15A and 15B show the frequency characteristics of an object 20 that vibrates due to rotation. Figure 15A shows the frequency characteristics of the displacement at a specific point A shown in Figure 8. Figure 15B shows the frequency characteristics of the difference [AD] between the displacement at specific point A and the displacement at specific point D shown in Figure 8.

[0080] Object 20 is a propeller, and the blade 21 forms a cantilever beam. Therefore, the blade 21 has a natural vibration mode corresponding to its size and material, and frequency components corresponding to the natural vibration mode appear in its vibration frequency characteristics.

[0081] As shown in Figure 15A, the natural frequencies of the first vibration mode of object 20 are visible in the out-of-plane displacement at specific point A, but the frequency components of higher-order vibration modes cannot be detected because they are buried in extraneous vibrations (noise) generated in the rotating object 20.

[0082] On the other hand, as shown in Figure 15B, when noise is suppressed, it becomes possible to detect not only the natural frequency of the first mode of object 20, but also the natural frequencies of the second and third modes. In this way, when the difference calculation unit 2 uses a specific point D shown in Figure 8 as a reference, calculates the difference between the displacement of specific point D and the displacements of specific points A, B, and C respectively, and suppresses noise, it becomes possible to evaluate the vibration state of object 20 more precisely, as shown in Figure 15B.

[0083] The difference calculation unit 2 may also calculate the displacement within the surface of the object 20 by image correlation calculation based on the correlation or change of images between frames of the time-series image. The difference calculation unit 2 can calculate the amount of displacement at a level of 1 / 100 of the pixel pitch of the imaging device 101 by using quadratic curve interpolation in the image correlation calculation. Alternatively, the difference calculation unit 2 may calculate the amount of displacement using the gradient method instead of image correlation calculation.

[0084] Figures 16A and 16B illustrate the out-of-plane displacement when there is an abnormality in the blade 21 of object 20. Figure 16A schematically shows a part of the object whose condition is to be determined. Figure 16B shows the relationship between the out-of-plane displacement of a part of the object shown in Figure 16A and the vibration frequency. Note that Figure 16A shows a state in which vibration occurs in the direction normal to the blade 21.

[0085] Furthermore, in the following explanation, the specifications of the blade 21 of object 20 are assumed to be as follows. In Figure 16A, 20A indicates the axis of rotation of the blade 21. Length: 700mm Width: 150mm Thickness: 3mm Material: steel Rotational speed: 2000 rpm

[0086] Assuming no abnormalities occur in the blade 21 when it is rotated at the above rotational speed, the natural frequencies of the first, second, and third modes in the direction normal to the object surface will be 10 Hz, 50 Hz, and 150 Hz, respectively. In this case, the natural frequencies of each mode are pre-registered in a storage device or similar.

[0087] Let's assume that an abnormality, such as damage, has occurred in a portion of the blade 21 of object 20 (the area enclosed by the dashed line in Figure 16A). In this case, the displacement distribution of the out-of-plane displacement of the blade 21 will be as shown in Figure 16B. For the measurement of the displacement portion, the distance between the imaging device 101 and object 20 (imaging distance) is set to 5m, and the focal length of the lens of the imaging device 101 is set to 200mm. The pixel pitch of the image sensor is 4μm, and the image sensor has a resolution of 0.1mm per pixel.

[0088] In Figure 16B, the solid line indicates that an abnormality has occurred in blade 21. do not have The diagram shows the case where an abnormality occurs in blade 21. On the other hand, the dashed line indicates the case where an abnormality occurs in blade 21. As shown in Figure 16B, when an abnormality occurs, a decrease in the natural frequency of each mode is observed compared to when no abnormality occurs.

[0089] Therefore, in Embodiment 1, the abnormality determination unit 5 determines whether an abnormality has occurred in the object 20 based on the difference in the out-of-plane displacement distribution between when an abnormality has occurred in the object 20 and when no abnormality has occurred.

[0090] Next, we will explain the analysis of in-plane displacement. Figures 17A, 17B, 17C, and 17D show the in-plane displacement when there is a crack in the blade 21. Figure 17A schematically shows a stationary state of a part of the object in which a crack has occurred. Figure 17B schematically shows a state in which a part of the object shown in Figure 17A is undergoing rotational motion. Figure 17C is a parallel to Figure 1 7 This figure shows the time change in in-plane displacement of a portion of the object shown in A. Figure 17D is the same as Figure 1 7 This figure shows the relationship between the in-plane displacement and position of a part of the object shown in A.

[0091] As shown in Figure 17A, assume that a crack 29 has formed in the blade 21 of object 20. In Figure 17A, points A and B are arbitrary points located between them, with the crack 29 of the blade 21 sandwiched between them. In this case, when the propeller object 20 rotates, the crack 29 of the blade 21 opens and closes due to the rotational motion of the blade 21, as shown in Figure 17B, and points A and B are displaced by the rotational motion of the blade 21. The in-plane displacements at points A and B then change over time in accordance with the rotational motion, as shown in Figure 17C. In other words, the opening and closing of the crack 29 due to rotational motion changes the in-plane displacements of points A and B.

[0092] Here, we assume a straight line passing through points A and B on the blade 21. In this case, the in-plane displacement along the straight line is as shown in Figure 17D. In Figure 17D, the solid line shows the in-plane displacement when no crack 29 occurs in the blade 21. In this case, the in-plane displacement distribution is continuous. On the other hand, in Figure 17D, the dashed line shows the in-plane displacement when a crack 29 occurs in the blade 21. In this case, the in-plane displacement distribution changes abruptly between points A and B.

[0093] As can be seen from Figure 17D, the in-plane displacement distribution differs depending on whether or not a crack 29 is present in the blade 21. Therefore, in Embodiment 1, the abnormality determination unit 5 determines whether or not an abnormality has occurred in the object 20 based on the difference in the in-plane displacement distribution between the case where a crack 29 is present in the object 20 and the case where it is not.

[0094] As described above, the abnormality determination unit 5 determines an abnormality in an object by analyzing the out-of-plane displacement and in-plane displacement of the object at each time point. Furthermore, the abnormality determination unit 5 may determine an abnormality in an object by analyzing only one of the out-of-plane displacement or in-plane displacement, or by analyzing both.

[0095] [Device operation] Next, the operation of the state determination device 100 in Embodiment 1 will be described. Figure 18 is a flowchart showing the operation of the state determination device 100 in Embodiment 1. Other drawings will be referred to as appropriate in the following description. In Embodiment 1, the state determination method is performed by operating the state determination device 100. Therefore, the description of the state determination method in Embodiment 1 will be replaced by the following description of the operation of the state determination device 100.

[0096] As shown in Figure 18, first, the fixed image acquisition unit 1 acquires a time-series image obtained by the imaging device 101 imaging the surface of the object 20 without controlling the rotation of the optical device 102 (S1).

[0097] Next, the fixed image acquisition unit 1 uses the pair of the m-th (≧1)th image and the (m+1)th image included in the time-series image to detect the rotation direction and rotation speed of the blade 21 of the rotating object 20, as explained in Figures 6A and 6B (S2). Note that the target blade may be any other blade.

[0098] Next, the fixed image acquisition unit 1 controls the optical device 102 based on the detected rotation direction and rotation speed (S3). Specifically, the fixed image acquisition unit 1 controls the rotating optical systems 32 and 33 of the optical device 102 so that the image incident on the optical device 102 rotates in the opposite direction to the blade 21 of the object 20 and at the same speed as the rotation speed of the blade 21.

[0099] Then, the fixed image acquisition unit 1 acquires the time-series images captured by the imaging device 101 through the rotating dove prisms 35 and 36 as time-series fixed images (S4). The acquisition of fixed images in step S4 is performed, for example, for a set time or until a predetermined number of frames is reached.

[0100] Next, when the fixed time-series images are acquired in step S4, the difference calculation unit 2 sets specific points and uses the acquired fixed time-series images to calculate the amount of temporal displacement for each specific point (S5).

[0101] Next, as explained in Figure 9, the difference calculation unit 2 uses one of the specified points for which the displacement amount has been determined as a reference, and calculates the difference between the displacement amount of each of the other specified points and the displacement amount of the reference specified point (S6).

[0102] Next, the out-of-plane displacement calculation unit 3 estimates the depth movement Δz using the difference in the displacement amounts calculated in step S6, and calculates the out-of-plane displacement (contribution in the z direction) from the depth movement Δz (S7).

[0103] Next, the in-plane displacement calculation unit 4 calculates the in-plane displacement of the object (S8) using the difference calculated in step S6 and the out-of-plane displacement calculated in step S7, as explained in Figure 14A.

[0104] After step S8 is executed, the fixed image acquisition unit 1 determines whether the processing in steps S5 to S8 has been completed for all fixed images acquired in S4 (S9). If the determination in step S9 indicates that the processing in steps S5 to S8 has not been completed for all fixed images (S9: No), step S5 is executed again for the fixed images for which processing has not been completed.

[0105] On the other hand, if the result of the determination in step S9 indicates that the processing in steps S5 to S8 has been completed for all fixed images (S9: Yes), then step S10 is executed.

[0106] In step S10, the abnormality determination unit 5 analyzes either or both of the out-of-plane displacement distribution and the in-plane displacement distribution of the blade 21 of the object 20 (S10). Then, based on the analysis results in step S10, the abnormality determination unit 5 determines whether or not an abnormality has occurred in the object 20 (S11).

[0107] After step S11 described above, the abnormality determination unit 5 may display the diagnostic result of the state of object 20 on a display device, or transmit it to a user's terminal device connected via a network or the like. This allows the user to look at the outputted diagnostic result and, for example, decide whether or not to repair or conduct a detailed investigation of object 20.

[0108] (Effects of Embodiment 1) According to the configuration of Embodiment 1, a single imaging device 101 can image a rotating object and determine abnormalities in the object, thus suppressing the need for a larger device. Furthermore, in Embodiment 1, the optical device 102 enables the acquisition of a fixed image with suppressed motion blur (motion blur) when photographing the object. Moreover, the effects of vibrations caused by rotation are also suppressed in the fixed image. Therefore, in Embodiment 1, sufficient accuracy is guaranteed for measuring in-plane and out-of-plane displacements using images, enabling accurate detection of abnormalities in rotating objects.

[0109] In addition, in Embodiment 1, the optical device 102 can be equipped with multiple dove prisms, allowing the dove prisms to rotate at high speed with less driving force. This enables appropriate suppression of motion blur according to the rotation speed of the object being subjected to state determination. Furthermore, in Embodiment 1, only one imaging device is used, which reduces the cost of imaging the object.

[0110] [program] The program in Embodiment 1 is any program that causes a computer to execute steps S1 to S11 shown in Figure 18. The computer is composed of a combination of hardware such as a CPU (for example, a multi-processor with multiple processor cores), a GPU (Graphics Processing Units), a DSP (Digital Signal Processors), an FPGA (Field-Programmable Gate Arrays), and an ASIC (Application Specific Integrated Circuit). By installing and executing this program on the computer, the state determination device and state determination method in Embodiment 1 can be realized. In this case, the computer's processor functions as a fixed image acquisition unit 1, a difference calculation unit 2, an out-of-plane displacement calculation unit 3, an in-plane displacement calculation unit 4, and an abnormality determination unit 5, and performs processing.

[0111] In addition to general-purpose PCs, other types of computers include smartphones and tablet devices.

[0112] Furthermore, the program in Embodiment 1 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a fixed image acquisition unit 1, a difference calculation unit 2, an out-of-plane displacement calculation unit 3, an in-plane displacement calculation unit 4, and an anomaly determination unit 5.

[0113] (Embodiment 2) Next, Embodiment 2 will be described below. In Embodiment 2, the state determination device is configured to track a rotating object within the field of view of the imaging device when the imaging device is mounted on a moving object such as a drone. It may also be applied when the rotating object is translating and the imaging device is fixed. Embodiment 2 differs from Embodiment 1 in this respect. The state determination device in Embodiment 2 includes, in addition to the optical device (first optical device) 102 described in Embodiment 1, an optical device (second optical device) for acquiring a fixed image that cancels out the translational movement of the object. 103 It has.

[0114] Figure 19 is a block diagram showing the configuration of the state determination device in Embodiment 2. Figure 20 shows the optical device 10 3 This is a diagram showing the configuration. Figure 21 is a schematic diagram showing a translationally moving object 20. In Figure 21, a time-series image of object 20 is shown. Big break The lines indicate images of object 20 in a specific frame. Thin The dashed lines indicate the image of the object in the next frame.

[0115] As shown in Figure 19, in Embodiment 2, the optical device 103 is positioned between the object 20 and the optical device 102. The optical device 103 comprises a first mirror 103a and a second mirror 103b. The first mirror 103a reflects the image of the object 20 and causes the reflected light to enter the second mirror 103b. The second mirror 103b further reflects the incident light and causes it to enter the optical device 102.

[0116] Furthermore, the optical device 103 performs so-called pan-tilt control (pan direction = x direction, tilt direction = y direction) to follow a rotating object by changing the angle of the first mirror 103a around the rotation axis 104a and the second mirror 103b around the rotation axis 104b which is perpendicular to the rotation axis 104a. For example, a servo motor can be used as the angle control mechanism for the first mirror 103a and the second mirror 103b.

[0117] Furthermore, the two axial directions in which the drive mechanism can move are perpendicular to the optical axis of the imaging device in the example shown in Figure 19. Therefore, the first mirror 103a can move in accordance with the movement of the object 20 as the object is translated by the drive mechanism.

[0118] In addition to the processing described in Embodiment 1, the fixed image acquisition unit 1 also controls the optical device 103 in Embodiment 2. Specifically, when the fixed image acquisition unit 1 acquires a time-series image (or fixed image) from the imaging device 101, it compares the portion of the object 20 near its rotation center (for example, the spinner 22) between frames to detect the translational direction and amount of movement of the object 20. Then, the fixed image acquisition unit 1 drives servo motors that control the angles of the first mirror 103a and the second mirror 103b in accordance with the detected translational direction and amount of movement.

[0119] As a result, in the image data output from the imaging device 101, the image of the object 20's rotation center and its vicinity will always be located at the same coordinates. Therefore, even if the object 20 moves beyond the field of view of the imaging device 101, this movement can be tracked. Accordingly, for example, the imaging device 101 can use a lens with a small field of view but high resolution, such as a telephoto lens, and according to Embodiment 2, high-precision displacement measurement over a wide area becomes possible.

[0120] In Embodiment 2, the optical device 103 is not limited to the configuration described above. Any device capable of capturing an image of an object by following its translation is acceptable. The optical device 103 may also have a function to optically track the object 20.

[0121] In Embodiment 2, the state determination device 100 also executes steps S1 to S11 shown in Figure 18. The state determination method in Embodiment 2 is also executed by operating the state determination device 100. Furthermore, the program in Embodiment 2 can be any program that causes the computer to execute steps S1 to S11 shown in Figure 18.

[0122] However, in Embodiment 2, in steps S1 and S4, the fixed image acquisition unit 1 also performs a process to make it track the optical device 103. Embodiment 2 differs from Embodiment 1 only in this respect.

[0123] (Effects of Embodiment 2) In Embodiment 2, as in Embodiment 1, the state determination device 100 can suppress vibrations caused by rotation and calculate the out-of-plane and in-plane displacements of the object itself, enabling accurate detection of abnormalities in the rotating object. Furthermore, as in Embodiment 1, since the rotating object is imaged by a single imaging device 101 to determine the abnormality of the object, the size of the device can be kept down.

[0124] Furthermore, in Embodiment 2, since the optical device 103 is used, even if an object is moving within a range greater than the field of view of the imaging surface of the imaging device 101, its image can be captured. Therefore, even when a telephoto lens with a narrow field of view is attached, it is possible to accurately determine abnormalities in the object 20.

[0125] [Physical configuration] Here, a computer that implements a state determination device by executing the programs in Embodiments 1 and 2 will be described using Figure 22. Figure 22 is a block diagram showing an example of a computer that implements a state determination device in Embodiments 1 and 2.

[0126] As shown in Figure 22, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0127] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0128] The CPU 111 loads the code set constituting the program in this embodiment, stored in the storage device 113, into the main memory 112, and performs various calculations by executing the code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0129] Furthermore, the programs in the embodiments are provided stored on a computer-readable recording medium 120. The programs in embodiments 1 and 2 may also be distributed over the internet connected via the communication interface 117.

[0130] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0131] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0132] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0133] Furthermore, the state determination device in Embodiments 1 and 2 can be implemented not by a computer on which the program is installed, but by using hardware corresponding to each part. Moreover, the state determination device may be partially implemented by a program and the remaining part by hardware.

[0134] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following descriptions.

[0135] (Note 1) A fixed image acquisition unit acquires each of the time-series images obtained by continuously photographing a rotating object with an imaging device via a first optical device that rotates and emits incident light, as a fixed image. A difference calculation unit calculates the displacement of multiple specific points of the object using each of the acquired fixed images, and using one of the multiple specific points as a reference specific point, calculates the difference between the displacement of the reference specific point and the displacement of the other specific points. An out-of-plane displacement calculation unit calculates the displacement of the object's surface on the imaging surface as it moves in the direction normal to the imaging surface of the imaging device, using the difference described above, An in-plane displacement calculation unit calculates the displacement of the object's surface on the imaging surface as an in-plane displacement resulting from the object's surface moving in a direction along the surface, using the difference and the out-of-plane displacement. An abnormality determination unit that identifies an abnormality in the object based on the distribution of the in-plane displacement or the temporal change in the out-of-plane displacement, A state determination device having the following features.

[0136] (Note 2) The state determination device described in Appendix 1, The fixed image acquisition unit detects the rotation direction and rotation speed of the object, and acquires the fixed image by controlling the rotation direction and rotation speed of the incident light in the first optical device based on the detected rotation direction and rotation speed. State determination device.

[0137] (Note 3) The state determination device described in Appendix 2, The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently around the optical axis of the incident light, and is further positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. The fixed image acquisition unit rotates each of the two or more prism elements in opposite directions to each other, and controls the rotation direction and rotation speed of the two or more prism elements so that the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the rotation speed of the object. State determination device.

[0138] (Note 4) A state determination device described in any one of the appendices 1 to 3, A second optical device is positioned between the object and the first optical device, which follows the translation of the object, captures an image of the object, and directs the captured image into the first optical device. The fixed image acquisition unit detects the translational direction and the amount of translational movement of the object, and controls the second optical device based on the detected translational direction and translational movement amount. State determination device.

[0139] (Note 5) A state determination device described in any one of the appendices 1 to 4, The out-of-plane displacement calculation unit uses the difference to determine the amount of depth movement of the surface of the object that has moved along the normal direction, and uses the amount of depth movement to calculate the out-of-plane displacement. State determination device.

[0140] (Note 6) The process involves continuously capturing images of a rotating object using an imaging device via a first optical device that rotates and emits incident light, and acquiring each of these time-series images as a fixed image. Using each of the acquired fixed images, the displacement amounts of multiple specific points of the object are calculated, and one of the multiple specific points is designated as a reference point. The difference between the displacement amount of the reference point and the displacement amounts of the other specific points is then calculated. Using the difference, the step of calculating the displacement of the object's surface on the imaging surface as an out-of-plane displacement due to the object moving in the direction normal to the imaging surface of the imaging device, Using the difference and the out-of-plane displacement, the step of calculating the displacement of the object's surface on the imaging surface as an in-plane displacement due to the object's surface moving in a direction along the surface, A step of identifying an abnormality in the object based on the distribution of the in-plane displacement or the temporal change in the out-of-plane displacement, A method for determining the "tight" state.

[0141] (Note 7) The state determination method described in Appendix 6, In the step of acquiring the fixed image, the rotation direction and rotation speed of the object are detected, and the fixed image is acquired by controlling the rotation direction and rotation speed of the incident light in the first optical device based on the detected rotation direction and rotation speed. A method for determining the state.

[0142] (Note 8) The state determination method described in Appendix 7, The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently around the optical axis of the incident light, and is further positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. In the step of acquiring the fixed image, each of the two or more prism elements is rotated in opposite directions to each other, and the rotation direction and rotation speed of the two or more prism elements are controlled so that the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the rotation speed of the object. A method for determining the state.

[0143] (Note 9) A method for determining the state described in any one of the appendices 6 to 8, A second optical device is positioned between the object and the first optical device, which follows the translation of the object, captures an image of the object, and directs the captured image into the first optical device. In the step of acquiring the image as a fixed image, the translational direction and amount of movement of the object during translation are detected, and the second optical device is controlled based on the detected translational direction and amount of translational movement. A method for determining the state.

[0144] (Note 10) A method for determining the state described in any one of the appendices 6 to 9, In the step of calculating the out-of-plane displacement, the difference is used to determine the amount of depth movement of the surface of the object that has moved along the normal direction, and the amount of depth movement is used to calculate the out-of-plane displacement. A method for determining the state.

[0145] (Note 11) On the computer, The process involves continuously capturing images of a rotating object using an imaging device via a first optical device that rotates and emits incident light, and acquiring each of these time-series images as a fixed image. Using each of the acquired fixed images, the displacement amounts of multiple specific points of the object are calculated, and one of the multiple specific points is designated as a reference point. The difference between the displacement amount of the reference point and the displacement amounts of the other specific points is then calculated. Using the difference, the step of calculating the displacement of the object's surface on the imaging surface as an out-of-plane displacement due to the object moving in the direction normal to the imaging surface of the imaging device, Using the difference and the out-of-plane displacement, the step of calculating the displacement of the object's surface on the imaging surface as an in-plane displacement due to the object's surface moving in a direction along the surface, A step of identifying an abnormality in the object based on the distribution of the in-plane displacement or the temporal change in the out-of-plane displacement, The program to execute Hmm.

[0146] (Note 12) As described in Appendix 11 program And, In the step of acquiring the fixed image, the rotation direction and rotation speed of the object are detected, and the fixed image is acquired by controlling the rotation direction and rotation speed of the incident light in the first optical device based on the detected rotation direction and rotation speed. program .

[0147] (Note 13) As described in Appendix 12 program And, The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently around the optical axis of the incident light, and is further positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. In the step of acquiring the fixed image, each of the two or more prism elements is rotated in opposite directions to each other, and the rotation direction and rotation speed of the two or more prism elements are controlled so that the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the rotation speed of the object. program .

[0148] (Note 14) As described in any one of the appendices 11 to 13. program And, A second optical device is positioned between the object and the first optical device, which follows the translation of the object, captures an image of the object, and directs the captured image into the first optical device. In the step of acquiring the image as a fixed image, the translational direction and amount of movement of the object during translation are detected, and the second optical device is controlled based on the detected translational direction and amount of translational movement. program .

[0149] (Note 15) The information described in any one of the appendices 11 through 14. program And, In the step of calculating the out-of-plane displacement, the difference is used to determine the amount of depth movement of the surface of the object that has moved along the normal direction, and the amount of depth movement is used to calculate the out-of-plane displacement. program . [Industrial applicability]

[0150] This invention is useful for determining the state of a rotating object to be measured, such as a wind turbine propeller, a turbine blade, or the wheels of a moving vehicle. [Explanation of symbols]

[0151] 1: Fixed image acquisition unit 2: Difference calculation part 3: Out-of-plane displacement calculation unit 4: In-plane displacement calculation unit 5: Abnormality determination section 20:Object 20A: Rotating shaft 21: Blade 22: Spina 25, 26: Surface 29: Crack 23, 24: Selection area 32, 33: Rotating Optical Systems 34: Optical axis 35, 36 Double Prism 37, 38: Rotary drive unit 100: State determination device 101: Imaging device 102, 103: Optical equipment 110: Computer 111: CPU 112: Main memory 113: Storage device 114: Input Interface 115: Display Controller 116: Data Reader / Writer 117: Communication Interface 118: Input devices 119: Display device 120: Recording media 121: Bus

Claims

1. A fixed image acquisition unit that acquires each of the time-series images obtained by continuously photographing a rotating object with an imaging device via a first optical device that rotates and emits incident light, as a fixed image, A difference calculation unit calculates the displacement of multiple specific points of the object using each of the acquired fixed images, and using one of the multiple specific points as a reference specific point, calculates the difference between the displacement of the reference specific point and the displacement of the other specific points. An out-of-plane displacement calculation unit calculates the displacement of the object's surface on the imaging surface as it moves in the direction normal to the imaging surface of the imaging device, using the difference described above, An in-plane displacement calculation unit calculates the displacement of the object's surface on the imaging surface as an in-plane displacement resulting from the object's surface moving in a direction along the surface, using the difference and the out-of-plane displacement. An abnormality determination unit that identifies an abnormality in the object based on either or both of the in-plane displacement distribution and the out-of-plane displacement distribution, It has, The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently and around the optical axis of the incident light with respect to an axis that penetrates each of the two inclined surfaces, and furthermore, it is positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. The aforementioned fixed image acquisition unit, With the two or more prism elements not rotating, the image layer values ​​are used to perform an image correlation calculation on the captured time-series images to determine the angle at which the correlation value is maximized. From the determined angle and the frame rate of the time-series images, the rotation speed and direction of the object are determined, and then, The rotation direction and rotation speed of the two or more prism elements are controlled such that adjacent prism elements rotate in opposite directions to each other, and the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the object. State determination device.

2. A state determination device according to claim 1, The fixed image acquisition unit detects the rotation direction and rotation speed of the object, and acquires the fixed image by controlling the rotation direction and rotation speed of the incident light in the first optical device based on the detected rotation direction and rotation speed. State determination device.

3. A state determination device according to claim 1, A second optical device is positioned between the object and the first optical device, which follows the translation of the object, captures an image of the object, and directs the captured image into the first optical device. The fixed image acquisition unit detects the translational direction and the amount of translational movement of the object, and controls the second optical device based on the detected translational direction and amount of translational movement. State determination device.

4. A state determination device according to claim 1, The out-of-plane displacement calculation unit uses the difference to determine the amount of depth movement of the surface of the object that has moved along the normal direction, and uses the amount of depth movement to calculate the out-of-plane displacement. State determination device.

5. A rotating object is continuously photographed by an imaging device via a first optical device that rotates and emits incident light, and each of the time-series images obtained is acquired as a fixed image. Using each of the acquired fixed images, the displacement amounts of multiple specific points of the object are calculated, and one of the multiple specific points is designated as a reference point. The difference between the displacement amount of the reference point and the displacement amounts of the other specific points is then calculated. Using the aforementioned difference, the displacement of the object's surface on the imaging surface as the object moves in the direction normal to the imaging surface of the imaging device is calculated as the out-of-plane displacement. Using the difference and the out-of-plane displacement, the displacement of the object's surface on the imaging surface due to the object's surface moving in a direction along the surface is calculated as the in-plane displacement. Based on either or both of the in-plane displacement distribution and the out-of-plane displacement distribution, an anomaly in the object is identified. The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently and around the optical axis of the incident light with respect to an axis that penetrates each of the two inclined surfaces, and furthermore, it is positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. In acquiring the aforementioned fixed image, With the two or more prism elements not rotating, the image layer values ​​are used to perform an image correlation calculation on the captured time-series images to determine the angle at which the correlation value is maximized. From the determined angle and the frame rate of the time-series images, the rotation speed and direction of the object are determined, and then, The rotation direction and rotation speed of the two or more prism elements are controlled such that adjacent prism elements rotate in opposite directions to each other, and the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the object. A method for determining the state.

6. On the computer, By continuously photographing a rotating object with an imaging device via a first optical device that rotates and emits incident light, each of the time-series images obtained is acquired as a fixed image. Using each of the acquired fixed images, the displacement amounts of multiple specific points of the object are calculated, and one of the multiple specific points is designated as a reference point, and the difference between the displacement amount of the reference point and the displacement amounts of the other specific points is calculated. Using the aforementioned difference, the displacement of the object's surface on the imaging surface of the imaging device as the object moves in the direction normal to the imaging surface of the imaging device is calculated as the out-of-plane displacement. Using the difference and the out-of-plane displacement, the displacement of the object's surface on the imaging surface due to the object's surface moving in a direction along the surface is calculated as the in-plane displacement. Based on either or both of the in-plane displacement distribution and the out-of-plane displacement distribution, an abnormality in the object is identified. The first optical device has two or more prism elements, and each of the two or more prism elements is configured to rotate independently and around the optical axis of the incident light with respect to an axis that penetrates each of the two inclined surfaces, and furthermore, it is positioned so that the optical axis of the incident light coincides with the normal to the imaging plane of the imaging device. In acquiring the aforementioned fixed image, With the two or more prism elements not rotating, the image layer values ​​are used to perform an image correlation calculation on the captured time-series images to determine the angle at which the correlation value is maximized. From the determined angle and the frame rate of the time-series images, the rotation speed and direction of the object are determined, and then, The rotation direction and rotation speed of the two or more prism elements are controlled such that adjacent prism elements rotate in opposite directions to each other, and the image captured by the imaging device through the two or more prism elements rotates in the opposite direction to the rotation direction of the object and at the same rotation speed as the object. program.

Images