Monitoring of a mechanism or its components

Abstract

A system for detecting the state of a mechanism includes a processing circuit. The mechanism includes a plurality of components at least some of which are movable relative to each other. At least one state is specified for the mechanism based on the alignment of the mechanism components. The processing circuit identifies reference points in at least two of the components within the image data of the mechanism. Based on the respective positions of the reference points, it determines whether the mechanism components are in one of the specified alignments (i.e., states) and outputs an indicator.

Description

Technical Field

[0001] The present disclosure generally relates to the monitoring of the status and functionality of a mechanism and / or its components and its maintenance based on prediction.

Background Art

[0002] Machine maintenance can include any work on a machine and / or its components that maintains the operation of the mechanical asset with minimal downtime. Machine maintenance can include both regularly scheduled services, daily inspections, and both scheduled and emergency repairs. Maintenance can also include the replacement or realignment of parts that have worn, been damaged, or are misaligned. Machine maintenance can be performed either before or after a failure occurs. Machine maintenance is important in any factory or facility that uses mechanical assets. It helps organizations meet production schedules, minimize downtime costs, and reduce the risk of accidents and injuries in the workplace.

[0003] Currently, industrial maintenance functions automatically over a certain period of time (scheduled maintenance) based on statistical data and / or historical data, a specific usage level (e.g., mileage or engine hours), or when a machine, part, or component fails (breakdown maintenance). This type of maintenance is wasteful and inefficient.

[0004] The landing gear is a complex subsystem of an aircraft (e.g., an unmanned aerial vehicle (UAV)) or a spacecraft (manned or unmanned), is positioned on the landing gear, and is used for takeoff and landing. The landing gear is extremely vulnerable to damage due to complex moving parts and factors such as corrosion, cracking, wear, tire problems, and others.

[0005] The landing gear undergoes routine maintenance, but failures of the landing gear in aircraft account for approximately 10% of all reported failures in the aerospace industry.

[0006] Additional background art includes Edgar A. Ossa et al. “Handbook of Materials Failure Analysis with Case Studies from the Aerospace and Automotive Industries”, 2016, Pages 167-190, Chapter 8 - Suspension and landing gear failures. SUMMARY OF THE INVENTION

[0007] According to some embodiments, there are provided a system, a method, and a computer program product for detecting the state of a component of a mechanical mechanism. According to some embodiments, there are provided a system, a method, and a computer program product for monitoring the health state and / or function and / or likelihood of failure of a mechanical mechanism and / or its components. According to some embodiments, there are provided a system and a method for monitoring the health state and / or function and / or likelihood of failure of a mechanism or its components. According to some embodiments, there are provided a system and a method for predictive maintenance of a mechanism.

[0008] The analysis of mechanisms that include a large number of joints (such as hinges, sliders, linear actuators, and rotary actuators, etc.) and components that undergo complex motion, such as landing gear, is very difficult.

[0009] Detection of the state of a mechanism and monitoring of the health state of a mechanism According to some embodiments of the present invention, there are provided a system and a method for detecting the state of a mechanism that includes a plurality of components. At least two of the components are movable relative to each other. The mechanism has at least one state, and each state is defined by a respective specified alignment of the mechanism components.

[0010] According to some embodiments, the system detects whether the mechanism is in a defined state by reference points identifiable in an image taken from a component. Optionally, each component whose relative position is required for detecting the mechanism state has at least one reference point. The system and method determine whether the reference points are arranged relative to each other to conform to a defined state and output an indicator of the state of the mechanism. If the alignment does not conform to the defined state of the reference points, the indicator indicates that the mechanism is not in the defined state.

[0011] The image is collected by one or more optical sensors, such as cameras. The optical sensor is positioned at a location having a field of view sufficient to detect whether the mechanism components are aligned according to a defined state and / or acquire images of all reference points necessary to evaluate the health state of the mechanism. Optionally, a single optical sensor has a field of view that includes a plurality of desired reference points. Alternatively or additionally, the desired reference points are imaged by a plurality of optical sensors, where each respective field of view of the optical sensors may include one or more of the reference points.

[0012] In some embodiments, multiple images including the same reference point are taken over time. Thus, the movement of the component can be tracked by following the movement of the reference point within the array of images. The array of images can be, for example, an array of videos or still images.

[0013] Some or all of the optical sensors may be part of the system, or none of them may be part of the system.

[0014] Optionally, the system includes one or more of the optical sensors.

[0015] The alignment of the reference points to match the defined state can be specified by any means known in the art. Examples include, but are not limited to, table look-up, arithmetic calculation of distance vectors, algorithms, and machine learning modeling of mechanism components based on reference points.

[0016] In some embodiments, an indicator of the state of the mechanism provides additional information regarding the state of the mechanism. For example, if the mechanism has multiple defined states, the indicator may include information regarding which defined state it is in. In another example, if the mechanism is not in a defined state, the indicator may include information regarding how far the reference points are misaligned from the alignment, which information may be classified as an intermediate state of the mechanism.

[0017] Many mechanisms include a locking component, such as a latch or a spring or any other component designed to prevent the mechanism from deviating from its state. Such components ensure that the mechanism remains locked until an action such as releasing the latch is taken, or that the mechanism is disposed to be movable. Optionally, the system and method detect whether the mechanism is in a locked state, which means that the mechanism is in a defined state and the locking mechanism is operative. In a locked state, at least a portion of the components is substantially static (although slight movement such as vibration may occur). Optionally, the mechanism has multiple locked states.

[0018] Optionally, detecting whether the mechanism is in a locked state includes determining whether the locking mechanism is operative from image data.

[0019] A mechanism is considered stateless if the components are not aligned in a manner defined as a state. Note that if the mechanism is not locked, the mechanism components may be moving or may be stationary.

[0020] For example, assume a gated arm defined to be in a locked state in both horizontal and vertical cases. When the arm is horizontal, the path is closed, and when the arm is vertical, the path is open. However, when the arm is moving between the open and closed positions or is sandwiched between the open and closed positions, the gate is considered to be in an unlocked state.

[0021] According to alternative or additional embodiments of the present invention, the image data and identifiable reference points within the image data can be used for monitoring the health state of the mechanism. During operation, the position and / or velocity and / or acceleration of the reference points in the array of images can be tracked for a plurality of components. The relative motion of the reference points is processed by a model of proper operation of the mechanism. When the relative motion conforms to the model, the mechanism operates properly within an acceptable error. Relative motion of the reference points that does not conform to the requirements of the model may be an indication of a failure and / or excessive wear of the mechanism and / or the need for maintenance.

[0022] The images can also be analyzed to obtain additional information for monitoring the health state of the mechanism.

[0023] One parameter that can provide information for trend analysis of the health state of the mechanism is the time taken for a component to move from one position to a second position. If the time increases, there may be an increase in friction, for example due to rust or corrosion. Alternatively, if the time decreases, there may be an increase in slipperiness due to thinning or warping of the component over time.

[0024] Another example of a parameter that can assist in determining the health state of the mechanism is the relative timing of the movement of the mechanism components. If the synchronization of the movement of different mechanism components is lost, maintenance or replacement may be required.

[0025] In some embodiments, the image is analyzed to determine the health state of a single component of the mechanism, such as loosening or rotation of a screw, formation of rust or corrosion, tension of an elastic element, leakage, cracks and / or other wear of the component. Optionally, the indicator includes information about defects and / or potential malfunctions in a single component.

[0026] Optionally, the health state of the mechanism is evaluated by co - analyzing the relative movement of a plurality of mechanism components and additional information of a single component. For example, the analysis checks the health state of components such as screws and hinges when a change in the time taken for the mechanism to transition from one state is detected.

[0027] Information regarding the health state of the mechanism and / or components of the mechanism is extremely important in fields such as prognostic health management (PHM), condition - based maintenance (CBM), and health & usage monitoring systems (HUMS).

[0028] In some embodiments, the mechanism is a subsystem of a vehicle. The optical sensor can be installed in the vehicle. Thus, the relative position between the optical sensor and the mechanism component does not change significantly during the movement of the vehicle.

[0029] In some embodiments, the optical sensor and its corresponding components (such as a printed circuit board and a light source) are very small so that the size, weight, and volume of the optical sensor installed in the vehicle do not interfere with the operation of the vehicle.

[0030] The optical sensors are of the same size or different sizes. Optionally, the diameter of the optical sensor is in the range of 0.5 - 10 mm, such as in the range of 0.65 - 3 mm or 0.65 - 5 mm.

[0031] Optionally, the optical sensor and the corresponding PCB and / or light source are shielded to prevent interference from the operation of the vehicle or the mechanism to or from the operation of the optical sensor.

[0032] In a further embodiment, the subsystem is configured to support the vehicle body between at least any of the specified states.

[0033] In one example, the mechanism is a landing gear of an aircraft or a spacecraft. The landing gear supports the aircraft fuselage while the aircraft is on the ground.

[0034] In a second example, the subsystem is a suspension system of a motor vehicle such as a passenger car or a truck. The suspension system includes elements such as tires, shock absorbers, linkages, axles, and other components that provide support between the vehicle chassis and the ground.

[0035] As used herein, according to some embodiments of the present invention, the terms "mechanical mechanism" and "mechanism" mean a system of objects that operate together to perform a function. Optionally, the mechanism is part of a machine.

[0036] As used herein, according to some embodiments of the present invention, the terms "component of a mechanism" and "component" mean one of the objects that form the mechanism.

[0037] As used herein, according to some embodiments of the present invention, the term "image data" means data based on an image captured by an optical sensor. The image data may include the image itself and / or data obtained by image processing (e.g., formatting the image into a data format suitable for a processing circuit, adjusting the contrast and / or brightness of the image, compensating for vibrations of the optical sensor, etc.).

[0038] According to some embodiments of the present invention, the image data includes an image of a mechanism component. Optionally, the image data includes additional images (e.g., images of other mechanisms such as static or passive components, images of the environment in which the mechanism operates, images of the machine of which the mechanism is a part, etc.).

[0039] As used herein, according to some embodiments of the present invention, the term "subsystem" means a self - contained system within a machine. Optionally, the self - contained system is operated by the operating system of the machine.

[0040] As used herein, according to some embodiments of the present invention, the term "state" means a specified alignment of a mechanical component.

[0041] As used herein, according to some embodiments of the present invention, the term "locked state" means that the mechanism is in one of the specified alignments and the locking mechanism is operating.

[0042] As used herein, according to some embodiments of the present invention, the term "unlock" means that the mechanical component is not in a locked state.

[0043] As used herein, according to some embodiments of the present invention, the term "optical sensor" means a device that detects an optical signal and outputs image data.

[0044] As used herein, according to some embodiments of the present invention, the term "reference point" means an element on a component that has a known position and / or orientation in the component that can be detected by image processing. Optionally, the position and / or orientation is relative to other components of the machine.

[0045] According to some embodiments, the term "reference point" is not limited to a particular size or shape of the reference point. For example, the reference point can be a circle, a line, an arrow, a rectangle, or any other shape. Note that the reference point can indicate a direction in addition to its position (e.g., an arrow).

[0046] As used herein, according to some embodiments of the present invention, the terms "relative motion" and "relatively movable components" refer to motion within a reference frame established by the mechanism position and orientation (or the position and orientation of specific mechanism components). Thus, a component of a machine can be spatially stationary, while other movable components in the mechanism may be in relative motion with respect to the stationary component.

[0047] Typically, a mechanism converts an input force and the motion of some mechanism components into an output force and the motion of other mechanism components. Mechanisms have various applications and are used in almost all machines. Non-limiting examples of mechanisms are 1) Actuators, 2) Linkages, 3) Brakes, 4) Gear trains, 5) Shafts, 6) Joints, 7) Cam and follower systems (e.g., belt and chain drives), 8) Cables, 9) Clutches, 10) Springs, and include.

[0048] One skilled in the art will be familiar with a number of types of mechanisms and the functions they perform.

[0049] Examples of reference points are 1) Markings (e.g., stickers or paint) attached to components, 2) Physical elements such as screws or joints of components, 3) Existing marks such as defects, natural lines or boundary lines of components, and may include, but are not limited to, these.

[0050] The effects of the present invention may include, but are not limited to, the following.

[0051] 1) During operation, the state of the mechanism, such as whether the mechanism is in a locked state, can be easily detected.

[0052] 2) Real-time indicators and / or alerts regarding the state and health of the mechanism can be provided.

[0053] 3) A mechanism that functions properly can be easily modeled based on the images of the mechanism components collected by an optical sensor. The model can be adjusted according to the specific mechanism based on the images of the components of the specific mechanism. The model can further be based on other types of data such as general specifications and knowledge about the operating conditions of the mechanism.

[0054] 4) The model can be retrained periodically based on images collected during the actual operation of general mechanisms and / or specific mechanisms.

[0055] 5) The health state of the mechanism can be easily evaluated from the image data and optionally provided to a PHM, CBM, HUMS system. For example, indications of the health state of the components can be provided, which are not limited to loosening or rotation of screws or bolts of the components, formation of rust or corrosion, tension of elastic elements, leakage, lubrication, wear, etc.

[0056] Some or all of the above can be achieved by the image data provided by an optical sensor arranged to collect images of reference points in the mechanism components. Optionally, the results of the analysis of the image data are verified and / or refined by data from other types of sensors, such as acoustic sensors, vibration sensors or temperature sensors. For example, a change in the movement of a mechanism component accompanied by a large noise detected by an acoustic sensor may indicate a serious malfunction, but the smaller the noise, the less serious the malfunction.

[0057] Monitoring the integrity and / or function of the mechanism According to some embodiments of the present invention, a system for monitoring the integrity and / or function of a mechanism is provided.

[0058] In some embodiments, the mechanism is a vehicle subsystem. In further embodiments, the subsystem is configured to support the vehicle body at least during a locked state. In still other embodiments, the mechanism is a landing gear of an aircraft or a spacecraft.

[0059] According to some embodiments, a system for monitoring the health of a mechanism includes at least one optical sensor, such as a camera, configured to be fixed to the mechanism or its component, in its vicinity, or in front of it, and at least one processor in communication with the at least one optical sensor. The system may be configured to provide an indication of the integrity and / or proper functioning of the mechanism or any of its components. Alternatively or additionally, the system may be configured to provide an indication of a potential failure in the mechanism or its component or its function. The system may further include a light source that provides illumination for the mechanism. According to some embodiments, the system is operable as a stand-alone system for monitoring the integrity and / or function of the mechanism. According to additional or alternative embodiments, the system may be operable as an additional or backup system for monitoring the integrity and / or function of the mechanism when other sensors (e.g., currently used pressure sensors) are malfunctioning.

[0060] According to some embodiments, the systems disclosed herein may further include other sensors, such as pressure sensors, temperature sensors, acoustic sensors, speed sensors, etc., in addition to one or more optical sensors (e.g., cameras). The combination of sensors may provide a more accurate indication regarding the mechanism, such as an indication of full lock.

[0061] According to some embodiments, the systems disclosed herein may further include a plurality of sensors (e.g., a plurality of cameras, or a combination of one or more cameras and other sensors), whereby each of the plurality of sensors is adapted to acquire data from each associated area, such as joints / movable components / elements of movable components / markings (e.g., lines).

[0062] According to some embodiments, one or more optical sensors (e.g., cameras) of a system for monitoring the mechanisms disclosed herein may have a field of view sufficient to capture all relevant mechanism components (e.g., all moving elements).

[0063] According to some embodiments, an operator / technician may be able to select target areas of a mechanism (e.g., joints / moving elements / lines, etc.) in order to conserve computer resources and monitor only these defined segments.

[0064] According to some embodiments, a processor may be executable to monitor the trajectory and / or speed of movement of a reference point (e.g., a predetermined reference point) in a mechanism or its components, at least based on signals received from at least one optical sensor. The integrity and functionality of the mechanism or its components may be determined by the processor by comparing each monitored trajectory and / or speed of the reference points to each pre-acquired (or pre-calculated) trajectory curve and / or speed of such reference points in a fully functional mechanism. For example, according to some embodiments, if one (or more) of the reference points deviates from a predetermined trajectory curve, the mechanism or its components may be determined to be malfunctioning. Additionally or alternatively, maintenance may be recommended. Additionally or alternatively, the time to failure may be calculated and indicated.

[0065] According to some embodiments, the integrity and functionality of a mechanism or its components may be determined in real time. According to some embodiments, the integrity and functionality of a mechanism or its components may be determined in real time from a remote location.

[0066] According to some embodiments, a method for determining the integrity and proper functionality of a complex mechanism, such as a landing gear or its components, utilizes at least one processor Determine one or more reference points in the moving parts of the mechanism or its components, For each reference point, assign an accurate / normal trajectory and / or an accurate / normal speed, During the operation of a complex mechanism such as a mechanism or its components, monitor the trajectory and / or speed of each of the reference points, Determine whether each of the monitored reference points moves along its respective assigned trajectory curve and / or moves at the assigned speed (or is within the margin / range defined by the assigned trajectory curve and / or speed of each reference point), and, If the trajectory and / or speed of at least one of the reference points deviates from its respective assigned (accurate / normal) trajectory curve and / or speed or its (predetermined) margin / range, Provide an indication of a suspected malfunction / failure / damage / defect of the mechanism, and / or, Provide an indication of a predicted malfunction / failure / damage / defect of the mechanism. This may be included.

[0067] According to some embodiments, the method may further include alerting the user of a suspected and / or predicted malfunction / failure / damage / defect of the mechanism or its components.

[0068] According to some embodiments, the assigned (accurate / normal) trajectory curve and / or speed of the reference point may be determined by one or more optical sensors (e.g., cameras) of the system.

[0069] According to some embodiments, the assigned (accurate / normal) trajectory curve and / or speed of the reference point may be determined by analyzing a plurality of images / video clips of a complex system such as a mechanism and determining the "permissible" range / margin within which the trajectory curve and / or speed of each reference point can still be defined as "normal".

[0070] According to some embodiments, the assigned (accurate / normal) trajectory curve and / or speed of the reference point can be determined by analyzing a plurality of images / video clips / data obtained from the failed mechanism (e.g., the failed mechanism or its components) to obtain the trajectory curve and / or speed values typical of the failure.

[0071] According to some embodiments, the deviation rate of the reference point from their respective assigned (correct / normal) trajectory curve and / or speed can be determined and utilized to predict the timeline until failure.

[0072] According to some embodiments, provided herein is a system for determining the position where the mechanism is fully lowered and locked. The system includes at least one optical sensor, such as a camera, configured to be fixed to the mechanism or its locking mechanism, in its vicinity, or in front of it, and at least one processor in communication with the optical sensor. According to some embodiments, the processor is executable to receive a signal from the at least one optical sensor (e.g., an image from the camera) and detect at least two markings (real or virtual) on at least two movable parts of the locking mechanism, and determine the full-lock position of the locking mechanism based on the relative position between the two markings. According to some embodiments, the markings can be any other mark not limited to lines, symbols, or arrow triangles. The markings can be, for example, two lines, and the relative position between the two lines indicating the full-lock position of the locking mechanism can be the parallel convergence of the two lines. According to some embodiments, the markings are configured as vernier scales, facilitating the calculation of the (angular and / or linear) accuracy level of the lock indication. According to some embodiments, the processor can further be configured to provide an indication of the full-lock of the locking mechanism of the mechanism.

[0073] According to some embodiments, at least two markings can be existing marks such as defects, natural lines, or boundary lines in mechanical / machine elements that are automatically identified and selected by an algorithm applied by the system or selected by an operator, for example, by an engineer's application.

[0074] According to some embodiments, at least two markings can be stickers attached to different elements of a movable part of a mechanism.

[0075] Determination of the locked position and detection of hard landing in a landing gear According to some embodiments, a method is provided for determining the position where a landing gear is fully lowered and locked. The method utilizes at least one processor to receive a signal (e.g., an image from a camera) from at least one optical sensor, detect at least two markings, e.g., lines (real or virtual), on at least two movable parts of a locking mechanism, and determine the full-lock position of the locking mechanism based on the relative position between at least two markings, e.g., lines, e.g., between their parallel convergence, including this.

[0076] The method can further be configured to provide an indication of full lock of the locking mechanism of the landing gear.

[0077] In an aircraft, particularly an unmanned aerial vehicle (UAV), an indication of touchdown and an indication of "hard landing" can be important for a pilot or a remote operator.

[0078] Accordingly, the present specification provides a system for monitoring / determining / predicting the touchdown of the wheels of a landing gear according to additional or alternative embodiments. The system includes at least one optical sensor, such as a camera, configured to be fixed to the landing gear or its wheels, in the vicinity thereof, or in front of it, and at least one processor in communication with the optical sensor. According to some embodiments, a system for determining the position where the landing gear is fully lowered and locked may also be configured to monitor / determine / predict the touchdown of the wheels of the landing gear.

[0079] According to some embodiments, the processor is executable to receive a signal from at least one optical sensor (an image of one or more wheels from the camera), measure the ground speed (e.g., by measuring the linear speed of different objects and the angular speed of one or more of the wheels), calculate the ground (linear) speed of one or more wheels using an algorithm theory, and provide a touchdown indication when the ground (linear) speed of one or more wheels is equal to the ground speed.

[0080] According to some embodiments, one or more of the following options may be utilized in the calculation of the ground speed.

[0081] 1. The system (e.g., the processor) determines the speed of the aircraft relative to the ground by other sensors / systems of the aircraft.

[0082] 2. The ground speed can be calculated in relation to the speed of the aircraft on the condition that the system determines the height of the aircraft from the ground by calculating the time it takes for any element on the ground to pass through a known number of pixels in the camera.

[0083] 3. The speed of the aircraft relative to the ground can be calculated by calculating the time it takes for a ground component of a known size to pass through a known number of pixels, provided that the size of the ground element (e.g., stripes on the runway) is known.

[0084] When the ground speed is known, it can be compared with the linear speed of the wheels. Since the position of the camera is fixed, the distance from the wheels is also known. As the wheels rotate, their angular speed can be calculated by measuring the time it takes for a mark on the wheel to pass a known number of pixels. If the diameter of the wheel or the measured value of the diameter by the camera is known, the linear speed of the wheel can be calculated from its angular speed.

[0085] According to an additional or alternative embodiment, the processor is executable to receive a signal from at least one optical sensor (e.g., an image of one or more wheels from a camera) and measure the distance between the ground and one or more of the wheels, and provide a touchdown indication if the distance is zero (or if the distance falls below a predetermined threshold).

[0086] According to an additional or alternative embodiment, the processor is executable to receive a signal from at least one optical sensor (e.g., a side image of one or more wheels from a camera) and measure the degree of flattening of one or more of the wheels, and provide a touchdown indication if the height of the bottom of the tire of one or more of the wheels is below a predetermined threshold (or less than the height of the top of the tire). Such "flattening" of the tire can indicate the weight of the aircraft on the wheel, and in other words, can indicate touchdown.

[0087] In this specification, according to an additional or alternative embodiment, a system for providing a "hard landing" indication is further provided. The processor of such a system is executable to receive a signal from at least one optical sensor (e.g., an image of the damper of the landing gear from a camera) and measure the degree of compression of the damper, and provide a "hard landing" indication if the degree of compression of the damper exceeds a predetermined threshold. The predetermined threshold can be calculated, for example, based on one or more standard deviation values from the damper compression value in an average landing. According to some embodiments, a system for determining when the landing gear is fully lowered and locked and / or monitoring / determining / predicting the touchdown of the wheels of the landing gear may be configured to provide a "hard landing" indication.

[0088] According to some embodiments, the processor is capable of receiving signals from at least one optical sensor, obtaining data related to the characteristics of at least one failure mode of a mechanism or its components (e.g., the trajectory and / or velocity of a reference point in a movable part of the monitored mechanism or its components), and identifying at least one change in the received signals (e.g., a deviation from the trajectory and / or velocity of one or more reference points in a functional / pre-calculated intact mechanism or its movable components) for the identified changes in the received signals.

[0089] Identification of Failure Modes According to some embodiments, the system monitors the health of components of an additional mechanism in addition to or instead of the movement of components. Optionally, the processor applies at least one identified change to an algorithm configured to analyze the identified changes in the received signals, classifies whether the identified changes in the received signals are related to a (optionally predetermined) failure mode of the mechanism or its components, labels the identified changes as defects based at least in part on the acquired data, and outputs a signal indicating that the identified changes are related to the failure mode for the identified changes classified as being related to the failure mode.

[0090] As used herein, according to some embodiments of the present invention, the term "identified change" in a received signal or image includes a change with respect to a baseline of the received image and / or a change with respect to the same area of other images of the optical sensor or similar components in other mechanisms.

[0091] According to some embodiments, for an identified defect, the processor may generate at least one model of the trend of the identified defect, where the trend may include the rate of change in the defect.

[0092] According to some embodiments, the system can be configured for smart maintenance of a mechanism or its components, performed by one or more algorithms configured to detect changes, identify defects, and determine whether the defects may develop into failures of the mechanism and / or components. Optionally, the system includes a suppressor that filters identified changes and / or defects not related to a predetermined failure mode of the image component. For example, the suppressor can reduce the number of false alarms by filtering defects indicating flies or optical lenses in the imaged component or dirt in the imaged component.

[0093] Advantageously, the system and method can enable visualization of inaccessible areas that require significant effort for inspection / maintenance by placing one or more optical sensors in, near, or in front of a mechanism or component that may not be visibly monitored.

[0094] Advantageously, the system and method can reduce the time and cost associated with grounding a machine (e.g., an aircraft). Also, the system and method can minimize the cost of unnecessary maintenance and the cost of unnecessary component replacement that automatically occur when a mechanism or its components are regularly replaced by the regular protocol maintenance of the mechanism.

[0095] Advantageously, the system can analyze the trend of defect development in a mechanism or its components by enabling trend identification and calculation, so that it is possible to predict a failure even before the normal behavior or operation of the mechanism changes. According to some embodiments, a system for monitoring potential failures in a mechanism or its components is provided, the system comprising at least one optical sensor configured to be fixed to the mechanism or its components or in the vicinity thereof, and at least one processor in communication with the optical sensor, the processor receiving signals from the at least one optical sensor, obtaining data related to the characteristics of at least one failure mode of the mechanism or its components, identifying at least one change in the received signals, and for the identified changes in the received signals, applying the at least one identified change to an algorithm configured to analyze the identified changes in the received signals and classify whether the identified changes in the received signals are related to a failure mode of the mechanism or its components, thereby labeling the identified changes as defects based at least in part on the obtained data, and for the identified changes classified as being related to a failure mode, outputting a signal indicating that the identified changes are related to a failure.

[0096] According to some embodiments, a computer-executable method for monitoring a mechanism or its components is provided, the method comprising receiving signals from at least one optical sensor fixed to the mechanism or its components or in the vicinity thereof, obtaining data related to the characteristics of at least one failure mode of the mechanism or its components, identifying at least one change in the received signals, and for the identified changes in the received signals, applying the at least one identified change to an algorithm configured to analyze the identified changes in the received signals and classify whether the identified changes in the received signals are related to a failure mode of the mechanism or its components based at least in part on the obtained data, and for the identified changes classified as being related to a failure mode, outputting a signal indicating that the identified changes are related to the failure mode.

[0097] According to some embodiments, for an identified defect, the method and / or system includes generating at least one model of the identified defect trend.

[0098] According to some embodiments, the trend includes a rate of change of the defect.

[0099] According to some embodiments, generating at least one model of the identified defect trend includes calculating a correlation between the rate of change of the defect and one or more environmental parameters.

[0100] According to some embodiments, for an identified defect, the method and / or system includes alerting a user of a predicted failure based at least in part on the generated model.

[0101] According to some embodiments, alerting a user of a predicted failure includes including any one or more of the time (or time range) of the predicted failure, the usage time of the mechanism, and the characteristics of the failure mode, or any combination thereof.

[0102] According to some embodiments, identifying at least one change in a signal includes identifying a change in the rate of change of the signal.

[0103] According to some embodiments, one or more environmental parameters include at least one of temperature, season or time of year, pressure, time, operating time of the mechanism or its components, operating duration of the mechanism or its components, authenticated user of the mechanism, GPS location, mode of operation of the mechanism or its components, or any combination thereof.

[0104] According to some embodiments, for an identified defect, the method and / or system outputs a prediction as to whether the identified defect is likely to cause a failure of the mechanism or its components based at least in part on the generated model.

[0105] According to some embodiments, predicting whether a failure is likely to occur in a mechanism or its component is at least partially based on known failure environment parameters.

[0106] According to some embodiments, a failure mode includes at least one of a change in dimension, a change in position, a change in color, a change in texture, a change in size, a change in appearance, a crack, structural damage, a fissure, a crack size, a critical crack size, a crack location, crack propagation, a specified pressure applied to the mechanism or its component, a change in the movement of one component relative to another component, a leakage amount, a leakage rate, a change in the leakage rate, an accumulated liquid amount, a change in the accumulated liquid amount, a formed bubble, a droplet, a liquid pool, a jet size, or any combination thereof.

[0107] According to some embodiments, for an identified defect, the method and / or system includes storing and / or using data related to the identified change for further consideration if the identified change is not classified as related to a failure mode, where the further consideration includes at least one of adding a failure mode, updating an algorithm configured to identify changes, and improving an algorithm configured to identify changes by training the algorithm to ignore identified changes in the future.

[0108] According to some embodiments, obtaining data related to the characteristics of at least one failure mode of a mechanism or its component includes data related to the location of the failure mode in the mechanism or its component and / or a specific type of failure mode.

[0109] According to some embodiments, obtaining data related to the characteristics of at least one failure mode of a mechanism or its component includes receiving input data from a user.

[0110] According to some embodiments, for an identified defect, the method and / or system includes analyzing a received signal, wherein obtaining data related to the characteristics of at least one failure mode of a mechanism or its component includes automatically receiving data from a database based at least in part on the received signal from at least one optical sensor.

[0111] According to some embodiments, obtaining data related to the characteristics of at least one failure mode of a mechanism or its component includes identifying previously unknown failure modes by applying the received signal to a machine learning algorithm configured to determine the failure mode of the mechanism or its component.

[0112] According to some embodiments, identifying at least one signal change includes analyzing the raw data of the received signal.

[0113] According to some embodiments, at least one signal includes at least one image, a part of an image, a set of images, or a video.

[0114] According to some embodiments, identifying at least one signal change includes analyzing the dynamic movement of a mechanism or its component, where the dynamic movement includes linear movement, rotational movement, periodic (repetitive) movement, damage, defect, crack size / length, crack growth rate, crack propagation, fissure, structural damage, defect diameter, notch, warp, expansion, deformation, abrasion, wear, corrosion, oxidation, spark, smoke, fluid flow rate, droplet size, fluid volume, liquid accumulation rate, change in texture, change in color / shape, formed bubbles, size of droplets, formation of liquid pools, increase in liquid pools, change in dimensions, change in position, change in color, change in texture, change in size, change in appearance, or any one or more of any combination thereof.

[0115] According to some embodiments, for an identified defect, the method and / or system includes identifying at least one segment to be monitored within the received signal, where at least one change in the signal is a change within at least one segment. According to some embodiments, at least one segment can be automatically identified. According to some embodiments, at least one segment may be manually identified by a user.

[0116] According to some embodiments, for an identified defect, the method and / or system includes monitoring at least one segment and detecting a shape change of at least one segment, a size of at least one segment, an occurrence rate of at least one segment in the received signal, or any combination thereof.

[0117] According to some embodiments, at least one segment includes a boundary of a surface defect.

[0118] According to some embodiments, at least one segment includes at least one boundary of an outer periphery of a liquid pool, an outer periphery of a droplet, an outer periphery of a saturation area (or material), or any combination thereof.

[0119] According to some embodiments, at least one segment includes a boundary of a spark.

[0120] According to some embodiments, at least one segment includes a boundary of a specific element of a mechanism or its component, and further includes identifying a geometry of at least one segment as a specific element of the mechanism or its component.

[0121] According to some embodiments, the specific element includes any one or more of a screw, a connector, a bolt, one or more vehicle components, one or more fuel tanks, an oil tank, a motor, a gearbox, a turbine component, a cable, a belt, a wire, a fastener, a cylinder, a blade, a nut, one or more flexible, semi-rigid, or rigid pipes / tubes, and any combination thereof. Each option is an individual embodiment.

[0122] According to some embodiments, the specific element includes a brake pad.

[0123] According to some embodiments, identifying the geometry includes analyzing any one or more of total intensity, scattered intensity, repair agent detection, line segment detection, line segment registration, edge segment curvature estimation, homography estimation, specific object identification, object detection, semantic segmentation, background model, change detection, detection by optical flow, or reflection detection, flame detection, or any combination thereof.

[0124] According to some embodiments, for the identified defect, the method and / or system includes outputting data related to the location optimal for the placement of the optical sensor, from which potential failure modes can be detected.

[0125] According to some embodiments, for the identified defect, the method and / or system includes at least one light source configured to illuminate the mechanism or its component, wherein classifying whether the identified change in the signal is related to the failure mode of the mechanism or its component is at least partially based on any one or more of the placement of the at least one light source, the duration of the illumination, the wavelength of the illumination, the intensity, the direction, and the frequency of the illumination.

[0126] According to some embodiments, the system is configured to monitor the failure modes of the screw, and identifying at least one change in the received signal includes identifying a change in the shape or rate of change of at least one segment, including identifying at least one segment within the received signal that includes the boundary around the visible portion of the screw, where the failure modes include loosening of the screw and / or rotation of the screw, and generating at least one model of the trend of the identified changes includes modeling the trend of the size and / or orientation of the segment to monitor whether the screw has loosened and / or rotated.

[0127] The implementation of the method and / or system of the embodiments of the present invention may include performing or accomplishing the selected tasks manually, automatically, or a combination thereof. Also, according to the actual apparatus and equipment of the embodiments of the method and / or system of the present invention, some selected tasks can be implemented by hardware, software, firmware, or a combination thereof using an operating system.

[0128] For example, the hardware for performing the selected tasks according to the embodiments of the present invention can be implemented as a chip or a circuit. For software, the selected tasks according to the embodiments of the present invention can be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In an exemplary embodiment of the present invention, one or more tasks according to the exemplary embodiments of the method and / or system described herein are executed by a data processor such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data, and / or non-volatile storage devices for storing instructions and / or data, such as magnetic hard disks and / or removable media. Optionally, a network connection is also provided. Optionally, a display and / or a user input device such as a keyboard or a mouse are also provided.

[0129] According to aspects of some embodiments of the present invention, a system for monitoring the health state of a mechanism is provided. The system identifies, within the image data of a plurality of components of the mechanism, the relative movement of reference points in at least two of the components of the mechanism, evaluates the health state of the mechanism by analyzing the relative movement with a model of proper operation of the mechanism, and outputs an indicator of the health state of the mechanism based on the evaluation, and includes a processing circuit configured as such.

[0130] According to some embodiments of the present invention, the indicator includes at least one of a maintenance instruction, an estimate of the time to failure, an alert for a detected failure, and an operation instruction in response to the detected failure.

[0131] According to some embodiments of the present invention, at least two of the plurality of components include a component that makes at least one rotational movement and a component that makes at least one linear movement.

[0132] According to some embodiments of the present invention, the analysis includes determining the alignment of the reference points when the reference points are static, and the indicator indicates whether the reference points are accurately aligned with each other or inaccurately aligned.

[0133] According to some embodiments of the present invention, the analysis includes comparing the relative trajectory of the reference points with a trajectory specified by a model, and the indicator includes an alert when the relative trajectory is outside the allowable range of the model.

[0134] According to some embodiments of the present invention, the analysis includes estimating the respective forces exerted on at least two of the plurality of components during the operation of the mechanism from the relative movement, and the indicator indicates the estimated force levels.

[0135] According to some embodiments of the present invention, the processing circuit is further configured to monitor at least one defect in a plurality of components, and the evaluation of the health state of the machine is further based on the monitoring of at least one defect.

[0136] According to some embodiments of the present invention, the processing circuit is further configured to detect a deformation of a component of the mechanism during operation of the mechanism from the image data, and the indicator indicates either a distortion or the absence of distortion of the component of the mechanism.

[0137] According to some embodiments of the present invention, the processing circuit is further configured to evaluate the health state of the mechanism by monitoring a change over time in the shape of at least one of the plurality of components.

[0138] According to some embodiments of the present invention, the evaluation of the health state of the mechanism is the force exerted on at least one of the plurality of components, and a defect in at least one of the plurality of components, and a deformation of at least one of the plurality of components, and a shape change of at least one of the plurality of components, and information from at least one non-imaging sensor, and is further based on at least one of them.

[0139] According to some embodiments of the present invention, the indicator includes maintenance requirements for at least one of the plurality of components of the mechanism, and the maintenance requirements are based on image data collected during a plurality of operations of the mechanism.

[0140] According to some embodiments of the present invention, the mechanism includes a vehicle subsystem.

[0141] According to some embodiments of the present invention, the subsystem is disposed between the vehicle body of the vehicle and the surface supporting the vehicle.

[0142] According to some embodiments of the present invention, the mechanism includes an aircraft landing gear.

[0143] According to some embodiments of the present invention, the indicator indicates one of the locked state and the unlocked state of the landing gear.

[0144] According to some embodiments of the present invention, the analysis includes estimating the hardness of the aircraft landing from the relative motion, and the indicator indicates the hardness of the aircraft landing.

[0145] According to some embodiments of the present invention, the model is trained by a training set of images collected during the operation of the mechanism and / or a similar mechanism.

[0146] According to some embodiments of the present invention, at least one of the reference points is automatically detected as a result of the training of the model.

[0147] According to some embodiments of the present invention, at least one of the reference points is specified by the user.

[0148] According to some embodiments of the present invention, the model is periodically retrained based on image data collected during further operation of the mechanism or a similar mechanism.

[0149] According to some embodiments of the present invention, the system further includes at least one optical sensor arranged to collect image data from at least one of a plurality of components of the mechanism.

[0150] According to some embodiments of the present invention, the system further includes a plurality of optical sensors arranged to provide image data of the same component among a plurality of components of the mechanism, where the image data of the same component among the plurality of components is processed as a three-dimensional image.

[0151] According to aspects of some embodiments of the present invention, a method for monitoring the health state of a mechanism, comprising: inputting image data of a plurality of components of the mechanism from at least one optical sensor; identifying relative movement of reference points in at least two of the plurality of components of the mechanism within the image data; evaluating the health state of the mechanism by analyzing the relative movement with a model of proper operation of the mechanism; outputting an indicator of the health state of the mechanism based on the evaluation; A method is provided.

[0152] According to some embodiments of the present invention, the indicator includes: a maintenance instruction; an estimate of time to failure; an alert for a detected failure; an operation instruction in response to a detected failure; and includes at least one of:

[0153] According to some embodiments of the present invention, at least two of the plurality of components include a component that makes at least one rotational movement and a component that makes at least one linear movement.

[0154] According to some embodiments of the present invention, the analysis includes determining the relative arrangement of the reference points when the reference points are static, and the indicator indicates whether the reference points are accurately arranged relative to each other or inaccurately arranged.

[0155] According to some embodiments of the present invention, the analysis includes comparing the relative trajectory of the reference points with a trajectory specified by a model, and the indicator includes an alert when the relative trajectory is outside the allowable range of the model.

[0156] According to some embodiments of the present invention, the analysis includes estimating the respective forces exerted on at least two of the plurality of components during operation of the mechanism from relative motion, and the indicator indicates the estimated force level.

[0157] According to some embodiments of the present invention, the method further includes monitoring at least one defect in the plurality of components, and the assessment of the health state of the mechanism is further based on the monitoring of the at least one defect.

[0158] According to some embodiments of the present invention, the method further includes detecting deformation of components of the mechanism from image data during operation of the mechanism, and the indicator indicates either distortion or lack of distortion of the components of the mechanism.

[0159] According to some embodiments of the present invention, the method further includes assessing the health state of the mechanism by monitoring the change over time of the shape of at least one of the plurality of components.

[0160] According to some embodiments of the present invention, the assessment of the health state of the mechanism is the force exerted on at least one of the plurality of components, and a defect in at least one of the plurality of components, and the deformation of at least one of the plurality of components, and the shape change of at least one of the plurality of components, and information from at least one non-imaging sensor, and is further based on at least one of.

[0161] According to some embodiments of the present invention, the indicator includes maintenance requirements for at least one of the plurality of components of the mechanism, and the maintenance requirements are based on image data collected during multiple operations of the mechanism.

[0162] According to some embodiments of the present invention, the mechanism includes a vehicle subsystem.

[0163] According to some embodiments of the present invention, the subsystem is disposed between the vehicle body of the vehicle and the surface supporting the vehicle.

[0164] According to some embodiments of the present invention, the mechanism includes an aircraft landing gear.

[0165] According to some embodiments of the present invention, the indicator indicates one of the locked state and the unlocked state of the landing gear.

[0166] According to some embodiments of the present invention, the analysis includes estimating the hardness of the aircraft landing from the relative motion, and the indicator indicates the hardness of the aircraft landing.

[0167] According to some embodiments of the present invention, the method further includes training the model with a training set of images collected during the operation of the mechanism and / or a similar mechanism.

[0168] According to some embodiments of the present invention, at least one of the reference points is automatically detected as a result of training the model.

[0169] According to some embodiments of the present invention, at least one of the reference points is specified by the user.

[0170] According to some embodiments of the present invention, the model is periodically retrained based on image data collected during further operation of the mechanism.

[0171] According to some embodiments of the present invention, the method further includes processing a plurality of images of the components of the mechanism as a three-dimensional image of the components, and the three-dimensional image serves as an input to the model.

[0172] According to some embodiments of the present invention, the analysis is further based on data input from non-imaging sensors.

[0173] According to aspects of some embodiments of the present invention, a system for detecting the state of a mechanism, wherein the mechanism includes a plurality of components and has at least one designated alignment of the components, and the system identifies reference points in at least two of the plurality of components within the image data of the mechanism, determines, based on the location of each of the reference points, whether the plurality of mechanism components are in one of the designated alignments, and outputs an indicator of the result of the determination. A system is provided that includes a processing circuit configured as described above.

[0174] According to some embodiments of the present invention, the indicator indicates one of the mechanism being in one of the designated alignments and the mechanism being stateless.

[0175] According to some embodiments of the present invention, the determination of whether the plurality of mechanism components are in one of the designated alignments is further based on the respective trajectories of the reference points determined from the image data.

[0176] According to some embodiments of the present invention, the determination of whether the plurality of mechanism components are in one of the designated alignments is based on a model of proper operation of the mechanism, and the model is trained by a training set of images collected during the operation of the mechanism and / or a similar mechanism.

[0177] According to some embodiments of the present invention, at least one of the reference points is automatically detected during training of the model.

[0178] According to some embodiments of the present invention, at least one of the reference points is specified by a user.

[0179] According to some embodiments of the present invention, the mechanism includes a vehicle subsystem.

[0180] According to some embodiments of the present invention, the subsystem is configured to support the vehicle body of a vehicle.

[0181] According to some embodiments of the present invention, the mechanism includes a landing gear of an aircraft, and at least one specified alignment of the components includes a locked state of the landing gear, where the landing gear is configured to be in the locked state during touchdown of the aircraft.

[0182] According to some embodiments of the present invention, the processing circuit is further configured to identify the touchdown of the aircraft by analyzing the image data and provide a touchdown indication.

[0183] According to some embodiments of the present invention, the processing circuit is further configured to detect the hardness of the aircraft landing from the image data based on the movement of each of at least two reference points on the landing gear during touchdown.

[0184] According to some embodiments of the present invention, the processing circuit is further configured to detect the hardness of the aircraft landing from the image data based on the deformation of the wheels of the landing gear during touchdown.

[0185] According to some embodiments of the present invention, the processing circuit is further configured to estimate a health state estimate of the mechanism based on changes in the trajectories of the reference points in at least two of the plurality of components from the image data collected during the operation of the mechanism, where the indicator includes the health state estimate.

[0186] According to some embodiments of the present invention, the processing circuit is further configured to estimate the wear degree in at least two of the plurality of components from the image data collected during the operation of the mechanism.

[0187] According to some embodiments of the present invention, the system further comprises at least one optical sensor arranged to collect image data of at least one of a plurality of components and provide the collected image data to a processing circuit.

[0188] According to some embodiments of the present invention, the processing circuit is configured to determine whether a plurality of mechanical components are in one of the specified alignments based on additional data inputs from non-imaging sensors.

[0189] According to some embodiments of the present invention, the image data is collected during operation of the mechanism, and the indicator is output in real time.

[0190] According to aspects of some embodiments of the present invention, a method for detecting the state of a mechanism, the mechanism including a plurality of components and having at least one specified alignment of the components, the method comprising: Inputting image data of a plurality of components of the mechanism from at least one optical sensor during operation of the mechanism; Identifying reference points in at least two of the plurality of components from the input image data; Determining whether a plurality of mechanical components are in one of the specified alignments based on the location of each of the reference points; Outputting an indicator of the result of the determination; Providing a method including.

[0191] According to some embodiments of the present invention, the indicator indicates one of the mechanism being in one of the specified alignments and the mechanism being stateless.

[0192] According to some embodiments of the present invention, the determination of whether a plurality of mechanical components are in one of the specified alignments is further based on the respective trajectories of the reference points.

[0193] According to some embodiments of the present invention, determining whether a plurality of mechanism components are in one of the specified alignments is based on a model of proper operation of the mechanism, and the model is trained by a training set of images collected during the operation of the mechanism and / or similar mechanisms.

[0194] According to some embodiments of the present invention, the method further includes automatically detecting at least one of the reference points during training of the model.

[0195] According to some embodiments of the present invention, at least one of the reference points is specified by data input via a communication interface.

[0196] According to some embodiments of the present invention, the mechanism includes a vehicle subsystem.

[0197] According to some embodiments of the present invention, the subsystem is configured to support the vehicle body on a surface.

[0198] According to some embodiments of the present invention, the mechanism includes an aircraft landing gear, and at least one specified alignment of the components includes a landing gear locked state, and the landing gear is configured to be in a locked state during touchdown of the aircraft.

[0199] According to some embodiments of the present invention, the method identifying a touchdown of the aircraft based on analysis of image data, and providing a touchdown indication, and further includes.

[0200] According to some embodiments of the present invention, the method further includes detecting the hardness of the aircraft landing based on the movement of each of at least two reference points on the landing gear during touchdown from the image data.

[0201] According to some embodiments of the present invention, the method further includes detecting the hardness of an aircraft landing based on deformation of the wheels of the landing gear during touchdown from image data.

[0202] According to some embodiments of the present invention, the method further includes estimating the health state of the mechanism based on the duration of a period during which the mechanism transitions between first and second specified alignments of components from image data collected during operation of the mechanism.

[0203] According to some embodiments of the present invention, the method further includes estimating a health state estimate of the mechanism based on changes in the respective trajectories of reference points in at least two of a plurality of components from image data collected during operation of the mechanism, wherein the indicator includes the health state estimate.

[0204] According to some embodiments of the present invention, the method further includes estimating the wear degree in at least two of a plurality of components from image data collected during operation of the mechanism.

[0205] According to some embodiments of the present invention, the method further includes inputting image data from at least one optical sensor configured to provide image data of at least one of a plurality of components.

[0206] According to some embodiments of the present invention, the determination is based on additional data input from non-imaging sensors.

[0207] According to aspects of some embodiments of the present invention, a computer program product including a computer-usable medium having computer-readable program code embodied therein for detecting the state of a mechanism, computer-readable program code for causing a computer to input image data of a plurality of components of the mechanism from at least one optical sensor during operation of the mechanism, and Computer-readable program code for causing a computer to identify reference points in at least two of a plurality of components from input image data, Computer-readable program code for causing a computer to determine whether a component of a mechanism is in one of a specified alignment based on the location of each reference point, Computer-readable program code for causing a computer to output an indicator of the determined alignment or misalignment result, A computer program product comprising:

[0208] According to aspects of some embodiments of the present invention, a computer program product comprising a computer-usable medium having computer-readable program code embodied therein for monitoring the health of a mechanism, Computer-readable program code for causing a computer to input image data of a plurality of components of a mechanism from at least one optical sensor, Computer-readable program code for causing a computer to identify relative motion of at least two reference points in a plurality of components of a mechanism from the input image data, Computer-readable program code for causing a computer to evaluate the health of a mechanism by analyzing the relative motion with a model of proper operation of the mechanism, Computer-readable program code for causing a computer to output an indicator of the health of a mechanism based on the evaluation, A computer program product comprising:

Brief Description of the Drawings

[0209] In this specification, some embodiments of the present invention are described by way of example only with reference to the accompanying drawings. Referring now to the drawings in detail, it is emphasized that the details shown are by way of example and are for purposes of exemplarily considering embodiments of the present invention. In this regard, the description associated with the drawings will make the implementation method of the embodiments of the present invention clear to those skilled in the art.

[0210] For clarity, some of the objects shown in the drawings are not drawn to scale. Also, two different objects in the same figure may be drawn at different scales.

[0211] In block diagrams and flowcharts, optional elements / components and optional steps may be included within dashed boxes.

[0212] A description of the drawings is set forth below.

Figure 1A

Figure 1B

Figure 1C

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 3E

Figure 3F

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9A

Figure 9B

Figure 10

Embodiments for Carrying Out the Invention

[0213] The principles, uses, and implementations of the teachings herein can be better understood with reference to the accompanying description and drawings. After reading the description and drawings presented herein, those skilled in the art will be able to implement the teachings herein without undue effort or experimentation. In the figures, the same reference numerals refer to the same parts throughout.

[0214] In the following description, various aspects of the present invention will be described. For purposes of interpretation, specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without all of the specific details presented herein. Further, some features known in the art may be omitted or simplified in order not to obscure the present invention.

[0215] Refer to FIGS. 1A - 1B, which are simplified block diagrams of a system for monitoring a mechanism or its components according to each embodiment of the present invention.

[0216] As described below, embodiments of the monitoring system include 1) Determining whether the mechanism is in a defined state or not (i.e., stateless), and optionally determining which defined state the mechanism is in, 2) Determining whether the mechanism is in a locked state or an unlocked state, 3) Monitoring the health state of the mechanism and / or its components, 4) Monitoring potential failures of the mechanism, 5) Determination of the integrity and proper functioning of the mechanism, can be employed for various purposes including but not limited to those.

[0217] According to the embodiment of FIG. 1A, a system 1 (also referred to herein as a monitoring system) for monitoring a mechanism or its components includes a processing circuit 2. The processor circuit includes one or more processors 3. The processor 3 processes the image data provided by the image data and performs the analysis described herein. The processor 3 also performs other tasks such as providing a graphical user interface (GUI) to the user and processing inputs from the GUI and / or other input / output means.

[0218] Optionally, the monitoring system 1 includes a memory 4 for internal storage of data used by the monitoring system 1. The stored data can include a) image data, b) program instructions, c) algorithms and rules for monitoring the mechanism, d) optionally, a model of the mechanism developed by machine learning from a training set of images of the mechanism or a similar mechanism, but not limited thereto. The model is developed according to the type implemented by the system. For example, the model can take as input an image of a component of the mechanism and output the state of the mechanism, the health state of the mechanism, an indicator of mechanism failure, etc.

[0219] Optionally, the processing circuit 2 further includes one or more interfaces 5 for inputting and / or outputting data. For example, the interface can function to input image data and / or communicate with other components in the machine and / or communicate with an external machine or system and / or provide a user interface.

[0220] In one example, indicators and information regarding the state, health state, etc. of the mechanism are provided to a HUMS, CBM or similar system via the interface 5.

[0221] In a second example, the lock status indicator may be provided to the aircraft navigation system such that when the landing gear is not locked, the aircraft control system does not allow touchdown of the aircraft or provides an indication of a critical failure.

[0222] According to the embodiment of FIG. 1B, system 1 further includes one or more optical sensors 6.1-6.n that provide image data for monitoring the mechanism. Optionally, the optical sensors 6.1-6.n provide the image data to the processor via a data bus 7.

[0223] According to some embodiments, the optical sensors 6.1-6.n may include cameras. According to some embodiments, the optical sensors 6.1-6.n may include electro-optical sensors. According to some embodiments, the optical sensors 6.1-6.n may include one or more of charge-coupled devices (CCDs) and complementary metal-oxide semiconductor (CMOS) sensors (or active pixel sensors), or any combination thereof. According to some embodiments, the optical sensors 6.1-6.n may include one or more of point sensors, distributed sensors, external sensors, internal sensors, omnidirectional beam sensors, diffuse reflection sensors, retroreflective sensors, or any combination thereof.

[0224] Optionally, the processing circuit 2 controls one or more light sources, where each light source illuminates at least a portion of the mechanism. Optionally, each light source is focused on a particular component or reference point, thereby allowing the intensity of the required light to be reduced. Alternatively or additionally, the light source is controlled by the user.

[0225] By controlling the light source, the processing circuit 2 and / or the user can improve the image characteristics to facilitate image processing and analysis. For example, the light source can be adjusted to increase the visibility of the reference point. Alternatively or additionally, the light source can be adjusted to facilitate the detection of defects and / or surface defects and / or structural defects by increasing the shadows that emphasize such areas.

[0226] According to some embodiments, the light source includes one or more of the light transmitted through a light bulb, a light emitting diode (LED), a laser, an electroluminescent wire, and an optical fiber wire or cable (e.g., from an LED coupled to an optical fiber cable). Other types of light sources may also be suitable.

[0227] Optionally, the processing circuit 2 1) the illumination direction of the light source, 2) the duration of illumination, 3) the frequency of illumination, 4) the illuminance, 5) the on / off switching of the light source, controls one or more of them.

[0228] According to some embodiments, the light source can emit visible light, infrared (IR) radiation, near-IR radiation, ultraviolet (UV) radiation, or light in any other spectral or frequency range.

[0229] According to some embodiments, the light source is a strobe or a light source configured to irradiate with short pulses. According to some embodiments, the light source can be configured to emit strobe light without using a global shutter sensor.

[0230] Optionally, the processing circuit 2 selects the optimal settings of the light source based on a predetermined algorithm. Optionally, the light source is controlled according to the environment in which the system is currently operating. For example, the light source can be turned on at night and turned off during the day.

[0231] Optionally, during operation, the processing circuit 2 dynamically changes the light source operation, for example, by emitting light at different times using different fibers of an optical fiber cable or by emitting light simultaneously from two or more fibers.

[0232] Optionally, the light source is part of the system 1.

[0233] According to some embodiments, one or more optical sensors may include one or more lenses and / or optical fiber sensors. According to some embodiments, the optical sensors 6.1 to 6.n may include a software correction matrix configured to generate an image from the acquired data. According to some embodiments, one or more optical sensors may include a focus sensor configured to enable detection of changes in the data acquired by the optical sensors. According to some embodiments, the focus sensor may be configured to enable the optical sensors to detect changes in one or more pixels of the acquired signal.

[0234] According to some embodiments, all or part of the processing circuit 2 is provided within the machine to be monitored and optionally within the same case as one or more of the optical sensors 6. This may be useful, for example, in the case of a machine such as an aircraft where connection to the cloud is more difficult or privacy is important. Alternatively, the image data may be transmitted to a remote location or the cloud where the processing circuit 2 is provided. In these cases, the processing circuit 2 may be used to analyze images of multiple machines and optionally analyze images of other machines using information obtained from one machine. In another alternative, the processing may be divided between a processing circuit within the machine to be monitored and additional circuitry outside the machine.

[0235] FIG. 1C is a simplified block diagram of a system for monitoring a mechanism or its components according to an exemplary embodiment of the present invention.

[0236] The types of mechanisms that can be monitored by system 100 can include, but are not limited to, any type of mechanism that transitions from one state to another, such as a landing gear, a mechanism of a landing gear, a component of a landing gear, two or more components of a landing gear, an independent component of a landing gear, an interconnected component of a landing gear, or any combination thereof.

[0237] According to some embodiments, system 100 can include one or more optical sensors 112 configured to be fixed to a mechanism or its component, or in its vicinity. According to some embodiments, system 100 can be configured to monitor a mechanism and / or its component in real time. According to some embodiments, system 100 can include at least one processor 102 that communicates with one or more optical sensors 112. According to some embodiments, processor 102 can be configured to receive signals (or data) from one or more optical sensors 112. According to some embodiments, processor 102 can include an embedded processor, a cloud computing system, or any combination thereof. According to some embodiments, processor 102 can be configured to process signals (or data) received from one or more optical sensors 112 (also referred to herein as received signals or received data). According to some embodiments, processor 102 can include an image processor 106 configured to process signals received from one or more optical sensors 112.

[0238] According to some embodiments, one or more optical sensors 112 may be configured to detect light reflected from the surface of a mechanism and / or its components. This can be advantageous because surfaces with different textures reflect light differently. For example, a matte surface has a lower reflectivity and may scatter (diffuse) light equally in all directions compared to a polished surface, which has a flat surface and reflects more light than an unpolished surface, reflecting most of the parallel light rays. A smooth and shiny polished surface absorbs very little light and can reflect more light, so the image detected from the light reflected from the polished surface can be sharper than the image detected from the light reflected from the unpolished surface. Thus, the surface texture of a crack, fissure, or any other surface defect may be different from the undamaged surface surrounding it (in other words, the original baseline surface), and for this reason, the different light reflections from the surface enable the detection of small defects. Also, this phenomenon can be enhanced by changing the wavelength, intensity, and / or direction of the light source of the system. According to some embodiments, as will be described in more detail elsewhere in this specification, the system may include one or more light sources configured to illuminate the mechanism and / or its components.

[0239] According to some embodiments, changing the direction of the light may include moving the light source. According to some embodiments, changing the direction of the light may include powering (or operating) the light source at different times while maintaining the positions of two or more light sources fixed, thereby changing the direction of the light illuminating the mechanism and / or its components. According to some embodiments, as will be described in more detail elsewhere in this specification, the system may include one or more light sources arranged such that the mechanism and / or its components are illuminated by their operation. According to some embodiments, the system may include a plurality of light sources, where each light source is arranged at a different location with respect to the mechanism and / or its components.

[0240] According to some embodiments, the wavelength, intensity, and / or direction of one or more light sources can be controlled by a processor. According to some embodiments, by changing the wavelength, intensity, and / or direction of one or more light sources, it becomes possible to detect surface defects on the surface of a mechanism and / or its components. According to some embodiments, one or more optical sensors 112 can detect minute dents and / or defects, such as one-third of a millimeter that may not be visible to the naked eye, by analyzing the reflected light.

[0241] According to some embodiments, one or more optical sensors 112 can include a camera. According to some embodiments, one or more optical sensors 112 can include an electro-optical sensor. According to some embodiments, one or more optical sensors 112 can include any one or more of a charge-coupled device (CCD) and a complementary metal-oxide-semiconductor (CMOS) sensor (or active pixel sensor), or any combination thereof. According to some embodiments, one or more optical sensors 112 can include any one or more of a point sensor, a distributed sensor, an external sensor, an internal sensor, an all-pass beam sensor, a diffuse reflection sensor, a retroreflective sensor, or any combination thereof.

[0242] According to some embodiments, one or more optical sensors can include one or more lenses and / or an optical fiber sensor. According to some embodiments, one or more optical sensors can include a software correction matrix configured to generate an image from the acquired data. According to some embodiments, one or more optical sensors can include a focus sensor configured to enable detection of changes in the data acquired by the optical sensor. According to some embodiments, the focus sensor can be configured to enable the optical sensor to detect changes in one or more pixels of the acquired signal.

[0243] According to some embodiments, system 100 may include one or more interfaces 114 that communicate with processor 102.

[0244] According to some embodiments, interface 114 may be configured to receive data from a user, where the data is associated with any one or more of an organization or its components, the type of the organization, the type of the system in which the organization operates, the mode of operation of the organization, the user of the organization, one or more environmental parameters, one or more failure modes of the organization, or any combination thereof.

[0245] According to some embodiments, user interface 114 may include any one or more of a keyboard, a display, a touch screen, a mouse, one or more buttons, or any combination thereof. According to some embodiments, user interface 114 may include a configuration file that can be generated automatically and / or manually by the user. According to some embodiments, the configuration file may be configured to identify at least one segment. According to some embodiments, the configuration file may be configured to enable the user to mark and / or select at least one segment.

[0246] According to some embodiments, system 100 may include a memory 104 configured to store data and / or instructions (or code) for execution by a data and / or processor 102. According to some embodiments, memory 104 may communicate (or be operably communicable) with processor 102. According to some embodiments, memory 104 may include a database 108 configured to store data associated with any one or more of system 100, a mechanism or its components, user input data, one or more training sets (or data sets for training one or more of the algorithms), or any combination thereof. According to some embodiments, memory 104 may include one or more models and / or algorithms 110 configured to be executed by processor 102 and stored therein. According to some embodiments, one or more models and / or algorithms 110 may be configured to analyze and / or classify received signals, as described in more detail elsewhere herein. According to some embodiments, one or more models and / or algorithms 110 may include one or more preprocessing techniques for preprocessing received signals, as described in more detail elsewhere herein. According to some embodiments, the model is trained by supervised or unsupervised learning techniques of machine learning.

[0247] According to some embodiments, one or more algorithms 110 may include a change detection algorithm configured to identify changes in received signals. According to some embodiments, one or more algorithms 110 and / or the change detection algorithm may be configured to receive signals from one or more optical sensors 112, obtain data related to the characteristics of at least one failure mode of a mechanism or its components, and / or identify at least one change in the received signals.

[0248] According to some embodiments, one or more algorithms 110 may include a classification algorithm configured to classify the identified changes. According to some embodiments, the classification algorithm may be configured to classify the identified changes as defects and, optionally, classify the severity of the defects. According to some embodiments, the classification algorithm may be configured to classify the identified changes as the normal performance (or movement) of a mechanism or its components.

[0249] According to some embodiments, one or more algorithms 110 may be configured to analyze a defect (or an identified change classified as a defect). According to some embodiments, one or more algorithms 110 may be configured to output a signal (or an alarm) indicating that the identified change is associated with a failure mode.

[0250] According to some embodiments, one or more algorithms 110 may be configured to execute, by the processor 102, a method for monitoring potential failures in a landing device or its components, such as the method shown in FIG. 4.

[0251] Reference points FIGS. 2A, 2C, and 2E show side views of three exemplary states of mechanism 2. In this example, there are five reference points P1 to P5 on mechanism 5. FIG. 2B shows the alignment of reference points P1 to P5 when mechanism 2 is in the first state. FIG. 2D shows the alignment of reference points P1 to P5 when mechanism 2 is in the second state. FIG. 2F shows the alignment of reference points P1 to P5 when mechanism 2 is in the third state.

[0252] As can be seen from FIGS. 2B, 2D, and 2F, the alignment of reference points P1 to P5 is different in each state. Therefore, the alignment of the reference points can be used to determine which state the mechanism is in and / or whether the mechanism is in a defined state.

[0253] Figure 2G shows the alignment of reference points P1 - P5 in state 1, and additional dashed lines indicate the angles and distances between the reference points. Figure 2H is a reduced version of Figure 2G. Note that the angles and relative distances between the reference points remain invariant regardless of the change in image size.

[0254] Figures 3A - 3F show the trajectories of the movement of reference points P1 - P5 during the proper operation of the mechanism. As can be seen from Figure 3A, reference point P1 moves along a circular curve. Figure 3B shows the position of reference point P1 as a curve on the x - axis and y - axis, indicating the allowable deviation from the curve.

[0255] Reference points P2 and P3 move along a straight - line horizontal trajectory during proper operation, and their curves on the x - axis and y - axis are shown in Figure 3C.

[0256] Reference point P4 moves along the curve shown in Figure 3D during proper operation, and its curves on the x - axis and y - axis and the allowable deviation from the curve are shown in Figure 3E.

[0257] Reference point P5 moves along a vertical straight line, and its curves on the x - axis and y - axis and the allowable deviation from the curve are shown in Figure 3F.

[0258] According to some embodiments, the alignment and / or trajectories of reference points in a component, such as those shown in Figures 2A - 2H and Figures 3A - 3F, can be used to monitor the alignment of the component itself. Optionally, a deviation of a reference point greater than the allowable deviation from a defined curve is interpreted as indicating a problem with the health of the mechanism.

[0259] Monitoring and Analysis of Potential Failures of the Mechanism According to some embodiments, a system is provided for monitoring potential failures in a mechanism or its components.

[0260] In some embodiments, the mechanism is a vehicle subsystem. In further embodiments, the subsystem is configured to support the vehicle body at least during a locked state, which is, for example, an automotive suspension that supports an automobile on the ground by connecting an automotive chassis to wheels. In yet another embodiment, the mechanism is a landing gear of an aircraft or a spacecraft.

[0261] According to some embodiments, the system may be configured to receive signals from at least one optical sensor disposed on or near the landing gear or its components, and may receive signals therefrom. According to some embodiments, the system may be configured to identify at least one change in the received signals. According to some embodiments, for the identified changes in the received signals, the system applies at least one identified change to an algorithm configured to analyze the identified changes in the received signals and classify whether the identified changes in the received signals are related to a failure mode of the landing gear or its components, thereby at least partially labeling the identified changes as defects based on the acquired data related to the failure mode of the landing gear and / or its components. According to some embodiments, for the identified changes classified as related to the failure mode, the system may output a signal indicating that the identified changes are related to the failure mode.

[0262] According to some embodiments, the system may be configured to generate at least one model of the identified defect trend, where the trend may include the rate of change of the defect.

[0263] According to some embodiments, the system may be configured to prevent failures of the landing gear and / or its components by identifying defects in real time and monitoring changes in the defects in real time.

[0264] Refer to FIG. 4 showing a flowchart of functional steps of a computer-executed method for monitoring potential failures in a mechanism or its components according to some embodiments of the present invention, and FIG. 5 showing a schematic block diagram of a method for monitoring potential failures in a mechanism or its components according to some embodiments of the present invention. According to some embodiments, method 200 of FIG. 4 may include one or more steps of block 300 of FIG. 5.

[0265] According to some embodiments, at step 202, the method may include identifying at least one change in the received signal. According to some embodiments, at step 204, the method may include identifying at least one change in the received signal. According to some embodiments, at step 206, the method may include analyzing the identified change in the received signal and classifying whether the identified change in the received signal is related to a failure mode of the mechanism or its components, thereby labeling the identified change as a defect. In some embodiments, the mechanism or its components are related to one or more predetermined failure modes provided by its manufacturer and / or user, and / or by monitoring and / or machine learning of the same or similar components.

[0266] According to some embodiments, at step 208, the method may include outputting a signal indicating that the identified change is related to a failure mode. According to some embodiments, at step 210, the method may include generating at least one model of the tendency of the identified defect. According to some embodiments, at step 212, the method may include alerting the user of a predicted failure based at least in part on the generated model.

[0267] According to some embodiments, as shown in FIG. 5 for example, the method may include signal acquisition 302, that is, the reception of one or more signals. According to some embodiments, the method may include receiving one or more signals from at least one optical sensor fixed to or in the vicinity of a mechanism or its components, for example, one or more sensors 112 of the system 100. According to some embodiments, the one or more signals may include one or more images. According to some embodiments, the one or more signals may include one or more portions of an image. According to some embodiments, the one or more signals may include a set of images such as image packets. According to some embodiments, the one or more signals may include one or more videos.

[0268] According to some embodiments, the method may include preprocessing (304) of the one or more signals. According to some embodiments, the preprocessing may include converting the one or more signals into electronic signals (for example, from optical signals to electrical signals). According to some embodiments, the preprocessing may include generating one or more images, one or more sets of images, and / or one or more videos from the one or more signals. According to some embodiments, the preprocessing may include dividing one or more images, one or more portions of one or more images, one or more sets of images, and / or one or more videos into a plurality of tiles. According to some embodiments, the preprocessing may include applying one or more filters to one or more images, one or more portions of one or more images, one or more sets of images, one or more videos, and / or a plurality of tiles. According to some embodiments, the one or more filters may include one or more noise reduction filters.

[0269] According to some embodiments, the method may include combining (or stitching) a plurality of signals obtained from two or more optical sensors. According to some embodiments, the method may include stitching the plurality of signals in real time.

[0270] According to some embodiments, the method may include identifying at least one segment within any one or more of a received signal, one or more images, one or more portions of one or more images, one or more sets of images, and / or one or more videos. According to some embodiments, the method may include monitoring the at least one (identified) segment. According to some embodiments, at least one change in the signal is a change within the at least one segment. According to some embodiments, at least one change in one or more images, one or more portions of one or more images, one or more sets of images, and / or one or more videos is a change within the at least one segment.

[0271] According to some embodiments, the user may mark segments that are monitored on an image and / or a portion of the image and / or at least a portion of the video. According to some embodiments, the user may input the location to be monitored. According to some embodiments, the algorithm may be configured to identify at least one segment within the location input by the user.

[0272] According to some embodiments, the method may include applying one or more signals, one or more images, one or more portions of one or more images, one or more sets of images, and / or one or more videos to a change detection algorithm 308 (e.g., one or more algorithms 110 of system 100) configured to detect changes therein. According to some embodiments, the change detection algorithm may include one or more learning models 322 of a mechanism (e.g., a landing device).

[0273] According to some embodiments, the method may include detecting whether the shape, size, occurrence rate of at least one segment, or any combination thereof in the received signal has changed. According to some embodiments, the method may include detecting whether the shape, size, and / or occurrence rate of at least one segment has changed over time. According to some embodiments, the method may include detecting whether the shape, size, and / or occurrence rate of at least one segment has changed over a specified period, such as 1 second, several seconds, 1 minute, 1 hour, 1 day, 1 week, several weeks, or any range therebetween.

[0274] According to some embodiments, at least one segment may include potential defects that need to be monitored, such as surface defects or loose screws. According to some embodiments, at least one segment may include an overview of by-products of the mechanism or its components, such as a fluid that may leak or a spark that may ignite. According to some embodiments, at least one segment may include the boundary of a surface defect. According to some embodiments, at least one segment may include at least one boundary around a liquid pool, around a droplet, around a saturated area (or material), or any combination thereof. According to some embodiments, at least one segment may include the boundary of a spark.

[0275] According to some embodiments, at least one segment may include the boundary of a particular element of a mechanism or its component. According to some embodiments, the method may include identifying the geometry of at least one segment as a particular element of a mechanism or its component. According to some embodiments, the particular element may include any one or more of a screw, a connector, a bolt, a brake pad, one or more vehicle components, one or more fuel tanks, an oil tank, a motor, a gearbox, a turbine component, a cable, a belt, a wire, a fastener, a cylinder, a blade, a nut, one or more flexible, semi-rigid, or rigid pipes / tubes, and any combination thereof. According to some embodiments, the method (or geometry identification) may include analyzing any one or more of total intensity, distributed intensity, repair agent detection, line segment detection, line segment registration, edge segment curvature estimation, homography estimation, specific object identification, object detection, semantic segmentation, background model, change detection, detection by optical flow, or reflection detection, flame detection, or any combination thereof.

[0276] According to some embodiments, the method may include obtaining data related to the characteristics of at least one failure mode of a mechanism or its component, or may include failure mode identification 306. According to some embodiments, the data related to the characteristics of at least one failure mode of a mechanism or its component may include the type of failure mode. According to some embodiments, the data related to the characteristics of at least one failure mode of a mechanism or its component may include the location or range of locations of the failure mode in the mechanism or its component, and / or a specific type of failure mode.

[0277] According to some embodiments, a failure mode may include one or more aspects that can fail in a mechanism or its components. According to some embodiments, as described in more detail herein, a failure mode may include a significant progression of an identified defect. According to some embodiments, a failure mode may include a change in dimension, a change in position, a change in color, a change in texture, a change in size, a change in appearance, a crack, a structural damage, a fissure, a crack size, a critical crack size, a crack position, a crack propagation, a specified pressure applied to the mechanism or its components, a change in the movement of one component relative to another component, a defect diameter, a notch, a warp, an expansion, a deformation, a wear, a abrasion, a corrosion, an oxidation, a spark, a smoke, a leakage amount, a leakage rate, a change in leakage rate, an accumulated liquid amount, a change in accumulated liquid amount, a formed bubble, a droplet, a size of a liquid pool, a formation of a liquid pool, an increase in a liquid pool, a jet flow, a fluid flow rate, a droplet size, a fluid volume, a liquid accumulation rate, a change in color / shape, a change in dimension, a change in position, a change in color, a change in size, a change in appearance, or any one or more of any combination thereof.

[0278] According to some embodiments, a method may include obtaining data related to characteristics of at least one failure mode of a machine or its components by receiving user input. According to some embodiments, a method may include obtaining data related to characteristics of at least one failure mode of a mechanism or its components by analyzing a received signal and detecting at least one segment related to the failure mode. According to some embodiments, a method may include obtaining data related to characteristics of at least one failure mode of a mechanism or its components by analyzing a received signal and detecting a potential failure mode. According to some embodiments, a method may include obtaining data related to characteristics of at least one failure mode of a mechanism or its components by analyzing a received signal and detecting one or more previously unknown failure modes.

[0279] According to some embodiments, obtaining data related to the characteristics of at least one failure mode of a mechanism or its components includes receiving input data from a user. According to some embodiments, the user can input data related to the failure mode of the mechanism or its components through a user interface. According to some embodiments, the method may include monitoring the mechanism and / or its components based at least in part on input data received from the user. According to some embodiments, the user can input the type of failure mode of the mechanism and / or the components of the mechanism. According to some embodiments, the user can input the type of failure mode related to a specific identified segment. According to some embodiments, the user can input the location of the failure mode. According to some embodiments, the user can identify one or more of at least one segment as being in a location where there is a high likelihood of failure and / or a high likelihood of occurrence of a defect.

[0280] According to some embodiments, the method may include automatically obtaining data related to the characteristics of at least one failure mode of a mechanism or its components. According to some embodiments, the method may include obtaining data related to the characteristics of at least one failure mode of a mechanism or its components without user input. According to some embodiments, the method may include analyzing a received signal and automatically retrieving data from a database, such as database 108. According to some embodiments, one or more algorithms may be configured to identify one or more failure modes in the database that may be related to an identified segment of the received signal of the mechanism and / or its components. According to some embodiments, the method may include searching for possible failure modes of the identified segment in the database. According to some embodiments, the method may include retrieving data from the database, where the data is related to possible failure modes of the identified segment.

[0281] According to some embodiments, the method may include obtaining data related to the characteristics of at least one failure mode of a mechanism or its components by identifying previously unknown failure modes. According to some embodiments, identifying previously unknown failure modes may include applying a received signal and / or an identified segment to a machine learning algorithm 324 configured to determine the failure mode of the mechanism or its components. According to some embodiments, the machine learning algorithm 324 may be trained to identify potential failure modes of the identified segment.

[0282] According to some embodiments, in step 204, the method may include identifying at least one change in the received signal and / or at least one identified segment. According to some embodiments, the method may include applying the received signal and / or at least one identified segment to a change detection algorithm, such as a change detection algorithm 308 configured to detect (or identify) at least one change in the received signal and / or at least one identified segment.

[0283] According to some embodiments, identifying at least one change in the signal includes identifying a change in the rate of change of the signal. For example, the algorithm may be configured to identify changes that occur regularly within the analysis signal, and the analysis signal may be able to "return" to a previous state (e.g., before the change in the analysis signal). According to some embodiments, the algorithm may be configured to identify a change in the occurrence rate of the identified change.

[0284] According to some embodiments, the term "analysis signal" as used herein can describe any one or more of the received signals, such as raw signals from one or more optical sensors, processed or pre-processed signals from one or more optical sensors, one or more images, one or more image packets, one or more portions of one or more images, one or more videos, one or more portions of one or more videos, at least one identified segment, at least a portion of an identified segment, or any combination thereof. According to some embodiments, identifying at least one change in the analysis signal can include analyzing the raw data of the received signal.

[0285] According to some embodiments, the change detection algorithm 308 can include any one or more of binary change detection, quantitative change detection, and qualitative change detection.

[0286] According to some embodiments, binary change detection can include an algorithm configured to classify the presence or absence of a change in the analysis signal. According to some embodiments, binary change detection can include an algorithm configured to compare two or more analysis signals. According to some embodiments, for a comparison indicating that the analyzed analysis signals are the same or essentially the same, the classifier labels the analysis signal as having no detected (or identified) change. According to some embodiments, for a comparison indicating that the analyzed analysis signals are different, the classifier labels the analysis signal as having a detected (or identified) change. According to some embodiments, two or more different analysis signals can have at least one different pixel. According to some embodiments, two or more identical analysis signals can have the same features and / or pixels. According to some embodiments, the algorithm can be configured to set a threshold number of different pixels above which two analysis signals can be considered different.

[0287] Advantageously, the change detection algorithm 308 enables rapid detection of changes in the analyzed signaling and can be very sensitive to small changes there. Further, the detection and warning of binary change detection can be performed within a single signal, e.g., within a few milliseconds, depending on the signal output rate of the optical sensor, or in the case of an optical sensor with a camera, within a single image frame, e.g., within a few milliseconds, depending on the frame rate of the camera.

[0288] According to some embodiments, the binary change detection algorithm may analyze, for example, an analysis signal to determine whether non - black pixels change to black over time, thereby indicating a possible change in the position of a mechanism or its components, perhaps due to deformation or a change in the position of components of another mechanism. According to some embodiments, when the binary change detection algorithm detects a change in the signal, a warning signal (or alert) may be generated to alert the facility or technician that maintenance may be required.

[0289] According to some embodiments, the binary change detection algorithm may be configured to determine the cause of the identified change by one or more machine learning models. According to some embodiments, the method may include determining the cause of the identified change by applying the identified change to a machine learning algorithm. For example, for black pixels that can change to a color other than black over time (or across consecutive analysis signals), the machine learning algorithm may output that the change indicates a change in the material of a mechanism or its components, e.g., due to overheating. According to some embodiments, the method may include generating a signal, such as an information signal or a warning signal, as needed. According to some embodiments, the warning signal may be a one - time signal or a continuous signal that may require some action, e.g., to reset the warning signal.

[0290] According to some embodiments, the method may include identifying at least one change in a signal by analyzing the dynamic movement of a mechanism or its components. According to some embodiments, the dynamic movement may include linear movement, rotational movement, periodic (repetitive) movement, damage, defect, crack size / length, crack growth rate, crack propagation, fracture, structural damage, defect diameter, notch, warp, expansion, deformation, wear, abrasion, corrosion, oxidation, spark, smoke, fluid flow rate, droplet size, fluid volume, liquid accumulation rate, change in texture, change in color / shape, formed bubbles, size of droplets, formation of liquid pools, increase in liquid pools, change in dimensions, change in position, change in color, change in texture, change in size, change in appearance, or any one or more of any combination thereof.

[0291] According to some embodiments, change detection may include quantitative change detection. According to some embodiments, quantitative change detection may include an algorithm configured to determine whether a change in magnitude has occurred in the analysis signal that exceeds a specific threshold. According to some embodiments, a change in magnitude that exceeds a specific threshold may include a cumulative change in magnitude that is independent of time, and / or a rate of change in magnitude (or multiple rates of change). For example, a value reflecting the change in magnitude may represent the number of changed pixels, the percentage of changed pixels, the sum of the numerical differences of one or more pixels within the field of view (or analysis signal), and combinations thereof. According to some embodiments, the quantitative change detection algorithm may output quantitative data related to the change in the analysis signal.

[0292] According to some embodiments, change detection may include a qualitative change detection algorithm. According to some embodiments, the qualitative change detection algorithm may include an algorithm configured to classify the analysis signal as indicative of a change in the mechanism and / or its components. According to some embodiments, the qualitative change detection algorithm may include a machine learning model configured to receive the analysis signal and classify the analysis signal into categories including at least those including a change in the behavior of the mechanism or its components and those not including a change in the mechanism or its components.

[0293] According to some embodiments, the change detection algorithm can be configured to analyze other more complex changes in the analysis signal generated by the optical sensor with the assistance of a machine learning model. According to some embodiments, the machine learning model can be trained to recognize complex and diverse changes. According to some embodiments, the machine learning model can identify complex changes for signals generated by an optical sensor that may exhibit some periodic instability, such as a signal that appears normal over a period of time and then begins to appear abnormal over a period of time before appearing normal again. The signal can then exhibit some abnormality that is similar but different from before, and the change detection algorithm can be configured to train itself over time to analyze the change and detect possible causes of the instability. According to some embodiments, the change detection algorithm can be configured to generate a warning signal or an information signal as needed to alert the user of changes in the machine and / or its components.

[0294] Refer to FIG. 6, which shows an exemplary schematic block diagram of a system for monitoring potential failures of a mechanism or its components according to some embodiments of the present invention, and FIG. 7, which shows an exemplary schematic block diagram of a system for monitoring potential failures of a mechanism or its components that communicates with a cloud storage module according to some embodiments of the present invention.

[0295] As shown in the exemplary systems of FIGS. 6 and 7, an optical sensor may receive one or more signals from a mechanism and / or its components, such as mechanism 502. According to some embodiments, the optical sensor may generate a signal, such as an image or video, and transmit the generated signal to an image processor 506. According to some embodiments, the image processor may process the signal generated by the optical sensor (e.g., optical sensor 504 of FIGS. 6 and 7) so that the data can be analyzed by a data analysis module 518 (or the algorithms described herein). According to some embodiments, the image processor 506 may include any one or more of an image / frame acquisition module 508, a frame rate control module 510, an exposure control module 512, a noise reduction module 514, and a color correction module 516. According to some embodiments, the data analysis module (or the algorithms described herein) may include a change detection algorithm, such as change detection algorithm 308. According to some embodiments, a user interface module 532 (described below) may issue a warning signal obtained from the signal analysis performed by the algorithm. According to some embodiments, any one or more of the signal and / or the algorithm may be stored in cloud storage 602. According to some embodiments, the processor may be disposed on a cloud, such as cloud computing 604, that may coexist with an embedded processor.

[0296] According to some embodiments, the data analysis module 518 may include any one or more of a binary (visual) change detector 520 (or a binary change detection algorithm as described in more detail elsewhere in this specification), a quantitative (visual) change detector 522 (or a quantitative change detection algorithm as described in more detail elsewhere in this specification), and / or a qualitative (visual) change detector 524 (or a qualitative change detection algorithm as described in more detail elsewhere in this specification). According to some embodiments, the qualitative (visual) change detector 524 may include any one or more of edge detection 526 and / or shape (deformation) detection 528. According to some embodiments, the data analysis module 518 may include and / or communicate with a user interface module 532. According to some embodiments, as described in more detail elsewhere in this specification, the user interface module 532 may include a monitor 534. According to some embodiments, the user interface module 532 may be configured to output an alarm and / or notification 536 / 326.

[0297] According to some embodiments, a change detection algorithm, such as change detection algorithm 308, may be implemented on an embedded processor or a processor in the vicinity of an optical sensor. Thus, a change detection algorithm, such as change detection algorithm 308, can enable rapid detection and prevent latency associated with transmitting data to a remote server (e.g., the cloud).

[0298] According to some embodiments, once a change is identified by a change detection algorithm, the identified change can be classified by a classification algorithm. According to some embodiments, in step 206, the method may include analyzing the identified change of the received signal (or the analyzed signal) and classifying whether the identified change of the received signal is related to a failure mode of a machine or its components, thereby labeling the identified change as a defect. According to some embodiments, the method may include applying the received signal (or the analyzed signal) to an algorithm configured to analyze the identified change of the received signal and classify whether the identified change of the received signal is related to a failure mode of a mechanism or its components based at least in part on the acquired data.

[0299] According to some embodiments, the method may include applying the identified change to an algorithm configured to match the identified change with the acquired data related to the failure mode. According to some embodiments, the algorithm may be configured to determine whether the identified change may develop into one or more failure modes. According to some embodiments, the algorithm may be configured to determine whether the identified change may develop into one or more failure modes based at least in part on the acquired data. According to some embodiments, the method may include labeling the identified change as a defect if the algorithm determines that the identified change may develop into one or more failure modes.

[0300] For example, an identified change in a surface defect and / or crack may be identified as a defect when the crack or defect reaches a specific size or length, and may be associated with a failure mode that is a critical crack size or a critical defect size.

[0301] For example, for an identified change in an increase in the formation rate of droplets formed at the bottom of a pipe, the defect may be identified as a leak, and the failure mode may be a predetermined formation rate of the droplets formed at the bottom of the pipe.

[0302] For example, if the identified change can include the texture or color of a component of the mechanism, the defect can be identified as corrosion, and the failure mode can be the amount of corrosion or the depth of corrosion within the component.

[0303] According to some embodiments, the defect can include structural damage, cracks, defects, a predetermined crack size and / or length, crack growth rate, crack propagation, fissures, defect diameter, cuts, warping, swelling, deformation, wear, abrasion, corrosion, oxidation, sparks, smoke, fluid flow rate, droplet formation, droplet size, volume of the fluid or droplet, droplet formation rate, liquid accumulation rate, change in texture, change in color / shape, size of the bubbles formed, formation of a liquid pool, increase in the liquid pool, change in the dimensions of at least a portion of the segment, change in the position of at least a portion of the segment, change in the color of at least a portion of the segment, change in the texture of at least a portion of the segment, change in the size of at least a portion of the segment, change in the appearance of at least a portion of the segment, linear movement of at least a portion of the segment, rotational movement of at least a portion of the segment, periodic (repetitive) movement of at least a portion of the segment, change in the speed of movement of at least a portion of the segment, or any one or more of any combination thereof.

[0304] According to some embodiments, the algorithm can identify defects by one or more machine learning models. According to some embodiments, as will be described in more detail elsewhere herein, the machine learning model can be trained over time to identify one or more defects. According to some embodiments, the machine learning model can be trained to identify previously unknown defects by analyzing the baseline behavior of the machine and / or its components.

[0305] Advantageously, by identifying defects by a machine learning model, it becomes possible to detect different types of defects, or similar defects that may appear differently in different machines or situations, or at different angles of an optical sensor. Thus, the machine learning model can increase the sensitivity of detection of one or more defects.

[0306] According to some embodiments, the system and / or one or more algorithms may include one or more suppressor algorithms 310 (also referred to herein as suppressor 310) for reducing the number of false alarms provided by the system. According to some embodiments, one or more suppressor algorithms may be configured to classify whether a detected defect is likely to develop into a failure, as shown, for example, in the failure mode branch 312 of FIG. 5. For example, suppressor 310 may indicate that a detected defect is not related to a failure mode associated with a component because it is due to dirt or insects on the optical lens or image component of the sensor. Thus, the suppressor filters out identified defects not related to a given failure mode to reduce the number of false alarms provided by the system.

[0307] According to some embodiments, one or more suppressor algorithms 310 may include one or more machine learning models 320. According to some embodiments, one or more suppressor algorithms 310 may classify a defect and / or a propagating defect as harmless.

[0308] According to some embodiments, at step 208, for an identified defect, the method may include outputting a signal, such as a warning signal, indicating that the identified change is related to a failure mode. According to some embodiments, the method may include storing the identified change in a database, thereby increasing a data set for training one or more machine learning models.

[0309] According to some embodiments, the method may include labeling data related to any one or more of the classifications shown in fault mode identification 306, change detection algorithm 308, suppressor 310, and branch of fault mode 312. According to some embodiments, the method may include supervised labeling 316, manual labeling of data by user input (or expertise).

[0310] According to some embodiments, if the identified change is not classified as related to a fault mode (as shown, for example, by arrow 350 in FIG. 5), it may be identified (or classified) as normal, that is, as normal behavior or operation of the mechanism or its components. According to some embodiments, for an identified change classified as normal, the method may include storing data related to the identified change, thereby adding the identified change to a database and increasing a data set for training one or more machine learning models (e.g., one or more of machine learning models 320 / 322 / 324). According to some embodiments, the method may include using the data related to the identified change for further consideration, where the further consideration includes at least one of adding a fault mode, updating an algorithm configured to identify changes, and improving the algorithm training for an algorithm configured to identify changes by ignoring the identified change in the future.

[0311] According to some embodiments, if the identified change is classified as being related to a fault mode (as shown, for example, by arrow 355 in FIG. 5), the method may include trend analysis and fault prediction 314. According to some embodiments, at step 210, the method may include generating at least one model of the identified defect trend. According to some embodiments, the method may include generating at least one model of the trend based on a plurality of analysis signals. According to some embodiments, the method may include generating at least one model of the trend by calculating the development of the identified change over time within the analysis signal. According to some embodiments, the trend may include a defect rate of change. According to some embodiments, the method may include generating at least one model of the identified defect trend by calculating the correlation between the defect rate of change and one or more environmental parameters. According to some embodiments, the one or more environmental parameters may include any one or more of temperature, season or seasonality, pressure, time, operating time of the mechanism or its components, duration of operation of the mechanism or its components, authenticated user of the mechanism (e.g., a particular pilot), GPS location (or location or country in the world), operating mode of the mechanism or its components, or any combination thereof.

[0312] According to some embodiments, the operating mode of the mechanism may include any one or more of the distance the aircraft has flown or traveled, frequency of movement, speed of movement, power consumption during operation, and change in power consumption during operation. According to some embodiments, generating at least one model of the identified defect trend by calculating the correlation between the defect rate of change and one or more environmental parameters may include considering different effects in the vicinity of the aircraft and / or its components. According to some embodiments, the method may include mapping different environmental parameters that affect the operation of the mechanism and / or components, where the environmental parameters may change over time.

[0313] According to some embodiments, at step 212, the method may include alerting the user of a predicted failure, at least in part based on the generated model. According to some embodiments, the method may include outputting a notification and / or alert 326 to the user. According to some embodiments, the method may include alerting the user of a predicted failure. According to some embodiments, the method may include alerting the user of a predicted failure by outputting any one or more of the predicted time (or time range) of the failure and the characteristics of the failure mode, or any combination thereof. According to some embodiments, the method may include outputting a prediction as to whether a defect identified at least in part based on the generated model is likely to cause a failure of the mechanism or its component. According to some embodiments, the prediction as to whether a failure is likely to occur in the mechanism or its component may be at least in part based on known future environmental parameters. According to some embodiments, the prediction as to whether a failure is likely to occur in the mechanism or its component may be at least in part based on a known schedule, such as a calendar.

[0314] According to some embodiments, a system for monitoring potential failures in a mechanism or its components, e.g., system 100, may include one or more light sources configured to illuminate at least a portion in the vicinity of the mechanism or its components. According to some embodiments, the one or more light sources may include any one or more of a light bulb, a light emitting diode (LED), a laser, a fiber optic light source, and an optical fiber cable. According to some embodiments, a user may input the location (or position) of the light source relative to one or more optical sensors, the illumination direction of the light source (in other words, the direction in which the light is directed), the duration of illumination of the light source, the wavelength, intensity, and / or frequency of the illumination. According to some embodiments, one or more algorithms may be configured to automatically position the one or more light sources. According to some embodiments, one or more algorithms may instruct the operating mode of the one or more light sources. According to some embodiments, one or more algorithms may instruct and / or manipulate any one or more of the illumination intensity of the one or more light sources, the number of powered light sources, the positions of the powered light sources, and the wavelength, intensity, and / or frequency of the illumination of the one or more light sources, or any combination thereof.

[0315] Advantageously, an algorithm configured to instruct and / or manipulate the one or more light sources may improve the clarity of the received signal by reducing darker areas (e.g., areas where light is not reflected and / or areas that have not been illuminated), and may fix (or optimize) the saturation of the received signal (or image).

[0316] According to some embodiments, one or more algorithms may be configured to detect and / or calculate the position, illumination duration, illumination wavelength, intensity, and / or frequency of one or more light sources relative to one or more optical sensors. According to some embodiments, one or more algorithms may be configured to detect and / or calculate the position, illumination duration, illumination wavelength, intensity, and / or frequency of one or more light sources relative to one or more optical sensors, at least partially based on an analysis signal. According to some embodiments, a processor may control the operation of one or more light sources. According to some embodiments, a processor may control any one or more of the illumination duration, illumination wavelength, intensity, and / or frequency of one or more light sources.

[0317] According to some embodiments, a method may include obtaining the position, illumination duration, illumination wavelength, intensity, and / or frequency of one or more light sources relative to one or more optical sensors. According to some embodiments, a method may include obtaining the position of one or more light sources by any one or more of user input, detection, and / or use of one or more algorithms. According to some embodiments, a method may include classifying whether a detected change in a (analysis) signal is related to a failure mode of a mechanism or its component, at least partially based on any one or more of the arrangement, illumination duration, illumination wavelength, intensity, and frequency of at least one light source.

[0318] According to some embodiments, a method may include outputting data related to an optimal location for placement (or positioning) of an optical sensor, from which potential failure modes can be detected. According to some embodiments, one or more algorithms may be configured to calculate at least one optimal location for placement (or positioning) of one or more optical sensors, at least partially based on the acquired data, data stored in a database, and / or user input data.

[0319] According to some embodiments, the light source may illuminate the mechanism and / or its components at one or more wavelengths from a wide spectral range of visible and invisible light. According to some embodiments, the light source may include a strobe, or a light source configured to irradiate with short pulses. According to some embodiments, the light source may be configured to emit strobe light without using a global shutter sensor.

[0320] According to some embodiments, the wavelength may include any one or more of lights in the ultraviolet region, the infrared region, or a combination thereof. According to some embodiments, one or more light sources may be mobile or movable. According to some embodiments, one or more light sources may perform operations such as changing the output wavelength during operation, changing the illumination direction during operation, and changing one or more lenses. According to some embodiments, the light source may be configured to vary illumination by using one or more optical fibers (FOs), for example, generating light at different times using different fibers, or combining two or more fibers simultaneously. According to some embodiments, the optical fiber may include one or more light sources attached thereto, such as LEDs. According to some embodiments, the light intensity and / or wavelength of the LED may be changed using one or more algorithms as described in more detail elsewhere in this specification.

[0321] Advantageously, illuminating the mechanism and / or its components may enable an optical sensor to detect defects and / or surface defects and / or structural defects by analyzing shadows and / or reflections. For example, a surface defect may generate a shadow that can be analyzed by one or more algorithms and detected as a surface defect.

[0322] Advantageously, while receiving optical signals from one or more optical sensors, illuminating the mechanism and / or its components to detect surface defects can enable the detection of defects and / or malfunctions that may not be visible to humans. According to some embodiments, the size of the defect and / or malfunction can range from 10 micrometers to 5 mm. According to some embodiments, the size of the defect and / or malfunction can be less than 10 micrometers.

[0323] Determination of the state of the mechanism Refer to FIG. 8A, which is a simplified flowchart of a method for detecting the state of a mechanism according to some embodiments of the present invention. Optionally, the method is implemented by any of the systems shown in FIGS. 1A - 1C, although other systems may also be used.

[0324] The mechanism includes a plurality of components, where at least some of the components are movable relative to each other. At least one state is defined for the mechanism. The state is defined by specifying the alignment of the mechanism components for a given state.

[0325] At 810, input image data of the components of the mechanism from at least one optical sensor. The image data is collected by the optical sensor during the operation of the mechanism.

[0326] At 820, reference points of at least two components that are movable relative to each other are identified within the input image data. The reference points can be identified within the image data by any technique known in the art, not limited to image segmentation and other techniques described herein.

[0327] At 830, use the location of each of the reference points to determine whether the mechanism components are in the alignment specified as the state (i.e., whether the mechanism is in the defined state or is stateless).

[0328] At 840, an indicator of whether the mechanism is in a defined state is output. If the mechanism has more than one state, the specified state can be indicated.

[0329] Optionally, the indicator signal is output only when the mechanism enters and exits the defined state and / or when it is currently in the defined state. Thus, by analyzing the indicator signal, it is possible to determine whether the mechanism is not in the defined state (i.e., is stateless) even without an explicit indication of the stateless situation.

[0330] Optionally, the determination of the state of the machine is further based on the analysis of the respective trajectories of the reference points. For example, in the analysis, reference points converging to a certain state can be tracked, and it can be predicted that the mechanism has entered a specific state.

[0331] Optionally, the determination of the state of the machine is model-based. The model can be developed by any means known in the art.

[0332] Optionally, the model is a trained model developed by machine learning techniques (e.g., supervised or unsupervised learning). The model is trained by a training set of images collected during the operation of the mechanism and / or a similar mechanism. Alternatively or additionally, the model includes algorithmic and / or rule-based decision-making and / or alignment of appropriate reference points and / or mapping of trajectories.

[0333] Optionally, the model is trained before the mechanism is put into use (e.g., during a pre-training period).

[0334] Optionally, the model is periodically retrained based on image data collected during the operation of the mechanism.

[0335] Optionally, at least one of the reference points is automatically detected when the model is trained and / or retrained. For example, areas of interest such as movable components can be emphasized during learning (which may be supervised) and features that can be used as reference points are identified within these areas by machine learning.

[0336] Optionally, the system is trained by inputting labeled image data of healthy and unhealthy movements.

[0337] In some embodiments, a three-dimensional (3D) model of the mechanism is created by combining images taken from different directions of the mechanism or components of the mechanism. The 3D model enables the model to determine reference points for learning and tracking the movement of the mechanism. Optionally, the 3D model is also used to determine the preferred positions of optical sensors for monitoring the mechanism.

[0338] Optionally, at least one reference point is specified by an external source via an interface. Optionally, the reference points are specified by the user through a GUI. Alternatively or additionally, the reference points are specified by an external mechanism and / or machine and / or system (e.g., over a network).

[0339] In some embodiments, the mechanism is a vehicle subsystem. In further embodiments, the subsystem is configured to support the vehicle body at least during a locked state.

[0340] Optionally, the mechanism is a landing gear of an aircraft or a spacecraft, and the method detects whether the landing gear is locked. This indicator can be very important for the safe operation of the aircraft by warning that the aircraft should not land because the landing gear is not locked.

[0341] Optionally, the method further includes analyzing the image data to detect and indicate an aircraft touchdown. The touchdown can be detected by a rapid transition of the mechanism from a defined state and / or to another defined state due to forces exerted on the mechanism during landing.

[0342] Optionally further, the method detects from the image data the level of stress on the mechanism during operation. High stress on the mechanism can cause an increased need for maintenance or failure. Conversely, if the mechanism is not subject to high mechanical stress, it may be possible to defer maintenance of the mechanism. Alternatively or additionally, the touchdown can be detected by one or more other types of sensors on the landing gear and / or inputs from the aircraft control system.

[0343] Optionally, the stress on the mechanism can be evaluated by the following 1) and 2).

[0344] 1) The movement of each of at least two reference points during operation. An unexpected rapid and / or large movement of the reference point can indicate that a large force has been applied to the mechanism.

[0345] 2) Deformation of components of the mechanism. For example, deformation can be detected if the distance between reference points on the same component changes.

[0346] Optionally, the method detects a hard landing of the aircraft based on an image of a component of the landing gear. Optionally further, the hardness of the landing is 1) The movement of each of at least two reference points on the landing gear during touchdown, and / or 2) The deformation of the landing gear wheels during touchdown, based on.

[0347] Refer to FIG. 8B, which is a simplified flowchart of a method for outputting an indicator of the health state of a mechanism according to some embodiments of the present invention. FIG. 8B shows an exemplary embodiment of 840 described above with reference to FIG. 8A.

[0348] At 841, check the result of the determination of 830 in FIG. 8A to detect whether the mechanism is in one of the specified alignments. If "yes", at 842, output an indicator that the mechanism is in a predefined state. Optionally, if there are multiple predefined alignments, the indicator includes which specific state the mechanism is in. If "no", at 843, output an indicator that the mechanism is not in a predefined state.

[0349] It is understood that the touchdown described with reference to FIGS. 8A-8B herein may be combined with any of the descriptions in the above "Determination of the Locking Position and Detection of Hard Landing in the Landing Gear" chapter.

[0350] Refer to FIGS. 9A-9B, which are simplified diagrams of the landing gear before and during touchdown. The landing gear 900 has six reference points A1 - A6. A4 rotates with the movement of the wheel. A1 and A3 move linearly relative to each other along the axis 920. FIG. 9A shows the alignment of the reference points A1 - A6 before landing. As seen in FIG. 9B, during touchdown, the alignment of the reference points A1 - A6 changes. For example, the distance between A1 and A3 is shortened by the force exerted on the shock absorber. The distance between A5 and A6 is shortened, indicating deformation of the wheel. These changes in distance can be used to determine the hardness of the landing. The harder the landing, the greater the change in the distance between A1 and A3 and / or the deformation of the wheel 910.

[0351] Optionally, the alignment and / or trajectory of the reference points are used to estimate the health state and / or wear degree of the mechanism. Optionally, the estimation is 1) the time it takes for the mechanism to transition between states, and 2) the change over time of the trajectory of the reference points, and 3) the change over time of the alignment of the reference points, and 4) the time of deployment of the mechanism, and based on at least one factor not limited to these.

[0352] Optionally, the method further includes inputting image data from at least one optical sensor. The optical sensor is arranged to provide image data that can be analyzed to detect reference points on the component.

[0353] Optionally, the state of the mechanism is determined based on additional data inputs from non-imaging sensors such as pressure sensors and / or acoustic sensors and / or vibration sensors and / or temperature sensors. The pressure, temperature, and other conditions under which the mechanism is operating can cause changes in the reference point alignment, which is compensated for when the conditions are known.

[0354] Monitoring the health of a mechanism Refer to FIG. 10, which is a simplified flowchart of a method for monitoring the health of a mechanism according to some embodiments of the present invention. Optionally, the method is implemented by any of the systems shown in FIGS. 1A - 1C, although other systems may also be used.

[0355] At 1010, input image data of the mechanism component from at least one optical sensor.

[0356] At 1020, identify the relative movement of reference points in at least two of the mechanism components within the image data.

[0357] At 1030, evaluate the health of the mechanism by analyzing the relative movement with a model of proper operation of the mechanism.

[0358] At 1040, output an indicator of the health of the mechanism, where the indicator is based on the evaluation performed at 1030.

[0359] The model can be developed by any means known in the art.

[0360] Optionally, the model is a trained model developed by machine learning techniques (e.g., supervised or unsupervised learning). The model is trained by a training set of images collected during the operation of the mechanism and / or a similar mechanism. Alternatively or additionally, the model includes algorithmic and / or rule-based decision-making and / or appropriate fiducial alignment and / or trajectory mapping.

[0361] Optionally, the model is trained prior to the practical use of the mechanism (e.g., during a pre-training period).

[0362] Optionally, the model is periodically retrained based on image data collected during the practical use of the mechanism.

[0363] Optionally, the movement of at least one mechanism component is a rotational movement and the movement of at least one mechanism component is a linear movement. An example is shown in FIGS. 9A-9B.

[0364] Optionally, the analysis includes determining the alignment of the fiducial points when the fiducial points are static, and the indicator indicates whether the fiducial points are accurately aligned or inaccurately aligned.

[0365] Optionally, the analysis includes comparing the relative trajectory of the fiducial points with the trajectory specified by the model, and the indicator includes an alert if the relative trajectory is outside the tolerance of the model.

[0366] Optionally, the images are analyzed to monitor the health of a single component of the mechanism, e.g., loosening or rotation of screws or rivets, formation of rust or corrosion, tension of elastic elements, leakage, lubrication, cracks and / or other wear of the component.

[0367] Optionally, the analysis includes one or more of the following 1)-4).

[0368] 1) Estimate the respective forces exerted on the mechanism components during the operation of the mechanism from the relative movement of the reference points. The indicator indicates the estimated force level.

[0369] 2) Detect the deformation of the components of the mechanism from the image data during the operation of the mechanism. The indicator indicates whether the component is distorted (optionally, to what extent it is distorted).

[0370] 3) Monitor the change over time of the shape of at least one mechanism component.

[0371] 4) Detect changes on the surface of the mechanism components, such as rust, dirt, dents, cracks and fissures.

[0372] Optionally, the method further includes inputting data from non-imaging sensors such as acoustic sensors, vibration sensors or temperature sensors.

[0373] Optionally, at 1025, obtain additional data (also referred to as mechanism health state related data) indicating the mechanism health state and / or the trend of the identified defect failure modes. This information is 1) the respective forces exerted on one or more of the mechanism components, and 2) the defects in at least one of the mechanism components, and 3) the deformation of at least one of the mechanism components, and 4) the shape change of at least one of the mechanism components, and 5) information from at least one non-imaging sensor (e.g., pressure sensor, temperature sensor, acoustic sensor, motion sensor, etc.), and may include one or more of, but is not limited to, these.

[0374] The additional health state related data is optionally used, together with the image data, for the evaluation of the mechanism health state. Alternatively or additionally, the health state related data input from the non-imaging sensors is used to analyze the severity of the identified defects.

[0375] The monitoring of the health state of the mechanism described with reference to FIG. 10 may, according to an embodiment of the present invention, be combined with any of the methods or processes described with respect to any of the above “identification of failure modes” or “monitoring and analysis of potential failures of the mechanism” chapters. For example, a defect may be identified by any of the change detection algorithms described with reference to FIGS. 4 or 5, and / or the trend of the failure mode may be analyzed by any of the processes described above.

[0376] Some embodiments may be carried out according to the teachings of PCT Patent Application Publication No. WO2022 / 162663, the entire contents of which are incorporated herein by reference.

[0377] Optionally, the indicator includes a maintenance instruction for at least one of a plurality of components of the mechanism. The maintenance instruction is based on image data collected during a plurality of operations of the mechanism and, optionally, other information (e.g., time since the last maintenance, environmental conditions under which the mechanism is operating, etc.). For example, the maintenance instruction may define the allowed usage time and / or usage amount before the mechanism is inspected and repaired as needed.

[0378] Optionally, the indicator includes an estimate of the failure time (e.g., the trend of the failure mode) for the mechanism and / or the machine including the mechanism. At the failure time, the mechanism (and possibly the entire machine) becomes inoperable.

[0379] The estimate of the failure time may be 1) machine learning of previous defects and / or failures, and 2) input from the manufacturer, and 3) one or more of environmental parameters, such as operating time, operating area, specific operator of the system, weather conditions, etc., and may be performed by one or more of the above.

[0380] Optionally, the indicator includes an alert for a detected malfunction. Since the detected malfunction may cause the mechanism to operate inaccurately, immediate action may be required. For example, the indicator may alert that landing needs to be aborted or may instruct a special way to operate the mechanism and / or machine until the malfunction is corrected. Further optionally, the indicator includes operation instructions in response to the detected failure.

[0381] Optionally, when a malfunction is detected, the indicator includes the severity of the malfunction. For example, the greater the deviation of the actual reference point trajectory from the expected trajectory, the more severe the malfunction. Optionally or additionally, the severity of the malfunction is analyzed based on the health status of the components of the system. Optionally or additionally, the severity of the malfunction is analyzed based on the environmental parameters under which the system is operating.

[0382] In some embodiments, the mechanism is a subsystem of a vehicle. In further embodiments, the subsystem is configured to support the vehicle body at least during a locked state.

[0383] Optionally, the mechanism is a landing gear of an aircraft or a spacecraft, and the method detects whether the landing gear is locked.

[0384] Optionally, the analysis includes estimating the hardness of the aircraft's landing from the relative movement of the reference points determined by image analysis, substantially as described above. The indicator may indicate the hardness of the aircraft's landing.

[0385] Optionally, at least one of the reference points is automatically detected when the model is trained and / or retrained.

[0386] Optionally, at least one of the reference points is specified by data input via a communication interface. Optionally, the reference points are specified by the user through a GUI. Alternatively or additionally, the reference points are specified by an external agency and / or machine and / or system.

[0387] Optionally, the method further includes processing a plurality of images of the same mechanism component taken from different directions as a three-dimensional image of the mechanism component. The three-dimensional image is used as an input to the model. Optionally, the three-dimensional image is output to the user for evaluation.

[0388] Optionally, the analysis is further based substantially on the input from the non-imaging sensors described above.

[0389] Optionally, the health state of the mechanism is evaluated by selecting a particular reference point as the focus for establishing a reference frame for the entire mechanism. The relative movement of some or all of the other components is determined with reference to the selected reference point. Thus, tracking of a single reference point in the established reference frame can be used to calculate the relative movement of the component in which the reference point is located.

[0390] In summary, some of the embodiments taught herein analyze images of components of a mechanism to obtain a large amount of information regarding the operation of the mechanism and the health state of the mechanism. Reference points associated with the mechanism components are identified within the image data. The relative positions and / or movements of the reference points are analyzed and indicators are output along with the results of the analysis. In some embodiments, the indicator indicates the state of the mechanism based at least on the analysis of the relative positions of the reference points at least at a given time or period. In some alternative or additional embodiments, the indicator includes information regarding the health state of the mechanism based at least on the analysis of the relative movement of the reference points.

[0391] In other uses, the information obtained may be used for preventive maintenance of the mechanism to prevent malfunctions or failures of the mechanism and / or to detect mechanism failures in real time so that immediate action can be taken.

[0392] During the term of the patents arising from this application, it is expected that numerous related mechanisms, machines, machine subsystems, optical sensors, cameras, models and modeling techniques, and image analysis techniques will be developed, and the terms mechanism, machine, subsystem, optical sensor, camera, model, reference point, identification of reference points, determination of reference point alignment, determination of reference point trajectory and similar terms are intended to empirically include all such new technologies.

[0393] The terms "comprises", "comprising", "includes", "including", "having" and their conjugations mean "including but not limited to...".

[0394] The term "consisting of" means "including and limited to...".

[0395] The term "consisting essentially of" means that a composition, method or structure may include additional components, steps and / or parts, but only to the extent that the additional components, steps and / or parts do not substantially change the basic and novel features of the claimed composition, method or structure.

[0396] As used herein, the singular forms "a", "an" and "the" include the plural forms as well, unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" can include a plurality of compounds, including mixtures thereof.

[0397] It is understood that certain features of the invention described in the context of separate embodiments may be provided in combination with a single embodiment. Conversely, various features of the invention described in the context of a single embodiment for brevity may be provided separately, or in any suitable sub-combination, or in any other described embodiment of the invention as appropriate. For example, the identification of defects is described in connection with multiple embodiments of the invention. The defects, failure modes, and tendencies of failure modes described in a particular embodiment may be identified and analyzed as defined in other embodiments described herein. Additionally, certain features such as touchdown analysis, tendency, or health state estimation are described in connection with a particular embodiment, but may be provided in connection with other embodiments of the invention not specifically mentioned in connection therewith. Specific features described in the context of various embodiments are not considered essential features of those embodiments unless the embodiment cannot be practiced without those elements.

[0398] Although the invention has been described in connection with its specific embodiments, it is evident that many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the spirit and broad scope of the appended claims.

[0399] All publications, patents, and patent applications mentioned herein are incorporated herein by reference to the same extent as if each individual publication, patent, and patent application were specifically and individually indicated to be incorporated by reference. Additionally, any citation or identification of a reference document in this application should not be construed as an admission that such reference document is available as prior art to the present invention.

[0400] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. In case of conflict, the present patent specification, including definitions, will control.

[0401] Although the steps of the method according to some embodiments may be described in a specific order, the methods of the present disclosure may include some or all of the described steps executed in a different order. The methods of the present disclosure may include some or all of the described steps. None of the specific steps in the disclosed methods are considered essential steps of the method unless explicitly indicated as such.

[0402] It is understood that the present disclosure is not necessarily limited in its application to the details of the construction and arrangement of the components and / or methods described herein. Other embodiments may be implemented and the embodiments may be executed in various ways.

[0403] The expressions and terms used herein are for purposes of description and should not be regarded as limiting. Any citation or identification of a reference document in this application should not be construed as an admission that such reference document is available as prior art to the present disclosure. To the extent that section headings are used, they should not necessarily be construed as limiting.

Claims

1. A system for monitoring the health status of the organization, In the image data of the plurality of components of the mechanism, the relative motion of at least two reference points above the plurality of components of the mechanism is identified. The evaluation of the health of the mechanism by analyzing the relative motion using a model of the proper operation of the mechanism, wherein the relative motion includes the relative trajectory of the reference point located above at least two of the plurality of components, Based on the above evaluation, the indicator of the health status of the mechanism is output, A system comprising a processing circuit configured to perform the following actions.

2. The system according to claim 1, wherein the analysis includes comparing the relative trajectory of the reference point with the trajectory defined by the model, and the indicator includes an alert when the relative trajectory is outside the acceptable range of the model.

3. The aforementioned indicator is Maintenance instructions and, Estimation of the time until failure, Detected fault alerts, Operational instructions in response to the detected fault, The system according to claim 1 or 2, comprising at least one of the following.

4. The system according to claim 1 or 2, wherein at least two of the plurality of components include at least one component that rotates and at least one component that moves in a linear motion.

5. The system according to claim 1 or 2, wherein the analysis includes determining the alignment of the reference points when the reference points are stationary, and the indicator indicates whether the reference points are properly aligned with respect to each other or not.

6. The system according to claim 1 or 2, wherein the analysis includes estimating the forces exerted on at least two of the plurality of components from the relative motion during the operation of the mechanism, and the indicator shows the level of the estimated forces.

7. The evaluation of the health status of the aforementioned mechanism is further, A force exerted on at least one of the aforementioned multiple components, A defect in at least one of the aforementioned plurality of components, A variation of at least one of the aforementioned plurality of components, A change in the shape of at least one of the aforementioned plurality of components, Information from at least one non-imaging sensor, The system according to claim 1 or 2, based on at least one of the following.

8. The mechanism includes a vehicle subsystem, The system according to claim 1 or 2, wherein the subsystem is disposed between the vehicle body and the surface supporting the vehicle.

9. The aforementioned mechanism includes the aircraft's landing gear, The system according to claim 8, wherein the indicator shows either the locked state or the unlocked state of the landing gear.

10. The system according to claim 9, wherein the analysis includes estimating the degree of stiffness of the aircraft's landing from the relative motion, and the indicator shows the stiffness of the aircraft's landing.

11. The system according to claim 1 or 2, wherein the model is trained using a training set of images collected during the operation of the mechanism and / or a similar mechanism.

12. The system according to claim 11, wherein at least one of the reference points is automatically detected as a result of the training of the model.

13. The system according to claim 1 or 2, further comprising at least one optical sensor arranged to collect image data from at least one of the plurality of components of the mechanism.

14. A method for monitoring the health status of an organization, The steps include inputting image data of multiple components of the mechanism from at least one optical sensor, The step of identifying the relative motion of a reference point located above at least two of the plurality of components of the mechanism in the image data, wherein the relative motion includes the relative trajectory of the reference point located above at least two of the plurality of components, A step of evaluating the health status of the mechanism by analyzing the relative motion using a model of the proper operation of the mechanism, Based on the evaluation, the step of outputting an indicator of the health status of the mechanism, Methods that include...

15. The aforementioned indicator is Maintenance instructions and, Estimation of the time until failure, Detected fault alerts, Operational instructions in response to the detected fault, The method according to claim 14, comprising at least one of the following.

Images