A system for and method of collision detection

Abstract

A system and method for collision detection. The method comprises receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings (FBGs), the plurality of FBGs being coupled to a manipulator, the plurality of FBGs distributed spaced apart from one another on the manipulator; determining at least one time-based feature from the single optical sensor signal; determining at least one frequency-based feature from the single optical sensor signal; providing the at least one time-based feature and the at least one frequency-based feature to a machine learning model; determine from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.

Description

A SYSTEM FOR AND METHOD OF COLLISION DETECTIONCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to the Singapore application no. 10202401109Y filed 16 April, 2024, the contents of which are hereby incorporated by reference in their entirety for all purposes.TECHNICAL FIELD

[0002] This application relates generally to the field of manipulator control, and more particularly, to a system and method of collision detection.BACKGROUND

[0003] Conventional approaches toward robotic manipulator collision detection mainly rely on the detection of contact or impact forces via sensors such as accelerometers, Inertial Measurement Units (IMUs), force sensors, torque sensors, etc. However, such approaches may only determine the instance of collision without providing further information relating to impact. Other approaches rely on image-based sensing or imaging system for collision detection which are typically limited to a controlled environment and are costly.SUMMARY

[0004] According to an aspect, disclosed herein a system. The system comprises: memory storing instructions; and a processor coupled to the memory and configured to process the stored instructions to implement: a module configured to perform a method of collision detection. The method of collision detection including: receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings (FBGs), the plurality of FBGs being coupled to a manipulator, theplurality of FBGs distributed spaced apart from one another on the manipulator; determining at least one time-based feature from the single optical sensor signal; determining at least one frequency-based feature from the single optical sensor signal; providing the at least one time based feature and the at least one frequency-based feature to a machine learning model; determine from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.

[0005] According to another aspect, disclosed herein a method of collision detection. The method including: receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings (FBGs), the plurality of FBGs being coupled to a manipulator, the plurality of FBGs distributed spaced apart from one another on the manipulator; determining at least one timebased feature from the single optical sensor signal; determining at least one frequency-based feature from the single optical sensor signal; providing the at least one time-based feature and the at least one frequency-based feature to a machine learning model; determining from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various embodiments of the present disclosure are described below with reference to the following drawings:

[0007] FIG. 1 is a schematic diagram showing a manipulator and a system for collision detection according to embodiments of the present disclosure;

[0008] FIG. 2 is a schematic diagram showing a series connection between a plurality of Fiber Bragg Gratings (FBGs) according to various embodiments;

[0009] FIG. 3 is a signal flow diagram showing the signal flow of an optical sensor signal to a machine learning model according to various embodiments;

[0010] FIG. 4 illustrates multiple center wavelengths and wavelength ranges of a plurality of FBGs according to various embodiments;

[0011] FIG. 5 is a process flowchart of a method of collision detection according to various embodiments;

[0012] FIG. 6 is a schematic diagram showing a manipulator and a system for collision detection according to embodiments of the present disclosure;

[0013] FIG. 7 is a schematic diagram showing a series connection between a plurality of Fiber Bragg Gratings (FBGs) according to various embodiments;

[0014] FIG. 9 shows schematically a series of time overlapping collisions according to various embodiments;

[0015] FIG. 10 is a flow chart for a method of collision detection according to various embodiments;

[0016] FIGs. 11 to 14 are images of an exemplary manipulator and system for collision detection;

[0017] FIG. 15 shows an exemplary manipulator and system for collision detection showing coupling point of a FBG sensor;

[0018] FIG. 16A is a flowchart illustrating an optical sensor signal preprocessing method according to various embodiments;

[0019] FIGs. 16B to 16E shows an exemplary optical sensor signal undergoing signal preprocessing;

[0020] FIG. 17 is a schematic of an exemplary manipulator and system for collision detection showing impact locations and the associated labeling (14 labels);[0021 J FIG. 18 shows confusion matrices corresponding to different algorithms and feature extraction processes for 14 classes of FIG. 17.

[0022] FIGs. 19A and 19B shows exemplary raw optical sensor signals of the system of FIG. 17;

[0023] FIGs. 20A to 20F shows exemplary optical sensor signals; and

[0024] FIG. 21 is a schematic diagram of a processor system.DETAILED DESCRIPTION

[0025] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0026] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0027] In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance as generally understood in the relevant technical field, e.g., within 10% of the specified value.

[0028] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0029] As used herein, terms “concurrently”, “simultaneously”, “at the same time”, or the like, may refer to events or actions that coincide or overlap within a period of time, regardlessof whether the events start at the same time instant, and regardless of whether the events end at the same time instant.

[0030] As used herein, the term “manipulator” may be used interchangeably with the terms “robotic manipulator”, “robotic arm”, “robot”, “end effector”, “linkage actuators”, “links”, etc. and may generally refer to a programmable machine configured to perform one or more tasks by manipulating objects or tools. In some examples, the “manipulator” may comprise multiple links and joints. Typically, links are rigid connecting components that connect different sections of the manipulator, while their rigidity provides a solid structure. Joints are used to provide flexibility to the manipulator, allowing the manipulator to perform desired movements within a working space. The joints may move in different ways, and may facilitate exemplary motions such as linear, rotary, and revolutionary motions.

[0031] Conventional approach towards manipulator collision detection relies on the detection of contact or impact forces via sensor-based systems or via imaging-based system using cameras or imaging devices. Such approaches arc typically limited to the amount of information provided relating to the impact.

[0032] The present disclosure comprises a system and a method of collision detection for a manipulator. Departing from conventional approaches, the proposed system and method not only detects an instance of a collision between the manipulator and an external object, but also detects a location of the collision. In various embodiments, the proposed system and method may also provide information relating to a material property of the external object and / or a contact interface (area of contact) between the manipulator and the external object, thus providing more information relating to the collision / impact allowing the determination of potential damages to the manipulator.

[0033] In various embodiments, the proposed system and method comprises an optical sensor such as a fiber optics sensor. In an exemplary embodiment, the optical sensor maycomprise a plurality of Fiber Bragg Gratings (FBGs) serially connected to one another. This enables a simple sensor implementation where the plurality of FBGs are connects in a single fiber optic, alleviating the complexity of conventional sensor lines or wiring, often present in electrical sensor systems.

[0034] By using FBGs or FBG sensors instead of other sensor types such as strain gauges, capacitor sensors, or piezoelectric sensors, the proposed system offers the benefits such as small size, corrosion resistance, and immunity to electromagnetic interference, making the FBGs well suited for embedding within a manipulator or robotic arm for collision monitoring. In addition, the small size of the FBG sensors allows easy integration into the manipulator, reducing the overall size and weight of the sensing system. Additionally, the immunity of the sensors to electromagnetic interference enables accurate and reliable measurement data even in noisy environments, such as at the joints of a robotic arm.

[0035] Further, due to the nature of the FBG sensors, multiple FBG sensors may be connected in scries on a single optical fiber, requiring only one interrogator for simultaneously measuring or receiving optical signals from all the FBG sensors on the single fiber. In contrast, other electrical sensors often require more complex wiring and multiple receivers to achieve simultaneous measurement of multiple sensors.

[0036] Referring to FIG. 1, in one aspect, the system 100 according to some embodiments of the present disclosure is configurable as a tool or apparatus for collision detection. The system 100 may include a module configured to perform a method of collision detection. The system 100 and / or method may be implementable on a dynamic actuatable structure for collision detection. In various exemplary embodiments, the system 100 and / or method may also be implementable on a generally static structure, such as a pressure vessel, for collision detection.

[0037] Referring to FIGs. 1 to 3s, in various embodiments, the system 100 and method may be implemented with a manipulator 200, such as a robotic arm, for collision detection. Therefore, the system 100 may be integrated into a manipulator system comprising a manipulator 200 and the system 100. According to various embodiments, the manipulator 200 may include a plurality of links 210 and a plurality of joints 220 coupled between each pair of links 210. The manipulator 200 may include an end effector 230 disposed on a distal end of the manipulator 200. As an example, the end effector 230 may be a gripper as illustrated in FIG. 1.

[0038] According to various embodiments, the system 100 may include an optical sensor 110 comprising a plurality of serially connected Fiber Bragg Gratings (FBGs) 120 or FBG sensors. Hence, the plurality of FBGs 120 may be connected serially along a common optical fibre. The optical sensor 1 10 may be in optical connection with an interrogator 130 which is further coupled to a processor or controller 140. In various embodiments, the plurality of FBGs 120 may be coupled to respective external surfaces or internal surfacese of the manipulator 200. The plurality of FBGs 120 may be distributed spaced apart from one another on the manipulator 200. In an exemplary embodiment, the plurality of FBGs 120 may be coupled to respective ones of the plurality of joints 220, hence allowing the plurality of FBGs 120 may be distributed spaced apart from one another. In alternative embodiments, the plurality of FBGs 120 may be connected parallelly along a plurality of parallel optical fibers. The plurality of parallel optical fibers may each comprise one or more FBGs. The plurality of parallel optical fibers may be in optical connection with the interrogator 130. In some embodiments, the plurality of parallel optical fibers may be further connected to a common optical fibre, and the common optical fibre is in optical connection with the interrogator 130.

[0039] In some embodiments, each the plurality of FBGs 120 may be coupled to a respective internal surface of the manipulator 200, such as within respective joint housings (not shown) of the manipulator 200.

[0040] In various embodiments, the system 100 may be configured to detect a collision 90 with an external object 80. In various embodiments, the system 100 may be configured to perform a method of collision detection. Further referring to FIG. 2, in the course of collision detection, the interrogator 130 may provide an optical source signal 112 to the optical sensor 110. The optical source signal 112 may pass through the optical sensor 110 and may form an optical sensor signal 1 14 to be received by the interrogator 130. As the optical sensor 1 10 comprises the plurality of FBGs connected in series, the interrogator 130 provides a single optical source signal 112 and receives a single optical sensor signal 114 at any one instant. The interrogator 130 of the system 100 may continuously provide the optical source signal 112 and receive the optical sensor signal 114. In various embodiments, the single optical sensor signal 1 14 may correspond to a vibration signal of the manipulator 200.

[0041] Referring to FIGs. 2 and 3, in various embodiments, responsive to the interrogator 130 receiving a single optical sensor signal 114 from the plurality of Fiber Bragg Gratings (FBGs) 120, the processor 140 of the system 100 may determine a time-domain optical sensor signal 115 from the single optical sensor signal 114 and determine a frequency -domain optical sensor signal 1 16 from the single optical sensor signal. Further, the processor 140 may determine at least one time-based feature 117 from the time-domain optical sensor signal 115. The at least one time -based feature 117 may include features computed and / or obtained from the time-domain optical sensor signal 115, such as a max value or a min value. Further, the processor 140 may determine at least one frequency-based feature 118 from the frequencydomain optical sensor signal 1 16. The at least one frequency-based feature 1 18 may include features computed and / or obtained from the frequency-domain optical sensor signal 116, such as a Crest indicator or a standard deviation. It may be understood that the processor 140 may determine at least one time-based feature 117 and at least one frequency-based feature 118 from the single optical sensor signal 114.

[0042] In various embodiments, the at least one time -based feature 117 may comprise at least one or more of: Max value, Mean value, Min value, Peak, to Peak, Variance, Clearance indicator, Crest indicator, Standard deviation, Mean square root, Skewness, and Kurtosis. In various embodiments, the at least one frequency -based feature 118 may comprise at least one or more of: Mean value, Center frequency. Mean square root, Variance, and Kurtosis. As it may be appreciated, the number of FBGs in the optical sensor 1 10 is not limited to the present exemplary embodiment. For example, the optical sensor 110 may include three FBGs to twenty FBGs. In other examples, the optical sensor 110 may include a range of two FBGs to a hundred FBGs.

[0043] In various embodiments, the processor 140 may provide at least one time -based feature 1 17 and at least one frequency-based feature 1 18 to a machine learning model 150. The processor 140 may further determine a characteristic of a collision 90 of the manipulator 200 with an external object 80 from an output 152 of the machine learning model 150. In various embodiments, the characteristic of the collision 90 may include an instance of a collision 90 of the manipulator 200 and a location of the collision on the manipulator 200, such as at location on linkage 210a. In various embodiments, the characteristic of the collision 90 may also comprise a direction of the collision 90 with the external object 80.

[0044] In various embodiments, the machine learning model 150 may be trained based on a collision dataset comprising various collision scenarios and collision characteristics. The collision dataset may include one or more time-based feature(s) 117 and one or more frequencybased feature(s) 1 18. In some embodiments, collision dataset may include raw collision data, and the time-based feature(s) 117 and frequency-based feature(s) 118 may be obtained from the raw collision data prior to training. In an exemplary embodiment, the collision dataset may include collision data between the manipulator 200 and a variety of external objects with different material properties, such as stiffness, hardness, strength, etc. In other exemplaryembodiment, the collision dataset may include collision data between the manipulator 200 and a variety of external objects with different geometries or shapes. In yet other exemplary embodiment, the collision dataset may include collision direction data comprising the manipulator 200 in collision with external objects with different impact directions. In yet other exemplary embodiment, the collision dataset may include collisions with impact locations away from the optical sensor 1 10, such as collisions between a robotic gripper finger and an external object.

[0045] As exemplary embodiments, the machine learning model 150 may include one or more of: a support vector machine (SVM), a convolution neural network (CNN), a decision tree model, a random forest model, etc.

[0046] In various embodiments, the single optical sensor signal 1 14, which comprises the time-domain optical sensor signal 115 and the frequency-domain optical sensor signal 116, may be processed prior to determining the respective time-based feature(s) 117 and the respective frequency- based fcaturc(s) 118.

[0047] In various embodiments, the single optical sensor signal 114 may be filtered based on a cutoff frequency (Fc), wherein the cutoff frequency corresponds to a signal shift due to a temperature change. In an example, a Butterworth high-pass filter may be used. In various embodiments, the sampling frequency (Fs) of the single optical sensor signal 114 may be set to 20 kHz and the cutoff frequency (Fc) of the highpass filter may be set to 10 Hz.

[0048] This filters out or removes the wavelength shift of the plurality of FBGs due to the temperature change. A reason is due to the wavelength shift caused by temperature change being very slow and hence low frequency, in comparison to frequency shifts due to vibration or collision.

[0049] In various embodiments, the single optical sensor signal 114 may be normalized using a normalizing frequency (Fn). The normalizing frequency may be determined based on aNyquist frequency of the plurality of FBGs 120. In an example, the normalizing frequency (Fn) may be set as the Nyquist frequency. In another example, the normalizing frequency (Fn) may be set to half of the sampling frequency (Fs). In various embodiments, the single optical sensor signal 114 may be normalized into a range of 0 to 1 by dividing the single optical sensor signal 114 by half of the sampling frequency (Fs / 2 = 10 kHz).

[0050] In various embodiments, the single optical sensor signal 1 14 may be denoised using a wavelet-based denoising method.

[0051] Referring to FIGs. 2 and 4, in an exemplary non-limiting embodiment, the optical sensor 110 may comprise five FBGs 120a / 120b / 120c / 120d / 120e defining a plurality of center wavelenths F1 / F2 / F3 / F4 / F5. Hence, each of the FBGs 120a / 120b / 120c / 120d / 120e may define a respective center wavelengths F1 / F2 / F3 / F4 / F5. Each of the plurality of center wavelengths F1 / F2 / F3 / F4 / F5 / F6 are unique from one another.

[0052] In various embodiments, the plurality of center wavelenths F1 / F2 / F3 / F4 / F5 may define a plurality of wavelength ranges R1 / R2 / R3 / R4 / R5. Hence, each of the center wavelengths F1 / F2 / F3 / F4 / F5 may define a respective wavelength range R1 / R2 / R3 / R4 / R5. The plurality of wavelength ranges R 1 / R2 / R3 / R4 / R5 are non-overlapping with one another. In other words, adjacent pairs of wavelength ranges e.g. R1 / R2 or R3 / R4 are non-overlapping with one another. As it may be appreciated, the number of FBGs in the optical sensor 110 is not limited to the present exemplary embodiment. For example, the optical sensor 110 may include three FBGs to twenty FBGs. In other examples, the optical sensor 110 may include a range of two FBGs to a hundred FBGs.

[0053] In various embodiments, each pair of the plurality of center wavelengths F1&F2 / F2&F3 / F3&F4 / F4&F5 are spaced apart from one another by a respective wavelength spacing S12 / S23 / S34 / S45. The wavelength spacings S12 / S23 / S34 / S45 are determined basedon the optical interrogator 130. The wavelength spacings S12 / S23 / S34 / S45 may be uniform from one another.

[0054] FIG. 5 illustrate a workflow of the manipulator 200 comprising the system 100 for collision detection. During manipulator operation 510, the system 100 performs collision monitoring 520 continuously. The process of collision monitoring 520 comprises continuously receiving the single optical sensor signal 1 14 and processing the single optical sensor signal 114 to obtain at least one time-based feature 117 and at least one frequency-based feature 118. The system 100 may continuously input the at least one time -based feature 117 and the at least one frequency-based feature 118 to the machine learning model 530. The machine learning model 530 may be pre-trained based on a database comprising a plurality of impact training data.

[0055] Responsive to impact detection 540 or determining the manipulator has experienced an impact / collision 90, the system 100 may send an emergency stop signal to the manipulator 200 to stop all current action(s) of the manipulator 200. Thereafter, the system 100 performs impact localization 550 to determine a location of the impact / collision 90 on the manipulator 200. Based on predetermined rules, the system 100 determines whether to pause 555 all manipulator actions to resolve the collision or to resume 557 manipulator actions.

[0056] FIGs. 6 and 7 illustrate another system 100 for collision detection implemented on a manipulator 200, according to various embodiments. The manipulator 200 may comprise a plurality of links 210, a plurality of joints 220 coupled between each pair of links 210, and an end effector 230 disposed on a distal end of the manipulator 200.

[0057] According to various embodiments, the system 100 may include an optical sensor 110 comprising a plurality of serially connected Fiber Bragg Gratings (FBGs) 120. The optical sensor 110 may be in optical connection with an interrogator 130 which is further coupled to a processor or processor 140. The plurality of FBGs 120 may be distributed spaced apart fromone another on the manipulator 200. Referring to FIG. 7, in various embodiments, the plurality of FBGs 120 may be coupled along a coupling length 240 of manipulator 200.

[0058] As shown in FIG. 7, the link 210a and the end effector 230 may be disposed on a noncoupling length 250, the non-coupling length 250 located away from the coupling length 240. In various embodiments, the system 100 may determine or detect a collision 90 with a respective location at the non-coupling length 250, i.e. located away from the coupling length 240 of the manipulator 200. This enables collision(s) which are away from the optical sensor 110 to be detected by the system 100.

[0059] In various embodiments, by training the machine learning model using collision data with different external object(s) of different material properties, the system 100 may determine a material property, such as a rigidity or a stiffness, of the external object 80 based on the output of the machine learning model.

[0060] In various embodiments, by training the machine learning model using collision data with different contact interfaces or different types of contact (c.g. point contact, line contact, surface contact), the system 100 may determine a contact interface (e.g. point contact, line contact, surface contact) between the external object 80 and the manipulator 200 based on the output of the machine learning model.

[0061] In various embodiments, the system 100 may determine a surface contour of the external object 80 in collision with the manipulator 200 based on the output of the machine learning model. In an exemplary embodiment shown in FIG. 7, the surface contour may be a sharp and pointed surface in collision with the manipulator 200. In other exemplary embodiments, the surface contour may be a blunt and rounded surface.

[0062] The system 100 utilizes FBGs 120 for measuring vibration signals of the manipulator200 or robotic arm. When an external object collides with or impacts the robotic arm, a high- frequency vibration signal may be generated. The frequency and amplitude of the "collisionsignal" is often larger / higher than that of the "inherent vibration signal" produced during the normal operation of the robotic arm. As such, the amplitude and frequency of the collision signal will vary depending on the colliding object. For instance, collisions with hard objects such as steel plates or stones produces a higher-frequency collision signals, in comparison to collisions with relatively softer objects such as wood and plastic, which in turn produces a lower-frequency signals. In similar sense, collisions with sharp objects and blunt objects also result in different vibration signals received by the FBGs on the manipulator 200. With the use of systematic training and data collection, and by analyzing the characteristics of the signals, the trained machine learning model may determine the external object in collision with the manipulator 200.

[0063] Referring to FIGs. 8 and 9, in various embodiments, the characteristic of the collision 90 may comprise a series of time overlapping collisions 90a / 90b of the manipulator 200. The series of time overlapping collisions 90a / 90b may correspond to a plurality of respective collision locations L1 / L2 on the manipulator 200. FIG. 9 shows an example of scries of time overlapping (Tai to Ta2 overlapping with Tbl to Tb2) collisions 90a / 90b of the manipulator 200. The series of time overlapping collisions 90a / 90b corresponds to respective collision locations L1 / L2.

[0064] FIG. 10 is a flowchart illustrating a method 700 of collision detection according to various embodiments. The method 700 of collision detection may include in frame 710: receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings (FBGs), the plurality of FBGs being coupled to a manipulator, the plurality of FBGs distributed spaced apart from one another on the manipulator; in frame 720: determining at least one time-based feature from the single optical sensor signal; in frame 730: determining at least one frequency-based feature from the single optical sensor signal; in frame 740: providing the at least one time-based feature and the at least one frequency -based feature to a machine learning model; and in frame750: determining from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.

[0065] In various embodiments, the method 700 may further comprise determining a material property of the external object based on the output of the machine learning model. In various embodiments, the method 700 may further comprise determining a contact interface between the external object and the manipulator based on the output of the machine learning model. In various embodiments, the method 700 may further comprise filtering the single optical sensor signal based on a cutoff frequency, wherein the cutoff frequency corresponds to a signal shift due to a temperature change. In various embodiments, the method 700 may further comprise normalizing the single optical sensor signal using a normalizing frequency, the normalizing frequency determined based on a Nyquist frequency of the plurality of FBGs. In various embodiments, the method 700 may further comprise denoising the single optical sensor signal using a wavelet-based denoising method.

[0066] In various embodiments, the method 700 may further comprise determining at least one time-based feature comprises: determining a time-domain optical sensor signal from the single optical sensor signal; and from the time-domain optical sensor signal, determining at least one or more of: Max value; Mean value; Min value; Peak to Peak; Variance; Clearance indicator; Crest indicator; Standard deviation; Mean square root; Skewness; and Kurtosis.

[0067] In various embodiments, the method 700 may further comprise determining at least one frequency-based feature comprises: determining a frequency -domain optical sensor signal from the single optical sensor signal; and from the frequency-domain optical sensor signal, determining at least one or more of: Mean value; Center frequency; Mean square root; Variance; and Kurtosis.

[0068] Exemplary Implementation

[0069] FIGs. 11 to 14 shows an exemplary implementation of the proposed system for collision detection on a robotic arm. The proposed system utilizes optical signals corresponding to movement and / or vibration signals for monitoring collisions and the respective collision locations on a manipulator or a robotic arm. Departing from conventional force sensors or imaging sensors, the proposed system employs distributed Fiber Bragg Grating vibration sensors (or FBGs) positioned at multiple points at various locations along the robotic arm. By way of analyzing the response characteristics of these FBG sensors and integrating machine learning models and / or frameworks, the proposed system enables precise determination of the locations of collisions based as well as other characteristics of the collision.

[0070] Additionally, as the FBG sensors are mounted on the respective covers of the robotic arm’s joints, the FBG sensors do not interfere with the original electrical circuits of the control board. This feature allows the proposed system to act as an “add-on” option for existing robotic arms, without the need to modify or replace the original electrical components, thus allowing the system to act or function as a “plug and play” add-on feature.

[0071] During the data processing phase, features extracted from both the time-domain optical sensor signal and frequency-domain optical sensor signal of the impact / collision signal combined into a feature set may be used for training the machine learning model. The process extracts various characteristics from the impact signals, improving the accuracy of detection, while also reducing the volume of data, thus facilitating faster decision-making. Furthermore, a visualization interface for the collision monitoring system was developed to enable real-time detection and display of collision locations. The proposed system represents a significant improvement over alternative monitoring methods and can contribute to the safe and effective operation of robotic arms in industrial settings.

[0072] Referring again to FIG. 5, illustrates an overall process of the system for collision detection implemented on a robotic arm. The system beings with collecting vibration signalsfrom the FBG sensors disposed at different locations on the robotic arm. The signals are preprocessed before feature extraction. The extracted features are then used as inputs to a trained machine learning model which predicts an instance of a collision and a location of the collision. Responsive to detecting a collision, the system applies instantaneous braking to the robotic arm in order to prevent further damage, which is a typical safety feature in robotic aims. However, in addition to collision detection and departing from conventional systems, the proposed system is able to identify the location(s) of the collision.

[0073] In addition, the proposed system may utilize the trained machine learning model or machine learning framework to analyze the characteristics of the collision and in turn improving the training model. During the training stage, by training the machine learning model with a large dataset of collision signals, the system is able to analyze other characteristics of the collision, such as the intensity of the collision, the type or material of external object(s) involved in the collision, for example, whether the external object is made of metal or wood, or is a human subject.

[0074] Characteristics of the collision, including the predicted collision location and other relevant information, may be useful information for a robotic controller or robotic planner to determine the robotic arm's path. This aids in further preventing / mitigating potentially damaging collisions of the robotic arm, thus reducing the risk to technicians / engineers during maintenance, and improving the overall safety and efficiency of the robotic arm operation.

[0075] System development

[0076] The system development and machine learning training process may be categorized into the following phases:

[0077] Data Collection: In the data collection phase, vibration sensor signal data are collected from the FBG sensors located at various positions on the robotic arm. This data may include detailed records of vibration amplitude, sampling frequency, and the specific locations ofcollisions. This information may be used for subsequent analyses and model training as it provides the foundational data necessary for identifying collision patterns and characteristics.

[0078] Data Pre-processing: Prior to the training of the machine learning model, the data collected undergoes a preprocessing phase, which includes: cleaning the data, such as removing erroneous records or filling in missing data values, and feature selection, which involves selecting the relevant or useful data features.

[0079] Feature Extraction: In addition, feature extraction is performed to extract features from the processed data. The feature extraction involves the application of various signal processing techniques and algorithms to extract key features from the raw data that help identify and predict collisions, such as time series analysis and frequency transformation. Feature extraction is performed prior to input to the machine learning model such that only critical information is retained as input to the machine learning model, thus reducing computational load during collision detection and improving data efficiency.

[0080] Model Selection: In this phase, a suitable machine learning algorithm or framework is selected for constructing the machine learning model for collision detection. As examples, the machine learning model may include exemplary models such as: Support Vector Machines (SVM), neural networks, decision trees, and random forests.

[0081] Training Phase: Upon determining or selecting the suitable machine learning algorithm or framework, the previously pre-processed and feature-enhanced dataset may be used to train the machine learning model. During training, the machine learning model attempts to learn and understand the complex relationships between different vibration signal characteristics and the collision characteristics (such as: locations, material, geometry) to improve collision detection accuracy and determining of collision characteristics.

[0082] Model Evaluation and Adjustment Phase: Upon the machine learning model is sufficiently trained, the next phase involves the evaluation of the machine learning modelperformance. This phase is typically performed utilizing machine learning statistical metrics such as cross-validation. Receiver Operating Characteristic (ROC) curves, accuracy, and recall rates. These evaluation metrics aids in uncovering and understanding the performance of the model in real- world applications and may also point out possible directions for improvement. If the performances of the machine learning do not meet expectations, adjustments may be made to the machine learning model and / or the feature extraction step, such as reselecting features for input to the machine learning model and / or tweaking algorithm parameters, such as hyperparameters .

[0083] Prediction Phase: After model evaluation, the machine learning model is ready to be used for collision detection. In this stage, newly collected vibration signal data act as input into the trained machine learning model, which utilizes the learned data patterns to predict the instance and the location of collision(s). The machine learning model may also provide other characteristics of the collision. The proposed system enables the monitoring and identifying of collision events in real-time, thus enhancing operational safety and efficiency.

[0084] System Description

[0085] The proposed system for collision detection utilizes multiple FBG sensors with different center wavelengths placed at different locations on a robotic arm, thus enabling comprehensive vibration monitoring. The FBG sensors may be connected by a single optical fiber, allowing simultaneous collection of responses from each FBG sensor with a single integrator.

[0086] FIG. 15 illustrates a schematic of the robotic arm, with boxes representing the installed FBG sensors. Six FBG sensors were installed and attached to 3D-printed covers using UV resin. During a collision, the impact signal is transmitted along the structure of the robotic arm as vibrations to various sensors. The response of the six sensors attached to different positions to the same vibration signal will differ depending on the collision / impact location. This difference includes variousfeatures such as vibration amplitude, frequency, waveform, and among others. Based on these features, based on analysis of the collision signals' characteristics in different segments and a trained machine learning model may be established. During application, with the FBG sensor responses as input, the trained machine learning model may detect the collision and collision characteristics, such as impact location.

[0087] FIG. 16A is a flow chart of an exemplary data processing workflow. As a first step of the data processing, a Butterworth high-pass filter may be applied to the optical sensor signal. The sampling frequency (Fs), which is the frequency used for data collection, may be set to 20 kHz. The cutoff frequency (Fc) of the high-pass filter may be set to 10 Hz. The data is normalized into a range of 0 to 1 by dividing by half of the sampling frequency (Fs / 2 = 10 kHz). This is done using the “scipy. signal butter” function which uses a normalized cutoff frequency from 0 to 1, where 1 is the Nyquist frequency. This process filters out the wavelength shift of FBGs due to the temperature change as the wavelength shift caused by temperature change is slow as compared to the shift caused by vibration or collision. Thereafter, the filtered signal undergoes wavelet signal denoising in removing noise in the signal. The signal is then converted into a time-domain optical sensor signal and a frequency-domain optical sensor signal, and feature extraction is performed on the time-domain optical sensor signal and / or the frequencydomain optical sensor signal.[OO88] FIGs. 16B to 16E shows the various exemplary data during preprocessing. FIG. 16B shows an exemplary FBG optical sensor signal collected by the interrogator during an impact event. It may be seen that there is a notifiable impact signal that can be observed at a time around 8.95 s. FIG. 16C shows the spectrogram of the optical signal presented in FIG. 16B. By comparing the spectrogram and FIG. 16B, it may be seen that the main vibration frequency of the impact signal is in a specific range.

[0089] A wavelet signal denoising process is applied to the filtered signal to further reduce the noise effect. The process may include using a wavelet-based denoising approach that employs Bayesian estimation and median thresholding to denoise the vibration signal at a 6- level of decomposition using the symmetric 5-tap wavelet. FIG. 16D shows the processed vibration signal. It may be seen that the main impact signal was extracted from the noise signal. In addition, FIG. 16E shows the Fast Fourier Transform (FFT) of the processed signal, with the FFT plot demonstrating the frequency characteristics of the impact signal being effectively preserved after noise reduction.

[0090] Feature Extraction

[0091] Various features may be extracted from the frequency-domain optical sensor signal and the time-domain optical sensor signal for input to the machine learning model. Performing feature extraction prior to instead of inputting raw optical sensor signal directly to the machine learning model improves collision detection performance. Training the machine learning or deep learning model directly using raw signals often yields a poor prediction result, due to the high data rate and information redundancy. Through the feature extraction process, the most discriminated characteristics of the signal may be identified and act as input to the machine learning model.

[0092] In the proposed system, both frequency-domain and time-domain optical sensor signals are used for extracting features of the impact signals. Examples of the features extracted are listed in Table 1, with a total of 11 features from the time-domain signal and 5 features from the frequency-domain signal.

[0093] The max value, mean value, min value in Table 1 are determined as: Eq. (1) Eq. (2) Eq. (3)where the x represents that value of the signal from both the “Time domain” and “Frequency domain”.

[0094] The peak-to-peak value is defined as:

[0095] Variance is a statistical measure that represents the dispersion or spread of a set of data points around their mean (average) value. It quantifies how much the numbers in the dataset deviate from the mean. Variance gives you an idea of the spread or scatter of your data, indicating whether the data points are closely clustered around the mean or spread out over a wide range, the variance is calculated as:where, x, represents each value of single,is the mean value, and V is the total number of values in the single.

[0096] The clearance indicator is a diagnostic tool commonly used in the analysis of vibration signals from rotating machinery. It helps to identify the presence of specific faultconditions, such as bearing clearances, by analyzing the vibration signal of a machine component. This is particularly useful in predictive maintenance and condition monitoring applications, where early detection of potential failures can prevent costly downtime and damage, where clearance indicator is calculated as:

[0097] Crest indicator is defined as:

[0098] Skewness is a statistical measure that describes the asymmetry of a distribution around its mean in any given dataset, ft provides an insight into the shape of the distribution of data points. Skewness can be positive, negative, or zero, which indicates the direction and degree of asymmetry of the distribution. The skewness is calculated as:where S is the standard deviation of the dataset.

[0099] Kurtosis is a descriptive statistic used to help measure how data disperse between a distribution's center and tails.where, N is the total number of observations in the dataset, xi represents each individual observation in the dataset, p is the mean of the signal.

[0100] Experimental Results[00101J FIG. 17 illustrates the division of regions and corresponding labels during the training of the machine learning model. The trained machine model may determine which region the collision occurs in based on the vibration signals collected by the sensors. At this stage, the robotic arm is divided into 13 regions, namely the 6 joints and the 7 regions on the "arm". These regions basically cover the entire range of the mechanical arm and are evenly distributed throughout its structure.

[0102] Table 2 shows the comparison between different algorithms and different feature extraction methods: TD indicates that only features extracted from the time domain are used; FS indicates that only features extracted from the frequency domain are used; and TD+FS indicates that both sets of features (a total of 16) are used simultaneously.Table 2: Overall Performance Evaluations of ML Models and Feature Extraction ProcessesFor 14 Classes.where TP is “true positive”, TN is “true negative”, FP is “false positive”; and FN is “false negative”.

[0103] FIG. 18 shows the corresponding prediction results, which show that the predictions made using TD+FS data and ANN are the best performing. Overall, the proposed collisiondetection method can effectively identify the location of collisions. Additionally, cases where no collision actually occurred (label 0 - No impact) were also included in the testing, and the system was able to distinguish between collisions and non-collisions.

[0104] FIGs. 19A and 19B shows two sets of vibration data collected by the 6 FBG sensors. The data clearly shows that when the impact happened at different locations of the robotic arm, and the vibration patterns of the 6 FBG sensors are different. By analyzing the data and applying it to the machine learning model, the collision location may be determined. FIGs. 20A to 20F shows exemplary optical sensor signal for the FBG sensors.

[0105] The proposed manipulator and the proposed system for performing a method of collision detection may be implemented by a processor system 900 as illustrated in the schematic block diagram of FIG. 21. Components of the processing system 900 may be provided within one or more computing device to carry out the functions of the modules or any other modules. One skilled in the art will recognize that the exact configuration or arrangement illustrated in FIG. 21 is provided by way of example only, c.g., each processing system provided may be different and the exact configuration of processing system 900 may vary.

[0106] In embodiments of the present disclosure, the processing system 900 may include a controller 901 and user interface 902. User interface 902 is configured to enable manual interactions between a user and the computing module as required. For this purpose, the processing system 900 includes the input / output components required for the user to enter instructions to provide updates to each of the modules. A person skilled in the art will recognize that components of user interface 902 may vary from embodiment to embodiment but may typically include one or more input devices 935 such as but not limited to a touchscreen, a keyboard, a joystick, a mouse, a microphone, etc. The user interface 902 also includes EEG sensors 933 that can be attached to the user’s head to sense the user’s brain activity. The userinterface 902 can also include a media player 940, which can be in the form of one or more playback devices, including but not limited to a display, a speaker, earphones, headsets, etc.

[0107] The controller 901 is configured to be in data communication with the user interface 902 via bus 915. The controller 901 includes memory 920 and processor 905 mounted on a circuit board to process instructions and data, e.g.. to perform the method of the present disclosure. The controller 901 includes an operating system 906, an input / output (I / O) interface 930 for communicating with user interface 902, and a communications interface, e.g., a network card 950. The network card 950 may, for example, be configured to send data from the controller 901 via a wired or wireless network to other processing devices or to receive data via the wired or wireless network. Wireless networks that may be utilized by the network card 950 include, but are not limited to, Wireless-Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks, Wide Area Networks (WAN), and etc.

[0108] Memory 920 and operating system 906 arc in data communication with central processing unit (CPU) 905 via bus 910. The memory 920 may include both volatile and nonvolatile memory. The memory 920 may include more than one of each type of memory, e.g., Random Access Memory (RAM) 923, Read Only Memory (ROM) 925, and a mass storage device 945. The mass storage device 945 may include one or more solid-state drives (SSDs). One skilled in the art will recognize that the memory described above includes non-transitory computer-readable media and shall be taken to include all computer-readable media except for a transitory, propagating signal. Typically, instructions are stored as program code in the memory but can also be hardwired. Memory 920 may include a kernel and / or programming modules such as a software application that may be stored in either volatile or non-volatile memory.[00109 J Herein, the term “processor” is used to refer generically to any device or component that can process computer-readable instructions, including for example, a microprocessor, microcontroller, programmable logic device, or other computational device. That is, processor 905 may be provided by any suitable logic circuitry for receiving inputs, processing them in accordance with instructions stored in memory, and generating outputs (for example to the memory components or media player 940). In the present disclosure, processor 905 may be a single core or multi-core processor with memory addressable space. In one example, processor 905 may be multi-core, comprising — for example — an 8 core CPU. In another example, it could be a cluster of CPU cores operating in parallel to accelerate computations.

[0110] Further, one skilled in the art will recognize that certain functional units in this description have been labelled as modules throughout the specification. The person skilled in the art will also recognize that a module may be implemented as circuits, logic chips or any sort of discrete component. Still further, one skilled in the art will also recognize that a module may be implemented in software which may then be executed by a variety of processor architectures. In embodiments of the disclosure, a module may also comprise computer instructions or executable code that may instruct a computer processor to carry out a sequence of events based on instructions received. In further embodiments, the module may comprise a combination of different types of modules or sub-modules. The choice of the implementation of the modules may be determined by a person skilled in the art and does not limit the scope of the claimed subject matter in any way.[001 11] All examples described herein, whether of apparatus, methods, materials, or products, are presented for the purpose of illustration and to aid understanding, and are not intended to be limiting or exhaustive. Modifications may be made by one of ordinary skill in the art without departing from the scope of the invention as claimed.

Claims

CLAIMS1. A system, comprising: a memory storing instructions; and a processor coupled to the memory and configured to process the stored instructions to implement: a module configured to perform a method of collision detection, the method including: receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings (FBGs), the plurality of FBGs being coupled to a manipulator, the plurality of FBGs distributed spaced apart from one another on the manipulator; determining at least one time-based feature from the single optical sensor signal; determining at least one frequency-based feature from the single optical sensor signal; providing the at least one time-based feature and the at least one frequency based feature to a machine learning model; determine from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.

2. The system as recited in claim 1 , wherein the characteristic of the collision comprises: an instance of a collision of the manipulator and a location of the collision on the manipulator.

3. The system as recited in claim 2, wherein the characteristic of the collision further comprises a direction of the collision with the external object.

4. The system as recited in any one of claims 2 and 3, wherein the plurality of FBGs are coupled along a coupling length of manipulator, and wherein the location of the collision is located away from the coupling length of the manipulator.

5. The system as recited in any one of claims 2 to 4, wherein the characteristic of the collision comprises: a series of time overlapping collisions of the manipulator and a plurality of respective collision locations of the series of time overlapping collisions on the manipulator.

6. The system as recited in any one of the above claims, wherein the plurality of FBGs are serially connected along one common optical fibre.

7. The system as recited in any one of the above claims, wherein the method further comprising: determining a material property of the external object based on the output of the machine learning model.

8. The system as recited in any one of the above claims, wherein the method further comprising: determining a contact interface between the external object and the manipulator based on the output of the machine learning model.

9. The system as recited in any one of the above claims, wherein each of the plurality ofFBGs defines a respective one of a plurality of center wavelengths, wherein the plurality of center wavelengths are unique from one another.

10. The system as recited in claim 9, wherein each of the plurality of center wavelengths define a respective one of a plurality of wavelength ranges, wherein the plurality of wavelength ranges are non-overlapping with one another.

11. The system as recited in any one of the above claims, wherein the single optical sensor signal corresponds to a vibration signal of the manipulator.

12. The system as recited in any one of the above claims, wherein the method further comprising: filtering the single optical sensor signal based on a cutoff frequency, wherein the cutoff frequency corresponds to a signal shift due to a temperature change.

13. The system as recited in claim 12, wherein the method further comprising: normalizing the single optical sensor signal using a normalizing frequency, the normalizing frequency determined based on a Nyquist frequency of the plurality of FBGs.

14. The system as recited in any one of claims 12 and 13, wherein the method further comprising: denoising the single optical sensor signal using a wavelet-based denoising method.

15. The system as recited in any one of the above claims, wherein each of the plurality of FBGs is coupled to a respective internal surface of the manipulator.

16. The system as recited in any one of the above claims, wherein the plurality of FBGs are coupled to respective ones of a plurality of joints of the manipulator.

17. The system as recited in any one of the above claims, wherein the machine learning model includes at least one of: a support vector machine (SVM), a convolution neural network (CNN), a decision tree model, a random forest model.

18. The system as recited in any one of the above claims, wherein the plurality of FBGs are coupled to an internal surface of the manipulator.

19. The system as recited in any one of the above claims, wherein determining at least one time-based feature comprises: determining a time-domain optical sensor signal from the single optical sensor signal; and from the time-domain optical sensor signal, determining at least one or more of: Max value; Mean value; Min value; Peak to Peak; Variance; Clearance indicator; Crest indicator; Standard deviation; Mean square root; Skewness; and Kurtosis.

20. The system as recited in any one of the above claims, wherein determining at least one frequency-based feature comprises: determining a frequency-domain optical sensor signal from the single optical sensor signal; and from the frequency-domain optical sensor signal, determining at least one or more of: Mean value; Center frequency; Mean square root; Variance; and Kurtosis.

21. A manipulator system, comprising the system as recited in any one of the above claims.

22. A method of collision detection, the method including: receiving a single optical sensor signal from a plurality of Fiber Bragg Gratings(FBGs), the plurality of FBGs being coupled to a manipulator, the plurality of FBGs distributed spaced apart from one another on the manipulator;determining at least one time -based feature from the single optical sensor signal; determining at least one frequency-based feature from the single optical sensor signal; providing the at least one time-based feature and the at least one frequency-based feature to a machine learning model; determining from an output of the machine learning model, a characteristic of a collision of the manipulator with an external object.

Images