Method for determining health of a robot, electronic device, computer-readable storage medium and computer program

Abstract

A method for determining a health condition of a robot is involved. The method includes acquiring image data and audio data of the robot during a motion (202). The method also includes determining a velocity profile of the robot based on the image data (204). The method also includes generating an audio spectrum of the audio data based on the velocity profile (206). The method also includes determining the health condition of the robot based on the audio spectrum. In this way, the health condition diagnostic mechanism is professional, fast and cost-effective. Neither expensive / heavy equipment nor complex operation is needed for the field engineer. Moreover, there is no operation interruption. Therefore, the health condition diagnostic mechanism is an easy-to-use solution that can be used by anyone to quickly check any robot at any time.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of robotics, and more specifically, to a method for determining the health status of a robot, an electronic device, a computer-readable storage medium, and a computer program. Background Technology

[0002] Robots play a vital role in modern industry because they can work faster, more precisely, and for longer periods than humans. As more robots operate autonomously, early detection of malfunctions that could lead to performance degradation or even factory downtime becomes increasingly important to minimize the impact on user operations. Given the high costs of production downtime, numerous solutions have been developed to monitor robot health.

[0003] For example, experts can go to the site to check the noise emitted by the robot, back up the controller settings, and record the noisy robot for further manual inspection. Alternatively, field engineers can upload the robot's motion data from the controller to a cloud / edge server for expert analysis. Summary of the Invention

[0004] In view of the above problems, exemplary embodiments of this disclosure propose a solution for health status diagnosis based on robot video.

[0005] In a first aspect of this disclosure, an exemplary embodiment provides a method for determining the health status of a robot. The method includes: acquiring video data during robot movement using a computing device. Here, the video data includes image data and audio data. The method further includes: determining a velocity profile of the robot based on the image data using the computing device. The method further includes: generating an audio spectrum of the audio data based on the velocity profile using the computing device. The method further includes: determining the robot's health status based on the audio spectrum using the computing device.

[0006] In a second aspect, an exemplary embodiment provides an electronic device. The electronic device includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform the method according to the first aspect of this disclosure.

[0007] In a third aspect, an exemplary embodiment provides a computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method according to the first aspect of this disclosure.

[0008] In a fourth aspect, an exemplary embodiment provides a computer program including instructions that, when executed by a computer, cause the computer to perform the method according to the first aspect of this disclosure. Attached Figure Description

[0009] The above and other objects, features, and advantages of the exemplary embodiments disclosed herein will become more readily understood from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several exemplary embodiments disclosed herein will be described by way of example and not limitation, in which:

[0010] Figure 1 A block diagram of a robot diagnostic system that can implement exemplary embodiments of the present disclosure is shown schematically;

[0011] Figure 2 A flowchart illustrating a method for determining the health status of a robot according to an embodiment of the present disclosure is shown schematically.

[0012] Figure 3 A flowchart illustrating a method for determining a robot velocity profile according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 4A A schematic diagram illustrating an exemplary process for determining a velocity profile using a machine learning model according to some embodiments of the present disclosure is shown.

[0014] Figure 4B A schematic diagram illustrating an exemplary process for training a machine learning model according to some embodiments of the present disclosure is shown.

[0015] Figure 5A A flowchart illustrating a method for generating an audio spectrum according to an embodiment of the present disclosure is shown schematically;

[0016] Figure 5B A flowchart illustrating a method for resampling audio data according to an embodiment of the present disclosure is shown schematically.

[0017] Figure 6 A flowchart illustrating a method for determining the health status of a robot according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 7 A schematic diagram illustrating an exemplary process for determining the health status of a robot according to some further embodiments of the present disclosure; and

[0019] Figure 8 A schematic diagram of an electronic device for implementing the method according to an embodiment of the present disclosure is shown.

[0020] In all the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0021] The principles of this disclosure will now be described with reference to some exemplary embodiments. It should be understood that these descriptions of embodiments are for illustrative purposes only and to assist those skilled in the art in understanding and implementing this disclosure, and do not imply any limitation on the scope of this disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0022] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0023] References to "an embodiment," "embodiment," "exemplary embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment must include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that those skilled in the art will recognize how such features, structures, or characteristics can be incorporated into other embodiments, whether explicitly described or not.

[0024] It should be understood that although the terms “first” and “second” may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “containing,” “comprise,” and / or “containing,” when used herein, specify the presence of the stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0026] As mentioned above, the usual practice is to send experts to the site and record video of the noisy robot. The experts then need to manually analyze the video based on their personal experience. However, because this requires the expert's specific expertise, less experienced engineers cannot automate the diagnostic process.

[0027] In view of the above, a mechanism for diagnosing robot health status via a terminal device is provided. In robot health status diagnosis, audio and image data contained in video data during robot movement are collected simultaneously. A velocity profile is determined based on the video of the robot's movement and used to calibrate the audio data, thereby generating an audio spectrum for determining the health status. The terminal device can automatically determine the health status by interpreting the audio spectrum.

[0028] In embodiments of this disclosure, the diagnostics are performed automatically by the terminal device. In the illustrated embodiments, operational data from the robot or controller cabinet is not required. Therefore, any mobile device can be used to perform robot health checks without the need for specially designed equipment. Furthermore, the audio spectral reference velocity is calibrated, allowing for automatic interpretation without expert input. Thus, the health check mechanism according to this disclosure is professional, fast, and cost-effective. Field engineers do not require expensive / bulky equipment or complex operations. Moreover, there is no operational interruption. Therefore, the health check mechanism is an easy-to-use solution that allows anyone to quickly check any robot at any time.

[0029] Reference Figures 1 to 8 Describes the framework according to embodiments of this disclosure. Figure 1 A block diagram schematically illustrates a robot diagnostic system 100 that can implement exemplary embodiments of the present disclosure. Figure 1 As shown, the robot diagnostic system 100 includes a robot 110. The robot 110 includes three joints or axes: a first joint 111, a second joint 112, and a third joint 113. The robot diagnostic system 100 also includes a controller cabinet 120 connected to the robot 110. The three joints of the robot 110 can be collaboratively driven by the controller cabinet 120 to perform specific operations.

[0030] The robot diagnostic system 100 also includes a computing device 130, illustrated as a terminal device for a field user. The computing device 130 includes a camera 131 and a microphone 132. During operation, the robot 110 can move along a pre-programmed trajectory and will inevitably emit sound. In this case, the computing device 130 can record video of the robot 110's movement using the camera 131 and record the sound emitted by the robot 110 using the microphone 132. This yields video data 140. The video data 140 includes video image data 141 captured by the camera 131 and audio data 142 recorded by the microphone 132. After obtaining the video data 140 during the robot 110's operation, the computing device 130 can process the video data 140 to normalize it, making the normalized data available for further analysis. For example, the video data 140 can be used to determine the health status of three joints of the robot 110. The computing device 130 can process the video data 140 to obtain a health status diagnostic result 150.

[0031] It should be understood that although computing device 130 is illustrated as a terminal device, such as a smartphone, computing device 130 may also include tablet computers, laptop computers, and similar devices capable of recording and providing computing resources at the same time.

[0032] Alternatively, the robot diagnostic system 100 may also include a monitoring camera 160. The monitoring camera 160 may be installed within the facility for security purposes and is capable of recording video / audio data. In some example embodiments, the monitoring camera 160 may continuously record sounds emitted by the robot 110 during operation and record video of the robot 110's motion. In these embodiments, video data 140 is acquired by the monitoring camera 160 and may be transmitted to the computing device 130 in real time or later (e.g., in response to a retrieval request from the computing device 130). In this case, the data acquisition function is provided by the monitoring camera 160, while the computing device 130 only provides computing resources. Since the data acquisition device is on-site, the computing device can be located on-site or remotely. Health status diagnoses can be performed anytime, anywhere.

[0033] In the illustrated embodiment, the computing device 130 is not connected to the controller cabinet 120 and can perform health status diagnoses without requiring specific operational data from the controller cabinet 120 or the robot 110. Furthermore, the video data 140 can be comprehensively processed and normalized, enabling automatic diagnosis based on reference data without expert intervention. The following will refer to… Figure 2-7 Describe in detail the health condition diagnosis mechanism.

[0034] Figure 2 A flowchart illustrating a method 200 for determining the health status of a robot according to an embodiment of this disclosure is shown schematically. For discussion purposes, reference will be made to... Figure 1 Describe method 200. For example, method 200 may be derived from... Figure 1 The computing device 130 is implemented in the middle.

[0035] At point 202, the computing device acquires video data during the robot's movement. In this case, the video data includes both image and audio data. For example, in Figure 1 In the illustrated embodiment, computing device 130 acquires image data 141 and audio data 142 of robot 110 during operation. In some example embodiments, image data 141 and audio data 142 may be recorded by computing device 130. In some alternative embodiments, image data 141 and audio data 142 may be recorded by an external imaging device (e.g., surveillance camera 160) and transmitted to computing device 130.

[0036] At position 204, the computing device determines the robot's velocity profile based on the image data. For example, in Figure 1 In the illustrated embodiment, computing device 130 determines the robot's velocity profile based on image data. In this case, the velocity profile may include the rotational speeds of all robot axes. With the rapid development of artificial intelligence (AI) technology, image processing and labeling can provide technical support for the identification and localization of robot key points (e.g., robot joints or axes). In some embodiments, to determine the velocity profile of robot 110 based on image data, computing device 130 may determine one or more sets of positions of one or more joints of the robot relative to multiple frames of the image data, based on the image data and according to a machine learning model. In this case, the machine learning model may be configured to identify the center point of the axis and determine the coordinates of the axis center point in a predefined coordinate system. After obtaining the coordinates of the axis center points in multiple frames of the video, the computing device may determine one or more velocity curves of one or more joints based on these coordinates.

[0037] In some example embodiments, the machine learning model can be trained to determine the 3D coordinates of robot joints in a camera coordinate system based on frames in a motion video. In some other embodiments, the machine learning model can be trained to determine the robot base coordinates and the 3D coordinates of robot joints based on frames in a motion video. In some still embodiments, the machine learning model can be trained to determine the robot base coordinates and the relative 3D coordinates of predefined keypoints of the robot with respect to the robot base based on frames in a motion video.

[0038] In some alternative embodiments, the machine learning model may not be trained to determine coordinates directly from the image. Instead, the machine learning model may first be trained to identify the robot's model or type. Then, the machine learning model may be trained to minimize the difference between the pose of the robot simulation model (e.g., a computer-aided design (CAD) model) and the robot image. For example, the computing device may identify the robot type based on video and a motion estimation model. The computing device then obtains a robot simulation model corresponding to that robot type. Finally, the computing device maps the simulation model to multiple frames of image data according to the motion estimation model to obtain the pose of the simulation model. Joint angles can be obtained directly from the simulation model. In this case, the angular velocity of the joints can be calculated from the obtained joint angles.

[0039] At position 206, the computing device generates the audio spectrum of the audio data based on the velocity profile. For example, in Figure 1 In the illustrated embodiment, computing device 130 generates the audio spectrum of the audio data based on the velocity profile. Typically, the velocities of the robot 110's axes are not constant during operation, and even the acceleration is not constant. In these cases, the collected audio data may not be interpretable using normalization rules. The audio data is calibrated in relation to the velocities of the robot's axes. The audio spectrum generation mechanism will be referenced below. Figures 5A-5B Detailed description.

[0040] At point 208, the computing device diagnoses the robot's health status based on the audio spectrum. For example, in Figure 1 In the illustrated embodiment, computing device 130 interprets the audio spectrum using normalized rules and determines whether each axis of the robot is healthy. For example, the calibrated audio spectrum of a normally functioning robot can be obtained. By comparing the calibrated audio spectrum of the healthy robot with the audio spectrum of the robot under test, a diagnostic result can be determined.

[0041] In the illustrated embodiment, no operational data from robot 110 or controller cabinet 120 is required. Therefore, any mobile device can be used to perform robot health checks without the need for specially designed equipment. Furthermore, the audio spectrum is calibrated with reference speed and can therefore be automatically interpreted without expert input. Thus, the health check mechanism according to this disclosure is professional, fast, and cost-effective. Field engineers do not need expensive / bulky equipment or complex operations. Moreover, there is no operational interruption. Therefore, the health check mechanism is an easy-to-use technology that allows anyone to quickly check any robot at any time.

[0042] Figure 3A flowchart of a method 300 for determining a robot velocity profile according to an embodiment of this disclosure is schematically shown. For discussion purposes, reference will be made to... Figure 1 Describe method 200. For example, method 200 can be described by... Figure 1 The computing device 130 is implemented in the middle.

[0043] At point 302, computing device 130 determines one or more 2D coordinates of one or more joints in a two-dimensional (2D) coordinate system based on one of multiple frames and according to a keypoint position estimation model. Computing device 130 may sample a subset of frames in the video to reduce computational cost. Alternatively, computing device 130 may select all frames in the video to ensure high accuracy. After multiple frames are sampled, computing device 130 inputs these frames into the keypoint position estimation model. The keypoint position estimation model outputs the identified robot keypoints, i.e., joints, and the coordinates of the identified keypoints in a specific 2D coordinate system.

[0044] At point 304, computing device 130 converts one or more 2D coordinates into one or more 3D coordinates of one or more joints in a three-dimensional (3D) coordinate system according to a coordinate transformation model. Computing device 130 inputs the 2D coordinates of the robot joints obtained from the keypoint position estimation model into the coordinate transformation model. The coordinate transformation model outputs the 3D coordinates of the robot joints based on the obtained 2D coordinates.

[0045] It should be understood that the machine learning model used in the actions performed at 302 and 304 can be replaced by any other machine learning model discussed above, and the relevant actions can be adapted to the selected machine learning model.

[0046] At 306, computing device 130 determines the angular change of the rotation angle of one of the one or more joints over multiple frames, based on one or more sets of 3D coordinates. In some example embodiments, the angular change can be calculated between every two adjacent frames over multiple frames.

[0047] In some example embodiments, the angles of each axis can be calculated using the Denavit-Hartenberg (DH) model. The DH model can be used to optimize 3D coordinates. The more accurate the estimated 3D coordinates, the more accurate the estimated joint angles. Therefore, the accuracy of the obtained rotational speed can be further improved using the DH model.

[0048] At 308, computing device 130 determines a time period of multiple frames. At 310, computing device 130 determines the rotational speed of a joint based on the angle change and the time period to obtain a velocity profile of the rotational speed within that time period. By dividing the angle change by the time period, the rotational speed of each axis can be calculated, and a velocity profile of each axis can be generated by plotting the rotational speed as the vertical axis and the time period as the horizontal axis.

[0049] In this embodiment, the velocity profile containing all axis velocity curves can be accurately calculated using the 3D coordinates output by the image processing machine learning model, based on known robot kinematics algorithms, without the need for input from the robot or controller cabinet.

[0050] Figure 4A A schematic diagram illustrating an example process 400A for determining a velocity profile using a machine learning model according to some embodiments of this disclosure is shown. Figure 4A As shown, in process 400A, multiple frames sampled from the video of robot 410 are input into keypoint position estimation model 420. Robot 410 includes three joints: first joint 411, second joint 412, and third joint 413.

[0051] After receiving multiple frames of images, the keypoint position estimation model 420 outputs an estimation result 430 of the "skeleton" of the robot 410. This estimation result 430 indicates that the center points of the first joint 411, the second joint 412, and the third joint 413 are identified. The coordinates of the center points of the three robot joints in coordinate system AB are obtained. As shown in the figure, the center point K1 of the first joint 411 has two-dimensional coordinates (a1, b1). The center point K2 of the second joint 412 has two-dimensional coordinates (a2, b2). The center point K3 of the third joint 413 has two-dimensional coordinates (a3, b3). The line connecting two center points represents the arm of the robot 410.

[0052] Then, the two-dimensional coordinates of the center points of the three robot joints are input into the coordinate transformation model 440. The coordinate transformation model 440 outputs the transformation result 450. This transformation result 450 indicates the three-dimensional coordinates of the robot joint center points in the three-dimensional coordinate system XYZ. As shown in the figure, the two-dimensional coordinates of the center point K1 of the first joint 411 are transformed into three-dimensional coordinates (x1, y1, z1). The two-dimensional coordinates of the center point K2 of the second joint 412 are transformed into three-dimensional coordinates (x2, y2, z2). The two-dimensional coordinates of the center point K3 of the third joint 413 are transformed into three-dimensional coordinates (x3, y3, z3). The line connecting the two center points represents the arm of the robot 410.

[0053] After obtaining the three-dimensional coordinates of the three robot joints across all frames, the rotational speed of each robot joint is calculated based on the angle changes and time intervals. For example, based on the obtained coordinates, the motion of center point K2 can be studied in a new coordinate system with center point K3 as the origin. This allows us to obtain the angle change of center point K2 relative to center point K3, thus yielding the rotational speed of center point K2 relative to center point K3. The rotational speed of center point K2 relative to center point K3 corresponds to the rotational speed of the arm between center points K2 and K3, and also to the rotational speed of the third joint 413 that drives the arm between center points K2 and K3. In this way, the rotational speed of the third joint 413 is obtained. Similarly, by studying the motion of center point K1 relative to center point K2, the rotational speed of the second joint 412 can be obtained.

[0054] Thus, the velocity profile 460 of robot 410 is obtained. Velocity profile 460 includes two velocity profiles for the rotational speed within that time period. That is, the horizontal axis represents the time period in seconds, and the vertical axis represents the rotational speed in radians per second. Velocity profile 461 corresponds to the third joint 413. Velocity profile 462 corresponds to the second joint 412.

[0055] It should be understood that the velocity profile of the first joint 411 can also be obtained by tracing a key point on the arm connected to the first joint 411. In some alternative embodiments, the key point to be traced may not be the center point of the joint. The key point may also be a point on the outer contour of the arm. These points can be selected such that the motion of these points can represent the motion of the joint.

[0056] Figure 4B A schematic diagram illustrating an example process 400B for training a machine learning model according to some embodiments of this disclosure is shown. Figure 4B As shown, in process 400B, a training dataset 470 containing the robot in different poses and from different viewpoints is input into the keypoint position estimation model 420. Since the training dataset 470 covers most of the viewpoints around the robot, extrinsic and intrinsic parameters of the camera are not required. Therefore, users can use any type of mobile device to capture video of the robot from any angle.

[0057] After receiving multiple frames of images, the keypoint location estimation model 420 outputs an estimation result 480 of the "skeleton" 482 of the robot 410. The estimation result 480 also shows the ground truth skeleton 481. The goal of training is to make the estimated skeleton 482 approximate the ground truth skeleton 481.

[0058] Then, the estimated skeleton 482 and the ground truth skeleton 481 are input into the coordinate transformation model 440. The coordinate transformation model 440 outputs the transformation result 490. The transformation result 490 includes the estimated 3D skeleton 492 and the ground truth 3D skeleton 491. The goal of training is to make the estimated 3D skeleton 492 approximate the ground truth skeleton 491.

[0059] Figure 5A A flowchart of a method 500A for generating an audio spectrum according to an embodiment of the present disclosure is illustrated schematically. For discussion purposes, reference will be made to... Figure 1 Describe method 500A. For example, method 500A may be derived from... Figure 1 The computing device 130 in the middle is implemented.

[0060] like Figure 5A As shown, at position 502, computing device 130 resamples the audio data based on the velocity profile to obtain resampled audio data. As mentioned above, the recorded audio data may not be directly interpretable. In the illustrated embodiment, the audio data is resampled at different intervals, allowing the resampled data to be further analyzed.

[0061] At position 504, computing device 130 interpolates the resampled audio data to obtain resampled audio data with uniform intervals. Resampled audio data is typically non-uniform. Resampled audio data with non-uniform sampling intervals can be transformed into a new audio dataset with uniform sampling intervals using any suitable interpolation method.

[0062] At position 506, computing device 130 performs a Fast Fourier Transform (FFT) on the resampled audio data to obtain the audio spectrum. After resampling, the FFT can be applied to the resampled and interpolated audio data to transform it from the time domain to the frequency domain. As a result, the audio spectrum is obtained. In the illustrated embodiment, the audio data is resampled and calibrated according to the robot joint speed, allowing the resampled data to be transformed into the frequency domain, thus facilitating audio data interpretation.

[0063] Figure 5B A flowchart illustrating a method 500B for resampling audio data according to an embodiment of this disclosure is shown schematically. For discussion purposes, reference will be made to... Figure 1 Describe method 500. For example, method 500 may be described by... Figure 1 The computing device 130 in the text implements this. Typically, the original sampling interval of audio data may be uniform. In order to transform the audio data to make it easier for subsequent analysis and processing, the computing device can resample the audio data at non-uniform intervals.

[0064] like Figure 5BAs shown, at 512, the computing device 130 determines the average value of the velocity values ​​in the velocity profile. At 514, the computing device 130 determines the ratio of the velocity values ​​to the average value. For example, the computing device 130 can calculate this ratio by dividing the velocity value by the average value.

[0065] At position 516, the computing device 130 determines the resampling interval based on the ratio and the original sampling interval of the audio data. For example, the computing device 130 can calculate the product of the ratio and the original sampling interval as the new resampling interval.

[0066] At 518, computing device 130 determines the resampling time point based on the resampling interval. For example, computing device 130 can calculate each resampling interval and sequentially accumulate the time of each resampling interval to determine the resampling time point to perform resampling. At 520, computing device 130 resamples the audio data at the resampling time point to obtain resampled audio data.

[0067] Figure 6 A flowchart illustrating a method 600 for determining the health status of a robot according to an embodiment of this disclosure is shown schematically. For discussion purposes, reference will be made to... Figure 1 Describe method 600. For example, method 600 can be described by... Figure 1 The computing device 130 in the middle is implemented.

[0068] like Figure 6 As shown, at 602, computing device 130 identifies the type of robot. At 604, computing device 130 obtains a reference amplitude corresponding to that robot type. In some example embodiments, this can be achieved by performing operations such as... Figure 2 The health diagnostic methods shown are used to obtain a reference range.

[0069] At 606, computing device 130 obtains one or more velocity values ​​associated with the resampled audio data from the velocity profile. At 608, computing device 130 determines one or more frequencies corresponding to the one or more velocity values. At 610, computing device 130 selects one or more amplitudes in the audio spectrum corresponding to the one or more frequencies. Since each axis has a separate velocity profile, the audio spectrum is calibrated with reference to each axis to obtain its respective audio spectrum. For example, when examining the first axis, the audio spectrum calibrated with reference to the velocity profile of the first axis will be selected. The frequencies corresponding to the velocity values ​​of the first axis can then be selected.

[0070] At 612, the computing device 130 determines whether the joint's range of motion exceeds a corresponding reference range. If the computing device 130 determines that the joint's range of motion exceeds the corresponding reference range, then method 600 proceeds to 614. At 614, the computing device 130 determines that the joint is abnormal or unhealthy. If the computing device 130 determines that the joint's range of motion does not exceed the corresponding reference range, then method 600 proceeds to 616. At 616, the computing device 130 determines that the joint is normal or healthy.

[0071] In the illustrated embodiment, the health status of each axis of the robot can be determined by comparing the amplitude peak of the corresponding frequency in the audio spectrum with a reference amplitude.

[0072] Figure 7 A schematic diagram illustrating an example process 700 for determining the health status of a robot according to other embodiments of this disclosure is shown. Figure 7 As shown, in process 700, a velocity profile 710 is illustrated. The velocity profile 710 includes a first velocity profile 711 for the robot's first joint, a second velocity profile 712 for the robot's second joint, and a third velocity profile 713 for the robot's third joint. These curves can be derived based on the motion of four key points on the robot. Based on the velocity profile 710, a first audio spectrum 720, a second audio spectrum 730, and a third audio spectrum 740 are obtained, respectively.

[0073] The first audio spectrum 720 is generated based on resampled audio data from the first velocity profile 711. In the first audio spectrum 720, frequency F12 is determined to correspond to a velocity value in the first velocity profile 711 (e.g., the average value of the first velocity profile 711). A corresponding reference amplitude 722 is read from memory and compared with the calculated amplitude 721. It can be seen that the calculated amplitude 721 is greater than the reference amplitude 722. Therefore, an abnormality or unhealthy condition of the first joint is diagnosed.

[0074] The second audio spectrum 730 is generated based on resampled audio data from the second velocity profile 712. In the second audio spectrum 730, frequency F23 is determined to correspond to a velocity value in the second velocity profile 712 (e.g., the average value of the first velocity profile 712). A corresponding reference amplitude 732 is read from memory and compared with the calculated amplitude 731. It can be seen that the calculated amplitude 731 is greater than the reference amplitude 732. Therefore, an abnormality or unhealthy condition of the second joint is diagnosed.

[0075] The third audio spectrum 740 is generated based on resampled audio data from the third velocity profile 713. In the third audio spectrum 740, frequency F34 is determined to correspond to the velocity value in the third velocity profile 713 (e.g., the average value of the first velocity profile 713). The corresponding reference amplitude 742 is read from memory and compared with the calculated amplitude 741. It can be seen that the calculated amplitude 741 is smaller than the reference amplitude 742. Therefore, the third joint is diagnosed as normal or healthy.

[0076] In some embodiments of this disclosure, a computing device is provided for implementing the methods 200, 300, 500A, 500B and 600 described above. Figure 8 A schematic diagram of an electronic device 800 for implementing a method according to an embodiment of the present disclosure is shown. The electronic device 800 may correspond to... Figure 1 The computing device 130 is included. The electronic device 800 includes at least one processor 810 and at least one memory 820. The at least one processor 810 may be coupled to the at least one memory 820. The at least one memory 820 contains instructions 822 that, when executed by the at least one processor 810, implement method 200, 300, 500A, 500B or 600.

[0077] In some embodiments of this disclosure, a computer-readable medium is provided for adjusting a robot path. The computer-readable medium stores instructions that, when executed on at least one processor, cause the at least one processor to perform the previously described method for managing a camera system, details of which will be omitted here.

[0078] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software executable by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0079] This disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions contained in program modules, which execute in a device on a target real or virtual processor to perform the functions described above. Figures 2 to 7A described process or method. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of a program module can be combined or split among program modules as desired. The machine-executable instructions used for a program module can execute on a local or distributed device. In a distributed device, program modules can reside on local and remote storage media.

[0080] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that, when executed by the processor or controller, it causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] The aforementioned program code can be embodied on a machine-readable medium, which can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0082] Furthermore, although the operations are depicted in a specific order, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or requiring the execution of all illustrated operations to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. On the other hand, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0083] Although the subject matter has been described in language specific to structural features and / or methodological behavior, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or behaviors described above. Rather, the specific features and behaviors described above are disclosed as examples of implementing the claims.

[0084] It should be understood that the detailed embodiments described above are merely illustrative or explanatory of the principles of this disclosure and do not limit the scope of this disclosure. Therefore, any modifications, equivalent substitutions, and improvements made without departing from the spirit and scope of this disclosure should be included within the protection scope of this disclosure. Furthermore, the claims of this disclosure are intended to cover all changes and modifications falling within the scope of the claims and their equivalents.

[0085] It should be understood that the summary portion is not intended to identify key or essential features of embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

Claims

1. A method for determining the health status of a robot, comprising: The video data of the robot during its movement is acquired by a computing device, and the video data includes image data and audio data; The computing device determines the robot's velocity profile based on the image data; The computing device generates the audio spectrum of the audio data based on the velocity profile; as well as The computing device determines the robot's health status based on the audio spectrum.

2. The method of claim 1, wherein determining the velocity profile of the robot based on the image data comprises: Based on the image data and according to a machine learning model, determine one or more sets of positions for one or more joints of the robot for multiple frames of the image data; as well as Based on the one or more position sets, one or more velocity curves of the one or more joints are determined as the velocity profile.

3. The method of claim 2, wherein determining the one or more positions of the one or more joints comprises: Based on one of the multiple frames and according to the key point position estimation model, determine one or more 2D coordinates of one or more joints in a two-dimensional 2D coordinate system; as well as According to the coordinate transformation model, the one or more 2D coordinates are converted into one or more 3D coordinates of the one or more joints in a three-dimensional 3D coordinate system.

4. The method of claim 2, wherein the key point location estimation model and the coordinate transformation model are trained using a collection of videos from multiple perspectives surrounding the robot.

5. The method of claim 2, wherein determining the one or more positions of the one or more joints comprises: Based on the video and motion estimation model, the type of the robot is identified; Obtain a simulation model of the robot that corresponds to the type of the robot; as well as Based on the motion estimation model, the simulation model is mapped to the multiple frames of the image data to obtain one or more pose sets for the one or more joints.

6. The method of claim 2, wherein determining one or more velocity profiles of the one or more joints comprises: Based on the one or more location sets, determine one or more 3D coordinate sets for the one or more joints; Based on the one or more 3D coordinate sets, determine the angular variation of the rotation angle of one of the one or more joints across the multiple frames; Determine the time period of the multiple frames; as well as Based on the angle change and the time period, the rotational speed of the joint is determined to obtain a speed curve of the rotational speed over the time period.

7. The method of claim 1, wherein generating the audio spectrum of the robot comprises: The audio data is resampled based on the velocity profile to obtain resampled audio data. The resampled audio data is interpolated to obtain resampled audio data with uniform intervals; as well as Perform a Fast Fourier Transform (FFT) on the resampled audio data to obtain the audio spectrum.

8. The method of claim 7, wherein resampling the audio data comprises: Determine the average value of the velocity values ​​in the velocity profile; Determine the ratio of the velocity value to the average value; The resampling interval is determined based on the ratio and the original sampling interval of the audio data; The resampling time point is determined based on the resampling interval; as well as The audio data is resampled at the resampling time point to obtain the resampled audio data.

9. The method of claim 1, wherein determining the health status of the robot comprises: Identify the type of the robot; Obtain the reference amplitude corresponding to the robot type; Obtain one or more velocity values ​​associated with the resampled audio data from the velocity profile; Determine one or more frequencies corresponding to the one or more speed values; Select one or more amplitudes corresponding to the one or more frequencies in the audio spectrum; In response to determining that the range of motion of a joint exceeds a corresponding reference range, the joint is determined to be abnormal.

10. The method of claim 9, wherein determining the health status of the robot further comprises: In response to determining that the second amplitude of the second joint does not exceed the corresponding reference amplitude, the second joint is determined to be normal.

11. The method of claim 9, wherein the reference amplitude represents the amplitude in the audio spectrum of a normally operating robot.

12. The method of claim 1, wherein the image data and the audio data are recorded by the camera and microphone of the computing device, and the computing device is not connected to the robot.

13. An electronic device (700), comprising: At least one processor (710); as well as At least one memory (720) stores instructions (721) that, when executed by the at least one processor, cause the device (700) to perform the method of any one of claims 1-12.

14. A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-12.

15. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-12.

Images