Robot detection method, device and system, electronic equipment and storage medium

Abstract

The invention provides a robot detection method, device and system, electronic equipment and a storage medium, and belongs to the technical field of signal processing, and the method comprises the steps: determining an operation mode of a to-be-detected robot, obtaining an operation audio and an operation video of the robot in the operation mode, and determining an audio anomaly detection result and a visual anomaly detection result of the robot; and iteratively executing until visual anomaly detection results and audio anomaly detection results of the robot in all the operation modes are obtained, so as to determine a comprehensive detection result. Wherein the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operation mode. According to the robot detection method provided by the invention, the visual detection and the audio detection are combined, the reference standard of the audio detection is dynamically adjusted according to the difference of the robot in different operation modes, some key stages of the robot can be detected in a targeted manner, and the detection precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a robot detection method, device, system, electronic equipment and storage medium. Background Art

[0002] In the current field of robot inspection, especially in the manufacturing and quality control process of automated equipment such as sweeping robots, it is crucial to ensure the operating performance and reliability of the robot.

[0003] Traditional detection methods rely on manual trajectory observation and audio detection. These methods have many limitations. For example, it is difficult to unify the detection effects and standards, which limits the consistency and reliability of the detection results. Manual observation cannot accurately observe and locate the trajectory, and it is difficult to capture subtle anomalies in the operation of the robot. Long-term listening and visual observation work can easily cause fatigue, affecting the quality of detection. Not only can it not meet the needs of large-scale and efficient detection, but it also causes a waste of manpower and material resources.

[0004] In view of this, there is an urgent need to provide a new detection method to meet the current requirements of fast, efficient and accurate robot offline detection. Summary of the invention

[0005] The present invention provides a robot detection method, device, system, electronic equipment and storage medium, which are used to solve the defects of low detection accuracy, large workload and poor real-time performance existing in current manual detection.

[0006] The present invention provides a robot detection method, comprising the following steps: S1, obtaining an operation image of the robot to be tested running in a test area, and determining an operation mode of the robot to be tested; S2, obtaining the running audio and running video of the robot to be tested in the running mode; S3, determining an audio anomaly detection result of the robot under test in the operating mode according to the real-time audio feature of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operating mode; S4, determining a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

[0007] According to a robot detection method provided by the present invention, determining the audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio includes: Determining at least one preset inspection item for inspecting the robot to be tested; Obtaining the frequency band interval of the preset inspection item; Sampling the running audio according to a preset sampling period to obtain at least one audio clip; Setting a bandpass filter according to the frequency band interval of the preset inspection item, so as to filter at least one of the audio segments using the bandpass filter to obtain target audio; Extracting the audio feature of the target audio as the real-time audio feature, and determining the detection result of the preset inspection item according to the comparison result between the real-time audio feature and the audio reference feature; The detection results of all preset inspection items are obtained to determine the audio anomaly detection result of the robot to be tested in the operating mode.

[0008] According to a robot detection method provided by the present invention, after setting a bandpass filter according to the frequency band interval of the preset inspection item to filter at least one of the audio segments using the bandpass filter to obtain the target audio, the method further includes: Performing short-time Fourier transform on the target audio to obtain an audio spectrum graph; Inputting the audio spectrogram into an audio anomaly detection model to obtain a model detection result output by the audio anomaly detection model; Determine the detection result of the preset inspection item by combining the model detection result and the comparison result between the real-time audio feature and the audio reference feature; The audio anomaly detection model is obtained after training with audio spectrogram samples, and each of the audio spectrogram samples is pre-labeled with an audio detection label of the preset inspection item.

[0009] According to a robot detection method provided by the present invention, extracting the audio features of the target audio as the real-time audio features includes: At least one of a sound intensity value and a sound coarseness value of the target audio is extracted as the real-time audio feature; the sound intensity value is determined according to the root mean square value of the intensity of the target audio, and the sound coarseness value is determined according to the modulation frequency of the target audio.

[0010] According to a robot detection method provided by the present invention, the step of determining at least one preset inspection item for detecting the robot to be detected and obtaining a frequency band interval of the preset inspection item includes: Determining the frequency band interval from a pre-constructed frequency band division table according to the preset check items; The frequency band division table records the frequency band intervals corresponding to different preset inspection items, and the frequency band interval of each preset inspection item is determined by the sound characteristic frequency of each preset inspection item in a normal state and a fault state.

[0011] According to a robot detection method provided by the present invention, the step of acquiring an operation image of the robot to be tested running in a test area and determining the operation mode of the robot to be tested includes: Inputting the acquired operation image into a target recognition model to obtain position information of the robot to be tested in the operation image output by the target recognition model; Using the mapping relationship between the image coordinate system where the running image is located and the space coordinate system where the test area is located, the actual position of the robot to be tested in the test area is determined according to the position information of the robot to be tested in the running image; The operating mode of the robot to be tested is determined according to the actual position and the preset distribution areas of different operating modes in the test area.

[0012] According to a robot detection method provided by the present invention, determining the visual anomaly detection result of the robot to be tested in the operation mode according to the operation video includes: Determine the real-time running trajectory of the robot to be tested in the running mode according to the running video; The visual anomaly detection result of the robot to be tested in the operation mode is determined according to the comparison result of the real-time operation trajectory and the reference operation trajectory.

[0013] According to a robot detection method provided by the present invention, determining the real-time running trajectory of the robot to be tested in the running mode according to the running video includes: A plurality of video frame images from the start to the end of the detection are intercepted from the running video according to a preset sampling frequency, and a SORT algorithm tracker is initialized; Inputting the starting frame image into the target recognition model, and outputting the recognized starting position information of the robot to be tested; Inputting the starting position information into the SORT algorithm tracker, obtaining the predicted position information of the robot to be tested in the next video frame image output by the SORT algorithm tracker, and matching the predicted position information with the starting position information; If the match is qualified, the state of the SORT algorithm tracker is updated using the predicted position information; Iteratively executing the step of inputting the next video frame image into the target recognition model, and updating the state of the SORT algorithm tracker using the predicted position information, until the traversal of all the video frame images is completed; Obtain the real-time running trajectory of the SORT algorithm tracker determined by all the predicted position information.

[0014] According to a robot detection method provided by the present invention, among the visual abnormality detection results and the audio abnormality detection results in all operating modes, if any visual abnormality detection result or audio abnormality detection result is abnormal, the comprehensive detection result is determined to be abnormal.

[0015] The present invention also provides a robot detection device, comprising: A pattern recognition unit, used to obtain an operation image of the robot to be tested running in a test area, and determine an operation mode of the robot to be tested; An information collection unit, used for acquiring the operation audio and video of the robot under test in the operation mode; an audio detection unit, configured to determine an audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio features and an audio reference feature, and the audio reference feature is determined according to the operating mode; A video detection unit, used to determine a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; An iterative control unit is used to control the cyclic operation from the pattern recognition unit to the video anomaly detection unit until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine the comprehensive detection results of the robot under test.

[0016] The present invention also provides a robot detection system, comprising: A detection box, wherein a test area is arranged at the bottom of the detection box, and a plurality of preset distribution areas for matching different operation modes of the robot to be tested are arranged in the test area; A sound collection device, used to collect operation videos and operation images of the robot to be tested in the test area; A video acquisition device, used to acquire the operating audio of the robot to be tested in the test area; The processor, the sound collection device and the video collection device are communicatively connected to the processor, and the processor executes any one of the robot detection methods described above when running.

[0017] According to a robot detection system provided by the present invention, there are multiple detection boxes; Each detection box is configured with at least one set of the sound collection device and the video collection device.

[0018] A robot detection system according to the present invention further includes a robot transmission mechanism communicatively connected to the processor; The processor controls the robot transport mechanism to operate so as to transport each of the robots to be tested into an idle testing box.

[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned robot detection methods is implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot detection method as described in any one of the above is implemented.

[0021] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the robot detection method described above is implemented.

[0022] The robot detection method, device, system, electronic device and storage medium provided by the present invention combine visual detection with audio detection, dynamically adjust the reference standard of audio detection according to the differences of the robot in different operating modes, and can perform targeted detection on some key stages of the robot, thereby effectively improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 This is one of the flow charts of the robot detection method provided by the present invention.

[0025] Figure 2 This is the second flow chart of the robot detection method provided by the present invention.

[0026] Figure 3 It is a structural schematic diagram of the robot detection device provided by the present invention.

[0027] Figure 4It is a schematic diagram of the layout of the detection box and internal devices provided by the present invention.

[0028] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0031] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0032] It should be noted that the present invention provides a method for offline detection of robots, wherein the robot can be an automated cleaning device such as a sweeping robot, or other types of automated mobile devices, such as a service robot, an industrial robot, an exploration robot, etc. These robots have the ability to operate under a variety of working conditions and emit different sound characteristics, and their operating trajectories and component states can be fully monitored and analyzed through visual and sound data. These robots contain multiple components that may fail, such as motors, drive wheels, working parts (such as cleaning brushes, clamping devices, etc.), driven wheels, vibration devices, etc., and each component will emit audio signals of different frequency bands when it fails. In addition, the robot to be detected is also suitable for combining deep learning models and using multi-microphone and camera hardware solutions to perform modular production line multi-station layout detection scenarios. For the convenience of description, the sweeping robot is used as an example in the subsequent embodiments, which is not regarded as a specific limitation on the scope of protection of the present invention.

[0033] Before leaving the factory, the robot vacuum cleaner needs to undergo a running test, which mainly includes observing whether its running track is normal, and whether there are defects such as running jams, harsh noises, and sound dragging during operation due to problems with parts, processes, and assembly. Existing detection methods mostly use manual track observation and manual listening detection.

[0034] Since manual listening detection cannot quantitatively record key acoustic properties such as decibels and roughness, and the detection effect and detection standards cannot be guaranteed to be unified, and long-term listening and visual observation work can easily cause fatigue, affecting the quality of detection. In addition, manual trajectory observation cannot accurately locate the trajectory, and cannot effectively combine the operating requirements under different operating modes to accurately determine abnormalities at key nodes such as emergency stop and start.

[0035] In view of this, the present invention provides a fully automated robot offline detection method, which can effectively overcome the above-mentioned defects. Figure 1-Figure 5 The robot detection method, device, system, electronic device and storage medium provided by the present invention are described in detail.

[0036] Figure 1 is one of the flow charts of the robot detection method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: S1, obtaining an operation image of the robot to be tested running in a test area, and determining an operation mode of the robot to be tested.

[0037] The test area is a standardized testing site, for example, it is set in a closed soundproof box, and the closed box is used to isolate the interference of external noise to eliminate the influence of external factors on the test results. It should be emphasized that in the robot detection method provided by the present invention, a plurality of preset distribution areas for matching the different operating modes of the robot to be tested will be set in advance in the entire detection area. Among them, the operating mode may include but is not limited to: Yanbian cleaning mode, obstacle avoidance mode, bow mode, step avoidance mode, high reflective area cleaning mode, low reflective area cleaning mode, cliff area avoidance mode, etc., which can be arranged in the test area according to the design requirements of the cleaning robot.

[0038] In step S1, during the entire process of performing off-line inspection of the robot to be tested, a video acquisition device installed in a soundproof box will be used to shoot a video of the test area where the robot to be tested is located to obtain the operation video of the robot to be tested, and then, by means of video frame extraction, any video frame is extracted from the operation video as the operation image of the robot to be tested running in the test area.

[0039] Of course, the video acquisition device may also be used to directly capture images of the test area where the robot to be tested is located, and obtain running images of the robot to be tested running in the test area.

[0040] Furthermore, the collected running images can be analyzed using an image recognition algorithm or a deep learning model to accurately locate the position information of the robot to be tested in the test area.

[0041] Since different preset distribution areas corresponding to different operation modes in the test area have been set when the test area is laid out, the operation mode of the robot under test can be determined based on the position information. For example, by determining that the robot under test is currently in the preset distribution area corresponding to the fault avoidance mode based on the position information of the robot under test, it can be determined that its operation mode is the obstacle avoidance mode.

[0042] S2, obtaining the operation audio and video of the robot to be tested in the operation mode.

[0043] As the robot under test is running, a sound collection device (such as an industrial microphone) and a video collection device (such as an industrial camera) placed in a silent box can be used to synchronously capture the running audio and running video of the robot under test in cleaning mode. The setting of multiple microphones and cameras can ensure that data is collected from multiple angles and positions, thereby improving the accuracy and comprehensiveness of the detection.

[0044] S3, determining the audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio.

[0045] The audio anomaly detection result is determined based on a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined based on the operating mode.

[0046] Furthermore, the collected running audio can be processed in real time to extract its corresponding audio features, wherein the audio features may include one or more of frequency, amplitude, energy distribution, harmonic content, etc., and the detection value of each audio feature can be accurately calculated.

[0047] These real-time audio features are compared with the preset audio reference features, and through the comparison it can be determined whether the sweeping robot makes abnormal sounds.

[0048] For example, if the maximum amplitude value or the maximum frequency value corresponding to the real-time audio feature is compared with the maximum allowable amplitude value and the maximum allowable frequency value corresponding to the audio reference feature of the robot to be tested determined according to the operating mode, if the maximum amplitude value is greater than the maximum allowable amplitude value, or the maximum frequency value is greater than the maximum allowable frequency value, then it can be determined that the audio anomaly detection result indicates that the robot to be tested is faulty.

[0049] The present invention takes into account that the working states of the various components of the robot to be tested, the interactions between them and the interactions with the external environment may be different in different operating modes, and these differences may directly lead to changes in the characteristics of the sound emitted.

[0050] For example, a sweeping robot has multiple operating modes such as sweeping, vacuuming, and mopping. In each operating mode, the speed, load, and working mode of key components such as the drive wheel motor and drum motor will be adjusted, resulting in different sound characteristics. In addition, environmental factors such as ground materials and obstacles encountered by the sweeping robot in different operating modes will also affect its sound characteristics. For example, in the sweeping mode, the speed of the drive wheel motor may be higher, and the frequency of the sound generated will increase accordingly; in the mopping mode, due to the increase in load, the working sound of the motor may become lower.

[0051] In view of this, when performing audio anomaly detection on the robot under test, the audio reference features used in the present invention are determined according to the operating mode of the robot under test, thereby achieving accurate monitoring and anomaly detection of the robot under test.

[0052] S4: Determine, based on the operation video, a visual anomaly detection result of the robot to be tested in the operation mode.

[0053] At the same time, the collected running videos can be processed, and deep learning models (such as YOLO v8) can be used to detect the target of the robot under test in each video frame image in the running video and mark the target box. Then, the real-time running trajectory is generated by the target box detected in each video frame image. Finally, the real-time running trajectory can be analyzed to determine whether the robot under test has a fault, and this result can be used as the visual anomaly detection result.

[0054] Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

[0055] As the robot under test continues to operate in the test area, the above steps S1 to S4 will be continuously iterated to obtain the visual anomaly detection results and audio anomaly detection results of the sweeping robot in all operating modes in the entire test area (which can be understood as completing the detection work of all operating modes planned in the entire test area), and finally complete the offline detection of the robot under test.

[0056] The robot detection method provided by the present invention combines visual detection with audio detection, dynamically adjusts the reference standard of audio detection according to the differences of the robot in different operating modes, and can perform targeted detection on some key stages of the robot, thereby effectively improving the detection accuracy.

[0057] Based on the content of the above embodiment, as an optional embodiment, determining the audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio includes: Determining at least one preset inspection item for inspecting the robot to be tested; Obtaining the frequency band interval of the preset inspection item; Sampling the running audio according to a preset sampling period to obtain at least one audio clip; Setting a bandpass filter according to the frequency band interval of the preset inspection item, so as to filter at least one of the audio segments using the bandpass filter to obtain target audio; Extracting the audio feature of the target audio as the real-time audio feature, and determining the detection result of the preset inspection item according to the comparison result between the real-time audio feature and the audio reference feature; The detection results of all preset inspection items are obtained to determine the audio anomaly detection result of the robot to be tested in the operating mode.

[0058] Figure 2This is the second flow chart of the robot detection method provided by the present invention. Figure 2 Taking the detection of the sweeping robot shown in the figure as an example, it is described in detail how to determine the audio anomaly detection result of the robot to be tested in the running mode according to the real-time audio features of the running audio.

[0059] There are many parts in a sweeping robot. Different types of parts emit fault sounds in different frequency bands, so it is important to divide the collected full-band operating audio into frequency bands in advance and then calculate the indicators.

[0060] First, according to the working characteristics, hardware composition and possible fault types of the sweeping robot, all inspection items for audio detection are determined (because they need to be determined in advance, they are called preset inspection items). For example, depending on the different detection components, these preset inspection items may include but are not limited to the detection of the following components: driving wheel motor, roller motor, driving wheel, roller, driven steering wheel, vibration plate, roller brush, etc.

[0061] Then, according to the different frequency bands of fault sounds emitted by different preset inspection items, the full-band sounds are divided in advance. The specific implementation process is as follows: First, by analyzing the working principle and historical fault data of each preset inspection item of the sweeping robot, the characteristic frequencies of the sounds emitted by different preset inspection items when a fault occurs are determined. Through experiments and tests, the sound data of different preset inspection items in normal and faulty states are collected to verify the results of theoretical analysis and determine the frequency band range of the fault sounds related to each preset inspection item.

[0062] Then, according to the results of the fault frequency band analysis, a frequency band division table is formulated to list the lower and upper limits of the fault characteristic frequencies of different components. For example, the characteristic frequency range of the BLDC type drive wheel motor fault mentioned in Table 1 below is 2860Hz to 12000Hz.

[0063] Table 1 List of frequency band demarcation results for each preset inspection item

[0064] Next, the running audio of the robot to be tested can be sampled according to the preset sampling period to obtain at least one audio segment, each of which contains sound information within a period of time. The selection of the sampling period can be determined according to the characteristics of the preset inspection items and the detection accuracy requirements. For example, if high-frequency sounds need to be accurately detected, the sampling period can be set relatively shorter.

[0065] Furthermore, the collected audio segments are band-pass filtered, that is, a band-pass filter is set according to the frequency band interval of the preset inspection items, so as to obtain the target audio by filtering at least one of the audio segments using the band-pass filter. The purpose of band-pass filtering is to extract the target audio of a specific frequency band interval from the audio segments of the full frequency band, and these specific frequency band intervals correspond to the fault characteristic frequency band intervals of different preset inspection items of the sweeping robot.

[0066] By applying a bandpass filter to the sampled audio clips, the target audio in a specific frequency band can be filtered out, and audio of other frequencies can be ignored. In this way, for different preset inspection items that may fail, the signals in the corresponding characteristic frequency bands can be extracted respectively, making the detection results more accurate. For example, when testing the preset inspection item of the drive wheel motor, the characteristic frequency range of the bandpass filter can be set to 2860HZ to 12000HZ.

[0067] It should be noted that the target audio can be obtained by performing bandpass filtering on any audio segment, or by performing bandpass filtering on multiple audio segments and then selecting an audio segment with the most obvious audio features as the target audio.

[0068] As an optional embodiment, after acquiring the target audio, it can also be further filtered using a Gaussian kernel. Gaussian kernel filtering is to further smooth the data on the basis of bandpass filtering to reduce noise and interference and improve the signal-to-noise ratio of the signal. Gaussian kernel filtering is a smoothing technology based on a Gaussian function, which uses the weighted characteristics of the Gaussian function to perform local averaging on the signal.

[0069] Optionally, the Gaussian kernel is defined as follows: ; in, Represents the value of the Gaussian kernel function, representing the data point and similarity or correlation between represents the test data point; represents a data point in the training dataset; is an exponential function; Represents data points and The square of the Euclidean distance between them measures the similarity between two points in the feature space; b represents the bandwidth parameter of the Gaussian kernel, also known as the standard deviation or scale parameter, which controls the width of the Gaussian function, that is, the smoothness of the function. A smaller b value will make the Gaussian function sharper, while a larger b value will make the function smoother, affecting the generalization ability of the model.

[0070] After the target audio is extracted, the audio features of the target audio are further extracted as real-time audio features. These audio features may include frequency, intensity, spectrum distribution, etc. Then, the real-time audio features are compared with the preset audio reference features to obtain the comparison results.

[0071] The audio reference features are determined according to the different operating modes of the robot to be tested, which are described in detail in the above embodiments and will not be elaborated here.

[0072] Based on the comparison result between the real-time audio feature and the preset audio reference feature, the detection result for a preset inspection item can be determined.

[0073] By using the above method, each preset inspection item is tested in turn, and the test results of all preset inspection items can be obtained. If the test result of a preset inspection item is abnormal, it can be considered that the robot under test has audio abnormalities in this operating mode. If the test results of all preset inspection items are normal, it can be considered that the operating status of the robot under test is good.

[0074] Combination Figure 2 As shown, as an optional embodiment, after a bandpass filter is set according to the frequency band interval of the preset inspection item, and the at least one audio segment is filtered by the bandpass filter to obtain the target audio, the method further includes: Performing short-time Fourier transform on the target audio to obtain an audio spectrum graph; Inputting the audio spectrogram into an audio anomaly detection model to obtain a model detection result output by the audio anomaly detection model; Determine the detection result of the preset inspection item by combining the model detection result and the comparison result between the real-time audio feature and the audio reference feature; The audio anomaly detection model is obtained after training with audio spectrogram samples, and each of the audio spectrogram samples is pre-labeled with an audio detection label of the preset inspection item.

[0075] The robot detection method provided by the present invention incorporates the application of short-time Fourier transform (STFT) and audio anomaly detection model, aiming to improve the accuracy of audio anomaly detection of preset inspection items (such as driving wheel motor, roller motor, driving wheel, roller, driven steering wheel, vibration plate, roller brush and other key components).

[0076] In various operating modes of the robot to be tested, the sound characteristics generated by each key component are complex and changeable. In order to accurately capture and identify anomalies in these audio characteristics, short-time Fourier transform technology is introduced to convert audio signals into spectrograms, so as to more intuitively analyze the changes of frequency components in the audio over time. At the same time, the pre-trained audio anomaly detection model is used to perform intelligent analysis on the spectrogram, and the comparison between real-time audio features and audio reference features is combined to achieve a comprehensive judgment of audio anomalies in preset inspection items, which mainly includes but is not limited to the following steps: First, corresponding bandpass filters are set according to the frequency band intervals of the preset inspection items (such as the characteristic frequency intervals of the drive wheel motor and the characteristic frequency intervals of the drum motor, etc.). Subsequently, at least one audio segment is filtered using these bandpass filters to obtain target audio that is closely related to the target inspection item.

[0077] Perform short-time Fourier transform on the target audio after bandpass filtering, convert the audio signal from time domain to frequency domain, and generate an audio spectrum. Short-time Fourier transform can reflect the frequency distribution of audio signals at different time points, providing rich information for subsequent audio anomaly detection.

[0078] The generated audio spectrogram is input into the pre-trained audio anomaly detection model, which can automatically identify abnormal features in the spectrogram and output the model detection results.

[0079] The audio anomaly detection model is trained using a large number of audio spectrogram samples annotated with preset inspection item audio detection labels.

[0080] Taking the inspection of the drive wheel motor as an example, a large number of audio samples under normal conditions and fault conditions are collected, and short-time Fourier transform is performed on the collected audio samples to form corresponding audio spectrum samples.

[0081] After data annotation is performed on the fault area on the audio spectrogram samples, the annotated audio spectrogram samples are introduced into a deep learning model (such as the YOLO V8 model) for training to obtain a trained audio anomaly detection model.

[0082] The robot detection method provided by the present invention, by introducing short-time Fourier transform technology and an audio anomaly detection model, realizes accurate detection of audio anomalies of preset inspection items in the robot detection method. This method not only improves the accuracy of detection, but also enhances the intelligence level of the system, providing a strong guarantee for the stable operation of the sweeping robot. At the same time, the method is also applicable to other types of robots, as long as these robots have detectable sound characteristics and can provide sufficient audio data for model training.

[0083] Based on the content of the above embodiment, as an optional embodiment, extracting the audio feature of the target audio as the real-time audio feature includes: At least one of a sound intensity value and a sound roughness value of the target audio is extracted as the real-time audio feature.

[0084] The sound intensity value is determined according to the root mean square value of the intensity of the target audio. The root mean square value of the intensity is also called the RMS value, which can be used as the effective value or equivalent value of the target audio and reflects the overall energy level of the target audio. Specifically, it can be represented by the following mathematical function: ; in, It represents the root mean square value of the signal, which is a measure of signal power and is often used in acoustic and vibration analysis; n represents the total number of signal samples; Represents the sample value of the nth signal sample.

[0085] The sound roughness value is determined according to the modulation frequency of the target audio, and the specific calculation process is as follows: First, determine the target audio frequency and perform spectrum analysis or other signal processing techniques to determine its corresponding modulation frequency. .

[0086] Then, yes Multiply Integrate in the range of z from 0 to 24. This integration process can capture the fluctuation characteristics of the target audio within the specified range. Indicates energy change or amplitude change within 1 / 2 octave Finally, multiply the integral result by a constant , and get the final roughness value.

[0087] The sound roughness value can be expressed using the following mathematical function: ; in, Indicates the calculated roughness value, which is used to measure the roughness of the target audio and is an important indicator of sound quality; is a constant used to adjust the dimension or scale of roughness to make it more suitable for specific application scenarios or standardization requirements; z represents the index of time, which depends on the application scenario; It is the modulation frequency to reflect the periodic changes or fluctuations of the target audio. In the calculation of roughness, it may be used to measure the degree of fluctuation of the target audio.

[0088] The robot detection method provided by the present invention provides an effective audio feature extraction method for the robot detection method by extracting the sound intensity value and the sound roughness value of the target audio as real-time audio features. This method is not only simple and easy to implement, but also can accurately reflect the sound characteristics of the robot during operation, providing reliable data support for subsequent audio analysis or abnormality detection.

[0089] In the actual detection process of the robot, different preset inspection items (such as motors, transmission mechanisms, sensors, etc.) often have different sound characteristic frequencies. In order to accurately capture these characteristic frequencies and effectively analyze them, the present invention pre-constructs a frequency band division table (as shown in Table 1) to record in detail the sound characteristic frequency range of each preset inspection item in normal and fault states.

[0090] As an optional embodiment, the present invention provides a specific implementation method of obtaining the frequency band interval of at least one preset inspection item for the robot to be tested according to the frequency band division table, including but not limited to the following steps: Determining the frequency band interval from a pre-constructed frequency band division table according to the preset check items; The frequency band division table records the frequency band intervals corresponding to different preset inspection items, and the frequency band interval of each preset inspection item is determined by the sound characteristic frequency of each preset inspection item in a normal state and a fault state.

[0091] First, based on a large amount of experimental data and accumulated experience, a frequency band division table was constructed. This frequency band division table lists in detail the sound characteristic frequency ranges of each key component (i.e., the inspection object of the preset inspection items) in the robot to be tested in normal and fault conditions. These frequency ranges are obtained by testing and analyzing a large number of samples and have high accuracy and reliability.

[0092] Before performing formal testing, you can first clarify the specific model and configuration of the robot to be tested to determine the key components that need to be tested (i.e., preset inspection items). The above information can usually be obtained by consulting the technical documentation of the robot to be tested, communicating with the manufacturer, or conducting a preliminary inspection.

[0093] After determining the preset inspection items of the robot to be tested, the frequency band intervals corresponding to these inspection items are found in the frequency band division table. Specifically, by finding the entries matching each preset inspection item in the frequency band division table and obtaining the characteristic frequency band intervals of its sound in the fault state, these frequency band intervals will serve as an important basis for subsequent audio collection and analysis.

[0094] After obtaining the frequency band intervals of each preset inspection item, corresponding bandpass filters can be set according to these intervals to extract target audio related to the target frequency band from the collected audio signal.

[0095] Then, the target audio can be further analyzed and processed, such as extracting audio features, performing anomaly detection, etc., which will not be described in detail here.

[0096] The robot detection method provided by the present invention, by constructing a frequency band division table, can quickly search the frequency band interval from the frequency band division table according to preset inspection items, thereby providing an accurate and efficient method for the robot detection method to determine the key components of the robot to be tested and their sound characteristic frequency ranges, which not only improves the accuracy and reliability of the detection, but also provides strong support for subsequent audio acquisition and analysis.

[0097] Based on the content of the above embodiment, as an optional embodiment, the obtaining of the running image of the robot to be tested running in the test area and determining the running mode of the robot to be tested includes: Inputting the acquired operation image into a target recognition model to obtain position information of the robot to be tested in the operation image output by the target recognition model; Using the mapping relationship between the image coordinate system where the running image is located and the space coordinate system where the test area is located, the actual position of the robot to be tested in the test area is determined according to the position information of the robot to be tested in the running image; The operating mode of the robot to be tested is determined according to the actual position and the preset distribution areas of different operating modes in the test area.

[0098] The following will explain in detail how to obtain the running image of the robot under test running in the test area and determine its running mode.

[0099] First, a video acquisition device or other image acquisition equipment captures the running image or running video of the robot to be tested in the test area (the running image is obtained by extracting frames from the running video).

[0100] The acquired running image is input into a pre-trained target recognition model, which can identify the robot under test in the running image and output its position information in the image coordinate system. This information usually includes the center point coordinates, bounding box, etc. of the robot under test.

[0101] Since the running image is captured on a two-dimensional plane, and the test area is a three-dimensional space, the present invention uses the mapping relationship between the image coordinate system and the space coordinate system to convert the position information of the robot to be tested in the running image into the actual position in the test area. Among them, the determination of the mapping relationship involves the calibration of the camera's intrinsic and extrinsic parameters, as well as geometric transformation methods such as perspective transformation, which can be achieved using some existing methods and will not be elaborated here.

[0102] After determining the actual position of the robot to be tested in the test area, the current operation mode of the robot can be determined based on the preset distribution areas of different operation modes in the test area. These preset distribution areas can be set in advance based on the robot's motion trajectory, work area and other information. For example, if the actual position of the robot to be tested is on the edge of the preset area, it can be determined that it is currently in the Yanbian cleaning mode; if the actual position of the robot is in the preset obstacle area, it can be determined that it is currently in the obstacle avoidance mode.

[0103] Based on the content of the above embodiment, as an optional embodiment, determining the visual anomaly detection result of the robot to be tested in the operating mode according to the operating video includes: Determine the real-time running trajectory of the robot to be tested in the running mode according to the running video; The visual anomaly detection result of the robot to be tested in the operation mode is determined according to the comparison result of the real-time operation trajectory and the reference operation trajectory.

[0104] In the process of testing the robot to be tested, in addition to determining the operation mode and position of the robot to be tested, the most important thing to pay attention to is its operation trajectory, so as to detect whether there are any visual abnormal behaviors through analysis of the operation trajectory. These visual abnormal behaviors may include deviation from the predetermined trajectory, collision with obstacles, jitter or unstable movement, etc. Through visual anomaly detection, the operation problems of the robot can be discovered and corrected in time to ensure that it can complete the task safely and efficiently.

[0105] First, the image frame sequence in the running video can be used to track the motion trajectory of the robot under test through image processing technology (such as optical flow method, feature point matching, etc.), including extracting the position information of the robot under test in each frame of the image and connecting them to form a continuous trajectory line. This trajectory line represents the real-time running trajectory of the robot under test in the running video.

[0106] In order to determine whether the real-time running trajectory is normal, a reference running trajectory is required as a comparison benchmark. The reference running trajectory is usually set in advance based on the task requirements, working environment, preset motion parameters and other information of the robot to be tested, combined with historical data. It can be a precise path line or an area containing multiple key points.

[0107] After the real-time running trajectory and the reference running trajectory are obtained, the two are compared and analyzed, for example, by calculating parameters such as position deviation, angle deviation, speed deviation, etc. between the two.

[0108] If the deviation between the real-time running trajectory and the reference running trajectory exceeds a preset threshold range, it can be considered that the robot under test has visual abnormal behavior.

[0109] In this way, based on the comparison results between the real-time running trajectory and the reference running trajectory, the visual anomaly detection results of the robot under test in the current running mode can be determined. If abnormal behavior is detected, the corresponding alarm information or fault prompt can also be output so that the operator can take timely intervention measures.

[0110] As an optional embodiment, the above-mentioned determination of the real-time running trajectory of the robot to be tested in the running mode according to the running video can be implemented in the following manner: A plurality of video frame images from the start to the end of the detection are intercepted from the running video according to a preset sampling frequency, and a SORT algorithm tracker is initialized; Inputting the starting frame image into the target recognition model, and outputting the recognized starting position information of the robot to be tested; Inputting the starting position information into the SORT algorithm tracker, obtaining the predicted position information of the robot to be tested in the next video frame image output by the SORT algorithm tracker, and matching the predicted position information with the starting position information; If the match is qualified, the state of the SORT algorithm tracker is updated using the predicted position information; Iteratively executing the step of inputting the next video frame image into the target recognition model, and updating the state of the SORT algorithm tracker using the predicted position information, until the traversal of all the video frame images is completed; Obtain the real-time running trajectory of the SORT algorithm tracker determined by all the predicted position information.

[0111] In the process of robot detection, accurately obtaining the real-time running trajectory of the robot is crucial for evaluating its performance, monitoring its behavior, and diagnosing potential problems. This embodiment can achieve continuous and accurate tracking of the robot's motion trajectory by extracting key information from the running video and using SORT combined with Kalman filtering for path tracking quality analysis. The specific implementation is as follows: First, a plurality of video frame images from the start to the end of the detection are captured from the running video according to a preset sampling frequency.

[0112] This sampling frequency can be set to 1 frame per second or faster to ensure that every detail of the robot's motion can be captured. At the same time, the SORT algorithm tracker is initialized to prepare for subsequent tracking tasks.

[0113] Next, the starting frame image is input into a pre-trained target recognition model to use the target recognition model to identify the robot to be tested in the starting frame image and output its starting position information, such as the center point coordinates of the robot to be tested, the size of the bounding box, etc.

[0114] Then, we input the starting position information into the SORT algorithm tracker. Based on this information, the SORT algorithm predicts the position of the robot in the next video frame. In order to verify the accuracy of the prediction, the predicted position information is matched with the starting position information. This matching process usually involves calculating parameters such as position deviation and shape similarity between the two. If the match is qualified, that is, the difference between the predicted position information and the starting position information is within an acceptable range, the state of the SORT algorithm tracker is updated using the predicted position information, indicating that the position of the robot in the next frame has been successfully tracked.

[0115] Then, the above process is iterated: the next video frame image is input into the target recognition model to obtain a new recognition result; the new recognition result is input into the SORT algorithm tracker to obtain new predicted position information; the new predicted position information is matched with the previous tracking result; if the match is qualified, the tracker status is updated.

[0116] This iterative process continues until all video frames are traversed, and the real-time running trajectory determined by all predicted position information of the SORT algorithm tracker can be obtained. This real-time running trajectory is a continuous path line composed of multiple points, which accurately reflects the motion trajectory of the robot under test in the running video.

[0117] The present invention provides an effective real-time running trajectory tracking means for the robot under test detection method by combining the target recognition model and the SORT algorithm tracker, which can accurately capture the motion trajectory of the robot under test in the running video.

[0118] It should be noted that, among the visual abnormality detection results and the audio abnormality detection results in all operating modes, if any one of the visual abnormality detection result or the audio abnormality detection result is abnormal, the comprehensive detection result is determined to be abnormal.

[0119] Figure 3 is a schematic diagram of the structure of the robot detection device provided by the present invention, such as Figure 3 As shown, it mainly includes but is not limited to the following components: The mode recognition unit 31 is used to obtain an operation image of the robot under test running in the test area and determine the operation mode of the robot under test; An information collection unit 32 is used to obtain the operation audio and operation video of the robot to be tested in the operation mode; an audio detection unit 33, configured to determine an audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio feature of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operating mode; A video detection unit 34 is used to determine a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; The iterative control unit 35 is used to control the cyclic operation from the pattern recognition unit to the video anomaly detection unit until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine the comprehensive detection results of the robot under test.

[0120] It should be noted that, when the robot detection device provided by the present invention is specifically operated, it can execute the robot detection method provided by any of the above embodiments, which will not be described one by one here.

[0121] The robot detection device provided by the present invention combines visual detection with audio detection, dynamically adjusts the reference standard of audio detection according to the differences of the robot in different operating modes, and can perform targeted detection on some key stages of the robot, thereby effectively improving the detection accuracy.

[0122] The present invention also provides a robot detection system, which mainly includes but is not limited to: A detection box, wherein a test area is arranged at the bottom of the detection box, and a plurality of preset distribution areas for matching different operation modes of the robot to be tested are arranged in the test area; A sound collection device, used to collect operation videos and operation images of the robot to be tested in the test area; A video acquisition device, used to acquire the operating audio of the robot to be tested in the test area; The processor, the sound collection device and the video collection device are communicatively connected to the processor, and the robot detection method provided by any of the above embodiments is executed when the processor is running.

[0123] Among them, the sound collection device can be a microphone, a pickup, etc., and the video collection device can be a camera, a camera, etc.

[0124] Generally speaking, one or more sound collection devices and video collection devices can be arranged in each detection box according to actual needs.

[0125] As an optional embodiment, there are multiple detection boxes; each detection box is configured with at least one set of the sound collection device and the video collection device.

[0126] When using multiple sound collection devices and video collection devices, by evenly distributing these sound collection devices and video collection devices in the detection box to completely cover the entire test area, it is possible to reduce the sound collection differences caused by the different positions of the robots to be tested, and effectively reduce the problem of the detection box being too high due to the insufficient field of view of a single camera.

[0127] For example, in a traditional single-microphone sound recording system, the position change of the robot under test may affect the sound capture effect. By using multiple microphones for sound recording, the sound recording differences caused by the different positions of the robot under test can be reduced. In addition, the multi-microphone system can enhance the sound from a specific direction through algorithms (such as beamforming technology) while suppressing noise from other directions, thereby improving the clarity and quality of the sound.

[0128] For example, when using a single camera, its field of view is limited and may not fully cover the entire running path of the robot under test. Arranging multiple cameras can provide a wider field of view coverage, ensuring that every movement and path of the robot under test can be monitored. In addition, a single camera may have blind spots, especially when the robot runs to certain parts or corners.

[0129] Figure 4 It is a schematic diagram of the layout of the detection box and internal devices provided by the present invention, such as Figure 4 As shown in the figure, by arranging 4 cameras, it is possible to monitor from different angles, reduce or eliminate these blind spots, and improve the comprehensiveness of the detection. Multiple cameras can provide more data points, through which the path and position of the robot under test can be calculated more accurately. This is crucial for detecting whether the running trajectory of the robot under test is normal.

[0130] The most critical point is that if a single camera is used, in order to cover the entire detection area, the camera may need to be placed at a higher position, which will increase the height of the detection box. By arranging multiple cameras, each camera only needs to cover part of the area, so the installation height of the camera can be lowered, reducing the overall height of the detection box. Compared with raising the height of the detection box to accommodate the field of view of a single camera, using multiple cameras can achieve comprehensive monitoring at a lower height, which may be more economical and save space.

[0131] In addition, multiple cameras can provide data from different perspectives, and more comprehensive motion information of the robot under test can be obtained by fusing these data. That is, this data fusion can improve the accuracy and reliability of path tracking.

[0132] As an optional embodiment, the robot detection system further includes a robot transmission mechanism communicatively connected to the processor; the processor controls the operation of the robot transmission mechanism to transport each of the robots to be tested into an idle detection box.

[0133] Specifically, the robot detection system provided by the present invention can modularly arrange multiple workstations of the production line. For example, eight detection boxes can be stacked in pairs and arranged on both sides of a gantry crane. A gantry crane can then be used to transmit the robot transmission mechanism to send the robots to be tested into idle detection boxes in sequence according to a preset distribution principle, and complete the detection according to the robot detection method provided in any of the above embodiments.

[0134] After the inspection is completed, another robot transport mechanism can be deployed to distinguish between robots with normal inspections and robots with abnormal inspections, and transport them to the next public area.

[0135] As an optional embodiment, the detection box used in the present invention is a soundproof box to isolate external noise interference and provide a more standardized test environment, which is of great significance for sound quality detection. For example, this effect can be achieved by laying sound insulation cotton on the inner and outer walls of the detection box respectively.

[0136] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the robot detection method, which includes: S1, obtaining an operation image of the robot to be tested running in a test area, and determining an operation mode of the robot to be tested; S2, obtaining the running audio and running video of the robot to be tested in the running mode; S3, determining an audio anomaly detection result of the robot under test in the operating mode according to the real-time audio feature of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operating mode; S4, determining a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

[0137] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the robot detection method provided by the above embodiments, the method comprising: S1, obtaining an operation image of the robot to be tested running in a test area, and determining an operation mode of the robot to be tested; S2, obtaining the running audio and running video of the robot to be tested in the running mode; S3, determining an audio anomaly detection result of the robot under test in the operating mode according to the real-time audio feature of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operating mode; S4, determining a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

[0139] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot detection method provided by the above embodiments is implemented, and the method comprises: S1, obtaining an operation image of the robot to be tested running in a test area, and determining the operation mode of the robot to be tested; S2, obtaining the running audio and running video of the robot to be tested in the running mode; S3, determining an audio anomaly detection result of the robot under test in the operating mode according to the real-time audio feature of the operating audio; the audio anomaly detection result is determined according to a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined according to the operating mode; S4, determining a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot detection method, characterized in that: include: S1, obtaining an operation image of the robot to be tested running in a test area, and determining an operation mode of the robot to be tested; S2, obtaining the running audio and running video of the robot to be tested in the running mode; S3, determining an audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio; The audio anomaly detection result is determined based on a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined based on the operation mode; S4, determining a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; Steps S1 to S4 are iteratively executed until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine a comprehensive detection result of the robot under test.

2. The robot detection method according to claim 1, characterized in that: Determining the audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio includes: Determining at least one preset inspection item for inspecting the robot to be tested; Obtaining the frequency band interval of the preset inspection item; Sampling the running audio according to a preset sampling period to obtain at least one audio segment; Setting a bandpass filter according to the frequency band interval of the preset inspection item, so as to filter at least one of the audio segments using the bandpass filter to obtain target audio; Extracting the audio feature of the target audio as the real-time audio feature, and determining the detection result of the preset inspection item according to the comparison result between the real-time audio feature and the audio reference feature; The detection results of all preset inspection items are obtained to determine the audio anomaly detection result of the robot to be tested in the operating mode.

3. The robot detection method according to claim 2, characterized in that: After setting a bandpass filter according to the frequency band interval of the preset inspection item, and using the bandpass filter to filter at least one of the audio segments to obtain the target audio, the method further includes: Performing short-time Fourier transform on the target audio to obtain an audio spectrum graph; Inputting the audio spectrogram into an audio anomaly detection model to obtain a model detection result output by the audio anomaly detection model; Determine the detection result of the preset inspection item by combining the model detection result and the comparison result between the real-time audio feature and the audio reference feature; The audio anomaly detection model is obtained after training with audio spectrogram samples, and each of the audio spectrogram samples is pre-labeled with an audio detection label of the preset inspection item.

4. The robot detection method according to claim 2, characterized in that: The extracting the audio feature of the target audio as the real-time audio feature includes: At least one of a sound intensity value and a sound coarseness value of the target audio is extracted as the real-time audio feature; the sound intensity value is determined according to the root mean square value of the intensity of the target audio, and the sound coarseness value is determined according to the modulation frequency of the target audio.

5. The robot detection method according to claim 2, characterized in that: The determining of at least one preset inspection item for the robot to be tested, and obtaining a frequency band interval of the preset inspection item, includes: Determining the frequency band interval from a pre-constructed frequency band division table according to the preset check items; The frequency band division table records the frequency band intervals corresponding to different preset inspection items, and the frequency band interval of each preset inspection item is determined by the sound characteristic frequency of each preset inspection item in a normal state and a fault state.

6. The robot detection method according to claim 1, characterized in that: The step of obtaining an operation image of the robot to be tested running in a test area and determining an operation mode of the robot to be tested includes: Inputting the acquired operation image into a target recognition model to obtain position information of the robot to be tested in the operation image output by the target recognition model; Using the mapping relationship between the image coordinate system where the running image is located and the space coordinate system where the test area is located, the actual position of the robot to be tested in the test area is determined according to the position information of the robot to be tested in the running image; The operating mode of the robot to be tested is determined according to the actual position and the preset distribution areas of different operating modes in the test area.

7. The robot detection method according to claim 6, characterized in that: Determining the visual anomaly detection result of the robot to be tested in the operating mode according to the operating video includes: Determine the real-time running trajectory of the robot to be tested in the running mode according to the running video; The visual anomaly detection result of the robot to be tested in the operation mode is determined according to the comparison result of the real-time operation trajectory and the reference operation trajectory.

8. The robot detection method according to claim 7, characterized in that: Determining the real-time running trajectory of the robot to be tested in the running mode according to the running video includes: A plurality of video frame images from the start to the end of the detection are intercepted from the running video according to a preset sampling frequency, and a SORT algorithm tracker is initialized; Inputting the starting frame image into the target recognition model, and outputting the recognized starting position information of the robot to be tested; Inputting the starting position information into the SORT algorithm tracker, obtaining the predicted position information of the robot to be tested in the next video frame image output by the SORT algorithm tracker, and matching the predicted position information with the starting position information; If the match is qualified, the state of the SORT algorithm tracker is updated using the predicted position information; Iteratively executing the step of inputting the next video frame image into the target recognition model, and updating the state of the SORT algorithm tracker using the predicted position information, until the traversal of all the video frame images is completed; Obtain the real-time running trajectory of the SORT algorithm tracker determined by all the predicted position information.

9. The robot detection method according to claim 1, characterized in that: Among the visual abnormality detection results and the audio abnormality detection results in all operating modes, if any one of the visual abnormality detection result or the audio abnormality detection result is abnormal, the comprehensive detection result is determined to be abnormal.

10. A robot detection device, characterized in that: include: A pattern recognition unit, used to obtain an operation image of the robot to be tested running in a test area, and determine an operation mode of the robot to be tested; An information collection unit, used for acquiring the operation audio and video of the robot under test in the operation mode; An audio detection unit, used to determine an audio anomaly detection result of the robot to be tested in the operating mode according to the real-time audio features of the operating audio; The audio anomaly detection result is determined based on a comparison result between the real-time audio feature and an audio reference feature, and the audio reference feature is determined based on the operation mode; A video detection unit, used to determine a visual anomaly detection result of the robot to be tested in the operating mode according to the operating video; An iterative control unit is used to control the cyclic operation from the pattern recognition unit to the video anomaly detection unit until the visual anomaly detection results and the audio anomaly detection results of the robot under test in all operating modes in the entire test area are obtained to determine the comprehensive detection results of the robot under test.

11. A robot detection system, characterized in that: include: A detection box, wherein a test area is arranged at the bottom of the detection box, and a plurality of preset distribution areas for matching different operation modes of the robot to be tested are arranged in the test area; A sound collection device, used to collect operation videos and operation images of the robot to be tested in the test area; A video acquisition device, used to acquire the operating audio of the robot to be tested in the test area; The processor, the sound collection device and the video collection device are communicatively connected to the processor, and the robot detection method described in any one of claims 1 to 9 is executed when the processor is running.

12. The robot detection system according to claim 11, characterized in that: There are multiple detection boxes; Each detection box is configured with at least one set of the sound collection device and the video collection device.

13. The robot detection system according to claim 11, characterized in that: Also included is a robotic transport mechanism communicatively coupled to the processor; The processor controls the robot transport mechanism to operate so as to transport each of the robots to be tested into an idle testing box.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the robot detection method according to any one of claims 1 to 9 is implemented.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot detection method according to any one of claims 1 to 9 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the robot detection method according to any one of claims 1 to 9 is implemented.

Images