Equipment fault detection method, robot and electronic equipment

Abstract

The invention provides an equipment fault detection method, a robot and electronic equipment, and the method comprises the steps: obtaining the sound data of target equipment, and carrying out the abnormality prediction of the sound data through a sound abnormality detection model, and obtaining a prediction result; obtaining a visible light image and an infrared image of the target equipment, and determining the real-time temperature of the target equipment according to the visible light image and the infrared image; and performing fault detection on the target equipment according to the prediction result and the real-time temperature of the target equipment. Mutual verification is carried out through combination of sound abnormity of the target equipment and real-time temperature detection determined by an image, fault detection is carried out on the target equipment according to a prediction result of sound data and the real-time temperature of the target equipment, and the multi-modal data of the real-time temperature determined by the sound data and the image can evaluate the health condition of the equipment more comprehensively. The probability of false alarm possibly caused by environmental noise by depending on sound alone is reduced, and the accuracy of fault detection on the equipment is greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial equipment monitoring and fault diagnosis, and in particular, to an equipment fault detection method, a robot and an electronic device. Background Art

[0002] At present, most equipment fault detection methods are based on sound data. For example, if the equipment to be detected is a belt conveyor, fixed microphones are usually arranged near each roller of the belt conveyor. Each microphone sends a sound to the terminal device, so that the terminal device can perform fault detection on the results of each microphone. In the actual practice, it is found that due to the noisy factory and the large number of sound sources, environmental sounds such as pumps, material ports, whistles, etc. often cause false detection. Therefore, the accuracy of equipment fault detection is currently low. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a device fault detection method, a robot and an electronic device, so as to improve the problem of low accuracy in fault detection of the device.

[0004] The embodiment of the present application provides a method for detecting equipment faults, including: obtaining sound data of a target equipment, and using a sound anomaly detection model to predict anomalies in the sound data to obtain prediction results; obtaining a visible light image and an infrared image of the target equipment, and determining the real-time temperature of the target equipment based on the visible light image and the infrared image; and performing fault detection on the target equipment based on the prediction results and the real-time temperature of the target equipment. In the implementation process of the above scheme, by combining the sound anomaly of the target equipment and the real-time temperature detection determined by the image to verify each other, the target equipment is detected for faults based on the prediction results of the sound data and the real-time temperature of the target equipment. This multimodal data of sound data and real-time temperature determined by the image can more comprehensively evaluate the health status of the equipment, effectively reducing the probability of false alarms due to environmental noise when relying solely on sound, thereby greatly improving the accuracy of equipment fault detection.

[0005] Optionally, in an embodiment of the present application, before determining the real-time temperature of the target device according to the visible light image and the infrared image, it also includes: determining that the predicted result is an abnormality. In the implementation of the above scheme, since the sound anomaly detection model is usually much lighter than the image processing model, the sound anomaly detection is performed first, and the image data is further acquired and processed only when the abnormality is detected, which can significantly reduce the consumption of computing resources. Furthermore, sound anomaly detection is usually faster than image processing, and it can quickly determine whether there is a preliminary anomaly, so that a preliminary judgment can be made more quickly, avoiding time-consuming image processing when there is no abnormality, thereby improving the efficiency of fault detection of the equipment.

[0006] Optionally, in an embodiment of the present application, the real-time temperature of the target device is determined based on the visible light image and the infrared image, including: using the target detection and recognition model to recognize the visible light image to obtain a recognition result, the recognition result including: the regional boundary box of the target device in the visible light image; obtaining the camera parameter matrix corresponding to the infrared image, and determining the conversion matrix based on the recognition result and the camera parameter matrix, the conversion matrix is ​​the matrix of the target device from the visible light image to the infrared image; converting the regional boundary box in the visible light image by the conversion matrix to obtain the regional coordinate box of the target device in the infrared image; calculating the temperature of the pixel points of the regional coordinate box in the infrared image to obtain the real-time temperature of the target device. In the implementation process of the above scheme, the model recognition result of the visible light image is used as auxiliary information for determining the real-time temperature of the target device by the infrared image, and the regional coordinate box of the target device in the infrared image is obtained, thereby reducing the error rate of directly identifying the area where the target device is located from the infrared image, overcoming the defect of low accuracy of fault detection of belt conveyor rollers using visible light images alone, and also overcoming the defect of low accuracy of fault detection of rollers using infrared images alone. Therefore, the regional coordinate frame of the target device in the infrared image obtained according to the visible light image is used as auxiliary information, which greatly improves the accuracy of the real-time temperature of the target device, thereby improving the accuracy of fault detection of the device.

[0007] Optionally, in an embodiment of the present application, the recognition result also includes: a registration identifier of the target device; using a target detection and recognition model to recognize the visible light image, including: using the target detection and recognition model to identify a regional bounding box from the visible light image; screening out a registered bounding box with the largest intersection-and-union ratio with the regional bounding box from multiple registered bounding boxes; determining the registration identifier corresponding to the registered bounding box with the largest intersection-and-union ratio as the registration identifier of the target device. In the implementation process of the above scheme, by using the target detection and recognition model to identify the regional bounding box from the visible light image, the position of the target device can be more accurately located, and combined with the intersection-and-union screening technology, the accuracy of recognition can be further improved, and the target device can be accurately distinguished and locked even in complex backgrounds or when similar devices coexist. Since the intersection-and-union screening technology can help the system select the most suitable match from multiple candidate objects, the system's ability to track the same device in continuous frames of the visible light image is further improved. Therefore, by combining the regional bounding box identified by the target detection model and the intersection-and-union screening technology to determine the registration identifier of the target device, not only the area where the target device is located can be identified, but also the registration identifier of the target device can be identified, effectively reducing misidentification caused by background interference or similar devices, thereby improving the accuracy of target recognition.

[0008] Optionally, in an embodiment of the present application, the conversion matrix is ​​determined according to the recognition result and the camera parameter matrix, including: obtaining the pixel width occupied by the target device from the visible light image; obtaining the physical width of the target device, and determining the real distance between the target device and the acquisition device of the visible light image according to the physical width of the target device and the pixel width occupied by the target device; calculating the camera parameter matrix and the real distance between the target device and the acquisition device of the visible light image to obtain the conversion matrix. In the implementation process of the above scheme, the real distance is determined by the physical width of the target device and the pixel width occupied, so that the coordinate transformation can be performed more accurately and the estimation error caused by the single pixel size can be reduced, which means that when the regional bounding box in the visible light image is converted to the infrared image, the position of the target device can be more accurately located, thereby improving the accuracy of fault detection. Furthermore, due to the influence of factors such as lighting conditions and lens distortion, relying solely on pixel information may lead to inaccurate distance estimation. By combining physical size information, the robustness of the system to these external factors can be enhanced, so that the system can maintain good performance in different environments, thereby improving the robustness of fault detection.

[0009] Optionally, in an embodiment of the present application, the recognition result also includes: the category probability of the target device; determining the transformation matrix according to the recognition result and the camera parameter matrix, including: judging whether the category probability of the target device is greater than a preset threshold; if so, determining the transformation matrix according to the recognition result and the camera parameter matrix. In the implementation process of the above scheme, by setting the category probability threshold, the coordinate transformation is performed only when the category probability of the target device is higher than the preset threshold. This ensures that only high-confidence detection results are used for further analysis, thereby reducing erroneous conversions caused by misidentification and improving the overall reliability of fault detection. Furthermore, in an industrial environment, false alarms not only waste resources, but may also cause unnecessary downtime and inspections. By setting the category probability threshold, the results with lower confidence can be filtered out, thereby significantly reducing the false alarm rate and improving the practicality of the system.

[0010] Optionally, in an embodiment of the present application, fault detection is performed on the target device based on the prediction result and the real-time temperature of the target device, including: if the prediction result is that there is an abnormality, and the real-time temperature of the target device is greater than a preset temperature threshold, the target device is added to a list of faulty devices, and the list of faulty devices is used to store devices with abnormal fault detection results. In the implementation process of the above scheme, by combining sound anomaly detection and temperature monitoring, the fault state of the device can be more accurately identified, effectively reducing the false alarms or missed alarms that may occur by relying on a single detection method, ensuring that the device is marked as a fault only when it is actually abnormal, thereby improving the accuracy of fault detection of the device.

[0011] Optionally, in an embodiment of the present application, adding the target device to the faulty device list includes: adding the registration identifier of the target device to the faulty device list; and / or, adding the regional boundary box of the target device to the faulty device list; and / or, adding the regional coordinate box of the target device to the faulty device list. In the implementation process of the above scheme, by providing detailed fault information (such as registration identifier, regional boundary box and / or regional coordinate box), the reviewer can quickly locate the specific device and its faulty area, reduce the search time, and improve the review efficiency.

[0012] Optionally, in an embodiment of the present application, fault detection is performed on the target device according to the prediction result and the real-time temperature of the target device, and it also includes: if the prediction result is abnormal, and the real-time temperature of the target device is less than or equal to the preset temperature threshold, then after the target device is found in the faulty device list, an early warning is issued to the target device. In the implementation process of the above scheme, even if the prediction result is abnormal, if the real-time temperature does not exceed the preset threshold, the system will not immediately mark the device as a fault, but will issue an early warning first, which can reduce the impact of false alarms on production and maintenance work and avoid unnecessary resource investment. Furthermore, by issuing an early warning when the prediction result is abnormal but the temperature has not exceeded the standard, an early warning mechanism can be implemented, which enables the system to remind relevant personnel before the fault is fully manifested, thereby buying more time for taking preventive measures. Early warning allows maintenance personnel to intervene before the equipment fails completely, reducing the number of emergency shutdowns caused by sudden failures, which helps to maintain the continuity and stability of production.

[0013] An embodiment of the present application also provides a robot, including: a sound data acquisition module, used to obtain sound data of a target device; a sound anomaly detection module, used to use a sound anomaly detection model to predict anomalies in the sound data and obtain prediction results; a device image acquisition module, used to obtain visible light images and infrared images of the target device; a real-time temperature determination module, used to determine the real-time temperature of the target device based on the visible light image and the infrared image; and a device fault detection module, used to perform fault detection on the target device based on the prediction results and the real-time temperature of the target device.

[0014] Optionally, in an embodiment of the present application, the real-time temperature determination module includes: a visible light recognition submodule, which is used to use a target detection and recognition model to recognize the visible light image and obtain a recognition result, wherein the recognition result includes: a regional bounding box of the target device in the visible light image; a conversion matrix determination submodule, which is used to obtain a camera parameter matrix corresponding to the infrared image, and determine the conversion matrix based on the recognition result and the camera parameter matrix, the conversion matrix is ​​a matrix that maps the target device from the visible light image to the infrared image; a bounding box conversion submodule, which is used to convert the regional bounding box in the visible light image through the conversion matrix to obtain the regional coordinate box of the target device in the infrared image; and a real-time temperature calculation submodule, which is used to calculate the temperature of the pixels in the regional coordinate box in the infrared image to obtain the real-time temperature of the target device.

[0015] Optionally, in an embodiment of the present application, the recognition result also includes: a registration identifier of the target device; a visible light recognition submodule, including: a bounding box recognition unit, used to use a target detection and recognition model to identify a regional bounding box from a visible light image; a bounding box screening unit, used to screen out a registered bounding box with the largest intersection-and-union ratio with the regional bounding box from multiple registered bounding boxes; and a registration identifier determination unit, used to determine the registration identifier corresponding to the registered bounding box with the largest intersection-and-union ratio as the registration identifier of the target device.

[0016] Optionally, in an embodiment of the present application, a conversion matrix determination submodule includes: a pixel width acquisition unit, used to acquire the pixel width occupied by the target device from the visible light image; a real distance determination unit, used to acquire the physical width of the target device, and determine the real distance between the target device and the visible light image acquisition device based on the physical width of the target device and the pixel width occupied by the target device; a conversion matrix acquisition unit, used to calculate the camera parameter matrix and the real distance between the target device and the visible light image acquisition device to obtain the conversion matrix.

[0017] Optionally, in an embodiment of the present application, the recognition result also includes: the category probability of the target device; a conversion matrix determination submodule, including: a category probability judgment unit, used to judge whether the category probability of the target device is greater than a preset threshold; a conversion matrix determination unit, used to determine the conversion matrix according to the recognition result and the camera parameter matrix if the category probability of the target device is greater than the preset threshold.

[0018] Optionally, in an embodiment of the present application, the device fault detection module includes: a faulty device adding sub-module, which is used to add the target device to a faulty device list if the prediction result is that there is an abnormality and the real-time temperature of the target device is greater than a preset temperature threshold. The faulty device list is used to store devices whose fault detection results are abnormal.

[0019] Optionally, in an embodiment of the present application, the faulty device adding submodule includes: a registration identifier adding unit, used to add the registration identifier of the target device to the faulty device list; and / or, a boundary box adding unit, used to add the regional boundary box of the target device to the faulty device list; and / or, a coordinate frame adding unit, used to add the regional coordinate frame of the target device to the faulty device list.

[0020] Optionally, in an embodiment of the present application, the equipment fault detection module also includes: a target equipment early warning submodule, which is used to issue an early warning to the target equipment after the target equipment is found in the faulty equipment list if the prediction result is an abnormality and the real-time temperature of the target equipment is less than or equal to a preset temperature threshold.

[0021] Optionally, in an embodiment of the present application, the device image acquisition module includes: a visible light camera and an infrared camera; the visible light camera is used to obtain a visible light image of the target device; and the infrared camera is used to obtain an infrared image of the target device.

[0022] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0023] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0024] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A schematic diagram of a process flow of a device fault detection method provided by an embodiment of the present application is shown;

[0027] Figure 2 A schematic diagram showing the recognition result of a visible light image provided by an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the structure of a robot provided by an embodiment of the present application is shown;

[0029] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0031] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally herein can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed for protection, but merely represents the selected embodiments of the embodiments of the present application.

[0032] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0033] It should be noted that the device fault detection method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a robot, a device terminal or a server having the function of executing a computer program. The robot here can be provided with a sound data acquisition module (such as a microphone), a device image acquisition module and a processor, etc., wherein the device image acquisition module may include: an ordinary camera for acquiring visible light images, and an infrared camera for acquiring infrared images. Device terminals include: smart phones, personal computers, tablet computers, personal digital assistants or mobile Internet devices, etc. A server refers to a device that provides computing services through a network, such as: x86 servers and non-x86 servers, and non-x86 servers include: mainframes, minicomputers and UNIX servers.

[0034] The following is an example of an application scenario in which the device fault detection method is applicable: the device fault detection method can be used to detect faults in the target device, where the target device is a device that generates sound data and heat during operation, and these heats will cause real-time temperature changes. The target device is, for example, a belt conveyor, a generator, ventilation equipment or large industrial equipment, etc. For ease of understanding and explanation, the embodiments of the present application are described in detail using the belt conveyor as an example. The sound emitted by the rollers of the belt conveyor when a fault occurs is indeed different from the sound emitted by the normal rollers when the rollers are running. Theoretically, a fixed-point microphone can be arranged near each roller of the belt conveyor, and the sound data collected by the microphone can be used to determine whether the equipment has a fault. However, there are many sources of sound in the noisy environment of the factory, and environmental sounds such as some pumps, drop ports, whistles, etc. often cause false detections. Therefore, the accuracy of fault detection of equipment is currently low.

[0035] For the above questions, please see Figure 1 The flowchart of the device fault detection method provided by the embodiment of the present application is shown; the main idea of ​​the device fault detection method is to verify each other by combining the sound anomaly of the target device and the real-time temperature detection determined by the image, and to detect the fault of the target device according to the prediction result of the sound data and the real-time temperature of the target device. This multimodal data of the sound data and the real-time temperature determined by the image can more comprehensively evaluate the health status of the device, thereby reducing the probability of false alarms due to environmental noise, and ultimately improving the accuracy of device fault detection. The implementation method of the above-mentioned device fault detection method may include:

[0036] Step S110: Acquire the sound data of the target device, and use the sound anomaly detection model to predict anomalies on the sound data to obtain prediction results.

[0037] The target device refers to the device that generates sound data and heat during operation, which will cause real-time temperature changes, such as belt conveyors, generators, ventilation equipment or large industrial equipment, etc. For ease of understanding and explanation, the following detailed explanation is given using the belt conveyor as an example.

[0038] Step S120: acquiring a visible light image and an infrared image of the target device, and determining a real-time temperature of the target device according to the visible light image and the infrared image.

[0039] It can be understood that the visible light image and infrared image of the above-mentioned target device can be obtained by extracting a frame of image from continuous frames in the video. For example, the visible light image of the target device can be an image extracted from a visible light video taken by an ordinary camera set on the robot, and the infrared image can be an image extracted from an infrared video taken by an infrared camera set on the robot.

[0040] After long-term observation, it is found that in the early stage of belt conveyor roller failure, the visible light image of the faulty roller during operation is basically the same as the visible light image of the normal roller during operation. Therefore, it is difficult to distinguish normal rollers from faulty rollers from the visible light image alone. When the roller failure gradually develops to the late stage, the roller sampling produces serious deformation and wear, so that the late-stage faulty rollers and normal rollers can be visually distinguished. Even so, when the failure of some late-stage faulty rollers lies in their internal abnormalities, it is difficult to distinguish the late-stage faulty rollers with internal abnormalities from normal rollers from the visual visible light image. Therefore, the use of visible light images alone to detect roller faults has a lag, resulting in a low accuracy rate for roller fault detection using visible light images alone.

[0041] In order to overcome the above defects, by determining the real-time temperature of the target device based on the visible light image and the infrared image, it is possible to detect abnormalities in the interior of the roller. Even if the interior of the roller fails and generates heat, resulting in abnormal temperature, the real-time temperature of the target device on the infrared image can be used to distinguish between late-stage fault rollers with internal abnormalities and normal rollers. Therefore, determining the real-time temperature of the target device based on the visible light image and the infrared image can effectively distinguish between late-stage fault rollers with internal abnormalities and normal rollers, thereby improving the accuracy of fault detection for belt conveyor rollers.

[0042] Step S130: performing fault detection on the target device according to the prediction result and the real-time temperature of the target device.

[0043] It is understandable that relying solely on sound data for fault detection may be affected by environmental noise, resulting in false alarms in the prediction results of sound data. For example, if the prediction result of sound data is that there is an abnormality, but the real-time temperature of the device has not changed significantly, then it may be a false alarm caused by environmental background noise. Therefore, by introducing temperature data as an auxiliary verification method, it can be ensured that a fault is only determined when multiple indicators show abnormalities at the same time. For example, if the prediction result is that there is an abnormality and the real-time temperature of the target device is greater than the preset temperature threshold, the target device is judged to have a fault. This comprehensive analysis of multimodal data of sound data and real-time temperature can improve the accuracy of equipment fault detection.

[0044] In the implementation process of the above scheme, the sound anomaly of the target device and the real-time temperature detection determined by the image are combined to verify each other, and the fault detection of the target device is performed based on the prediction result of the sound data and the real-time temperature of the target device. This multimodal data of sound data and real-time temperature determined by the image can more comprehensively evaluate the health status of the device, effectively reducing the probability of false alarms due to environmental noise when relying solely on sound, thereby greatly improving the accuracy of fault detection of the device.

[0045] It is understandable that the execution order of the above-mentioned steps S110 and S120 should not be understood as a limitation of the embodiments of the present application. Step S110 may be executed first, and then step S120. Of course, step 120 may also be executed first, and then step S110. In some embodiments, step S110 and step S120 may also be executed concurrently. In other embodiments, if the prediction result in step S110 is that there is no abnormality, then the target device may be directly judged as not having a fault. In this case, step S120 may not be executed, which can effectively improve the detection efficiency of fault-free devices. Therefore, the execution order of the above-mentioned steps S110 and S120 should not be understood as a limitation of the embodiments of the present application.

[0046] For example, for the more specific execution order above: assuming that obtaining the sound data of the target device in the above step S110 is sub-step A, using the sound anomaly detection model to predict the anomaly of the sound data to obtain the prediction result in the above step S110 is sub-step B, and obtaining the visible light image and infrared image of the target device in the above step S120 is sub-step C, and determining the real-time temperature of the target device according to the visible light image and the infrared image in the above step S120 is sub-step D. There are many execution orders for the above sub-steps A, B, C and D, including but not limited to the following:

[0047] In the first case, the execution order of sub-steps A, B, C, and D is usually executed on a server or server cluster with abundant computing and storage resources. Acquiring and processing sound data, visible light images, and infrared images in sequence can fully utilize the advantages of multimodal data and provide more comprehensive fault detection results. In an environment with sufficient resources, a large amount of sound and image data can be saved and processed, providing rich data support for subsequent advanced analysis and fault mode identification.

[0048] Of course, it can also be executed on the front-end equipment on site (such as robots or robot vehicles in factories, etc.), in which case it can be before obtaining the visible light image and infrared image of the target device, and it can also include: determining that the prediction result is an abnormality, that is, obtaining the visible light image and infrared image of the target device only after determining that the prediction result is an abnormality. In the implementation process of the above scheme, since the amount of image data is usually large, if the image data is frequently collected and stored, it will cause pressure on the storage system. The above scheme only collects images when the sound detects an abnormality, which can reduce unnecessary storage requirements and effectively save computing resources and storage resources.

[0049] Alternatively, before determining the real-time temperature of the target device based on the visible light image and the infrared image, it may also include: determining that the prediction result is an abnormality, that is, the image may be collected while the sound is collected, but the real-time temperature of the target device is determined based on the visible light image and the infrared image only after determining that the prediction result is an abnormality. Furthermore, sound abnormality detection is usually faster than image processing, and it can quickly determine whether there is a preliminary abnormality, so that a preliminary judgment can be made more quickly, avoiding time-consuming image processing when there is no abnormality, thereby improving the efficiency of fault detection of the device.

[0050] The second case is the execution order of sub-steps A, C, B and D, or the execution order of sub-steps A, C, D and B, or the execution order of sub-steps C, A, B and D, or the execution order of sub-steps C, A, D and B. In this case, sub-steps A and C are usually executed on the front-end equipment on site (such as robots or robot cars in factories), while sub-steps B and D in this case are executed on servers or server clusters with abundant computing resources and storage resources. The front-end equipment (such as robots or robot cars) only needs to be responsible for data collection tasks, which can reduce its computing burden, because these devices usually have limited computing resources, and servers or server clusters have abundant computing resources and storage space, which can efficiently process large amounts of data, perform complex data analysis and model operations, thereby effectively optimizing the resource allocation between the front-end equipment and the server (cluster). Furthermore, the front-end equipment can quickly collect sound and image data, reduce the delay of data transmission, and while the front-end equipment collects data, the server can start processing the previously uploaded data, realize parallel processing, and improve the overall response speed. In addition, the division of labor and cooperation between front-end devices and servers supports a distributed processing architecture, which can more easily expand the system scale, thereby supporting the scalability of the distributed architecture.

[0051] In the third case, sub-steps A and C are executed simultaneously and concurrently, and / or, sub-steps B and D are executed simultaneously and concurrently. That is, A and C are executed simultaneously in no particular order, and similarly, B and D are executed simultaneously in no particular order. This case is similar to the second case above. Sub-steps A and C are usually executed on the front-end equipment on site (such as robots or robot vehicles in factories), while sub-steps B and D are executed on servers or server clusters with abundant computing resources and storage resources, so they will not be described in detail.

[0052] As an optional implementation of the above step S110, for example: the sound data of the target device is obtained through a sound collection device (such as a microphone set on a robot), and the feature extraction module in the sound anomaly detection model is used to extract features of the sound data (such as 10 seconds of audio) to obtain sound features, where the sound features can be the Mel spectrum features of the 10-second audio. Then, the classification network in the sound anomaly detection model is used to predict the anomaly of the sound features to obtain the prediction results; wherein, the classification network in the sound anomaly detection model here can use the Whisper large model backbone, and use the open source large model Whisper fine-tuning as the backbone, which effectively improves the recognition effect and robustness of sound anomaly detection.

[0053] Optionally, before using the sound anomaly detection model, the sound anomaly detection model can also be trained. When training the model, the training data can be preprocessed first. For example, if the positive and negative samples in the training data are seriously unbalanced, the collected sample data can also be enhanced. If the sound data in the training data is concentrated in the frequency band above 2000Hz, but the energy is far lower than the 0-2000Hz base frequency, the sound data can be obtained by short-time Fourier transform (STFT), and then the 0-2000Hz data is eliminated by using a filter. The waveform is transformed by inverse short-time Fourier transform (ISTFT) to obtain pure fault audio without base frequency. Then, the normal audio and the pure fault audio are superimposed according to the preset signal-to-noise ratio to obtain a large amount of synthetic fault audio, which ensures the balance of the training samples.

[0054] After long-term observation, it is found that there are many reasons why the heating of the roller of the belt conveyor affects the real-time temperature. For example, when the belt conveyor is transporting materials, the materials will press on the roller of the belt conveyor, resulting in long-term friction and heat generation, which will affect the real-time temperature of the roller and become abnormal. It should be noted that the infrared camera will be affected by many factors when collecting infrared images, resulting in poor quality of the collected infrared images. For example, when the temperature difference between the heating components of the roller and the environment is too small, it is difficult to identify the roller in the infrared image even with the target detection and recognition model of deep learning, and it is also difficult to see the roller clearly in the infrared pseudo-color image through the human eye. For example, the texture information in the infrared image is weak, and the infrared image is easily affected by the ambient temperature. The large temperature difference between day and night causes the recognition results of the infrared image collected during the day to be different from the recognition results of the infrared image collected at night. Therefore, the error of using the target detection and recognition model to recognize the infrared image is high, especially when there are other high-temperature objects in the environment of the roller, which will lead to many invalid recognition results. Therefore, the accuracy of using infrared images alone to detect roller faults is also low.

[0055] In order to overcome the above defects, the model recognition results of the visible light image are used as auxiliary information for determining the real-time temperature of the target device using the infrared image, which can effectively distinguish whether the temperature abnormality is caused by an abnormality inside the roller and heat generated, or by heat generated in the external environment of the roller (for example, there are other high-temperature objects in the environment). The specific process of using the visible light image as auxiliary information for determining the real-time temperature of the device is described in detail below.

[0056] As an optional implementation of the above step S120, the implementation of determining the real-time temperature of the target device according to the visible light image and the infrared image may include:

[0057] Step S121: Use the target detection recognition model to recognize the visible light image to obtain a recognition result, which includes: a regional boundary box of the target device in the visible light image.

[0058] See also Figure 2 A schematic diagram of the recognition result of the visible light image provided by an embodiment of the present application is shown; specifically, assuming that the target device is a roller of a belt conveyor, the recognition result may include: detecting a regional boundary box where the target device is located from the visible light image, the probability that the target device in the regional boundary box is the roller of the belt conveyor is 0.95, and the registration identification (Identification, ID) of the target device is 170 in the figure.

[0059] Optionally, before the target detection and recognition model is used to recognize the visible light image, the target detection and recognition model can also be trained using a training data set, for example: a training data set is obtained, the training data set includes: sample images and sample labels, the sample images may include target devices, and the sample labels include the regional bounding box of the target device in the sample image, and the category of the target device. The untrained target detection and recognition model is trained using the above sample images and sample labels to obtain the above target detection and recognition model.

[0060] Step S122: Obtain a camera parameter matrix corresponding to the infrared image, and determine a conversion matrix based on the recognition result and the camera parameter matrix. The conversion matrix is ​​a matrix that the target device uses to convert and map the visible light image to the infrared image.

[0061] It is understandable that the infrared camera for collecting infrared images and the ordinary camera for collecting visible light images can be set very close to each other and can keep the relative position stable. For example, an ordinary camera for collecting visible light images is set at the left eye of the robot, and an infrared camera for collecting infrared images is set at the right eye of the same robot. Then, the infrared camera is calibrated to obtain the camera parameter matrix of the infrared camera. Similarly, the ordinary camera is calibrated to obtain the camera parameter matrix of the ordinary camera. Among them, the determination process of the above-mentioned conversion matrix is ​​relatively complicated, so the determination process of the conversion matrix will be described in detail below.

[0062] Step S123: transforming the regional boundary frame in the visible light image through a transformation matrix to obtain a regional coordinate frame of the target device in the infrared image.

[0063] The implementation of the above step S123 is, for example: obtaining the regional boundary box in the visible light image, calculating the coordinates of the four vertices of the regional boundary box, where the coordinates of the four vertices include: (xmin, ymin), (xmax, ymin), (xmin, ymax) and (xmax, ymax). Using the transformation matrix, the new coordinates of each vertex are calculated to obtain the new coordinates of the four vertices, and then the regional coordinate box of the target device in the infrared image is determined according to the new coordinates of the four vertices.

[0064] Step S124: Calculate the temperature of the pixels of the regional coordinate frame in the infrared image to obtain the real-time temperature of the target device.

[0065] The implementation method of the above step S124 is, for example, to use the following formula:

[0066] t=percentile(irimg[box],80%),t list =[t1,t2,t3,…,t n ]

[0067] Calculate the temperature of the pixels in the infrared image of the regional coordinate frame to obtain the real-time temperature of the target device.

[0068] Where t is the real-time temperature of the target device (for example, the real-time temperature of the roller), percentile is the quantile function, irimg is the temperature value in the infrared image, box is the coordinate of the target device in the regional coordinate box in the infrared image, and t list To record the temperature array of the target device in the area coordinate frame of the infrared image, when the real-time temperature data stream enters, each real-time temperature is stored in the list. When the cumulative number exceeds the preset number (for example, 5), t can be calculated. list The median value of median , and t list The median value of median Determine the real-time temperature of the target device.

[0069] In the implementation process of the above scheme, the model recognition result of the visible light image is used as auxiliary information for determining the real-time temperature of the target device by the infrared image, and the regional coordinate frame of the target device in the infrared image is obtained, thereby reducing the error rate of directly identifying the area where the target device is located from the infrared image, overcoming the defect of low accuracy of fault detection of belt conveyor rollers using visible light images alone, and also overcoming the defect of low accuracy of fault detection of rollers using infrared images alone. Therefore, the regional coordinate frame of the target device in the infrared image obtained from the visible light image as auxiliary information greatly improves the accuracy of the real-time temperature of the target device, thereby improving the accuracy of fault detection of the device.

[0070] As an optional implementation of the above step S121, the above recognition result may also include: a registration identification (ID) of the target device; the above implementation of using the target detection and recognition model to recognize the visible light image may include:

[0071] Step S121a: Use the target detection and recognition model to identify the region bounding box from the visible light image. It is understandable that the above-mentioned visible light image can be taken by a device traveling along a preset track, or by a device moving along a preset path, such as a robot that cruises or inspects along a pre-set hanging track in a factory, or an industrial camera that moves along a preset path (including an ordinary camera that captures visible light images and / or an infrared camera that captures infrared images).

[0072] An implementation example of the above step S121a is: use a YOLO series target detection and recognition model to recognize the visible light image to obtain a recognition result, wherein the above YOLO series target detection and recognition model can adopt a YOLOv5 model, a YOLOv6 model, a YOLOv7 model, a YOLOv8 model or a YOLOv8-ST target detection network model, etc., and the above recognition result can include a regional bounding box of the target device in the visible light image.

[0073] Step S121b: Filter out a registered bounding box having the largest intersection-over-union ratio with the regional bounding box from the multiple registered bounding boxes.

[0074] An implementation example of the above step S121b is as follows: the formula IOU=(A∩B) / (A∪B) can be used to query and filter out the registered bounding box with the largest intersection over union ratio with the regional bounding box from multiple registered bounding boxes in the database, where IOU represents the intersection over union (IOU) between the registered bounding box and the regional bounding box, A represents the registered bounding box in the database, B represents the regional bounding box identified from the visible light image, and A and B are detection boxes in the visible light image captured by an image acquisition device (such as a camera set on a robot or an industrial camera moving along a preset path) at different times at the same position.

[0075] Step S121c: Determine the registration identifier corresponding to the registration bounding box with the largest intersection-over-union ratio as the registration identifier of the target device.

[0076] The implementation method of the above step S121c is, for example: obtaining the registration identifier corresponding to the registration bounding box with the largest intersection-over-union ratio from the above database, and determining the registration identifier corresponding to the registration bounding box with the largest intersection-over-union ratio as the registration identifier of the target device. Among them, the above multiple registration bounding boxes are bounding boxes of registration identifiers that are manually marked in advance. In the marking project, 10 or 20 rollers can be marked together. The multiple registration bounding boxes here can be obtained from a database, and the database can store registration bounding boxes and registration identifiers, as well as the corresponding relationship between registration bounding boxes and registration identifiers.

[0077] As an optional implementation of the above step S122, the above implementation of determining the conversion matrix according to the recognition result and the camera parameter matrix may include:

[0078] Step S122a: Obtain the pixel width occupied by the target device from the visible light image.

[0079] The implementation of the above step S122a is, for example, to calculate the region bounding box of the target device identified from the visible light image to obtain the pixel width occupied by the target device. Specifically, assuming that the target device is a roller of a belt conveyor, the pixel width of the roller bearing can be counted from the region bounding box.

[0080] Step S122b: Acquire the physical width of the target device, and determine the actual distance between the target device and the visible light image acquisition device according to the physical width of the target device and the pixel width occupied by the target device.

[0081] Step S122c: Calculate the camera parameter matrix and the actual distance between the target device and the device for acquiring the visible light image to obtain a conversion matrix.

[0082] For example, the implementation of the above steps S122b to S122c is as follows: the physical width of the target device can be manually measured in advance or provided by the manufacturer. In this case, when in use, the physical width of the target device can be directly read from the configuration file or database. Then, the physical width of the target device and the pixel width occupied by the target device are calculated, so as to determine the actual distance between the target device and the visible light image acquisition device. Then, the following formula is used:

[0083]

[0084] The camera parameter matrix and the actual distance between the target device and the device for acquiring the visible light image are calculated to obtain a transformation matrix, which is also called a 3D affine matrix.

[0085] Wherein, z is the actual distance between the devices collecting visible light images, F is the focal length of the camera in the camera parameter matrix, d is the physical width of the target device (e.g. the bearing width of the roller), D is the pixel width of the target device in the visible light image (e.g. the pixel diameter of the bearing of the roller in the visible light image), rx and ry are the converted values ​​of the x-axis and y-axis coordinates of the point in the visible light image coordinate system, x and y are the x-axis and y-axis coordinates of the point in the visible light image coordinate system, crx and cry are the x-axis and y-axis coordinates of the center point of the visible light image, fr is the focal length of the ordinary camera collecting the visible light image, x1, y1 and z1 are the coordinate values ​​of the x-axis, y-axis and z-axis in the infrared image coordinate system, R is the rotation matrix in the transformation matrix from the physical coordinates of the visible light to the physical coordinates of the infrared camera, T is the translation vector in the transformation matrix from the physical coordinates of the visible light to the physical coordinates of the infrared camera, fr1 is the focal length of the infrared camera, x ir and ir They represent the x-axis coordinate and y-axis coordinate of a point in the infrared image coordinate system, respectively. crx1 and cry1 represent the x-axis coordinate and y-axis coordinate of the center point of the infrared image, respectively.

[0086] As an optional implementation of the above step S122, the above recognition result may also include: category probability of the target device; the above implementation of determining the conversion matrix according to the recognition result and the camera parameter matrix may include:

[0087] Step S122d: Determine whether the category probability of the target device is greater than a preset threshold.

[0088] Step S122e: If the category probability of the target device is greater than a preset threshold, a conversion matrix is ​​determined according to the recognition result and the camera parameter matrix.

[0089] The implementation method of the above steps S122d to S122e is, for example: first use an executable program to determine whether the category probability of the target device is greater than a preset threshold, and then, if the category probability of the target device is greater than the preset threshold, determine the conversion matrix based on the recognition result and the camera parameter matrix. The implementation method of determining the conversion matrix here refers to the implementation method of the above steps S122a to S122c, so it will not be repeated. If the category probability of the target device is less than or equal to the preset threshold, continue to collect the sound data of the target device, and execute the above step S110 based on the sound data. In the implementation process of the above scheme, by setting the category probability threshold, coordinate transformation is performed only when the category probability of the target device is higher than the preset threshold. This ensures that only high-confidence detection results will be used for further analysis, thereby reducing erroneous conversions caused by misidentification and improving the overall reliability of fault detection.

[0090] As an optional implementation of the above step S130, the above implementation of performing fault detection on the target device according to the prediction result and the real-time temperature of the target device may include:

[0091] Step S131: If the prediction result is that there is an abnormality, and the real-time temperature of the target device is greater than the preset temperature threshold, the target device is added to a faulty device list, which is used to store devices whose fault detection results are abnormal.

[0092] It is understandable that the above preset temperature threshold can be calculated based on historical temperature data, for example, using the formula t median +thresold calculates the above preset temperature threshold, where t median is the temperature array t of the target device's regional coordinate frame in the infrared image list The median temperature value calculated by the preset number (for example, 5) stored in , thresold is the manually set temperature threshold (which can be set to 10 degrees). Therefore, if the real-time temperature of the target device t>t median +thresold, the target device is judged to be faulty, that is, the target device is added to the faulty device list. The faulty device list is used to store devices with abnormal fault detection results.

[0093] As an optional implementation of the above step S131, the above implementation of adding the target device to the faulty device list may include: adding the registration identifier of the target device to the faulty device list, and / or, adding the regional boundary box of the target device to the faulty device list, and / or, adding the regional coordinate frame of the target device to the faulty device list. That is to say, after the above target device is added to the faulty device list, it can be manually reviewed whether the target device is truly faulty, and when manually reviewing, the review can be performed based on any one of the registration identifier of the target device, the regional boundary box and / or the regional coordinate frame, or any combination of two or more.

[0094] As an optional implementation of the above step S130, the above implementation of performing fault detection on the target device according to the prediction result and the real-time temperature of the target device may also include:

[0095] Step S132: If the prediction result shows that an abnormality exists and the real-time temperature of the target device is less than or equal to the preset temperature threshold, an early warning is issued to the target device after the target device is found in the faulty device list.

[0096] The implementation method of the above step S132 is, for example: if the prediction result is that there is an abnormality, and the real-time temperature of the target device is less than or equal to the preset temperature threshold, the historical alarm information and historical temperature trend of the target device are found according to the registration identifier of the target device, and then the target device is warned according to the historical alarm information and historical temperature trend. For example, if the target device is found in the list of faulty devices, and there is a trend that the historical temperature increases by a magnitude greater than the preset threshold, it means that the target device has an alarm history and is deteriorating, and the target device can be warned at this time. On the contrary, if the prediction result is that there is no abnormality, it is determined that the fault detection result is that there is no fault, and then the above step S110 is continued to be executed to continue the device fault detection.

[0097] See also Figure 3 The schematic diagram of the structure of the robot provided in the embodiment of the present application is shown; the embodiment of the present application provides a robot 200, including:

[0098] The sound data acquisition module 210 is used to acquire the sound data of the target device. The sound data acquisition module may be a microphone, so the sound data of the target device may be acquired by the microphone.

[0099] The sound anomaly detection module 220 is used to use the sound anomaly detection model to perform anomaly prediction on the sound data to obtain a prediction result.

[0100] The device image acquisition module 230 is used to acquire the visible light image and infrared image of the target device.

[0101] The real-time temperature determination module 240 is used to determine the real-time temperature of the target device according to the visible light image and the infrared image.

[0102] The device fault detection module 250 is used to perform fault detection on the target device according to the prediction result and the real-time temperature of the target device.

[0103] As an optional implementation of the above robot, the real-time temperature determination module includes:

[0104] The visible light recognition submodule is used to use the target detection recognition model to recognize the visible light image and obtain the recognition result. The recognition result includes: the regional boundary box of the target device in the visible light image.

[0105] The conversion matrix determination submodule is used to obtain the camera parameter matrix corresponding to the infrared image, and determine the conversion matrix based on the recognition result and the camera parameter matrix. The conversion matrix is ​​the matrix that the target device converts and maps from the visible light image to the infrared image.

[0106] The bounding box conversion submodule is used to convert the regional bounding box in the visible light image through a conversion matrix to obtain the regional coordinate box of the target device in the infrared image.

[0107] The real-time temperature calculation submodule is used to calculate the temperature of the pixel points of the regional coordinate frame in the infrared image to obtain the real-time temperature of the target device.

[0108] As an optional implementation of the robot, the recognition result further includes: a registration identifier of the target device; and a visible light recognition submodule including:

[0109] The bounding box recognition unit is used to recognize the region bounding box from the visible light image using the target detection recognition model.

[0110] The bounding box screening unit is used to screen out a registered bounding box having the largest intersection-over-union ratio with the regional bounding box from multiple registered bounding boxes.

[0111] The registration identifier determining unit is used to determine the registration identifier corresponding to the registration bounding box with the largest intersection-over-union ratio as the registration identifier of the target device.

[0112] As an optional implementation of the above robot, the conversion matrix determination submodule includes:

[0113] The pixel width acquisition unit is used to acquire the pixel width occupied by the target device from the visible light image.

[0114] The real distance determination unit is used to obtain the physical width of the target device and determine the real distance between the target device and the visible light image acquisition device according to the physical width of the target device and the pixel width occupied by the target device.

[0115] The conversion matrix obtaining unit is used to calculate the camera parameter matrix and the real distance between the target device and the visible light image acquisition device to obtain the conversion matrix.

[0116] As an optional implementation of the robot, the recognition result further includes: a category probability of the target device; a conversion matrix determination submodule, including:

[0117] The category probability judgment unit is used to judge whether the category probability of the target device is greater than a preset threshold.

[0118] The conversion matrix determination unit is used to determine the conversion matrix according to the recognition result and the camera parameter matrix if the category probability of the target device is greater than a preset threshold.

[0119] As an optional implementation of the above robot, the equipment fault detection module includes:

[0120] The faulty device adding submodule is used to add the target device to the faulty device list if the prediction result is an abnormality and the real-time temperature of the target device is greater than the preset temperature threshold. The faulty device list is used to store devices with abnormal fault detection results.

[0121] As an optional implementation of the above robot, the faulty device is added with a submodule, including:

[0122] The registration identification adding unit is used to add the registration identification of the target device to the fault device list.

[0123] And / or, a bounding box adding unit, used to add a regional bounding box of the target device to the faulty device list.

[0124] And / or, a coordinate frame adding unit, used to add the regional coordinate frame of the target device to the faulty device list.

[0125] As an optional implementation of the above robot, the equipment fault detection module further includes:

[0126] The target device early warning submodule is used to issue an early warning to the target device after finding the target device in the faulty device list if the prediction result shows that there is an abnormality and the real-time temperature of the target device is less than or equal to the preset temperature threshold.

[0127] As an optional implementation of the above robot, the device image acquisition module includes: a visible light camera and an infrared camera.

[0128] The visible light camera is used to obtain a visible light image of the target device.

[0129] Infrared cameras are used to obtain infrared images of target devices.

[0130] It should be understood that the robot corresponds to the above-mentioned device fault detection method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the robot can be referred to in the above description, and the detailed description is appropriately omitted here. The robot includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the robot.

[0131] See also Figure 4 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0132] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0133] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0134] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the robot human embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed robot and method can also be implemented in other ways. The robot embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the robots, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0136] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0137] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A device fault detection method, characterized in that: include: Acquire sound data of a target device, and use a sound anomaly detection model to perform anomaly prediction on the sound data to obtain a prediction result; Acquire a visible light image and an infrared image of the target device, and determine a real-time temperature of the target device according to the visible light image and the infrared image; Fault detection is performed on the target device according to the prediction result and the real-time temperature of the target device.

2. The method according to claim 1, characterized in that Before determining the real-time temperature of the target device according to the visible light image and the infrared image, the method further includes: It is determined that the predicted result is abnormal.

3. The method according to claim 1, characterized in that The determining the real-time temperature of the target device according to the visible light image and the infrared image includes: Using a target detection recognition model to recognize the visible light image, obtaining a recognition result, wherein the recognition result includes: a region boundary box of the target device in the visible light image; Acquire a camera parameter matrix corresponding to the infrared image, and determine a conversion matrix according to the recognition result and the camera parameter matrix, wherein the conversion matrix is ​​a matrix that the target device converts and maps from the visible light image to the infrared image; The regional boundary frame in the visible light image is transformed by the transformation matrix to obtain the regional coordinate frame of the target device in the infrared image; The temperature of the pixels of the regional coordinate frame in the infrared image is calculated to obtain the real-time temperature of the target device.

4. The method according to claim 3, characterized in that The recognition result also includes: a registration identifier of the target device; and the use of the target detection and recognition model to recognize the visible light image includes: Using a target detection and recognition model to identify the region bounding box from the visible light image; Filtering out, from a plurality of registered bounding boxes, a registered bounding box having a maximum intersection-over-union ratio with the region bounding box; The registration identifier corresponding to the registration bounding box with the largest intersection-over-union ratio is determined as the registration identifier of the target device.

5. The method according to claim 3, characterized in that: The determining of the conversion matrix according to the recognition result and the camera parameter matrix comprises: Acquire a pixel width occupied by the target device from the visible light image; Acquire the physical width of the target device, and determine the real distance between the target device and the visible light image acquisition device according to the physical width of the target device and the pixel width occupied by the target device; The camera parameter matrix and the actual distance between the target device and the device for acquiring the visible light image are calculated to obtain the conversion matrix.

6. The method according to claim 3, characterized in that The recognition result also includes: the category probability of the target device; and determining the conversion matrix according to the recognition result and the camera parameter matrix includes: Determine whether the category probability of the target device is greater than a preset threshold; If so, the transformation matrix is ​​determined according to the recognition result and the camera parameter matrix.

7. The method according to any one of claims 1 to 6, characterized in that: The performing fault detection on the target device according to the prediction result and the real-time temperature of the target device includes: If the prediction result is abnormal, and the real-time temperature of the target device is greater than a preset temperature threshold, the target device is added to a faulty device list, and the faulty device list is used to store devices with abnormal fault detection results.

8. The method according to any one of claims 1 to 5, characterized in that: The performing fault detection on the target device according to the prediction result and the real-time temperature of the target device further includes: If the prediction result shows that there is an abnormality, and the real-time temperature of the target device is less than or equal to a preset temperature threshold, then after the target device is found in the faulty device list, an early warning is issued to the target device.

9. A robot, characterized in that: include: A sound data acquisition module is used to obtain sound data of the target device; A sound anomaly detection module, used to use a sound anomaly detection model to perform anomaly prediction on the sound data to obtain a prediction result; A device image acquisition module, used to acquire a visible light image and an infrared image of the target device; A real-time temperature determination module, used to determine the real-time temperature of the target device according to the visible light image and the infrared image; The device fault detection module is used to perform fault detection on the target device according to the prediction result and the real-time temperature of the target device.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

12. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images