Equipment inspection and management method, equipment and storage medium based on image recognition

Abstract

The present application discloses an equipment inspection and management method based on image recognition, which includes: digitally filing the target area and the equipment in the area, obtaining the basic information, maintenance information and location information of the equipment, and setting a corresponding ID for each device; establishing a digital twin model for the target area and the equipment in the area; obtaining the image and location information of the faulty equipment uploaded by the inspection device, and determining the faulty equipment ID based on image recognition and / or the digital twin model; generating a work order based on the faulty equipment ID and generating a corresponding fault description. The present invention also discloses an equipment inspection and management device and storage medium based on digital twin and image recognition. The equipment inspection and management method based on image recognition provided by the present invention realizes rapid identification and fault detection of equipment, shortens fault response time, and reduces losses caused by equipment failure.

Description

Technical Field

[0001] The present application relates to the technical field of equipment management, and more specifically, to equipment inspection and management methods, equipment and storage media based on image recognition. Background Art

[0002] The traditional equipment inspection method mainly relies on manual observation and recording, which was more common in early industrial production environments. However, with the continuous improvement of industrial automation levels and the complexity of production environments, traditional inspection methods have gradually exposed many problems. First, the efficiency of manual observation and recording is relatively low, especially in large factories, where the number of equipment is large, the inspection routes are long, and each inspection takes a long time, making it difficult to effectively record the failure of specific equipment. Secondly, manual recording has a high possibility of error. Inspection personnel are prone to omissions or misrecording during the recording process, and these human factors will lead to inaccurate inspection results.

[0003] With the development of science and technology, the types and complexity of factory equipment are increasing. Traditional equipment inspection methods can no longer meet the needs of modern factories for fault detection and emergency response. Therefore, improving the existing manual inspection process, improving the efficiency and accuracy of inspection, and better adapting to the needs of fault detection in complex environments have become issues that need to be solved urgently. Summary of the invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides an equipment inspection and management method, equipment and storage medium based on image recognition, which will solve at least one of the problems existing in the above-mentioned background technology.

[0005] To achieve the above object, according to a first aspect of the present invention, a method for equipment inspection and management based on image recognition is provided, comprising the following steps:

[0006] Digitally archive the target area and the equipment in the area, obtain the basic information, maintenance information and location information of the equipment, and set a corresponding ID for each device;

[0007] Establish a digital twin model for the target area and the equipment within the area;

[0008] Obtain the image and location information of the faulty device uploaded by the inspection device, and determine the faulty device ID based on image recognition and / or the digital twin model;

[0009] Generate a work order based on the faulty device ID and the corresponding fault description;

[0010] The step of obtaining the image and location information of the faulty device uploaded by the inspection device and determining the faulty device ID based on image recognition and / or the digital twin model specifically includes:

[0011] Based on the positioning system and Bluetooth beacon positioning submodule, obtain the location information of the inspection equipment;

[0012] Based on the shooting elevation angle, shooting focal length and size of the faulty device, the distance between the faulty device and the inspection device is calculated to obtain the location information of the faulty device.

[0013] Based on the location information of the faulty device, determine the type of the faulty device through image recognition, and obtain the ID of the faulty device in the device database;

[0014] The positioning system and the Bluetooth beacon positioning submodule are used to obtain the location information of the inspection device. After the inspection device takes an image of the faulty device, the following steps are also included:

[0015] The inspection device is rotated at a first angle in situ to capture a first environment image, and then rotated at the first angle again to capture a second environment image. After repeating the above steps n times, n environment images are obtained.

[0016] Based on image recognition and digital twin models, the location data of the inspection equipment that takes each environmental image is obtained respectively;

[0017] Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection device is obtained and fused with the location data of the inspection device that takes each environmental image to obtain the updated location information of the inspection device.

[0018] Furthermore, the above-mentioned equipment inspection and management method based on image recognition, the location information of the inspection equipment is obtained based on the positioning system and the Bluetooth beacon positioning submodule, and the location data of the inspection equipment that takes each environmental image is obtained and fused, specifically including:

[0019] Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection equipment and the location data of the inspection equipment that takes each environmental image are obtained, and the updated location information of the inspection equipment is obtained.

[0020]

[0021] in The location information of the updated inspection equipment. and To obtain the location information and corresponding weight of the inspection equipment based on the positioning system and Bluetooth beacon positioning submodule, and is the location and corresponding weight of the inspection equipment in each environment image.

[0022] Furthermore, the above-mentioned equipment inspection and management method based on image recognition, the distance between the faulty equipment and the inspection equipment is calculated based on the shooting elevation angle, shooting focal length and size of the faulty equipment of the inspection equipment, and the location information of the faulty equipment is obtained, specifically including:

[0023] The size of the faulty device in the image is obtained based on image recognition, and the distance between the target device and the inspection device is obtained based on the shooting focal length and the actual size of the device in the basic information of the device.

[0024]

[0025] Based on the location information of the inspection equipment, obtain the location information of the faulty equipment

[0026]

[0027]

[0028] The orientation of the inspection device is obtained based on the built-in sensor of the inspection device, and the specific location of the faulty device is obtained according to the position of the inspection device, the height of the device, the horizontal distance between the inspection device and the device, and the orientation angle of the inspection device.

[0029] Furthermore, the above-mentioned equipment inspection and management method based on image recognition, after obtaining the specific location of the faulty equipment, further includes:

[0030] The inspection device moves a first distance in a first angle direction to a first inspection device position, and obtains position information of the first inspection device position;

[0031] Taking an image of the faulty device at the first inspection device position, and obtaining direction and distance information of the faulty device relative to the first inspection device position at the first inspection device position based on image recognition;

[0032] The inspection device moves a second distance in a second angle direction to a second inspection device position, and obtains position information of the second inspection device position;

[0033] Taking an image of the faulty device at the second inspection device position, and obtaining direction and distance information of the faulty device relative to the second inspection device position at the second inspection device position based on image recognition;

[0034] Based on the position of the first inspection device, the direction and distance information of the faulty device relative to the position of the first inspection device, the position of the second inspection device, and the direction and distance information of the faulty device relative to the position of the second inspection device, the position information of the first faulty device is obtained by triangulation;

[0035] Repeat the above steps until k faulty device location information is obtained;

[0036] After fusing the specific location of the faulty device and the location information of k faulty devices, the updated specific location of the faulty device is obtained.

[0037] Furthermore, in the above-mentioned equipment inspection and management method based on image recognition, the specific location of the faulty equipment and the location information of k faulty equipment are fused and processed to obtain the updated specific location of the faulty equipment, which specifically includes:

[0038] S1 obtains an initial estimate, an initial uncertainty value and a system noise parameter, wherein the initial estimate is the specific location of the faulty device, the initial uncertainty value is a preset confidence level of the quantified initial estimate, and the system noise parameter is a preset device accuracy parameter;

[0039] S2 updates the uncertainty value based on the location information of one of the faulty devices and the system noise parameter;

[0040] S3 generates an error residual based on the location information of one of the faulty devices and the uncertainty value, and corrects the initial estimated value according to the residual to obtain an updated estimated value;

[0041] S4 executes steps S2 and S3 for each piece of faulty device location information in turn, gradually revising the initial estimated value, and obtaining the updated specific location of the faulty device.

[0042] Furthermore, the above-mentioned equipment inspection and management method based on image recognition, after obtaining the location of the faulty equipment, further includes:

[0043] Obtain a digital twin model within a first distance range of the location of the inspection device and the location of the faulty device;

[0044] The model display angle is adjusted based on the shooting elevation angle of the inspection equipment, and the digital twin model is scaled accordingly based on the size of the faulty device in the image of the faulty device uploaded by the inspection equipment, so that the image of the faulty device coincides with the corresponding model in the digital twin model, and the faulty device ID is obtained.

[0045] According to the second aspect of the present invention, there is also provided an equipment inspection and management device based on digital twins and image recognition, which includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of any one of the above methods.

[0046] According to the third aspect of the present invention, a storage medium is also provided, which stores a computer program that can be executed by an equipment inspection and management device based on digital twins and image recognition. When the computer program runs on the equipment inspection and management device based on digital twins and image recognition, the equipment inspection and management device based on digital twins and image recognition executes the steps of any one of the above methods.

[0047] According to a fourth aspect of the present invention, there is also provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0048] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0049] The equipment inspection and management method based on image recognition provided by the present invention determines the type and location of faulty equipment based on image recognition and / or a digital twin model, generates a work order according to the type and location of the faulty equipment, and generates a corresponding fault description. This can improve the efficiency and accuracy of equipment inspections, reduce the cost and risk of relying on manual inspections, achieve rapid identification and fault detection of equipment, achieve temporary emergency disposal of equipment, shorten fault response time, and reduce losses caused by equipment failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of the process of the equipment inspection and management method based on image recognition provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a positioning method for an equipment inspection and management method based on image recognition provided in an embodiment of the present application;

[0053] FIG3 is a schematic diagram of an equipment inspection and management device based on image recognition provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0056] Figure 1 A flow chart of a device inspection and management method based on image recognition provided in an embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0057] 101. Digitally archive the target area and the equipment in the area, obtain the basic information, maintenance information and location information of the equipment, and set a corresponding ID for each device.

[0058] Specifically, the target area can be a factory park or within a company. The equipment in the target area is digitally archived, and the basic information, maintenance information, and location information of each device are stored in a cloud server or local database. A corresponding ID is set for each device to facilitate query and update at any time.

[0059] It is possible to collect multi-angle photos of the equipment in the target area and the materials that need to be inspected, as well as normal or faulty photos of other similar products, and use object detection and image segmentation models to train large AI models in vertical fields.

[0060] Commonly used object detection and image segmentation models include the R-CNN series, YOLO series, SSD, RetinaNet or other models. You can choose the model type according to actual application requirements.

[0061] 102. Establish a digital twin model for the target area and the equipment within the area.

[0062] Specifically, data is collected from the physical entities of the target area and the equipment in the target area. The collected data may include sensor data, surveillance camera data, and historical data. The collected data is cleaned, integrated, and stored. After the data collection is completed, a digital twin model of the target area is established based on the actual situation. The methods for establishing the model include mathematical modeling, computer simulation, machine learning, etc. The established model is verified to ensure the accuracy and reliability of the established model.

[0063] If the type or location of the equipment in the target area changes, the digital twin model is updated to ensure the consistency between the digital twin model and the actual situation.

[0064] 103. Obtain the image and location information of the faulty device uploaded by the inspection device, and determine the faulty device ID based on image recognition and / or the digital twin model.

[0065] Specifically, the inspection device may include a user-held inspection device, such as a mobile phone or a professional inspection device, or may be a fully automatic inspection device such as an inspection robot.

[0066] During the inspection process, the inspection device will turn on the positioning system and Bluetooth module. When a faulty device is found, the inspection device will take a photo of the faulty device and upload it. The uploaded data includes the picture of the faulty device and the specific location information of the device when the picture is taken. After receiving the relevant data of the uploaded faulty device, the faulty device is identified through the image recognition model, and the device ID of the corresponding device is searched in the database.

[0067] If the faulty device cannot be identified from the image recognition model alone, the pre-created digital twin model is called to overlap the digital twin model with the image of the device, the device that actually failed is found, and the corresponding device ID is searched in the database.

[0068] 104. Generate a work order based on the faulty equipment ID and generate a corresponding fault description.

[0069] Specifically, the work order is generated together with the equipment ID information and photos. The image recognition model can identify the specific situation of the equipment failure and give a description of the failure. Based on the fault work order, corresponding processing can be carried out, realizing temporary emergency disposal of the equipment and shortening the fault response time.

[0070] The image recognition-based equipment inspection and management method provided in the embodiment of the present application determines the type and location of faulty equipment based on image recognition and / or a digital twin model, generates a work order according to the type and location of the faulty equipment, and generates a corresponding fault description. This can improve the efficiency and accuracy of equipment inspections, reduce the cost and risk of relying on manual inspections, achieve rapid identification and fault detection of equipment, achieve temporary emergency disposal of equipment, shorten fault response time, and reduce losses caused by equipment failures.

[0071] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application obtains the image and location information of the faulty device uploaded by the inspection device, and determines the faulty device ID based on image recognition and / or the digital twin model, specifically including:

[0072] Based on the positioning system and Bluetooth beacon positioning submodule, obtain the location information of the inspection equipment;

[0073] Based on the shooting elevation angle, shooting focal length and size of the faulty device, the distance between the faulty device and the inspection device is calculated to obtain the location information of the faulty device.

[0074] Based on the location information of the faulty device, the type of the faulty device is determined through image recognition, and the ID of the faulty device is obtained from the device database.

[0075] Specifically, during the inspection process, the specific location of the inspection equipment is determined through Bluetooth beacons and positioning systems. When a faulty device is found, the inspection equipment takes a photo of the faulty device. When taking the photo, information from sensors such as the gyroscope in the inspection equipment is obtained, and the shooting elevation angle of the inspection equipment, the focal length of the camera during shooting, and the size of the faulty device identified by the image recognition large model are obtained. The distance between the faulty device and the inspection equipment is calculated, and then the location information of the photographed faulty device is calculated based on the specific location information of the inspection equipment.

[0076] Based on the location information of the faulty device, the type of the faulty device is determined in combination with the recognition technology of the image recognition large model, and the corresponding device ID is searched in the database.

[0077] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application obtains the location information of the inspection device based on the positioning system and the Bluetooth beacon positioning submodule, and after the inspection device captures the image of the faulty device, it also includes:

[0078] The inspection device is rotated at a first angle in situ to capture a first environment image, and then rotated at the first angle again to capture a second environment image. After repeating the above steps n times, n environment images are obtained.

[0079] Based on image recognition and digital twin models, the location data of the inspection equipment that takes each environmental image is obtained respectively;

[0080] Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection device is obtained and fused with the location data of the inspection device that takes each environmental image to obtain the updated location information of the inspection device.

[0081] Specifically, after taking the image of the faulty device, the inspection device rotates a certain angle on the spot to take the first environmental image, and then rotates a certain angle on the spot to take the second environmental image. After repeating this process n times, the environmental images taken at multiple angles at the inspection device position are obtained. For each environmental image, the corresponding position of the shooting position of the inspection device in the digital twin model is obtained based on image recognition. Due to the positioning accuracy error of the positioning system and the Bluetooth beacon positioning submodule, or due to various interference factors, such as high-rise buildings, dense trees, tunnels or underground garages blocking the propagation of GPS signals, or weather conditions such as rain, snow, haze, etc., resulting in weak GPS signals, or obstacles, etc., resulting in weak Bluetooth signals, resulting in poor positioning effect after taking pictures, and it is impossible to accurately determine whether the positioning effect is available. The inspection device position obtained based on the environmental image is used as an auxiliary positioning means, and the location information of the inspection device obtained based on the positioning system and the Bluetooth beacon positioning submodule is fused with the location data of the inspection device that takes each environmental image, to obtain the updated location information of the inspection device.

[0082] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application obtains the location information of the inspection device based on the positioning system and the Bluetooth beacon positioning submodule, and fuses the location data of the inspection device obtained for taking each environmental image, specifically including:

[0083] Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection equipment and the location data of the inspection equipment that takes each environmental image are obtained, and the updated location information of the inspection equipment is obtained.

[0084]

[0085] in The location information of the updated inspection equipment. and To obtain the location information and corresponding weight of the inspection equipment based on the positioning system and Bluetooth beacon positioning submodule, and is the location and corresponding weight of the inspection equipment in each environment image.

[0086] Specifically, the inspection equipment position obtained based on the environmental image is used as an auxiliary positioning means, and the location information of the inspection equipment obtained based on the positioning system and the Bluetooth beacon positioning submodule is weighted averaged with the location data of the inspection equipment obtained for each environmental image to obtain the updated location information of the inspection equipment.

[0087]

[0088] The weight and It can be determined according to the actual application situation.

[0089] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application calculates the distance between the faulty device and the inspection device based on the shooting elevation angle, shooting focal length and size of the faulty device of the inspection device, and obtains the location information of the faulty device, specifically including:

[0090] Based on image recognition, the size of the faulty device in the image is obtained. Based on the shooting focal length and the actual size of the device in the basic information of the device, the distance between the target device and the inspection device is obtained.

[0091]

[0092] Based on the location information of the inspection equipment, obtain the location information of the faulty equipment

[0093]

[0094]

[0095] The orientation of the inspection device is obtained based on the built-in sensor of the inspection device, and the specific location of the faulty device is obtained according to the position of the inspection device, the height of the device, the horizontal distance between the inspection device and the device, and the orientation angle of the inspection device.

[0096] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application, after obtaining the specific location of the faulty device, further includes:

[0097] The inspection device moves a first distance in a first angle direction to a first inspection device position, and obtains position information of the first inspection device position;

[0098] Taking an image of the faulty device at the first inspection device position, and obtaining direction and distance information of the faulty device relative to the first inspection device position at the first inspection device position based on image recognition;

[0099] The inspection device moves a second distance in a second angle direction to a second inspection device position, and obtains position information of the second inspection device position;

[0100] Taking an image of the faulty device at the second inspection device position, and obtaining direction and distance information of the faulty device relative to the second inspection device position at the second inspection device position based on image recognition;

[0101] Based on the position of the first inspection device, the direction and distance information of the faulty device relative to the position of the first inspection device, the position of the second inspection device, and the direction and distance information of the faulty device relative to the position of the second inspection device, the position information of the first faulty device is obtained by triangulation;

[0102] Repeat the above steps until k faulty device location information is obtained;

[0103] After the specific location of the faulty device and the location information of k faulty devices are fused, the updated specific location of the faulty device is obtained.

[0104] Specifically, the triangulation positioning method is a commonly used method in the positioning field, and the specific steps will not be repeated here.

[0105] The specific location of the first inspection position is obtained, and the faulty equipment is photographed at the first inspection position. The distance and direction between the inspection equipment and the first inspection position are confirmed based on image recognition. The inspection equipment moves a certain distance in a predetermined direction to reach the second inspection position, and the direction and distance of movement are recorded. The specific location of the second inspection position is obtained, and the faulty equipment is photographed at the second inspection position. The distance and direction between the inspection equipment and the second inspection position are confirmed based on image recognition. At this time, the position of the faulty equipment can be determined by the triangulation positioning method based on the above data to obtain the precise location of the faulty equipment.

[0106] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application obtains the updated specific location of the faulty device after fusing the specific location of the faulty device and the location information of k faulty devices, specifically including:

[0107] S1 obtains an initial estimate, an initial uncertainty value and a system noise parameter, wherein the initial estimate is the specific location of the faulty device, the initial uncertainty value is a preset confidence level of the quantified initial estimate, and the system noise parameter is a preset device accuracy parameter;

[0108] S2 updates the uncertainty value based on the location information of one of the faulty devices and the system noise parameter;

[0109] S3 generates an error residual based on the location information of one of the faulty devices and the uncertainty value, and corrects the initial estimated value according to the residual to obtain an updated estimated value;

[0110] S4 executes steps S2 and S3 for each piece of faulty device location information in turn, gradually revising the initial estimated value, and obtaining the updated specific location of the faulty device.

[0111] Specifically, in processing the calculation of obtaining a determined target position based on multiple measurement values, the present application adopts a Kalman filter technology to fuse multiple measurement results and finally obtain a high-precision position of the target point.

[0112] According to the specific situation of the shooting equipment, the initial position estimate, initial uncertainty value and system noise parameters are obtained. The initial position estimate is the specific position of the faulty equipment. The initial uncertainty value is used to quantify the credibility of the initial estimate and can be set in advance based on experience. The system noise parameters include measurement noise and process noise, which correspond to measurement errors and dynamic uncertainty, respectively. The specific values ​​are determined by the accuracy of the shooting equipment and the actual application requirements.

[0113] The target point is predicted, and the target point is the updated position of the faulty device to be obtained. Based on the position information of one of the faulty devices and the system noise parameters, the uncertainty value is updated to further improve the accuracy of subsequent corrections.

[0114] An error residual is calculated based on the location information of one of the faulty devices and the uncertainty value, and the initial estimated value is corrected according to the residual to obtain an updated estimated value.

[0115] For each faulty device location information, execute steps S2 and S3 in turn, and gradually correct the initial estimate until the correction based on k faulty device location information is completed. Then, take the estimate of the last iteration as the final position of the target point, output a high-precision positioning result, and obtain the specific position of the updated faulty device.

[0116] The data before fusion can also be processed based on outlier processing, Gaussian filtering or extended Kalman filtering before fusion, which can improve the positioning accuracy. This method can effectively fuse multiple measurement data, eliminate the influence of random noise, and ultimately achieve high-precision measurement and determination of the target point position.

[0117] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application, after obtaining the location of the faulty device, further includes:

[0118] Obtain a digital twin model within a first distance range of the location of the inspection device and the location of the faulty device;

[0119] The model display angle is adjusted based on the shooting elevation angle of the inspection device, and the digital twin model is scaled accordingly based on the size of the faulty device in the image of the faulty device uploaded by the inspection device, so that the image of the faulty device coincides with the corresponding model in the digital twin model;

[0120] Based on the confirmation information input by the user, the ID of the faulty device is obtained.

[0121] Specifically, when the faulty device cannot be directly determined based on the location of the device, for example, when there are multiple devices of the same type near the faulty device, the digital twin model within the first distance range of the location of the inspection device and the location of the faulty device is obtained according to the specific location of the inspection device. The first distance range can be selected according to the actual application situation. In one embodiment, the first distance range can be selected within 3m around the inspection device and the faulty device. Obtaining a partial digital twin model can avoid loading all models and improve the loading speed while ensuring the normal use of functions.

[0122] The acquired part of the digital twin model is displayed on the display module of the inspection device. When displaying, the model display angle is adjusted based on the shooting elevation angle of the inspection device. The digital twin model is scaled accordingly based on the size of the faulty device in the image of the faulty device uploaded by the inspection device, so that the image of the faulty device coincides with the corresponding model in the digital twin model, and the displayed model is consistent with the shooting angle of the inspection device. The inspection device selects the corresponding device according to the actual faulty device, thereby determining the faulty device information and obtaining the ID of the faulty device.

[0123] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application digitally files the target area and the equipment in the area, specifically including:

[0124] Set the fields of the equipment files in the target area. The fields of the equipment files include one or more of the equipment name, model, manufacturer, installation date and usage status;

[0125] Obtain paper files of the equipment in the target area uploaded by the scanner, including equipment purchase contracts, warranty cards and user manuals;

[0126] Store the equipment file fields and paper files uploaded by the scanner in a cloud server or local database to complete the digital archiving of the target area and the equipment within the area;

[0127] Establish a digital twin model for the target area and the equipment in the area, specifically including collecting the geographic information system, building information model and equipment model in the target area, constructing the digital twin model in the target area, and setting a corresponding ID for each device.

[0128] Specifically, select a smart park management platform suitable for the target area, which has an equipment file management function. Set the fields of the equipment file in the target area, such as one or more of the equipment name, model, manufacturer, installation date or usage status. Check and register all the equipment in the target area one by one, collect the purchase contract, warranty card, instruction manual and other relevant information of the equipment, clean and organize the collected data to ensure the accuracy and consistency of the information. For paper files, such as equipment purchase contracts and instruction manuals, use a high-resolution scanner to scan them, upload the scanned files to the smart park management platform, and associate them with the corresponding equipment. Use the positioning system to locate the installed equipment, and collect the equipment's latitude and longitude information, altitude and other information and merge them with the above content for management. Store the final data obtained in the cloud server or local database, set a corresponding id for each device, and complete the digital archiving of the target area and the equipment in the area.

[0129] Optionally, in the equipment inspection and management method based on image recognition provided in the embodiment of the present application, the inspection device is an intelligent terminal device with a shooting function.

[0130] Specifically, the inspection device may be an intelligent terminal device, such as a smart phone, a tablet computer, etc., or a dedicated inspection device with shooting and display functions.

[0131] Optionally, the device inspection and management method based on image recognition provided in the embodiment of the present application, after generating a work order according to the faulty device ID, includes:

[0132] Determine the fault condition of the faulty equipment. If it is a routine fault, directly submit a work order or call forwarding maintenance; if it is an emergency fault, temporarily deal with the equipment, and then submit a work order to arrange the subsequent process.

[0133] Specifically, a work order is generated together with the faulty equipment information and the captured image of the faulty equipment, and a fault description is given for reference based on the trained model. Under normal circumstances, for common faults, you can directly submit the work order or call forwarding maintenance without making any modifications. When encountering an emergency fault, the intelligent control system can be used to temporarily deal with the equipment, such as powering off, shutting down, reducing the load, etc. After that, the system will automatically generate a work order and arrange subsequent processing procedures. Optionally, an intelligent control system and work order processing module can be developed to realize emergency handling and subsequent process automation.

[0134] The following is a specific example to illustrate the specific process of the equipment inspection and management method based on image recognition provided by this application:

[0135] In this embodiment, the inspection device is an automatic inspection robot, which can move within the park according to a preset inspection route, and can locate and report the device when a faulty device is found.

[0136] When the inspection equipment was conducting an inspection in a park, it discovered that a camera was faulty. The camera's indicator light was off and the camera was located high. The inspection equipment was unable to get close to inspect in time, nor could it quickly and accurately confirm the equipment's specific number and other relevant information.

[0137] The inspection device opens the inspection function page of the inspection software, turns on GPS and Bluetooth, and uses Bluetooth beacons and GPS to determine the exact location information L1 of the inspection device. Confirm area A with L1 as the center and a radius of 1m around it, and query the server to obtain the device status in area A.

[0138] When the exact location of the inspection device cannot be accurately determined based on GPS and Bluetooth, the inspection device rotates 60° clockwise in situ, takes the first environmental image, and then rotates 60° to take the second environmental image. Repeat the above steps 6 times to obtain 6 environmental images. Based on image recognition and digital twin models, the location data of the inspection device that takes each environmental image is obtained. The location information of the inspection device obtained based on the positioning system and Bluetooth beacon positioning submodule is merged with the location data of the inspection device that takes each environmental image to obtain the updated location information of the inspection device.

[0139] Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection equipment and the location data of the inspection equipment that takes each environmental image are obtained, and the updated location information of the inspection equipment is obtained.

[0140]

[0141] in The location information of the updated inspection equipment. and To obtain the location information and corresponding weight of the inspection equipment based on the positioning system and Bluetooth beacon positioning submodule, and is the location and corresponding weight of the inspection equipment in each environment image.

[0142] If there is only one registered device in the current area, obtain the relevant data of the registered device, display the device details, and execute the steps to generate a work order.

[0143] If there are multiple devices in the current area, the shooting page will be automatically opened. At this time, the built-in gyroscope and other sensors of the inspection device are used to calculate the pitch angle of the device according to the sensor direction and the device direction to obtain the elevation angle R of the inspection device. Specifically, the SensorManager can be used to obtain the pitch angle of the device; the image recognition function is used to identify the size of the device in the image, and the distance between the target device and the inspection device is calculated by the following formula based on the focal length of the camera and the actual size of the device in the pre-collected device information.

[0144]

[0145] like Figure 2 As shown in (a), the location information of the device being photographed is calculated based on the location information of the current inspection device.

[0146]

[0147]

[0148] Obtaining the orientation of the inspection device through the built-in sensor of the inspection device involves obtaining data from an accelerometer and a magnetometer, and using these data to calculate the orientation angle θ (clockwise compared to the true north direction) of the inspection device.

[0149] like Figure 2 As shown in (b), the specific position of the equipment is obtained according to the position of the inspection equipment, the height of the equipment, the horizontal distance between the inspection equipment and the equipment, and the orientation angle of the inspection equipment.

[0150] When the specific location of the faulty equipment cannot be determined or the equipment location accuracy needs to be improved, the inspection equipment can be moved 5m to the east to the first inspection equipment position, the location information of the first inspection equipment position can be obtained, and an image of the faulty equipment can be photographed at the first inspection equipment position. Based on image recognition, the direction and distance information of the faulty equipment relative to the first inspection equipment position can be obtained at the first inspection equipment position.

[0151] The inspection device moves 5m to the south to the second inspection device position, obtains the position information of the second inspection device position, takes an image of the faulty device at the second inspection device position, and obtains the direction and distance information of the faulty device relative to the second inspection device position at the second inspection device position based on image recognition.

[0152] Based on the position of the first inspection device, the direction and distance information of the faulty device relative to the first inspection device, the position of the second inspection device and the direction and distance information of the faulty device relative to the second inspection device, the position information of the first faulty device is obtained by triangulation.

[0153] Repeat the above steps until the location information of four faulty devices is obtained. After the specific location of the faulty device and the location information of the four faulty devices are integrated, the updated specific location of the faulty device is obtained.

[0154] S1 obtains an initial estimate, an initial uncertainty value and a system noise parameter, wherein the initial estimate is the specific location of the faulty device, the initial uncertainty value is a preset confidence level of the quantified initial estimate, and the system noise parameter is a preset device accuracy parameter;

[0155] S2 updates the uncertainty value based on the location information of one of the faulty devices and the system noise parameter;

[0156] S3 generates an error residual based on the location information of one of the faulty devices and the uncertainty value, and corrects the initial estimated value according to the residual to obtain an updated estimated value;

[0157] S4 executes steps S2 and S3 for each piece of faulty device location information in turn, and gradually corrects the initial estimated value until the updated specific location of the faulty device is obtained after correction based on the four pieces of faulty device location information.

[0158] The captured image is sent to the server, and the device database is searched based on image recognition to determine the type C of the faulty device.

[0159] When the positioning error is large, for example, when the error of the device location information L is 1m, the area B with the calculated position of the faulty device as the center and a radius of 1m is used as the target area. If there is only one device in the area, the device-related information is directly pulled; if the search results have multiple devices after limiting the location area and device type, that is, multiple devices are arranged closely, the pre-established digital twin model is loaded in the APP. When loading, it is not completely loaded, but loaded according to the location information of the inspection device and the faulty device. Partial loading can avoid loading a large model and slow loading speed. When loading, the viewing angle of the model after loading is directly adjusted according to the shooting elevation angle R of the inspection personnel's inspection equipment, and the digital twin model is scaled accordingly based on image recognition to ensure that the viewing angle is consistent with the inspection equipment, and then the possible equipment is highlighted for the inspection equipment, and finally the faulty equipment is determined to determine the equipment information.

[0160] Combined with the equipment information, a work order is generated together with photos, and a fault description is given for reference based on the trained model. Under normal circumstances, common faults can be directly submitted as work orders or forwarded to maintenance without any modification. When encountering emergency faults, the intelligent control system can temporarily deal with the equipment, such as powering off, shutting down, reducing load, etc., after which the system will automatically generate a work order and arrange subsequent processing procedures.

[0161] This application also provides a device inspection and management device based on digital twins and image recognition, such as Figure 3 As shown, including:

[0162] The equipment digital archiving module is used to digitally archive the target area and the equipment in the area, obtain the basic information, maintenance information and location information of the equipment, and set the corresponding ID for each device;

[0163] Digital twin model module, used to establish digital twin models for the target area and the equipment within the area;

[0164] An image recognition and positioning module, used to obtain the image and location information of the faulty device uploaded by the inspection device, and determine the faulty device ID based on image recognition and / or the digital twin model;

[0165] The emergency response and work order processing module is used to generate a work order based on the faulty equipment ID and generate the corresponding fault description.

[0166] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0167] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0173] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0174] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for equipment inspection and management based on image recognition, characterized in that: The steps include: Digitally archive the target area and the equipment in the area, obtain the basic information, maintenance information and location information of the equipment, and set a corresponding ID for each device; Establish a digital twin model for the target area and the equipment within the area; Obtain the image and location information of the faulty device uploaded by the inspection device, and determine the faulty device ID based on image recognition and / or the digital twin model; Generate a work order based on the faulty device ID and the corresponding fault description; The step of obtaining the image and location information of the faulty device uploaded by the inspection device and determining the faulty device ID based on image recognition and / or the digital twin model specifically includes: Based on the positioning system and Bluetooth beacon positioning submodule, obtain the location information of the inspection equipment; Based on the shooting elevation angle, shooting focal length and size of the faulty device, the distance between the faulty device and the inspection device is calculated to obtain the location information of the faulty device. Based on the location information of the faulty device, determine the type of the faulty device through image recognition, and obtain the ID of the faulty device in the device database; Wherein, the positioning system and the Bluetooth beacon positioning submodule are used to obtain the location information of the inspection equipment. After the inspection equipment takes an image of the faulty equipment, the following is also included: The inspection device is rotated at a first angle in situ to capture a first environment image, and then rotated at the first angle again to capture a second environment image. After repeating the above steps n times, n environment images are obtained. Based on image recognition and digital twin models, the location data of the inspection equipment that takes each environmental image is obtained respectively; The location information of the inspection device obtained based on the positioning system and the Bluetooth beacon positioning submodule is integrated with the location data of the inspection device that takes each environmental image to obtain the updated location information of the inspection device; The location information of the inspection device is obtained based on the positioning system and the Bluetooth beacon positioning submodule, and the location data of the inspection device that takes each environmental image is obtained and fused, specifically including: Based on the positioning system and the Bluetooth beacon positioning submodule, the location information of the inspection equipment and the location data of the inspection equipment that takes each environmental image are obtained, and the updated location information of the inspection equipment is obtained. in The location information of the updated inspection equipment. and To obtain the location information and corresponding weight of the inspection equipment based on the positioning system and Bluetooth beacon positioning submodule, and is the location and corresponding weight of the inspection equipment in each environment image.

2. The equipment inspection and management method based on image recognition according to claim 1, characterized in that: The method of calculating the distance between the faulty device and the inspection device based on the shooting elevation angle, shooting focal length and size of the faulty device of the inspection device and obtaining the location information of the faulty device specifically includes: The size of the faulty device in the image is obtained based on image recognition, and the distance between the target device and the inspection device is obtained based on the shooting focal length and the actual size of the device in the basic information of the device. Based on the location information of the inspection equipment, obtain the location information of the faulty equipment The orientation of the inspection device is obtained based on the built-in sensor of the inspection device, and the specific location of the faulty device is obtained according to the position of the inspection device, the height of the device, the horizontal distance between the inspection device and the device, and the orientation angle of the inspection device.

3. The equipment inspection and management method based on image recognition as claimed in claim 2, characterized in that: After obtaining the specific location of the faulty device, it also includes: The inspection device moves a first distance in a first angle direction to a first inspection device position, and obtains position information of the first inspection device position; Taking an image of the faulty device at the first inspection device position, and obtaining direction and distance information of the faulty device relative to the first inspection device position at the first inspection device position based on image recognition; The inspection device moves a second distance in a second angle direction to a second inspection device position, and obtains position information of the second inspection device position; Taking an image of the faulty device at the second inspection device position, and obtaining direction and distance information of the faulty device relative to the second inspection device position at the second inspection device position based on image recognition; Based on the position of the first inspection device, the direction and distance information of the faulty device relative to the position of the first inspection device, the position of the second inspection device, and the direction and distance information of the faulty device relative to the position of the second inspection device, the position information of the first faulty device is obtained by triangulation; Repeat the above steps until k faulty device location information is obtained; After fusing the specific location of the faulty device and the location information of k faulty devices, the updated specific location of the faulty device is obtained.

4. The equipment inspection and management method based on image recognition as claimed in claim 3, characterized in that: After fusing the specific location of the faulty device and the location information of k faulty devices, obtaining the updated specific location of the faulty device specifically includes: S1 obtains an initial estimate, an initial uncertainty value and a system noise parameter, wherein the initial estimate is the specific location of the faulty device, the initial uncertainty value is a preset confidence level of the quantified initial estimate, and the system noise parameter is a preset device accuracy parameter; S2 updates the uncertainty value based on the location information of one of the faulty devices and the system noise parameter; S3 generates an error residual based on the location information of one of the faulty devices and the uncertainty value, and corrects the initial estimated value according to the residual to obtain an updated estimated value; S4 executes steps S2 and S3 for each piece of faulty device location information in turn, gradually revising the initial estimated value, and obtaining the updated specific location of the faulty device.

5. The equipment inspection and management method based on image recognition according to claim 4, characterized in that: After obtaining the location of the faulty device, it also includes: Obtain a digital twin model within a first distance range of the location of the inspection device and the location of the faulty device; The model display angle is adjusted based on the shooting elevation angle of the inspection equipment, and the digital twin model is scaled accordingly based on the size of the faulty device in the image of the faulty device uploaded by the inspection equipment, so that the image of the faulty device coincides with the corresponding model in the digital twin model, and the faulty device ID is obtained.

6. Equipment inspection and management based on digital twins and image recognition, characterized by: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: It stores a computer program that can be executed by an equipment inspection and management device based on digital twins and image recognition. When the computer program runs on the equipment inspection and management device based on digital twins and image recognition, the equipment inspection and management device based on digital twins and image recognition executes the steps of the method described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images