Intelligent inspection method and system based on AR helmet

Abstract

The invention relates to the technical field of intelligent inspection, and provides an intelligent inspection system based on an AR helmet, which comprises a display module, an image processing module, an information processing module, a command input module, a cloud database, a remote communication module and a remote transmission module, and the information processing module is used for processing images in the mine captured by the plurality of cameras and transmitting the images to the display module, and the information processing module is remotely and wirelessly connected with a sensor. According to the system and the method, staff can obtain real-time navigation, early warning and warning functions from a map and a task target in an AR helmet in the working process, work is guided by the task, guidance and help are synchronously obtained in the working process, and information of an event coping mode is quickly obtained under the help of a cloud database; therefore, the employee does not need to take off the helmet or distract attention, and can quickly handle the situation, so that the working efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent inspection technology, and in particular to an intelligent inspection method and system based on an AR helmet. Background Art

[0002] At present, my country ranks first in the world in coal production. 95% of my country's coal mining is underground mining, which is much more dangerous than open-pit mining. Faced with the severe situation of coal mine safety production at this stage, high-tech and advanced practical technologies are used to transform traditional industries, improve scientific and technological content, promote product upgrading, and improve product quality and economic benefits. With the rapid increase in production in recent years, the staffing has been relatively reduced, and the labor intensity of employees has been continuously increased. Under the condition of fixed time and workload for patrol inspection, daily inspection, and power outage inspection, inexperienced colleagues are prone to make the patrol inspection of substations and equipment inspections superficial, and the patrol inspections are not detailed and in place.

[0003] Instead of handheld terminals, AR remote control terminals are used to confirm on-site operations step by step, and intelligent guidance of operations is achieved in a virtual-real way. The reliance on employees' operating experience is eliminated, and operational risks are reduced. The reliance of traditional operational activities on theoretical learning, operational practice, and human experience is broken, the standards and norms of operational activities are improved, and the safety risks brought by operations are further reduced. Although this inspection method can reduce the safety risks brought by operations, new employees still need on-site guidance from instructors, which leads to inaccurate guidance and greatly reduces the flexibility of work. Therefore, an intelligent inspection method and system based on AR helmets is needed. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an intelligent inspection method and system based on an AR helmet, which solves the problem that new employees still need to know when using an AR helmet for inspection, which greatly reduces the flexibility of their work.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] An intelligent inspection system based on an AR helmet includes a display module, an image processing module, an information processing module, a command input module, a cloud database, a remote communication module and a remote transmission module. The image processing module is remotely and wirelessly connected to multiple cameras to process and transmit the internal images of the mine captured by the multiple cameras to the display module. The information processing module is remotely and wirelessly connected to a sensor to transmit the collected internal temperature, humidity and gas concentration data of the mine to the display module. The cloud database is used to store records of miners' activities and provide training and teaching material data. The command input module is used to send instructions received by the AR helmet to working equipment in the mine. The remote transmission module is used to transmit video data in other AR helmets for apprentices to learn. The remote communication module is used to conduct voice calls between multiple employees to improve the efficiency of the entire team.

[0007] Preferably, the display module includes an image display unit, a task display unit, a gas concentration display unit, a temperature and humidity display unit and an operation frame, the image display unit is electrically connected to a feature point marking unit, the image display unit is electrically connected to an image processing module, the task instruction display unit is electrically connected to a map route display unit, and the gas concentration display unit and the temperature and humidity display unit are both electrically connected to an information processing module.

[0008] Preferably, the cloud database includes a historical image storage unit, an operation guide storage unit and a task record storage unit, and the operation guide unit stores mine work guidelines and emergency response plans.

[0009] Preferably, the image processing module includes an image stitching unit, an image correction unit and a light-dark contrast correction contrast unit which are electrically connected to each other.

[0010] Preferably, the information processing module includes a gas classification unit and a temperature and humidity threshold unit, and the gas classification unit is electrically connected to the concentration threshold unit.

[0011] Preferably, the command input module includes a voice control input unit, a manual input unit and a remote input unit, and the remote input unit is electrically connected to the remote transmission module.

[0012] An intelligent inspection method based on an AR helmet comprises the following steps:

[0013] Step 1: When the employee uses the AR helmet, the camera transmits the captured image data of the inside of the mine to the image processing module through wireless transmission for image processing. The sensor processes the collected temperature, humidity and gas concentration data of the inside of the mine through wireless transmission. The image data, temperature data, humidity data and gas concentration data are projected to the display module for human eyes to read;

[0014] Step 2: Perform inspection within the visible range, and wirelessly transmit the command to control the machine through the command input module to control the machine to perform inspection work. During the inspection process, the map and task instructions in the display module will cooperate with the processing solution data in the cloud database to assist the inspection work;

[0015] Step 3: When the employees communicate and report their work status, their voices will be wirelessly transmitted to the remote communication module in another AR helmet through the remote communication module in the AR helmet using the remote transmission module to realize the conversation;

[0016] Step 4: The entire process will be recorded while using the AR helmet and uploaded to the cloud database for accident analysis, work process improvement, resource optimization, and re-decision making on mine management.

[0017] Preferably, the wireless transmission in step 1, step 2 and step 3 selects any one of wireless transmission modes such as Wi-Fi, LTE, 5G, etc.

[0018] Preferably, the specific steps of the image processing in step 1 are:

[0019] S1. The images transmitted to the image processing module are spliced ​​by the image splicing unit first, and the images captured by multiple adjacent cameras are spliced. Then, the image correction unit is used to repair the spliced ​​parts to make the transition more natural and reduce the offset.

[0020] S2. After the spliced ​​image is optimized, the image is processed twice, and the dark and bright parts of the captured image inside the mine are adjusted using the light and dark contrast correction unit, the brightness and glare of the bright part are reduced, the brightness of the dark part is increased, and the contour line of the object is increased to improve recognition;

[0021] S3, the processed image is transmitted to the image display unit, and the feature point marking unit selects the feature points of the inspection area;

[0022] S4. When selecting feature points in the inspection area, feature points related to the task will be marked with a dialog box next to the selected area in the image, and the dialog box will be filled with text of precautions related to the feature point and emergency handling methods in the operation guide storage unit.

[0023] Preferably, the display module in step 1 is also used for training new employees, and the specific steps are:

[0024] S1. Use the remote transmission module to synchronously transmit the image in the AR helmet of the experienced employee to the display module of the AR helmet of the new employee, so that the new employee can watch the work process of the experienced employee for learning;

[0025] S2. New employees can also project the teaching images in the historical image storage unit in the cloud database to the display module for viewing and learning;

[0026] S3. When new employees are performing their work, experienced employees can use the remote transmission module to manipulate the remote input unit, perform remote auxiliary operations, and send commands to the new employees’ AR helmets to replace the operation of the machines.

[0027] S4. The learning process of newcomers in S1, S2 and S3 can be taught by experienced employees through voice communication modules.

[0028] The present invention provides an intelligent inspection method and system based on an AR helmet, which has the following beneficial effects:

[0029] 1. The present invention enables employees to obtain real-time navigation, early warning and alarm functions from the maps and task objectives in the AR helmet during work, obtain task guidance at work, and receive guidance and help synchronously during the work process. With the help of the cloud database, employees can quickly obtain information on how to deal with situations, so that employees do not need to take off their helmets or divert their attention, and can quickly deal with situations, thereby improving work efficiency and accuracy.

[0030] 2. The present invention displays the images, temperature, humidity and gas concentration data inside the mine in real time on the display module of the AR helmet, and with the help of clear images processed by the image processing module and marked feature points, employees can quickly obtain all the information inside the mine, thereby further improving work efficiency.

[0031] 3. The present invention utilizes the data in the cloud database in conjunction with the image display unit. New employees can learn by watching the work processes and historical images of experienced employees. At the same time, they can receive remote assisted operation and voice communication teaching when performing their work, which effectively promotes the skill improvement and training effect of new employees. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a diagram of an intelligent inspection system based on an AR helmet according to the present invention;

[0033] Figure 2 A schematic diagram of a display module system of an AR helmet-based intelligent inspection system of the present invention;

[0034] Figure 3 A schematic diagram of a cloud database system of an AR helmet-based intelligent inspection system of the present invention;

[0035] Figure 4It is a schematic diagram of an image processing module system of an AR helmet-based intelligent inspection system of the present invention;

[0036] Figure 5 It is a schematic diagram of an information processing module of an intelligent inspection system based on an AR helmet of the present invention;

[0037] Figure 6 This is a schematic diagram of a command input module system of an AR helmet-based intelligent inspection system of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example:

[0040] As one aspect of this application, please see the attached Figure 1 -Attached Figure 5 An embodiment of the present invention provides an intelligent inspection system based on an AR helmet, including a display module, an image processing module, an information processing module, a command input module, a cloud database, a remote communication module and a remote transmission module. The image processing module is remotely and wirelessly connected to multiple cameras to process and transmit the internal images of the mine captured by the multiple cameras to the display module. The information processing module is remotely and wirelessly connected to a sensor to transmit the collected internal temperature, humidity and gas concentration data of the mine to the display module. The cloud database is used to store records of miners' activities and provide training and teaching materials data. The command input module is used to send the instructions received by the AR helmet to the working equipment in the mine. The remote transmission module is used to transmit video data in other AR helmets for apprentices to learn. The remote communication module is used to conduct voice calls between multiple employees to improve the efficiency of the entire team.

[0041] The display module consists of an image display unit, a task display unit, a gas concentration display unit, a temperature and humidity display unit and an operation frame. The image display unit is electrically connected to a feature point marking unit. The image display unit is electrically connected to the image processing module. The task instruction display unit is electrically connected to a map route display unit to display a map and task objectives in the operation frame of the display module when wearing an AR helmet, and to provide work task guidance so that employees can directly view relevant information when operating equipment or performing tasks without taking off helmets or diverting attention, thereby improving work efficiency and accuracy. The gas concentration display unit and the temperature and humidity display unit are both electrically connected to the information processing module.

[0042] The cloud database consists of a historical image storage unit, an operation guide storage unit and a task record storage unit. The operation guide unit stores mine work guidelines and emergency response plans.

[0043] The image processing module is composed of an image splicing unit, an image correction unit and a brightness and darkness contrast correction contrast unit which are electrically connected to each other.

[0044] The information processing module includes a gas classification unit and a temperature and humidity threshold unit, and the gas classification unit is electrically connected to the concentration threshold unit.

[0045] The command input module consists of a voice-controlled input unit, a manual input unit and a remote input unit, and supports manual control and voice-controlled control. The remote input unit is electrically connected to the remote transmission module to receive control from other AR helmets to assist new employees in their work.

[0046] Based on the above-mentioned intelligent inspection system based on an AR helmet, as another aspect of the present application, an intelligent inspection method based on an AR helmet includes the following steps:

[0047] Step 1: When the employee uses the AR helmet, the camera transmits the captured image data of the inside of the mine to the image processing module through wireless transmission for image processing. The sensor processes the collected temperature, humidity and gas concentration data of the inside of the mine through wireless transmission. The image data, temperature data, humidity data and gas concentration data are projected to the display module for human eyes to read;

[0048] The specific steps of image processing are:

[0049] S1. The images transmitted to the image processing module are spliced ​​by the image splicing unit first, and the images captured by multiple adjacent cameras are spliced. Then, the image correction unit is used to repair the spliced ​​parts to make the transition more natural and reduce the offset.

[0050] S2. After the spliced ​​image is optimized, the image is processed twice, and the dark and bright parts of the captured image inside the mine are adjusted using the light and dark contrast correction unit, the brightness and glare of the bright part are reduced, the brightness of the dark part is increased, and the contour line of the object is increased to improve recognition;

[0051] S3, the processed image is transmitted to the image display unit, and the feature point marking unit selects the feature points of the inspection area;

[0052] S4. When selecting feature points in the inspection area, feature points related to the task will be marked with a dialog box next to the selected area in the image, and the dialog box will be filled with text of precautions related to the feature point and emergency handling methods in the operation guide storage unit.

[0053] The gas concentration data is processed as follows: the gas classification unit in the information processing module classifies the gas collected by the sensor, and the concentration threshold unit divides the gas concentration, which is converted into visual data and transmitted to the gas concentration display unit, which displays the gas concentration, temperature, ventilation and other key environmental parameters in the mine, helping miners to make correct decisions and actions quickly.

[0054] Step 2: Perform inspection within the visible range, and wirelessly transmit the command to control the machine through the command input module to control the machine to perform inspection work. During the inspection process, the map and task instructions in the display module will cooperate with the processing solution data in the cloud database to assist the inspection work;

[0055] Step 3: When the employees communicate and report their work status, their voices will be wirelessly transmitted to the remote communication module in another AR helmet through the remote communication module in the AR helmet using the remote transmission module to realize the conversation;

[0056] Step 4: The entire process will be recorded while using the AR helmet and uploaded to the cloud database for accident analysis, work process improvement, resource optimization, and re-decision making on mine management.

[0057] The wireless transmission in step 1, step 2 and step 3 may be any one of Wi-Fi, LTE, 5G and other wireless transmission methods.

[0058] The display module is also used for the training of new employees. The specific steps are as follows:

[0059] S1. Use the remote transmission module to synchronously transmit the image in the AR helmet of the experienced employee to the display module of the AR helmet of the new employee, so that the new employee can watch the work process of the experienced employee for learning;

[0060] S2. New employees can also project the teaching images in the historical image storage unit in the cloud database to the display module for viewing and learning;

[0061] S3. When new employees are performing their work, experienced employees can use the remote transmission module to manipulate the remote input unit, perform remote auxiliary operations, and send commands to the new employees’ AR helmets to replace the operation of the machines.

[0062] The learning process of newcomers in S4, S1, S2 and S3 can be taught by experienced staff through voice communication modules.

[0063] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent inspection system based on AR helmet, characterized in that: It includes a display module, an image processing module, an information processing module, a command input module, a cloud database, a remote communication module and a remote transmission module. The image processing module is remotely wirelessly connected to multiple cameras to process and transmit the internal images of the mine captured by the multiple cameras to the display module. The information processing module is remotely wirelessly connected to a sensor to transmit the collected internal temperature, humidity and gas concentration data of the mine to the display module. The cloud database is used to store records of miners' activities and provide training and teaching materials data. The command input module is used to send the instructions received by the AR helmet to the working equipment in the mine. The remote transmission module is used to transmit video data in other AR helmets for apprentices to learn. The remote communication module is used to conduct voice calls between multiple employees to improve the efficiency of the entire team.

2. The intelligent inspection system based on AR helmet according to claim 1 is characterized in that: The display module includes an image display unit, a task display unit, a gas concentration display unit, a temperature and humidity display unit and an operation frame. The image display unit is electrically connected to a feature point marking unit. The image display unit is electrically connected to an image processing module. The task instruction display unit is electrically connected to a map route display unit. The gas concentration display unit and the temperature and humidity display unit are both electrically connected to an information processing module.

3. The intelligent inspection system based on AR helmet according to claim 1 is characterized in that: The cloud database includes a historical image storage unit, an operation guide storage unit and a task record storage unit. The operation guide unit stores mine work guidelines and emergency response plans.

4. The intelligent inspection system based on AR helmet according to claim 1 is characterized in that: The image processing module includes an image splicing unit, an image correction unit and a light-dark contrast correction contrast unit which are electrically connected to each other.

5. The intelligent inspection system based on AR helmet according to claim 1 is characterized in that: The information processing module includes a gas classification unit and a temperature and humidity threshold unit, and the gas classification unit is electrically connected to the concentration threshold unit.

6. The intelligent inspection system based on AR helmet according to claim 1 is characterized in that: The command input module includes a voice control input unit, a manual input unit and a remote input unit, and the remote input unit is electrically connected to the remote transmission module.

7. An intelligent inspection method based on an AR helmet, using an intelligent inspection system based on an AR helmet as claimed in any one of claims 1 to 6, characterized in that: The following steps are involved: Step 1: When the employee uses the AR helmet, the camera transmits the captured image data of the inside of the mine to the image processing module through wireless transmission for image processing. The sensor processes the collected temperature, humidity and gas concentration data of the inside of the mine through wireless transmission. The image data, temperature data, humidity data and gas concentration data are projected to the display module for human eyes to read; Step 2: Perform inspection within the visible range, and wirelessly transmit the command to control the machine through the command input module to control the machine to perform inspection work. During the inspection process, the map and task instructions in the display module will cooperate with the processing solution data in the cloud database to assist the inspection work; Step 3: When the employees communicate and report their work status, their voices will be wirelessly transmitted to the remote communication module in another AR helmet through the remote communication module in the AR helmet using the remote transmission module to realize the conversation; Step 4: The entire process will be recorded while using the AR helmet and uploaded to the cloud database for accident analysis, work process improvement, resource optimization, and re-decision making on mine management.

8. The intelligent inspection method based on AR helmet according to claim 7 is characterized in that: The wireless transmission in step 1, step 2 and step 3 uses any one of the wireless transmission modes such as Wi-Fi, LTE, 5G, etc.

9. The intelligent inspection method based on AR helmet according to claim 7, characterized in that: The specific steps of the image processing in step 1 are: S1. The images transmitted to the image processing module are spliced ​​by the image splicing unit first, and the images captured by multiple adjacent cameras are spliced. Then, the image correction unit is used to repair the spliced ​​parts to make the transition more natural and reduce the offset. S2. After the spliced ​​images are optimized, the images are processed twice, and the dark and bright parts of the captured images of the interior of the mine are adjusted using the light and dark contrast correction unit, the brightness and glare of the bright parts are reduced, the brightness of the dark parts is increased, and the contour lines of the objects are increased to improve recognition; S3, the processed image is transmitted to the image display unit, and the feature point marking unit selects the feature points of the inspection area; S4. When selecting feature points in the inspection area, feature points related to the task will be marked with a dialog box next to the selected area in the image, and the dialog box will be filled with text of precautions related to the feature point and emergency handling methods in the operation guide storage unit.

10. The intelligent inspection method based on AR helmet according to claim 7, characterized in that: The display module in step 1 is also used for training new employees, and the specific steps are: S1. Use the remote transmission module to synchronously transmit the image in the AR helmet of the experienced employee to the display module of the AR helmet of the new employee, so that the new employee can watch the work process of the experienced employee for learning; S2. New employees can also project the teaching images in the historical image storage unit in the cloud database to the display module for viewing and learning; S3. When new employees are performing their work, experienced employees can use the remote transmission module to manipulate the remote input unit, perform remote auxiliary operations, and send commands to the new employees’ AR helmets to replace the operation of the machines. S4. The learning process of newcomers in S1, S2 and S3 can be taught by experienced employees through voice communication modules.

Images