Collaborative management system based on augmented reality

Abstract

The invention discloses a collaborative management system based on augmented reality, and belongs to the technical field of AR glasses. The collaborative management system based on augmented reality comprises AR glasses, a rear platform and a cloud, and is characterized in that the AR glasses, the rear platform and the cloud are connected through a wireless network, and the rear platform comprises a real-time information receiving and processing module, a remote assistance module, an SOS emergency response module and a log recording and backtracking module. The cloud comprises a data storage and management module, a data analysis and decision support module and a computing resource module. The problems that the transmission delay problem is prominent, the remote cooperation efficiency is low, an emergency response mechanism is lacked and environmental perception is insufficient are solved. According to the invention, through real-time information transmission and interaction among the AR glasses, the rear platform and the cloud, the system response speed and the cooperation efficiency are improved, efficient interaction between rear personnel and field operation personnel is supported, and the environment perception capability and the safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AR glasses, and in particular to a collaborative management system based on augmented reality. Background Art

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing, augmented reality (AR) technology is increasingly being used in the industrial field. However, although traditional AR glasses systems can provide basic visual information enhancement functions, they have many limitations in actual industrial application scenarios. First, the problem of information transmission delay is prominent. Traditional AR glasses systems usually rely on local processing and simple data transmission, and cannot achieve real-time and high-speed data exchange, which may lead to decision-making errors or safety accidents in complex industrial environments. Secondly, remote collaboration is inefficient and lacks effective remote assistance functions. It is difficult for rear support personnel to obtain on-site information in real time and provide accurate guidance, resulting in on-site operators having to wait for rear personnel to arrive at the scene when encountering complex problems, which seriously affects work efficiency. Third, there is a lack of emergency response mechanism. Traditional AR glasses systems cannot notify staff and start emergency processes in the first place, and their ability to deal with emergencies is insufficient, increasing potential risks and losses. Fourth, the lack of environmental perception capabilities. Traditional AR glasses systems usually only have basic visual information collection functions, cannot monitor environmental parameters such as temperature and humidity in real time, and cannot perform fire reconnaissance and temperature detection, which is particularly insufficient in high-risk environments.

[0003] In order to solve the above problems, the present invention proposes a collaborative management system based on augmented reality. Summary of the invention

[0004] The purpose of the present invention is to provide a collaborative management system based on augmented reality, which realizes real-time information transmission and interaction between AR glasses, rear platform and cloud through wireless network connection, thereby improving the response speed and collaborative efficiency of the system. The rear platform provides multiple communication methods and real-time video marking tools to support efficient interaction between rear personnel and on-site workers, enhance the accuracy of remote collaboration, and is equipped with an SOS emergency response module to effectively deal with emergencies. AR glasses are equipped with multiple sensors and infrared cameras to monitor environmental parameters in real time, thereby improving the environmental perception capability and safety of the system and solving the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A collaborative management system based on augmented reality, comprising AR glasses, a rear platform and a cloud, wherein the AR glasses, the rear platform and the cloud are connected via a wireless network;

[0007] The AR glasses include:

[0008] The first camera is used to synchronize the real situation on site and perform visual object recognition and analysis for edge computing;

[0009] The second camera has an infrared temperature measurement function and is used for fire reconnaissance and temperature sensing to detect target objects;

[0010] The third camera is used for hand motion recognition and application control;

[0011] Waveguide diffractive lenses for displaying augmented reality information;

[0012] An interactive unit, used for the wearer to view and process the information displayed by the waveguide diffraction lens and issue operation instructions;

[0013] The information displayed by the waveguide diffraction lens is viewed and processed, specifically, a third camera is used to capture and recognize gestures to implement an interactive function, and the user gestures are captured and recognized, including:

[0014] S1, read in the pre-trained palm detection model and hand key point detection model;

[0015] S2, hand image captured by the gesture recognition camera;

[0016] S3, using a palm detection model to locate the palm area in the hand image;

[0017] S4, using the hand key point detection model to obtain the key point position of the hand;

[0018] S5, using the position information of the key points, calculate the finger angle through an algorithm;

[0019] S6, judging the gesture type by comparing the calculated finger angle with a preset threshold;

[0020] The rear platform comprises:

[0021] Real-time information receiving and processing module, used to receive video, audio and sensor data transmitted by AR glasses, and process and distribute the received data;

[0022] The remote assistance module is used to provide text, voice and video communication methods, support real-time interaction between rear-line personnel and on-site workers, and display real-time video transmitted back by AR glasses; it provides marking tools to allow rear-line personnel to mark key areas in the video and guide on-site workers.

[0023] As a preferred solution of the present invention, the cloud comprises:

[0024] Data storage and management module, used for centralized storage and management of data;

[0025] The data analysis and decision support module conducts in-depth analysis of the received video, audio, and sensor data and extracts key features. It uses algorithms to analyze the data, identify potential dangers, and generate analysis reports for reference by rear personnel.

[0026] The computing resource module is used to provide computing power and support complex data processing and big data analysis tasks.

[0027] Furthermore, the data analysis and decision support module is further used to:

[0028] Clean and preprocess the incoming data to remove noise and invalid data;

[0029] Extract key frames and object features from videos, extract features from audio, and extract statistical features of temperature, humidity, and altitude;

[0030] Perform fault diagnosis on sensor data to identify equipment failures and abnormal conditions, conduct real-time analysis of video and audio to identify potential safety hazards, analyze the behavior of on-site operators, and identify irregular operations;

[0031] Summarize the analysis results into reports, including but not limited to fault diagnosis reports, safety warning reports and behavior analysis reports, use charts and graphs to present the analysis results, and generate specific recommendations and measures based on the analysis results;

[0032] Analyze and summarize historical data and extract long-term trends and patterns.

[0033] As a preferred solution of the present invention, the remote assistance module includes:

[0034] The voice and text recognition unit recognizes and processes the received voice and text information in real time to extract key information; it supports multi-language recognition and converts the recognition results into text for analysis and storage;

[0035] The image recognition unit recognizes and processes the received picture information and video information in real time, locates, recognizes and classifies the target object in the image, and extracts the key features of the target object, including but not limited to the location, category, and attribute information;

[0036] The image annotation unit is used to process and automatically annotate the received images to improve the recognition accuracy of the images; it supports automatic and manual annotation simultaneously, and performs manual correction if the automatic annotation fails or is inaccurate; and it stores the annotation results for subsequent analysis and use.

[0037] As a preferred solution of the present invention, the AR glasses further include:

[0038] A sensing unit, used to collect environmental data, the sensing unit includes but is not limited to a temperature sensor, a humidity sensor and an altitude sensor;

[0039] Beidou and GPS units for positioning and navigation;

[0040] Compatible mounting unit, used to install the waveguide diffractive lens on existing industrial safety helmets or fire fighting helmets without affecting the use of people wearing myopia glasses;

[0041] Optical machine, headphones and controller to support voice and video calls and information display.

[0042] As a preferred solution of the present invention, the rear platform further comprises:

[0043] The display and display interface are used to centrally display the audio, video, text information and sensor data transmitted by the AR glasses. The functional divisions of the display interface include:

[0044] Every ID uplink information window, by clicking each ID you can zoom in on the front view and display video, infrared data, and sensor data;

[0045] Front-end action execution buttons, providing ACT, POI, RT Log position and action execution buttons;

[0046] Downlink command module, used for chat entry and SOS downlink window, provides general commands, voice, subtitles and SOS flash screen entry;

[0047] Downstream video and graphics PiP are dragged into the window to support split-screen display of video and graphics.

[0048] As a preferred solution of the present invention, the cloud also includes a cross-platform support module for providing an API interface for information exchange between different devices.

[0049] As a preferred solution of the present invention, the rear platform further comprises:

[0050] The SOS emergency response module is used to receive the SOS signal sent by the AR glasses, initiate the emergency response process, notify staff and departments, coordinate emergency resources, and provide emergency rescue guidance to help on-site operators deal with emergencies;

[0051] The logging and backtracking module is used to record and backtrack all activity logs.

[0052] As a preferred solution of the present invention, the calculation of the finger angle in step 5 comprises the following steps:

[0053] S501, calculating a first vector from the wrist to the base of the finger;

[0054] S502, calculating a second vector from the base of the finger to the fingertip;

[0055] S503: Calculate the angle between the first vector and the second vector.

[0056] As a preferred solution of the present invention, the palm detection model and the hand key point detection model both adopt the DHRNet network model, the final stage of feature fusion of the DHRNet network model is stage D, and in the stage D, the highest resolution branch and the higher resolution branch after transposed convolution respectively calculate the loss function of the predicted heat map, and the loss value of the final heat map is the sum of the losses of the above two branches;

[0057] Specifically, the 1 / 4 resolution heat map containing i key points and the final 1 / 2 resolution heat map after the transposed convolution operation are the learning targets of the network; the two-dimensional Gaussian distribution function is used to generate the real heat map h i (x,y):

[0058]

[0059] Among them, (x, y) represents the position coordinates of any pixel in the real heat map, (x i ,y i ) is the center of the Gaussian function, which represents the true position coordinates of the i-th key point, and σ represents the standard deviation of the Gaussian function preset for each key point;

[0060] Prediction heat map h i * (x,y) is calculated by the following formula:

[0061]

[0062] The loss function is divided into classification loss and regression loss, and the mean square error MSE loss function is used for calculation. The formula is as follows:

[0063]

[0064] Among them, N b represents the number of key points of the hand, and h (i，b) (x, y) represent the predicted heat map and the real heat map of the bth heat map respectively;

[0065] The highest resolution branch loss value in stage D is used as Loss 1 / 4 Indicates that the higher resolution branch loss value after transposed convolution is used by Loss 1 / 2 Indicates that the total loss value L of the final key point detection heat map is:

[0066] L=Loss 1 / 4 +Loss 1 / 2 .

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention realizes real-time transmission and interaction of information by wirelessly connecting AR glasses with the rear platform and the cloud, which is a significant improvement over traditional AR glasses that only have a one-way information display function.

[0069] 2. The present invention sets up an SOS emergency response module to receive SOS signals from AR glasses, quickly initiates the emergency response process, notifies staff and departments, coordinates emergency resources, and provides emergency rescue guidance.

[0070] 3. The present invention provides a marking tool through the remote assistance module, allowing rear personnel to mark key areas in the video, effectively guiding on-site workers and improving on-site work efficiency. By integrating a voice and text recognition unit, an image recognition unit and an image annotation unit, the efficiency and accuracy of remote collaboration are improved.

[0071] 4. The present invention is equipped with a second camera with an infrared temperature measurement function, which can be used for fire reconnaissance and temperature sensing detection of target objects, thereby enhancing the environmental perception capability and safety of the system.

[0072] 5. The present invention enables the wearer to view and process the information displayed by the waveguide diffraction lens and issue operation instructions through the interactive unit. It also supports the zooming operation of video or graphic content, as well as the management and adjustment of notes and SOS notifications on the rear platform, thereby improving the user's operating experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the system module of the present invention;

[0074] Figure 2 It is a schematic diagram of the structure of the AR glasses of the present invention;

[0075] Figure 3 It is a schematic diagram of the cloud module of the present invention;

[0076] Figure 4 It is a schematic diagram of the rear platform module of the present invention;

[0077] Figure 5 It is a schematic diagram of the structure of the remote assistance module of the present invention;

[0078] Figure 6 It is a structural diagram of the existing HRNet network model;

[0079] Figure 7It is a structural diagram of the DHRNet network model of the present invention. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] In order to solve the problems of prominent information transmission delay, low efficiency of remote collaboration, lack of emergency response mechanism and insufficient environmental perception in existing technologies, please refer to Figure 1-5 , this embodiment provides the following technical solutions:

[0082] A collaborative management system based on augmented reality includes AR glasses, a rear platform and a cloud. The AR glasses, the rear platform and the cloud are connected via a wireless network to achieve real-time transmission and interaction of information, thereby improving the response speed and collaborative efficiency of the system.

[0083] The rear platform includes:

[0084] Real-time information receiving and processing module, used to receive video, audio and sensor data transmitted by AR glasses, and process and distribute the received data;

[0085] The remote assistance module is used to provide text, voice and video communication methods, support real-time interaction between rear-line personnel and on-site operators, and improve the diversity and effectiveness of communication; display the real-time video sent back by AR glasses to facilitate rear-line experts to observe the on-site situation; provide marking tools to allow rear-line personnel to mark key areas in the video and guide on-site operators, which enhances the intuitiveness and accuracy of guidance;

[0086] The SOS emergency response module is used to receive the SOS signal sent by the AR glasses, initiate the emergency response process, notify staff and departments, coordinate emergency resources, and provide emergency rescue guidance to help on-site operators deal with emergencies. The rapid response mechanism improves the ability to deal with emergencies and reduces potential risks and losses;

[0087] The logging and backtracking module is used to record and backtrack all activity logs.

[0088] The remote assistance module further includes:

[0089] The voice and text recognition unit recognizes and processes the received voice and text information to extract key information; it supports multi-language recognition and converts the recognition results into text for analysis and storage, which is convenient for subsequent processing and application;

[0090] An image recognition unit recognizes and processes the received image information and video information, locates, recognizes and classifies the target object in the image, and extracts the key features of the target object, including but not limited to location, category, and attribute information;

[0091] The image annotation unit is used to process and automatically annotate the received images to improve the recognition accuracy of the images; it supports both automatic and manual annotation and provides flexible annotation methods. If the automatic annotation fails or is inaccurate, manual correction can be performed to ensure the accuracy of the annotation; the annotation results are stored for subsequent analysis and use;

[0092] The rear platform also includes a display and a display interface, which is used to centrally display the audio, video, text information and sensor data transmitted by the AR glasses. The functional divisions of the display interface include:

[0093] Every ID uplink information window, by clicking each ID you can zoom in on the front view and display video, infrared data, and sensor data;

[0094] Front-end action execution buttons, providing ACT, POI, RT Log position and action execution buttons;

[0095] Downlink command module, used for chat entry and SOS downlink window, provides general commands, voice, subtitles and SOS flash screen entry;

[0096] Downlink video and graphics PiP are dragged into windows to support split-screen display of video and graphics. Users can freely adjust the size and position of the window, which improves the flexibility and convenience of operation; loading animations and prompt information are provided during the user operation process to enhance the user experience; keyboard shortcuts are supported to improve operational efficiency, especially in emergency situations, users can use shortcuts to quickly perform key actions.

[0097] The cloud includes:

[0098] Data storage and management module, used for centralized storage and management of data to ensure data security and reliability;

[0099] The data analysis and decision support module conducts in-depth analysis of the received video, audio and sensor data and extracts key features. It uses algorithms to analyze the data, identify potential dangers, and generate analysis reports for reference by rear personnel.

[0100] The computing resource module is used to provide computing power, support complex data processing and big data analysis tasks, and improve the processing power and efficiency of the system;

[0101] The cross-platform support module is used to provide an API interface for information exchange between different devices, facilitate third-party application integration, support a variety of terminal devices, and improve the compatibility and scalability of the system.

[0102] The data analysis and decision support module is further used to:

[0103] Clean and preprocess the incoming data to remove noise and invalid data;

[0104] Extract key frames and object features from videos, extract features from audio, and extract statistical features of sensor data such as temperature, humidity, and altitude;

[0105] Perform fault diagnosis on sensor data to identify equipment failures and abnormal conditions, conduct real-time analysis of video and audio to identify potential safety hazards, analyze the behavior of on-site operators, and identify irregular operations;

[0106] Summarize the analysis results into reports, including but not limited to fault diagnosis reports, safety warning reports, and behavior analysis reports, use charts and graphs to present the analysis results, and generate specific suggestions and measures based on the analysis results to help rear-end personnel develop response strategies;

[0107] Analyze and summarize historical data and extract long-term trends and patterns.

[0108] AR glasses include:

[0109] The first camera is used to synchronize the real situation on site and perform visual object recognition and analysis for edge computing, which improves the real-time and accuracy of on-site information. Through edge computing, data can be processed quickly locally, reducing data transmission delays and improving response speed.

[0110] The second camera has an infrared temperature measurement function, which is used for fire reconnaissance and temperature detection of target objects, enhancing the system's environmental perception and safety. The infrared camera can detect high-temperature areas, discover potential fire risks in advance, and improve accident prevention capabilities.

[0111] The third camera is used for hand motion recognition and application control;

[0112] Waveguide diffractive lenses for displaying augmented reality information;

[0113] Compatible mounting unit, used to install the waveguide diffractive lens on existing industrial safety helmets or fire fighting helmets without affecting the use of people wearing myopia glasses;

[0114] Beidou and GPS units are used for positioning and navigation. They can update the wearer's location, time and movement speed in real time, and transmit this data to the cloud for reading and analysis, improving the accuracy and reliability of location management. It helps rear personnel to grasp the location of on-site personnel in real time and conduct effective scheduling and management. At the same time, it provides real-time navigation guidance to help the wearer quickly find the target location in a complex environment, improving work efficiency and safety. AR technology is used to overlay navigation information in the wearer's field of view, making navigation more intuitive and convenient.

[0115] A sensing unit, used to collect environmental data; the sensing unit includes but is not limited to a temperature sensor, a humidity sensor and an altitude sensor;

[0116] An interactive unit, used for the wearer to view and process the information displayed by the waveguide diffraction lens and issue operation instructions;

[0117] Optical machine, headphones and controller to support voice and video calls and information display.

[0118] The interaction unit is further used for:

[0119] Zoom in and out the video or graphic content displayed by the AR glasses;

[0120] Click the notes and SOS notifications on the back platform, adjust the display position and size of the notes and SOS notifications by clicking or dragging, and put them into the mailbox icon in the lower left corner;

[0121] The information displayed by the waveguide diffraction lens is viewed and processed, specifically, the gestures recognized by the third camera are used to zoom, drag and click, and gestures are used to connect, confirm, return, switch modes, view information and communicate with the backend. Notes are text instructions made by the backend based on the scene, such as scene annotations, operation instructions, important information and notes such as equipment number, operation time, etc., which are transmitted to AR glasses to guide the actions of on-site personnel, and multi-channel communication is achieved by combining video and voice.

[0122] Working principle: The industrial intelligent AR glasses collaborative system consists of three parts: AR glasses, rear platform and cloud. Through wireless network connection, real-time transmission and interaction of information are realized, which improves the response speed and collaborative efficiency of the system. AR glasses are equipped with a first camera to synchronize the real situation on site and perform visual object recognition and analysis for edge computing; the second camera has an infrared temperature measurement function for fire reconnaissance and temperature detection, and the third camera is used to recognize gestures and convert them into operation instructions. The waveguide diffraction lens displays augmented reality information, the interactive unit supports the wearer to view, process and issue operation instructions for the displayed information, and the optical machine, headset and controller support voice and video calls. Beidou and GPS units provide positioning navigation, and the sensor unit collects environmental data. The rear platform includes a real-time information receiving and processing module, which receives and processes the video, audio and sensor data transmitted by AR glasses; the remote assistance module provides text, voice and video communication methods, supports real-time interaction between rear personnel and on-site operators, displays real-time video and provides marking tools to guide on-site operations; the SOS emergency response module receives the SOS signal, starts the emergency response process, notifies staff and departments, and coordinates emergency resources; the log recording and backtracking module records all activity logs. The cloud includes a data storage and management module to ensure data security and reliability; the data analysis and decision support module conducts in-depth analysis of the incoming data and generates analysis reports for reference by rear personnel; the computing resource module provides computing power to support complex data processing; the cross-platform support module provides an API interface for information exchange between different devices to improve the compatibility and scalability of the system.

[0123] The information displayed by the waveguide diffraction lens is viewed and processed, specifically, a third camera is used to capture and recognize gestures to implement an interactive function, and the user gestures are captured and recognized, including:

[0124] S1, read in the pre-trained palm detection model and hand key point detection model;

[0125] S2, hand image captured by the gesture recognition camera;

[0126] S3, using a palm detection model to locate the palm area in the hand image;

[0127] S4, using the hand key point detection model to obtain the key point position of the hand;

[0128] S5, using the position information of the key points, calculate the finger angle through an algorithm;

[0129] S6. Determine the gesture type by comparing the calculated finger angle with a preset threshold.

[0130] The calculation of the finger angle in step S5 includes the following steps:

[0131] S501, calculating a first vector from the wrist to the base of the finger;

[0132] S502, calculating a second vector from the base of the finger to the fingertip;

[0133] S503: Calculate the angle between the first vector and the second vector.

[0134] Wherein, step S6 comprises the following steps:

[0135] S601, setting the thumb angle threshold to 53°, the first angle threshold to 65°, and the second angle threshold to 49°;

[0136] S602: If the angle of the thumb is greater than the thumb angle threshold, then proceed to step S603; otherwise, proceed to step S605;

[0137] S603: If the angles of the remaining four fingers are greater than the first angle threshold, it is determined to be a fist gesture, otherwise, the process goes to step S604;

[0138] S604, judging the remaining four fingers as numbers 1, 2, 3, and 4 according to the number of fingers whose angles are smaller than the second angle threshold;

[0139] S605: If the angle of the thumb is smaller than the second angle threshold, and the angles of the other four fingers are larger than the first angle threshold, it is determined to be a thumb gesture, otherwise, the process goes to step S606;

[0140] S606: If the angles of the thumb and the little finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 6, otherwise, the process goes to step S607;

[0141] S607: If the angles of the thumb and index finger are less than the second angle threshold, and the angles of the other three fingers are greater than the first angle threshold, it is determined to be the number 8, otherwise, the process goes to step S608;

[0142] S608: If the angles of the thumb, index finger, middle finger, ring finger, and pinky finger are all smaller than the second angle threshold, it is determined that the palm is open.

[0143] In order to reduce the complexity of the model while ensuring the accuracy of the model, the palm detection model and the hand key point detection model of the present invention both adopt the DHRNet network model, which is an improvement on the existing HRNet network model.

[0144] like Figure 6As shown in the figure, first, the HRNet network model downsamples the input image through two convolution operations, reducing the resolution of the input image by 4 times to 1 / 4 of the initial size, and then enters the main network. The main part of the HRNet network consists of four stages, among which stage 1 is composed of 4 BottleNeck residual units. After stage 1, there is a series of exchange structures and stage structures. Each time a network passes through an exchange structure, a scale branch is added through a downsampling operation. In addition, in order to obtain multi-scale information, each stage in the network will fuse features of different scales. Stages 2, 3, and 4 are stacked repeatedly 1, 4, and 3 times respectively. The last exchange unit in stage 4 only retains the feature layer with the highest resolution, and finally predicts the hand points on the high-resolution feature map.

[0145] By observing the HRNet network structure, we can find that with the increase of low-resolution sub-networks, the number of network branches and convolution modules also increases, which leads to an exponential increase in the number of network parameters and the computational overhead of the model as the order increases. However, in practical applications, both high speed and high accuracy are often required.

[0146] like Figure 7 As shown in the figure, the DHRNet network model retains the first three stages (stage 1, stage 2 and stage 3), and changes stage 4 to stage D. Stage D fuses multi-scale features through transposed convolution and greatly reduces the number of model parameters. It also adds another transposed convolution to the highest resolution feature map of its output to generate a higher resolution feature map for predicting heat maps.

[0147] In order to better handle scale changes and improve the network's ability to perceive learning errors, a multi-resolution supervision strategy is used during training. Specifically, the highest resolution branch (1 / 4 resolution of the input) in stage D and the higher resolution branch (1 / 2 resolution of the input) after transposed convolution are used to calculate the loss function of the predicted heat map. The final heat map loss value is the sum of the losses of the above two branches.

[0148] In stage D, the 1 / 4 resolution heat map containing i key points and the final 1 / 2 resolution heat map after the transposed convolution operation are the learning targets of the network. The real heat map h is generated using a two-dimensional Gaussian distribution function. i (x,y):

[0149]

[0150] Among them, (x, y) represents the position coordinates of any pixel in the real heat map, (x i ,y i) is the center of the Gaussian function, represents the true position coordinates of the ith key point, and σ represents the standard deviation of the Gaussian function preset for each key point.

[0151] Prediction heat map h i * (x,y) is calculated by the following formula:

[0152]

[0153] The loss function is divided into classification loss and regression loss, and the mean square error MSE loss function is used for calculation. The formula is as follows:

[0154]

[0155] Among them, Nb represents the number of key points of the hand, and h (i，b) (x, y) represents the predicted heat map and the true heat map of the b-th heat map, respectively.

[0156] The highest resolution (1 / 4 resolution of the input) branch loss value in stage D is used by Loss 1 / 4 Indicates that the higher resolution (1 / 2 resolution of the input) branch loss value after transposed convolution is used by Loss 1 / 2 Indicates that the total loss value L of the final key point detection heat map is:

[0157] L=Loss 1 / 4 +Loss 1 / 2

[0158] Finally, experiments are provided: the improved DHRNet network is experimentally verified on the COCO dataset, as follows:

[0159] 1. The COCO key point detection dataset designs the target key point similarity (OKS) indicator to indicate the similarity between the true value and the predicted hand key points. The OKS calculation method is as follows:

[0160]

[0161] Among them, i represents the i-th hand key point, d i represents the Euclidean distance between the predicted position and the true position of the i-th hand key point, s represents the scale of the target, and s 2 is the area of ​​the target detection box, ki represents the normalization factor of the hand point, which is a constant used to control the attenuation. i Indicates the visibility of the hand point, which can be 0, 1, or 2. 0 means that the hand point is not labeled, 1 means that the hand point is labeled but is not visible due to occlusion, and 2 means that the hand point is labeled and visible. i>0) means when v i When >0, the value is 1, otherwise it is 0.

[0162] 2. Average accuracy AP:

[0163]

[0164] Among them, p represents the target number, T is the preset OKS threshold, and only when the OKS value is greater than the threshold, it indicates that the prediction is correct. δ is used to represent the relationship between OKS and the threshold T. If OKS is greater than T, the value is 1.

[0165] AP n It represents the accuracy when T=n, AP M Represents the average accuracy of medium-scale targets, AP L It represents the average accuracy of large-scale targets, and AR represents the average recall.

[0166] First, the aspect ratio of the hand detection box height to width is adjusted to 4:3, and then the box is cropped from the image and adjusted to a fixed input size. In order to reduce computational cost and experimental cycle, the input size of this experimental network is set to 256 pixels x 192 pixels. In addition, the data augmentation method adopts random rotation, random scale transformation and flipping, with the rotation range of [-45°, 45°] and the random scale transformation range of [0.65, 1.35]. It should be noted that 0.65 and 1.35 have no units, they represent the scale factor relative to the original image size.

[0167]

[0168] Experimental results show that improving stage 4 to stage D can effectively reduce the number of network parameters and calculations, and the accuracy is basically the same as the original one.

[0169] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0170] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A collaborative management system based on augmented reality, including AR glasses, a rear platform and a cloud, characterized in that: The AR glasses, the rear platform and the cloud are connected via a wireless network; The AR glasses include: The first camera is used to synchronize the real situation on site and perform visual object recognition and analysis for edge computing; The second camera has an infrared temperature measurement function and is used for fire reconnaissance and temperature sensing to detect target objects; The third camera is used for hand motion recognition and application control; Waveguide diffractive lenses for displaying augmented reality information; An interactive unit, used for the wearer to view and process the information displayed by the waveguide diffraction lens and issue operation instructions; The information displayed by the waveguide diffraction lens is viewed and processed, specifically, a third camera is used to capture and recognize gestures to implement an interactive function, and the user gestures are captured and recognized, including: S1, read in the pre-trained palm detection model and hand key point detection model; S2, hand image captured by the gesture recognition camera; S3, using a palm detection model to locate the palm area in the hand image; S4, using the hand key point detection model to obtain the key point position of the hand; S5, using the position information of the key points, calculate the finger angle through an algorithm; S6, judging the gesture type by comparing the calculated finger angle with a preset threshold; The rear platform comprises: Real-time information receiving and processing module, used to receive video, audio and sensor data transmitted by AR glasses, and process and distribute the received data; The remote assistance module is used to provide text, voice and video communication methods, support real-time interaction between rear-line personnel and on-site workers, and display real-time video transmitted back by AR glasses; it provides marking tools to allow rear-line personnel to mark key areas in the video and guide on-site workers.

2. The collaborative management system based on augmented reality according to claim 1, characterized in that: The cloud comprises: Data storage and management module, used for centralized storage and management of data; The data analysis and decision support module conducts in-depth analysis of the received video, audio, and sensor data and extracts key features. It uses algorithms to analyze the data, identify potential dangers, and generate analysis reports for reference by rear personnel. The computing resource module is used to provide computing power and support complex data processing and big data analysis tasks.

3. The collaborative management system based on augmented reality according to claim 2, characterized in that: The data analysis and decision support module is further used for: Clean and preprocess the incoming data to remove noise and invalid data; Extract key frames and object features from videos, extract features from audio, and extract statistical features of temperature, humidity, and altitude; Perform fault diagnosis on sensor data to identify equipment failures and abnormal conditions, conduct real-time analysis of video and audio to identify potential safety hazards, analyze the behavior of on-site operators, and identify irregular operations; Summarize the analysis results into reports, including but not limited to fault diagnosis reports, safety warning reports and behavior analysis reports, use charts and graphs to present the analysis results, and generate specific recommendations and measures based on the analysis results; Analyze and summarize historical data and extract long-term trends and patterns.

4. The collaborative management system based on augmented reality according to claim 1, characterized in that: The remote assistance module includes: The voice and text recognition unit recognizes and processes the received voice and text information in real time to extract key information; it supports multi-language recognition and converts the recognition results into text for analysis and storage; The image recognition unit recognizes and processes the received picture information and video information in real time, locates, recognizes and classifies the target object in the image, and extracts the key features of the target object, including but not limited to the location, category, and attribute information; The image annotation unit is used to process and automatically annotate the received images to improve the recognition accuracy of the images; it supports automatic and manual annotation simultaneously, and performs manual correction if the automatic annotation fails or is inaccurate; and it stores the annotation results for subsequent analysis and use.

5. The collaborative management system based on augmented reality according to claim 1, characterized in that: The AR glasses further include: A sensing unit, used to collect environmental data, the sensing unit includes but is not limited to a temperature sensor, a humidity sensor and an altitude sensor; Beidou and GPS units for positioning and navigation; Compatible mounting unit, used to install the waveguide diffractive lens on existing industrial safety helmets or fire fighting helmets without affecting the use of people wearing myopia glasses; Optical machine, headphones and controller to support voice and video calls and information display.

6. The collaborative management system based on augmented reality according to claim 1, characterized in that: The rear platform also includes: The display and display interface are used to centrally display the audio, video, text information and sensor data transmitted by the AR glasses. The functional divisions of the display interface include: Every ID uplink information window, by clicking each ID you can zoom in on the front view and display video, infrared data, and sensor data; Front-end action execution buttons, providing ACT, POI, RT Log position and action execution buttons; Downlink command module, used for chat entry and SOS downlink window, provides general commands, voice, subtitles and SOS flash screen entry; Downstream video and graphics PiP are dragged into the window to support split-screen display of video and graphics.

7. The collaborative management system based on augmented reality according to claim 1, characterized in that: The cloud also includes a cross-platform support module for providing an API interface for information exchange between different devices.

8. The collaborative management system based on augmented reality according to claim 1, characterized in that: The rear platform also includes: The SOS emergency response module is used to receive the SOS signal sent by the AR glasses, initiate the emergency response process, notify staff and departments, coordinate emergency resources, and provide emergency rescue guidance to help on-site operators deal with emergencies; The logging and backtracking module is used to record and backtrack all activity logs.

9. The collaborative management system based on augmented reality according to claim 1, characterized in that: The calculation of the finger angle in step 5 includes the following steps: S501, calculating a first vector from the wrist to the base of the finger; S502, calculating a second vector from the base of the finger to the fingertip; S503: Calculate the angle between the first vector and the second vector.

10. The collaborative management system based on augmented reality according to claim 1, characterized in that: The palm detection model and the hand key point detection model both adopt the DHRNet network model, the last stage of feature fusion of the DHRNet network model is stage D, and in the stage D, the highest resolution branch and the higher resolution branch after transposed convolution respectively calculate the loss function of the predicted heat map, and the loss value of the final heat map is the sum of the losses of the above two branches; Specifically, the 1 / 4 resolution heat map containing i key points and the final 1 / 2 resolution heat map after the transposed convolution operation are the learning targets of the network; the two-dimensional Gaussian distribution function is used to generate the real heat map h i (x,y): Among them, (x, y) represents the position coordinates of any pixel in the real heat map, (x i ,y i ) is the center of the Gaussian function, which represents the true position coordinates of the i-th key point, and σ represents the standard deviation of the Gaussian function preset for each key point; Prediction heat map h i * (x,y) is calculated by the following formula: The loss function is divided into classification loss and regression loss, and the mean square error MSE loss function is used for calculation. The formula is as follows: Among them, N b represents the number of key points of the hand, and h (i，b) (x, y) represent the predicted heat map and the real heat map of the bth heat map respectively; The highest resolution branch loss value in stage D is used as Loss 1 / 4 Indicates that the higher resolution branch loss value after transposed convolution is used by Loss 1 / 2 Indicates that the total loss value L of the final key point detection heat map is: L=Loss 1 / 4 +Loss 1 / 2 。

Images