Virtual reality application method, system, terminal and storage medium for operation and maintenance industry
Through virtual reality technology and deep learning, accurate perception of operation and maintenance scenarios is achieved, the problem of operation and maintenance inspection is not intelligent and convenient enough, and the inspection efficiency and equipment management accuracy are improved.
Patent Information
- Application Number
- CN202410261622.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-03-07
AI Technical Summary
During the operation and maintenance industry inspection, personnel need to check the equipment information one by one, which is time-consuming and labor-intensive, and lacks intelligent and convenient solutions.
Using virtual reality technology combined with deep learning, by obtaining actual scene data and camera positioning information, a spatial model of the real environment is built, and the device data panel is added to the model, to achieve accurate superposition of virtual information and provide real-time device status and operation data.
It improves operation and maintenance efficiency, helps operation and maintenance personnel to discover problems in a timely manner and take measures, reduces the time and labor intensity of inspections, and enhances the understanding and management of equipment status.
Smart Images

Figure CN118092654B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual reality technology, and in particular to a virtual reality application method, system, terminal and storage medium for operation and maintenance industry. Background Art
[0002] Augmented Reality (AR) is a technology that enhances the perception of reality by superimposing virtual information on the real world. Unlike virtual reality, AR does not completely immerse users in a virtual environment, but rather superimposes virtual information on the real environment, allowing users to perceive both real and virtual content at the same time. Deep learning is a branch of machine learning algorithms that handles complex pattern recognition tasks through the training and optimization of neural network models. At present, deep learning technology has achieved remarkable results in various fields, such as image recognition, speech recognition, and natural language processing.
[0003] In the related technologies, during the operation and maintenance inspection process, inspectors need to check the information of the equipment one by one, which requires a high level of professionalism. At the same time, in many inspection scenarios, inspectors need to subjectively judge whether the equipment is abnormal, and the inspection operation is time-consuming and laborious. Combining virtual reality technology and deep learning technology can bring more convenience to the operation and maintenance industry. Summary of the invention
[0004] In order to solve the problem that the operation and maintenance industry is not smart, convenient, time-consuming and labor-intensive when conducting inspections, the present application provides a virtual reality application method, system, terminal and storage medium for the operation and maintenance industry.
[0005] In a first aspect of the present application, a virtual reality application method for operation and maintenance industry is provided, comprising:
[0006] Obtain actual scene data and camera positioning information;
[0007] According to the positioning information of the camera, the actual scene data is processed using a preset algorithm to construct a spatial model of the real environment;
[0008] Adding a preset equipment data panel to the spatial model;
[0009] Real-time scene data is acquired, and after matching the real-time scene data with the spatial model, a preset algorithm is used to place the device panel information at the corresponding position of the actual scene.
[0010] By adopting the above technical solutions, the deep learning model can achieve accurate perception of the actual operation and maintenance scene, so that virtual information can be accurately superimposed on the actual scene. This helps the operation and maintenance personnel to understand the equipment status and operation more clearly. It can obtain on-site data in real time and combine it with virtual information to provide the operation and maintenance personnel with real-time equipment status and operation data. This helps to find problems in time, make decisions, and take necessary measures, thereby improving the efficiency of operation and maintenance.
[0011] In a possible implementation, the positioning information of the camera includes a position and a posture, and obtaining the actual scene data and the positioning information of the camera includes:
[0012] Obtain point cloud data and image information of the actual scene;
[0013] Calculating depth information of the actual scene based on the point cloud data;
[0014] Performing bilateral filtering on the depth information to obtain noise-reduced depth data;
[0015] According to the camera internal parameters, the three-dimensional coordinates of each pixel point are projected and reflected. k ;
[0016] Calculate the vertex normal map N k ;
[0017] N k The number of rows and columns are halved to obtain a three-layer pyramid of vertex / normal map;
[0018] According to the three-layer pyramid, the position and posture of the camera are calculated.
[0019] In a possible implementation, the actual scene data is processed using a preset algorithm according to the positioning information of the camera to construct a spatial model of the real environment, including:
[0020] Calculating a projection matrix of the camera according to the positioning information of the camera; the positioning information includes the posture and position of the camera;
[0021] Calculate the corresponding position of each pixel in the three-dimensional space according to the point cloud data and the projection matrix of the camera;
[0022] Combining the corresponding position of each pixel point in the three-dimensional space with the depth information to obtain a depth image, wherein each pixel point in the depth image includes spatial position information and distance information;
[0023] Based on the depth image and the camera positioning information, a preset algorithm is used to construct a spatial model of the real environment;
[0024] The space model is optimized.
[0025] In a possible implementation, real-time scene data is acquired, and after matching the real-time scene data with a space model, a preset algorithm is used to place the device panel information at a corresponding position in the actual scene, including:
[0026] Acquire ground feature point cloud and image data at a first time point;
[0027] Extracting robust feature points from the image data at the first time point, and obtaining position information of the feature points;
[0028] Obtaining camera positioning information based on position information of feature points in the image data at the first time point;
[0029] Matching the feature points of the ground feature point cloud and the feature points of the image data at the first time point with the existing map feature points, determining the positions of the feature points of the ground feature point cloud and the feature points of the image data in three-dimensional space, and constructing a three-dimensional map of the environment at the first time point;
[0030] Aligning the model and texture information of the preset virtual panel with the three-dimensional map of the environment at the first time point;
[0031] By matching the position and posture of the virtual panel with the position and posture of the camera, the virtual panel is superimposed on the corresponding position in the real environment.
[0032] In a possible implementation, matching the real-time scene data with the space model further includes:
[0033] Acquire ground feature point cloud and image data at a second time point;
[0034] A three-dimensional map of the environment at the second time point is obtained based on the camera positioning information at the first time point, the three-dimensional map of the environment, and the ground feature point cloud and image data at the second time point.
[0035] In a possible implementation, information interaction is also implemented based on user behavior information:
[0036] Acquire user behavior information, where the behavior information includes action information and audio information;
[0037] Matching corresponding interaction instructions according to the wake-up word in the action information or audio information;
[0038] According to the interactive instruction, a corresponding feedback program is called.
[0039] In a possible implementation, the interaction of the voice function includes:
[0040] Get the user's audio information;
[0041] Recognize the audio information into text information and input it into a deep learning model to obtain data feedback text information of the on-site equipment;
[0042] The data feedback text information is converted into audio using speech synthesis technology and then output.
[0043] In a second aspect of the present application, a virtual reality application system for operation and maintenance is provided, comprising:
[0044] The acquisition module is used to obtain the actual scene data and the positioning information of the camera;
[0045] A model building module, used to process the actual scene data using a preset algorithm according to the positioning information of the camera to build a spatial model of the real environment;
[0046] A data adding module, used for adding a preset equipment data panel to the space model;
[0047] The reality matching display module is used to obtain real-time scene data, match the real-time scene data with the spatial model, and then use a preset algorithm to place the device panel information at the corresponding position of the actual scene.
[0048] In a third aspect of the present application, a terminal is provided, which has the characteristic of using virtual reality to improve the efficiency of operation and maintenance.
[0049] The third application objective of this application is achieved through the following technical solutions:
[0050] A terminal includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the above-mentioned virtual reality application method for operation and maintenance industry.
[0051] In a fourth aspect of the present application, a computer storage medium is provided, which can store corresponding programs and has the characteristic of facilitating the use of virtual reality to improve the efficiency of operation and maintenance.
[0052] The fourth application objective of this application is achieved through the following technical solutions:
[0053] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned virtual reality application methods for operation and maintenance industry.
[0054] In summary, the present application includes at least one of the following beneficial technical effects: accurate perception of actual operation and maintenance scenarios is achieved through deep learning models, so that virtual information can be accurately superimposed on actual scenarios. This helps operation and maintenance personnel understand the equipment status and operation status more clearly. It is possible to obtain field data in real time and combine it with virtual information to provide operation and maintenance personnel with real-time equipment status and operation data. This helps to identify problems in a timely manner, make decisions, and take necessary measures, thereby improving operation and maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a flowchart of a virtual reality application method for operation and maintenance industry according to one embodiment of the present application.
[0056] Figure 2 It is a flowchart of one step of a virtual reality application method for operation and maintenance industry according to one embodiment of the present application.
[0057] Figure 3 It is a schematic diagram of a virtual reality application system for operation and maintenance industry according to one embodiment of the present application.
[0058] Figure 4 It is a structural diagram of a terminal according to an embodiment of the present application.
[0059] Explanation of the reference numerals: 201, acquisition module; 202, model building module; 203, data adding module; 204, reality matching display module; 301, CPU; 302, ROM; 303, RAM; 304, bus; 305, I / O interface; 306, input part; 307, output part; 308, storage part; 309, communication part; 310, drive; 311, removable medium. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0061] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0062] The following is combined with Figures 1 to 4 This application is described in further detail.
[0063] In order to be able to highly accurately superimpose virtual information with actual operation and maintenance scenarios, thereby achieving real-time monitoring, maintenance and management. By using AI technology to help answer questions related to the site and equipment, and to analyze and make decisions on some situations, this application provides a virtual reality application method for the operation and maintenance industry.
[0064] Reference Figure 1 , a virtual reality application method for operation and maintenance industry, comprising the following steps:
[0065] S101: Deep model training for operation and maintenance scenarios.
[0066] In order to achieve the above technical effects, before conducting operation and maintenance inspections, a large amount of operation and maintenance scenario data needs to be collected for model training. Operation and maintenance scenario data includes equipment status, environmental information, etc. Taking the operation and maintenance of the server room as an example, the data that can be displayed include: 1. Server information: the model, brand, operating status, maintenance record and other information of each server in the room. 2. Network equipment data: the model, port connection status, etc. of network equipment in the room such as switches and routers. 3. Power data: data on power usage and power supply status of each device in the room. 4. Environmental data such as temperature and humidity: temperature and humidity data of each area in the room. 5. Fault alarm information: data on fault alarm information of equipment in the room. 6. Common equipment operations and related documents on problem handling.
[0067] Use deep learning models to process the collected operation and maintenance scenario data, and adjust the model structure and parameters for different operation and maintenance scenarios and tasks. In the above server room operation and maintenance example, common operation and maintenance scenarios include: 1. Equipment inspection: inspect various equipment in the room, such as servers, network equipment, power equipment, etc., to check whether the equipment is operating normally, whether there are faults or abnormal conditions. 2. Temperature and humidity monitoring: monitor environmental parameters such as temperature and humidity in the room to ensure that the room environment meets the requirements, and take timely measures to adjust and repair. 3. Power supply monitoring: monitor the power supply equipment in the room, check whether the power supply equipment is operating normally and whether the voltage is stable, and ensure the normal operation of the equipment in the room. 4. Network equipment management: manage and maintain network equipment such as switches and routers in the room, check the operating status of network equipment, whether the connection is normal, and deal with network problems in a timely manner. 5. Security monitoring: monitor the security equipment in the room (such as cameras, access control systems, etc.) to ensure that the safety and protection measures of the room are effective and deal with security incidents in a timely manner. 6. Data backup and recovery: Back up important data in the computer room and conduct data recovery tests regularly to ensure data integrity and availability. 7. Resource management: Manage and configure various resources in the computer room, such as IP addresses, storage space, etc., to ensure the rational use and allocation of resources. 8. Fault handling: Timely discover and handle faults in computer room equipment, and quickly restore the normal operation of equipment in an emergency.
[0068] This step is mainly to increase the relevance between the operation and maintenance inspection content and the data model. The data model can classify problems according to the operator. For example, when inspecting equipment, the model results are closer to the data related to the equipment parameters. When encountering fault handling, the model results are closer to the relevant content of the operation manual of some equipment. For example, when inspecting the server room, the data related to the operation and maintenance of the server room is used for training, so that the model can obtain the capabilities related to the operation and maintenance of the server room.
[0069] After obtaining the operation and maintenance scenario data related to the operation and maintenance scenario, it is also necessary to perform spatial scanning and reconstruction of the actual scene.
[0070] S102: After spatially scanning and reconstructing the actual scene, virtual reality information display is realized.
[0071] First, on-site personnel need to collect actual scene data, and then use the actual scene data to reconstruct the space of the actual scene, and then add the corresponding operation and maintenance scene data to the reconstructed space model. When the user wears AR glasses and enters the actual scene, the AR glasses scan the actual scene data at the user's location, match the actual scene data with the space model, determine the user's location information, and put the corresponding operation and maintenance scene data to the corresponding location of the actual scene. In this way, the corresponding operation and maintenance data in the actual scene can be displayed at different locations.
[0072] Reference Figure 2 After spatial scanning and reconstruction of the actual scene, the information display of virtual reality includes:
[0073] S1021: Acquire actual scene data and camera positioning information.
[0074] First, the on-site personnel use AR glasses that support spatial recognition, turn on the scene scanning function in the software, and scan the on-site environment to obtain image information and depth information. AR glasses obtain depth information by using camera sensors. Ways to obtain depth information include: camera sensors use structured light sensors or time-of-flight sensors. Structured light sensors emit a series of light spots or light patterns into the environment, and then use the camera to sense the position changes of these light spots or light patterns to calculate the scene depth information. Time-of-flight sensors measure the propagation time of light by sending and receiving infrared light pulses to calculate depth information.
[0075] The depth information is then processed to obtain the positioning information of the camera in the AR glasses. The specific processing method is: bilateral filtering is performed on the original depth map corresponding to the depth information to obtain the denoised depth map Dk after denoising. The purpose of bilateral filtering is to maintain clear boundaries. General filtering is weighted averaging in the spatial domain. The closer the pixel is to the center point, the higher the weight. Bilateral filtering is based on the weighted averaging of the value domain, that is, the closer the pixel gray value is to the gray value of the center pixel, the higher the weight. Near the boundary, the grayscale value is very different, so although the pixels on both sides of the boundary are close together in the spatial domain, due to the large difference in grayscale values, the weights for each other are very low, so a clear boundary can be maintained.
[0076] After getting the denoised depth map Dk, we can back-project the three-dimensional coordinates of each pixel based on the camera intrinsic parameter K. This is the Vertex map Vk. In the formula, u is the pixel coordinate, is the corresponding homogeneous coordinate. The normal vector of each vertex can be easily obtained by cross-producting the adjacent vertices. Then kThe depth map is downsampled, and the number of rows and columns are reduced by half, resulting in a three-layer pyramid of the vertex / normal map. Downsampling uses mean downsampling, that is, the depth values of four adjacent pixels on the depth map are averaged into one value. The purpose of constructing a three-layer pyramid is to calculate the camera position and posture from coarse to fine, which has the effect of accelerating calculation.
[0077] S1022: Based on the positioning information of the camera, a preset algorithm is used to process the actual scene data to construct a spatial model of the real environment.
[0078] Constructing a spatial model of the real environment includes: calculating the projection matrix, generating a depth image, establishing a spatial model, and optimizing the model.
[0079] The first is the projection matrix calculation: In order to accurately reconstruct the environment, it is necessary to calculate the projection matrix of the camera, which describes the geometric relationship between the camera and the scanned environment. This is achieved by calibrating the relative position and posture between the camera and the reference object.
[0080] The second is depth image generation: Based on the position changes of light spots or light patterns sensed by the camera and the camera's projection matrix, the corresponding position of each pixel in three-dimensional space can be calculated. Combined with the depth information measured by the depth sensor, a complete depth image can be generated, in which each pixel contains spatial position and distance information.
[0081] Then the reconstruction algorithm builds a spatial model: Based on the complete depth image and camera position information, different algorithms can be used for spatial reconstruction. Spatial reconstruction algorithms include voxelization, triangulation, point cloud reconstruction, etc. These algorithms convert the data in the depth image into a three-dimensional shape and generate a spatial model that represents the real environment.
[0082] Finally, the spatial model is processed and optimized: The generated spatial model may contain some noise or incomplete parts. In order to improve the quality, the model can be processed and optimized, such as removing noise, filling holes, smoothing surfaces, etc.
[0083] S1023: Add the preset equipment data panel to the space model.
[0084] Get the model and texture information of the virtual panel. The model and texture information of the virtual panel can be customized by the user. The virtual panel user displays the corresponding device data panel. Associate the device in the space model with the data panel of the corresponding device, so that when the user wears AR glasses to view the corresponding device, the data panel of the corresponding device can be quickly retrieved and displayed on the virtual panel.
[0085] S1024: Acquire real-time scene data, match the real-time scene data with the space model, and use a preset algorithm to place the device panel information at a corresponding position in the actual scene.
[0086] After the on-site personnel wear AR glasses, they open the software to scan the environment and compare it with the previous spatial model. They use SLAM (simultaneous localization and mapping) technology to place the device panel information at the corresponding position in the real scene, and then they can view and use it. The specific implementation steps are: 1. Data acquisition: SLAM technology first collects the perception data of the environment through sensors, including ground feature point clouds obtained by lidar, image data obtained by cameras, acceleration and angular velocity information obtained by inertial measurement units (IMUs), etc. 2. Feature extraction: In image data, the SLAM algorithm extracts some robust feature points or feature descriptors to analyze the movement of the camera and the structure of the environment. Common features include corner points, edges, etc. In point cloud data, ground feature points can be extracted through lidar scanning data. 3. Motion estimation: By analyzing the position changes of feature points or feature descriptors, the movement of machines or mobile devices, such as displacement and rotation, can be estimated. Motion estimation methods for computing device motion include visual odometers and inertial navigation. 4. Data association: Associate data from different sensors, that is, match image feature points or ground point cloud feature points with existing map feature points to determine their positions in three-dimensional space. The data association method can use feature descriptors and nearest neighbor search algorithms. 5. Map construction: In the SLAM process, in addition to estimating the movement of the machine or mobile device, it is also necessary to build a three-dimensional map of the environment. By matching the estimated position and posture with the sensor data, the map can be continuously updated and optimized. 6. Closed-loop detection: In the SLAM process, errors accumulate over time, causing map drift. The goal of closed-loop detection is to detect whether a machine or mobile device has passed through a location that has already been detected. When a closed loop is detected, the previous trajectory and map can be optimized.
[0087] S103: Implementing interactive functions according to user behavior information.
[0088] Obtain the user's behavior information, where the behavior information includes action information and audio information. Match the corresponding interaction instructions according to the wake-up words in the action information or audio information. Call the corresponding feedback program according to the interaction instructions.
[0089] The implementation steps of voice interaction are: Select appropriate voice recognition technology to recognize voice content as text. Input text content into the existing deep learning model to obtain data feedback text information of field equipment. Integrate speech synthesis technology to realize voice feedback of the system to operation and maintenance personnel. Implement a security mechanism for voice interaction to ensure that the system only responds to legal and authorized instructions.
[0090] Effects of a virtual reality application method for the operation and maintenance industry in the embodiment of the present application: 1. Highly accurate virtual information superposition: The system realizes accurate perception of the actual operation and maintenance scene through a deep learning model, so that virtual information can be accurately superimposed on the actual scene. This helps the operation and maintenance personnel to understand the equipment status and operation status more clearly. 2. Real-time data display and analysis: The system can obtain on-site data in real time and combine it with virtual information to provide real-time equipment status and operation data for the operation and maintenance personnel. This helps to find problems in time, make decisions, and take necessary measures to improve the efficiency of operation and maintenance. 3. Convenience of voice interaction: The voice interaction function realized by the AI algorithm enables on-site operation and maintenance personnel to interact with the system through natural language without relying on manual input or touch screen. This improves the convenience and efficiency of operation, especially in busy or special environments. More practical. 4. The intuitiveness of augmented reality: Using AR technology, the system can intuitively superimpose virtual information on the actual scene, so that the operation and maintenance personnel can intuitively understand the structure and operation status of the equipment. This intuitiveness helps to quickly locate and solve problems. 5. Improve work safety: Operation and maintenance personnel can obtain equipment status and operation information through the system without directly contacting dangerous on-site equipment. This helps reduce the risk of accidents and improve work safety. 6. Intelligent data analysis: The system can perform intelligent data analysis through deep learning algorithms, identify anomalies or trends in equipment operation, and provide operation and maintenance personnel with deeper data insights, which helps preventive maintenance and optimize operation and maintenance strategies.
[0091] This application provides a virtual reality application system for the operation and maintenance industry, which adopts the following technical solutions:
[0092] Reference Figure 3 , a virtual reality application system for operation and maintenance industry, comprising:
[0093] The acquisition module 201 is used to acquire actual scene data and camera positioning information.
[0094] The model building module 202 is used to process the actual scene data according to the positioning information of the camera using a preset algorithm to build a spatial model of the real environment.
[0095] The data adding module 203 is used to add a preset device data panel to the space model.
[0096] The reality matching display module 204 is used to obtain real-time scene data, match the real-time scene data with the space model, and then use a preset algorithm to place the device panel information at the corresponding position of the actual scene.
[0097] Figure 4 A schematic diagram of the structure of a terminal suitable for implementing an embodiment of the present application is shown.
[0098] like Figure 4 As shown, the terminal includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part to the random access memory (RAM) 303. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0099] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.
[0100] In particular, according to the embodiments of the present application, the above reference process Figure 1 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the system of the present application are executed.
[0101] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, register file (RF), etc., or any suitable combination of the above.
[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0103] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module 201, a model building module 202, a data adding module 203, and a reality matching display module 204. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.
[0104] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the terminal described in the above embodiment; or may exist independently and not be assembled into the terminal. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the virtual reality application method for the operation and maintenance industry described in the present application.
[0105] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A virtual reality application method for operation and maintenance industry, comprising the following steps: S101: Operation and maintenance scenario deep model training: Before the operation and maintenance inspection, a large amount of operation and maintenance scenario data is collected for model training; the operation and maintenance scenario data includes equipment status and environmental information; Use deep learning models to process the collected operation and maintenance scenario data, and adjust the model structure and parameters for different operation and maintenance scenarios and tasks to increase the relevance of the operation and maintenance inspection content and the data model; The operation and maintenance scenario data model is classified according to the problem. When inspecting equipment, the model results are closer to the data related to the equipment parameters. When troubleshooting, the model results are closer to the relevant content of the operation manual of some equipment. When inspecting the server room, the data related to the operation and maintenance of the server room is used for training, so that the model can obtain the capabilities related to the operation and maintenance of the server room. S102: After obtaining the operation and maintenance scenario data related to the operation and maintenance scenario, the actual scene is spatially scanned and reconstructed. After the actual scene is spatially scanned and reconstructed, virtual reality information display is realized. First, on-site personnel are required to collect actual scene data, and then use the actual scene data to achieve spatial reconstruction of the actual scene, and then add the corresponding operation and maintenance scene data trained by the deep model to the reconstructed spatial model. When the user wears AR glasses to enter the actual scene, the AR glasses scan the actual scene data at the user's location, match the actual scene data with the spatial model, determine the user's location information, and put the corresponding operation and maintenance scene data trained by the deep model to the corresponding position of the actual scene, thereby achieving the display of the corresponding operation and maintenance data in the actual scene at different locations, specifically: S1021: Obtaining the actual scene data and the positioning information of the camera; first, the on-site personnel use AR glasses that support spatial recognition, turn on the scene scanning function in the software, and scan the on-site environment to obtain image information and depth information; AR glasses obtain depth information by using camera sensors. The methods of obtaining depth information include: the camera sensor uses a structured light sensor or a time-of-flight sensor; the structured light sensor emits a series of light spots or light patterns into the environment, and then senses the position changes of these light spots or light patterns through the camera to calculate the scene depth information; the time-of-flight sensor measures the propagation time of light by sending and receiving infrared light pulses to calculate the depth information; the depth information is then processed to obtain the positioning information of the camera in the AR glasses; S1022: according to the positioning information of the camera, the actual scene data is processed using a preset algorithm to construct a spatial model of the real environment; constructing the spatial model of the real environment includes: calculating the projection matrix, generating a depth image, establishing a spatial model and optimizing the model; S1023: the preset operation and maintenance scene trained by the depth model is The device data panel of the data is added to the spatial model; the model and texture information of the virtual panel are obtained, and the model and texture information of the virtual panel are customized by the user, and the virtual panel user displays the device data panel of the corresponding operation and maintenance scene data trained by the deep model; the device in the spatial model is associated with the data panel of the operation and maintenance scene data of the corresponding device trained by the deep model, so that when the user wears AR glasses to view the corresponding device, the data panel of the operation and maintenance scene data of the corresponding device trained by the deep model can be quickly retrieved and displayed on the virtual panel; S1024: obtain real-time scene data, match the real-time scene data with the spatial model, and use the preset algorithm to place the device panel information to the corresponding position of the actual scene; after the on-site personnel wear AR glasses, open the software to scan the environment and compare it with the previous spatial model, use SLAM (simultaneous localization and mapping) technology to place the device data panel information to the corresponding position in the real scene, and then it can be viewed and used; S103: Implementing interactive functions according to user behavior information; obtaining user behavior information, wherein the behavior information includes action information and audio information; matching corresponding interactive instructions according to wake-up words in the action information or audio information; and calling corresponding feedback programs according to the interactive instructions.
2. The virtual reality application method for operation and maintenance industry according to claim 1, characterized in that: Acquire real-time scene data, match the real-time scene data with the spatial model, and then use a preset algorithm to place the device panel information at the corresponding position of the actual scene, including: Acquire the ground feature point cloud and image data at a first time point; Extract robust feature points in the image data at the first time point, and obtain the location information of the feature points; Obtain the positioning information of the camera based on the location information of the feature points in the image data at the first time point; Match the feature points of the ground feature point cloud and the image data at the first time point with the existing map feature points to determine the positions of the feature points of the ground feature point cloud and the image data in three-dimensional space, and construct a three-dimensional map of the environment at the first time point; Align the preset model and texture information of the virtual panel with the three-dimensional map of the environment at the first time point; Superimpose the virtual panel on the corresponding position in the real environment by matching the position and posture of the virtual panel with the position and posture of the camera.
3. The virtual reality application method for operation and maintenance industry according to claim 2, characterized in that: Matching the real-time scene data with the spatial model also includes: obtaining a ground feature point cloud and image data at a second time point; obtaining a three-dimensional map of the environment at the second time point based on the camera's positioning information at the first time point, the three-dimensional map of the environment, and the ground feature point cloud and image data at the second time point.
4. An application system based on the virtual reality application method for operation and maintenance industry according to any one of claims 1 to 3, characterized in that: include: An acquisition module (201) is used to acquire actual scene data and camera positioning information; A model building module (202) is used to process the actual scene data using a preset algorithm according to the positioning information of the camera to build a spatial model of the real environment; a data adding module (203) is used to add a preset device data panel to the spatial model; The reality matching display module (204) is used to obtain real-time scene data, match the real-time scene data with the space model, and then use a preset algorithm to place the device panel information at a corresponding position in the actual scene.
5. A terminal comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.
