VR-based drone inspection method, system, storage medium and program product

Abstract

A VR-based unmanned aerial vehicle inspection method, system, storage medium and program product, in which a low-polygon terrain model is calculated, subdivided and smoothed to obtain a three-dimensional terrain image; a panoramic image sequence is matched and registered with the three-dimensional terrain image to obtain a point cloud map; a virtual mining area environment map is generated; image quality parameters are calculated; when the image quality parameters are lower than a preset threshold, a semi-real and semi-virtual fusion display screen is generated; the semi-real and semi-virtual fusion display screen is transmitted to a VR helmet for display, and the attitude information of the unmanned aerial vehicle is synchronized to the VR helmet; when fatigue characteristics are determined according to brain wave data, the unmanned aerial vehicle is controlled to switch to a preset autonomous flight mode; a virtual anti-collision boundary is constructed; an optimized flight path is obtained according to the virtual anti-collision boundary, and the unmanned aerial vehicle is controlled to perform inspections according to the optimized flight path. The continuity and reliability of the inspection task are improved, and the safety risk of the inspection task is reduced.

Description

Technical Field

[0001] The present application belongs to the field of data processing methods specifically suitable for management, and in particular, relates to a VR-based drone inspection method, system, storage medium and program product. Background Art

[0002] With the continuous expansion of mine production scale and the continuous improvement of safety requirements, drone inspection technology plays an increasingly important role in mine safety monitoring. The traditional drone inspection method mainly relies on operators to remotely operate through ground station monitoring screens. This method has problems such as limited operator viewing angle, poor adaptability to complex terrain environments, and low inspection efficiency. It is difficult to meet the refined inspection needs in complex mine environments.

[0003] In related technologies, panoramic cameras can be mounted on drones to collect real-time images and transmit image data to VR devices, allowing operators to observe the inspection scene in real time from a first-person perspective, improving spatial perception and operation accuracy during the inspection process. At the same time, the solution also supports superimposing the three-dimensional model of the inspection target in the VR field of view, providing operators with a more intuitive inspection reference.

[0004] However, since VR display screens completely rely on real-time transmitted image data, when environmental conditions change drastically and image quality deteriorates, operators are prone to visual fatigue and dizziness, which reduces the continuity and reliability of inspection tasks and increases the safety risks of inspection tasks. Summary of the invention

[0005] The present application provides a VR-based drone inspection method, system, storage medium, and program product for improving the continuity and reliability of inspection tasks and reducing the safety risks of inspection tasks.

[0006] In the first aspect, the present application provides a VR-based drone inspection method to obtain the drone's location information, attitude information, and a panoramic image sequence taken in real time by a panoramic camera carried by the drone;

[0007] Based on the position information and posture information, a low polygon terrain model within the current viewing angle is calculated, and the low polygon terrain model is subdivided and smoothed to obtain a three-dimensional terrain image;

[0008] Match and register the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map;

[0009] Generate virtual mining environment map based on point cloud map;

[0010] Calculate image quality parameters based on the virtual mining environment map and the flight speed of the UAV;

[0011] When the image quality parameter is lower than a preset threshold, the weight of the texture image is reduced and the weight of the 3D model rendering is increased to generate a semi-real and semi-virtual fusion display screen;

[0012] The semi-real and semi-virtual fusion display screen is transmitted to the VR helmet for display, and the attitude information of the drone is synchronized to the VR helmet;

[0013] Obtaining brain wave data of the operator, and when it is determined based on the brain wave data that the operator has fatigue characteristics, controlling the drone to switch to a preset autonomous flight mode, the preset autonomous flight mode is an obstacle avoidance priority mode, and the drone flies according to a preset flight path;

[0014] Collect visual odometry data and inertial navigation data from drones;

[0015] Construct a virtual anti-collision boundary based on visual odometer data and inertial navigation data combined with a virtual mining area environment map;

[0016] The preset flight path is optimized according to the virtual anti-collision boundary to obtain the optimized flight path, and the UAV is controlled to perform inspections along the optimized flight path.

[0017] By adopting the above technical solution, the position information, attitude information and panoramic image sequence of the UAV are obtained, and a three-dimensional terrain image is generated by combining the low-polygon terrain model. Then, the panoramic image sequence is matched and registered with the three-dimensional terrain image to obtain a point cloud map, and then a virtual mining area environment map is generated, so that the system can accurately grasp the real environment of the UAV. The image quality parameters are calculated according to the virtual mining area environment map and the flight speed of the UAV. When the image quality parameters are lower than the preset threshold, a semi-real and semi-virtual fusion display screen is generated by reducing the proportion of texture images and increasing the proportion of three-dimensional model rendering. Fatigue characteristics are detected in time by detecting the operator's brain wave data, and the preset autonomous flight mode is automatically switched when fatigue characteristics are detected, reducing the potential safety hazards caused by operator fatigue. During the autonomous flight process, a virtual anti-collision boundary is constructed based on visual odometer data and inertial navigation data, and the preset flight path is optimized accordingly, ensuring that the UAV can complete the inspection task safely and efficiently. By organically combining VR display technology, image quality adaptive adjustment, fatigue detection and autonomous obstacle avoidance technologies, the continuity and reliability of the inspection task are improved, and the safety risk of the inspection task is reduced.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, matching and registering the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map specifically includes:

[0019] Extract feature points of each frame image in the panoramic image sequence to obtain a feature point set;

[0020] Align the feature point set with the vertex data in the three-dimensional terrain image to obtain the initial posture transformation matrix;

[0021] According to the initial posture transformation matrix, the feature point set is projected into the three-dimensional terrain image space to form a three-dimensional feature point cloud;

[0022] The 3D feature point cloud is fused with the 3D terrain image to obtain a point cloud map.

[0023] By adopting the above technical solution, the feature points of each frame image in the panoramic image sequence are extracted, and the feature point set is aligned with the vertex data in the three-dimensional terrain image to obtain an initial pose transformation matrix, and then the feature point set is projected into the three-dimensional terrain image space using the matrix to form a three-dimensional feature point cloud. Finally, the three-dimensional feature point cloud is fused with the three-dimensional terrain image to obtain a point cloud map, which improves the accuracy of the point cloud map and the matching degree between the VR display screen and the real environment, so that the operator can more accurately perceive the environmental conditions around the drone.

[0024] In conjunction with some embodiments of the first aspect, in some embodiments, the image quality parameter is calculated according to the virtual mining environment map and the flight speed of the drone, specifically including:

[0025] Extract texture density values, depth gradient values ​​and edge feature quantities from the virtual mining environment map;

[0026] Calculate the image motion blur coefficient according to the flight speed of the drone;

[0027] Calculate image clarity score based on texture density value, depth gradient value and number of edge features;

[0028] The image clarity score is weightedly modified in combination with the image motion blur coefficient to obtain the image quality parameter.

[0029] By adopting the above technical solution, the texture density value, depth gradient value and number of edge features in the virtual mining environment map are extracted, and the image quality is comprehensively evaluated in combination with the image motion blur coefficient calculated by the UAV flight speed. The image clarity score calculated based on the texture density value, depth gradient value and number of edge features reflects the static quality characteristics of the image, while the weighted correction through the image motion blur coefficient takes into account the impact of dynamic factors on image quality. By comprehensively considering the static image characteristics and the impact of dynamic blur, the system can reduce the visual fatigue of the operator due to poor image quality while ensuring the effective information transmission of the display screen, thereby improving the accuracy of UAV remote control and the work comfort of the operator.

[0030] In conjunction with some embodiments of the first aspect, in some embodiments, before transmitting the semi-real and semi-virtual fused display image to the VR helmet for display, the method further includes:

[0031] Collect the motion parameters of the VR helmet and the operator's sight focus data;

[0032] Calculate the rate of change of the field of view angle and the acceleration of the head rotation according to the motion parameters;

[0033] When the rate of change of the viewing angle exceeds a first dynamic threshold or the acceleration of the head rotation exceeds a second dynamic threshold, dynamic blur compensation is performed on the semi-real and semi-virtual fusion display image;

[0034] Divide the attention area and non-attention area of ​​the semi-real and semi-virtual fusion display screen according to the sight focus data;

[0035] The focus area is rendered with high precision, and the non-focus area is rendered with low precision to generate a partitioned rendering display screen.

[0036] By adopting the above technical solution, the motion parameters of the VR helmet and the operator's line of sight focus data are collected, and dynamic blur compensation is performed on the display screen when the field of view angle change rate or the head rotation acceleration exceeds the corresponding dynamic threshold, and the display screen is partitioned and rendered according to the line of sight focus data. This display optimization solution based on the visual characteristics of the human eye adopts differentiated rendering strategies for different areas, which not only improves the display quality of the focus area, but also reduces the occupation of computing resources. The dynamic blur compensation mechanism can reduce the dizziness of the operator during rapid head turning and improve the viewing comfort of VR display. By combining motion compensation and partition rendering, the real-time performance of the system is improved while ensuring the display effect, allowing operators to monitor the flight status and surrounding environment of the drone more naturally and comfortably through the VR helmet.

[0037] In combination with some embodiments of the first aspect, in some embodiments, dynamic blur compensation is performed on the semi-real and semi-virtual fusion display picture, specifically including:

[0038] Obtaining the temporal resolution parameters and spatial resolution parameters of the human eye retina;

[0039] The maximum perceptible motion speed is calculated based on the temporal resolution parameter;

[0040] Determine the clarity attenuation coefficient within different viewing angles based on the spatial resolution parameter;

[0041] Compare the maximum perceptible motion speed with the current picture motion speed;

[0042] When the current picture motion speed is greater than the maximum perceptible motion speed, a compensation coefficient is calculated according to the excess ratio, where the excess ratio is the ratio of the difference between the current picture motion speed and the maximum perceptible motion speed to the maximum perceptible motion speed;

[0043] The compensation coefficient and the clarity attenuation coefficient are used to perform motion compensation on the image.

[0044] By adopting the above technical solution, the temporal resolution parameters and spatial resolution parameters of the human eye retina are introduced, and a compensation mechanism based on the physiological characteristics of the human eye is established. The maximum perceptible motion speed of the human eye is calculated based on the temporal resolution parameters, and the clarity attenuation coefficient within different viewing angles is determined based on the spatial resolution parameters, so that the compensation process matches the perception characteristics of the human eye. When the image motion speed exceeds the maximum perceptible speed of the human eye, the system can adaptively calculate the compensation coefficient based on the excess ratio, and perform targeted compensation on the image in combination with the clarity attenuation coefficient, which can reduce the degree of image motion blur while maintaining the continuity of the image, improve the visual comfort of operators when watching VR display images, and reduce visual fatigue.

[0045] In conjunction with some embodiments of the first aspect, in some embodiments, before constructing a virtual anti-collision boundary based on the visual odometer data and the inertial navigation data in combination with the virtual mining area environment map, the method further includes:

[0046] Collect meteorological and lighting parameters of the drone's surrounding environment;

[0047] Calculate the atmospheric turbulence intensity and visibility based on meteorological parameters; determine the ambient light intensity and shadow distribution based on lighting parameters;

[0048] Adjust the weight of visual odometry data according to the intensity of atmospheric turbulence and visibility;

[0049] The inertial navigation data is compensated and calibrated according to the ambient light intensity and shadow distribution.

[0050] By adopting the above technical solutions, the meteorological parameters and lighting parameters of the drone's surrounding environment are collected, and environmental characteristics such as atmospheric turbulence intensity, visibility, ambient light intensity and shadow distribution are calculated, thus achieving environmental adaptability optimization of visual odometer data and inertial navigation data. When atmospheric turbulence and visibility conditions are poor, the system reduces the impact of environmental factors on visual measurement accuracy by adjusting the weight of visual odometer data. Compensation calibration of inertial navigation data based on ambient light intensity and shadow distribution can reduce navigation errors caused by lighting changes, improve the reliability of visual odometer and inertial navigation, and make the subsequent construction of virtual anti-collision boundaries more accurate.

[0051] In conjunction with some embodiments of the first aspect, in some embodiments, trajectory optimization is performed on a preset flight path according to a virtual anti-collision boundary to obtain an optimized flight path, specifically including:

[0052] Extracting a contour feature point sequence of a virtual anti-collision boundary, and calculating a curvature value between adjacent feature points in the contour feature point sequence;

[0053] Determine the turning area and the straight area of ​​the virtual anti-collision boundary according to the curvature value;

[0054] Increase the sampling point density of the preset flight path in the turning area and reduce the sampling point density in the straight area to obtain the initial waypoint sequence;

[0055] Calculate the minimum spacing constraint between adjacent points in the initial waypoint sequence based on the minimum turning radius of the drone;

[0056] Based on the minimum spacing constraint, the density of the initial waypoint sequence is optimized, and the waypoints that do not meet the minimum spacing constraint are eliminated to obtain a sparse waypoint sequence;

[0057] Smoothing the sparse waypoint sequence to obtain a continuous and smooth path curve;

[0058] Calculate the tangential acceleration and normal acceleration at any point on the path curve;

[0059] The path curve whose tangential acceleration and normal acceleration satisfy the dynamic constraints of the UAV is taken as the optimized flight path.

[0060] By adopting the above technical solution, the curvature value of the virtual anti-collision boundary contour feature point sequence is analyzed, the turning area and the flat area are identified, and the waypoint sampling density is adjusted accordingly to make the waypoint distribution more reasonable. The waypoint sequence is density optimized and smoothed in combination with the minimum turning radius constraint of the UAV to obtain a continuous smooth path curve that meets the dynamic constraints, which can improve the efficiency of path planning while ensuring flight safety. Since the optimized flight path meets the tangential acceleration and normal acceleration constraints of the UAV, the UAV can smoothly perform turning maneuvers and slow down drastic attitude changes. By adopting a trajectory optimization strategy that uniformly considers geometric constraints and dynamic constraints, the maneuverability and control accuracy of the UAV in complex environments are improved, making the execution of inspection tasks safer and more reliable.

[0061] In a second aspect, an embodiment of the present application provides a VR-based drone inspection system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0062] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a system, causes the system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0063] In a fourth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a system, the system executes the method described in any possible implementation manner in the first aspect.

[0064] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0065] 1. The present application provides a VR-based drone inspection method, which obtains the location information, attitude information and panoramic image sequence of the drone, generates a three-dimensional terrain image in combination with a low-polygon terrain model, and then matches and registers the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map, and then generates a virtual mining area environment map, so that the system can accurately grasp the real environment of the drone. The image quality parameter is calculated according to the virtual mining area environment map and the flight speed of the drone. When the image quality parameter is lower than the preset threshold, a semi-real and semi-virtual fusion display screen is generated by reducing the proportion of texture images and increasing the proportion of three-dimensional model rendering. By detecting the brain wave data of the operator, fatigue characteristics are timely detected, and when fatigue characteristics are detected, the preset autonomous flight mode is automatically switched to reduce the potential safety hazards caused by operator fatigue. During the autonomous flight process, a virtual anti-collision boundary is constructed based on visual odometer data and inertial navigation data, and the preset flight path is optimized accordingly, ensuring that the drone can complete the inspection task safely and efficiently. By organically combining VR display technology, image quality adaptive adjustment, fatigue detection and autonomous obstacle avoidance, the continuity and reliability of the inspection task are improved, and the safety risk of the inspection task is reduced.

[0066] 2. The present application provides a VR-based drone inspection method, which collects the motion parameters of the VR helmet and the operator's line of sight focus data, performs dynamic blur compensation on the display screen when the field of view angle change rate or the head rotation acceleration exceeds the corresponding dynamic threshold, and performs partition rendering on the display screen according to the line of sight focus data. This display optimization solution based on the visual characteristics of the human eye adopts differentiated rendering strategies for different areas, which not only improves the display quality of the focus area, but also reduces the occupation of computing resources. The dynamic blur compensation mechanism can reduce the operator's dizziness during rapid head turning and improve the viewing comfort of VR display. By combining motion compensation and partition rendering, the real-time performance of the system is improved while ensuring the display effect, allowing operators to monitor the flight status and surrounding environment of the drone more naturally and comfortably through the VR helmet.

[0067] 3. The present application provides a VR-based UAV inspection method to analyze the curvature value of the virtual anti-collision boundary contour feature point sequence, identify the turning area and the flat area, and adjust the waypoint sampling density accordingly to make the waypoint distribution more reasonable. The waypoint sequence is density optimized and smoothed in combination with the minimum turning radius constraint of the UAV to obtain a continuous smooth path curve that meets the dynamic constraints, which can improve the efficiency of path planning while ensuring flight safety. Since the optimized flight path meets the tangential acceleration and normal acceleration constraints of the UAV, the UAV can smoothly perform turning maneuvers and slow down drastic attitude changes. By adopting a trajectory optimization strategy that takes both geometric constraints and dynamic constraints into consideration, the maneuverability and control accuracy of the UAV in complex environments are improved, making the execution of inspection tasks safer and more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of a VR-based drone inspection method in an embodiment of the present application.

[0069] Figure 2 It is a flow chart of a display optimization method based on human visual characteristics in an embodiment of the present application.

[0070] Figure 3 It is a schematic diagram of the physical device structure of a VR-based drone inspection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0072] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0073] The following uses an embodiment and combines Figure 1 , a VR-based drone inspection method in an embodiment of the present application is described:

[0074] See also Figure 1 , which is a flow chart of a VR-based drone inspection method in an embodiment of the present application.

[0075] S101, obtaining the position information and attitude information of the drone and a panoramic image sequence taken in real time by a panoramic camera carried by the drone;

[0076] In this step, the system uses the GPS positioning module, IMU inertial measurement unit and panoramic camera on the drone to obtain the drone's position coordinates (such as longitude and latitude, altitude), attitude parameters (such as roll, pitch, yaw angles) and panoramic image sequences of the surrounding environment in real time during flight.

[0077] Specifically, the GPS positioning module carried by the drone receives GPS satellite signals and combines them with satellite ephemeris data to calculate the real-time position coordinates of the drone using the principle of triangulation. At the same time, the IMU inertial measurement unit on the drone uses sensors such as accelerometers and gyroscopes to measure acceleration and angular velocity data during flight, and estimates the real-time attitude angle through attitude solution algorithms (such as extended Kalman filtering). In addition, the drone can also be equipped with a barometer to measure the air pressure value and calculate the current flight altitude based on the air pressure-altitude relationship to assist in vertical positioning.

[0078] The panoramic camera of the drone can use a fisheye lens or a multi-camera stitching method to collect real-time environmental images within the 360-degree field of view around the drone to obtain a continuous panoramic image sequence. In order to ensure the timeliness and continuity of image acquisition, the panoramic camera can set a suitable exposure time and frame rate (such as 0.1s exposure, 10fps), and transmit the image data back to the ground in real time through a high-speed data transmission interface (such as USB 3.0).

[0079] S102, calculating a low polygon terrain model within the current viewing angle based on the position information and the posture information, and performing subdivision and smoothing processing on the low polygon terrain model to obtain a three-dimensional terrain image;

[0080] In this step, the system needs to determine the current viewing angle according to the position and attitude of the drone, and load the low polygon (Low Poly) terrain model of the area from the terrain database. The Low Poly model simplifies the terrain with fewer polygonal patches, has high rendering efficiency, and can meet the needs of real-time drawing. In order to improve the display quality, the system performs subdivision smoothing on the Low Poly model, increases the number of polygonal patches, and optimizes the vertex position to make the terrain surface smoother and more natural.

[0081] The system can use CLOD (Continuous Level of Detail) technology to dynamically adjust the level of detail of terrain blocks according to the viewpoint position and direction, and realize multi-resolution rendering related to view distance. Use high-resolution models for nearby terrain blocks and low-resolution models for distant terrain blocks, which reduces the complexity of drawing while ensuring display quality. When performing terrain subdivision, the system can use the Catmull-Clark subdivision algorithm to generate a smoother and more realistic terrain surface by recursively subdividing each polygonal patch.

[0082] S103, matching and registering the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map;

[0083] The system matches and aligns the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map. Specifically: the feature points of each frame in the panoramic image sequence are extracted to obtain a feature point set; the feature point set is aligned with the vertex data in the three-dimensional terrain image to obtain an initial pose transformation matrix; the feature point set is projected into the three-dimensional terrain image space according to the initial pose transformation matrix to form a three-dimensional feature point cloud; the three-dimensional feature point cloud is fused with the three-dimensional terrain image to obtain a point cloud map.

[0084] In this step, the system needs to match and register the panoramic image sequence taken by the drone with the 3D terrain image generated in the previous step to obtain a point cloud map containing rich texture and color information. The matching and registration process needs to find the corresponding position and direction of each frame in the image sequence in the 3D terrain image, and project the image onto the terrain surface to form 3D point cloud data with texture.

[0085] In order to achieve accurate matching and registration, the system can adopt an image registration method based on feature points. First, extract the feature points of each frame in the panoramic image sequence, such as SIFT, SURF, etc., to obtain a feature point set. Then, align the feature point set with the vertex data in the three-dimensional terrain image, and estimate the initial pose transformation matrix from the image to the terrain space through feature point matching. Next, project the image feature points into the three-dimensional terrain image space according to the transformation matrix to form a three-dimensional feature point cloud. Finally, fuse the three-dimensional feature point cloud with the three-dimensional terrain image, and use point cloud registration algorithms such as ICP (Iterative Closest Point) to further optimize the pose transformation and obtain a precisely registered point cloud map.

[0086] S104, generating a virtual mining area environment map based on the point cloud map, and calculating image quality parameters according to the virtual mining area environment map and the flight speed of the UAV;

[0087] The system generates a virtual mining environment map based on the point cloud map, and calculates image quality parameters according to the virtual mining environment map and the flight speed of the UAV. Specifically: the texture density value, depth gradient value and number of edge features in the virtual mining environment map are extracted; the image motion blur coefficient is calculated according to the flight speed of the UAV; the image clarity score is calculated based on the texture density value, depth gradient value and number of edge features; the image clarity score is weightedly corrected in combination with the image motion blur coefficient to obtain the image quality parameters.

[0088] In this step, the system needs to generate a realistic virtual mining environment map based on the point cloud map obtained in the previous step. The virtual environment map not only contains static terrain information, but also needs to simulate dynamic elements of the mining area, such as working machinery and transportation vehicles. At the same time, the system needs to calculate a comprehensive parameter reflecting the image quality based on the flight speed of the drone and the complexity of the virtual environment map for subsequent semi-real and semi-virtual fusion display.

[0089] In order to generate high-quality virtual mining environment maps, the system can use physically based rendering (PBR) technology to simulate the interaction between light and object surfaces in the real world and calculate lighting effects that conform to physical laws, such as diffuse reflection, specular reflection, subsurface scattering, etc. Using PBR materials, different objects in the mining environment can be given a real material appearance. At the same time, the system can also introduce HDR (High Dynamic Range) rendering technology to simulate a wider range of brightness and enhance the contrast and detail expression of the picture.

[0090] When calculating image quality parameters, the system needs to take multiple factors into consideration. First, extract the texture density value in the virtual mining environment map, that is, the number of texture pixels per unit area. The higher the texture density, the better the image clarity. Secondly, calculate the depth gradient value of the virtual environment map, that is, the relative depth difference between the foreground and background objects. The larger the gradient value, the stronger the three-dimensional sense of the image. Thirdly, count the number of edge features in the virtual environment map, such as contour lines, shadow edges, etc. The more features there are, the richer the details of the image. Finally, the system also needs to estimate the degree of motion blur of the image based on the flight speed of the drone. The faster the speed, the easier it is for the image to blur.

[0091] S105, when the image quality parameter is lower than a preset threshold, reducing the weight of the texture image and increasing the weight of the 3D model rendering to generate a semi-real and semi-virtual fusion display image;

[0092] In this step, the system needs to dynamically adjust the weight of the virtual scene and the real image in the fused display according to the image quality parameters calculated in the previous step. When the image quality parameters are lower than the preset threshold, it indicates that the quality of the real image currently captured is poor, and there may be problems such as overexposure, underexposure, and motion blur. Directly displaying the real image will affect the user experience. Therefore, the system needs to reduce the weight of the real texture image and increase the rendering weight of the virtual 3D model to generate a semi-real and semi-virtual fused display.

[0093] In order to achieve seamless fusion of real images and virtual scenes, the system can adopt a fusion strategy based on image semantic segmentation. First, semantic segmentation is performed on the real image to identify different semantic regions such as the sky, ground, and buildings. Then, according to the quality parameters of different semantic regions, their weights in the fused image are adjusted separately. For example, for the sky area, due to the lack of significant texture and edge features, the image quality parameters are usually low, so the real image weight of the sky area can be reduced, while the rendering weight of the virtual sky can be increased. For the ground and building areas, the weights of the real image and the virtual model are dynamically adjusted according to parameters such as texture density and depth gradient to ensure that while maintaining realism, there will be no obvious quality problems.

[0094] When fusing virtual scenes and real images, the system also needs to consider the geometric consistency and lighting consistency between the two. Through feature point matching and pose estimation, the virtual scene and the real image can be geometrically registered to ensure the spatial consistency of the two. At the same time, the system can use the lighting estimation algorithm in the image, such as lighting estimation based on spherical harmonics, to obtain the ambient lighting information of the real scene, and apply it to the rendering of the virtual scene to achieve consistency in lighting tone between the virtual scene and the real scene.

[0095] In order to further improve the realism of the fusion display, the system can also introduce depth-based image fusion technology. Use a depth camera or a binocular camera to obtain the depth map of the real scene and fuse it with the depth buffer of the virtual scene. According to the depth value, the occlusion relationship between virtual and real objects is determined to achieve realistic foreground and background fusion. At the same time, the system can also use the depth information to perform three-dimensional reconstruction of the real image, obtain a three-dimensional model of the real scene, and fuse it with the virtual scene more finely to improve the realism and immersion of the fusion display.

[0096] S106, transmitting the semi-real and semi-virtual fusion display image to the VR helmet for display, and synchronizing the attitude information of the drone to the VR helmet;

[0097] In this step, the system needs to transmit the semi-real and semi-virtual fusion display screen generated in the previous step to the VR helmet worn by the operator in real time for display. At the same time, the system also needs to synchronize the attitude information of the drone, including pitch angle, roll angle, yaw angle, etc., to the VR helmet in real time, so that the display perspective of the helmet is consistent with the perspective of the drone, providing an immersive first-person perspective experience.

[0098] In order to achieve low-latency, high-quality image transmission, the system can adopt the H.265 high-definition video encoding standard and use its excellent compression performance to reduce the transmission bandwidth while ensuring image quality. At the same time, the system can also introduce adaptive bit rate control technology to dynamically adjust encoding parameters and transmission strategies according to network conditions to ensure smooth transmission and reliability. On the VR helmet side, the system needs to use an efficient decoder, such as GPU-based hardware decoding, to ensure low-latency, high-frame-rate image decoding and rendering.

[0099] In order to synchronize the attitude information of the drone with the display perspective of the VR helmet, the system can use the inertial measurement unit (IMU) on the helmet to detect the position and angle changes of the helmet in real time, and integrate it with the attitude information of the drone to dynamically adjust the display perspective of the helmet. By introducing prediction algorithms and Kalman filters, the impact of network transmission delays and IMU measurement errors can be reduced, and smooth and coherent perspective synchronization can be achieved.

[0100] S107, obtaining brain wave data of the operator, and when it is determined according to the brain wave data that the operator has fatigue characteristics, controlling the drone to switch to a preset autonomous flight mode;

[0101] The system obtains the operator's brain wave data. When it is determined that the operator has fatigue characteristics based on the brain wave data, the system controls the drone to switch to a preset autonomous flight mode. The preset autonomous flight mode is an obstacle avoidance priority mode, and the drone flies according to a preset flight path.

[0102] In this step, the system needs to obtain the operator's brain wave data in real time, and determine whether the operator has fatigue characteristics by analyzing the frequency, amplitude and other characteristics of the brain waves. When fatigue characteristics are detected, the system automatically switches the drone to the preset autonomous flight mode to ensure flight safety. In autonomous flight mode, the drone performs flight missions according to the preset flight path and obstacle avoidance strategy, reducing dependence on the operator.

[0103] In order to obtain brain wave data, the system can integrate brain wave sensors, such as EEG (electroencephalogram) electrodes, on the VR helmet to collect the potential signal on the operator's scalp surface. The collected EEG signals need to be preprocessed, such as removing power frequency interference and electrooculogram interference, to extract effective brain wave components. Then, the system uses signal processing methods such as time-frequency analysis and wavelet analysis to extract the frequency domain and time domain characteristics of brain waves, such as the relative energy ratio of Alpha waves and Beta waves, and the amplitude change trend of Theta waves and Delta waves.

[0104] By collecting and analyzing the brain wave data of a large number of people, the system can establish a characteristic model of brain waves in a fatigue state. By comparing the real-time collected brain wave characteristics of the operator with the fatigue model, it can be determined whether the current operator is in a fatigue state. In order to improve the accuracy of fatigue detection, the system can also introduce machine learning algorithms, such as support vector machines (SVM), convolutional neural networks (CNN), etc., to automatically learn and optimize the fatigue discrimination model through training sample data.

[0105] Once fatigue characteristics are detected, the system will trigger the drone's mode switch. The strategy of autonomous flight mode can be preset according to the application scenario and mission requirements. Generally, autonomous flight mode needs to take into account both flight safety and mission completion. Its core is the obstacle avoidance algorithm and path planning algorithm. The system can use technologies such as visual obstacle avoidance and lidar obstacle avoidance to detect obstacles around the drone in real time and plan a safe obstacle avoidance path. At the same time, the system also needs to consider mission constraints, such as coverage area, flight duration, etc., to optimize the flight path and improve flight efficiency while meeting obstacle avoidance and mission constraints.

[0106] S108, collecting visual odometer data and inertial navigation data of the UAV, and constructing a virtual anti-collision boundary based on the visual odometer data and the inertial navigation data in combination with a virtual mining area environment map;

[0107] In this step, the system needs to obtain the visual odometer data and inertial navigation data of the drone, and build a virtual anti-collision boundary in the virtual environment in combination with the constructed virtual mining environment map. The virtual anti-collision boundary represents the spatial range in which the drone can safely fly in the current state, and is an important reference for obstacle avoidance algorithms and path planning.

[0108] Visual odometry is a technology that uses continuous images taken by an onboard camera to estimate the motion posture changes of a drone. By tracking and matching the feature points in the image sequence, the relative posture transformation between adjacent images can be calculated, and the motion trajectory of the drone can be obtained by integrating these relative postures. Visual odometry can provide relatively accurate short-term position estimates, but the error will accumulate over time. The inertial navigation system uses IMU to measure the acceleration and angular velocity of the drone, and obtains position, velocity and attitude information through integration. Inertial navigation is sensitive to short-term motion, but is not affected by the external environment.

[0109] In order to obtain robust and accurate state estimation, the system needs to fuse visual odometer and inertial navigation. Common fusion schemes include loose coupling and tight coupling. The loose coupling scheme performs weighted averaging or Kalman filtering on the estimation results of the two methods to obtain the final state estimation. The tightly coupled scheme considers both visual measurement and IMU measurement during the state estimation process, builds a unified optimization model, and jointly solves state quantities such as posture, velocity, and IMU bias to achieve deeper information fusion.

[0110] After obtaining a high-precision state estimate of the drone, the system can construct a virtual anti-collision boundary in the virtual mining environment map. The specific method is to locally crop the virtual environment map with the current position of the drone as the center to obtain a local environment model centered on the drone. Then, combined with the drone's kinematic parameters such as speed and acceleration, the drone's motion trajectory is simulated in the local environment model, and the trajectory is collided with obstacles in the environment model to obtain a safe flight space range, namely the virtual anti-collision boundary.

[0111] S109, optimizing the trajectory of the preset flight path according to the virtual anti-collision boundary to obtain an optimized flight path, and controlling the UAV to perform inspections according to the optimized flight path.

[0112] The system optimizes the trajectory of the preset flight path according to the virtual anti-collision boundary to obtain the optimized flight path. Specifically: extract the contour feature point sequence of the virtual anti-collision boundary and calculate the curvature value between adjacent feature points in the contour feature point sequence; determine the turning area and the straight area of ​​the virtual anti-collision boundary according to the curvature value; increase the sampling point density of the preset flight path in the turning area and reduce the sampling point density in the straight area to obtain the initial waypoint sequence; calculate the minimum spacing constraint between adjacent points in the initial waypoint sequence according to the minimum turning radius of the drone; optimize the density of the initial waypoint sequence based on the minimum spacing constraint, eliminate the waypoints that do not meet the minimum spacing constraint, and obtain a sparse waypoint sequence; smooth the sparse waypoint sequence to obtain a continuous and smooth path curve; calculate the tangential acceleration and normal acceleration at any point on the path curve; and use the path curve whose tangential acceleration and normal acceleration meet the dynamic constraints of the drone as the optimized flight path. Finally, the drone is controlled to perform inspections according to the optimized flight path.

[0113] In this step, the system needs to optimize the trajectory of the preset flight path under the constraints of the virtual anti-collision boundary to obtain a safe, smooth and efficient flight path, and finally control the drone to perform the inspection task according to the optimized path. The path optimization problem is a classic motion planning problem that requires comprehensive consideration of multiple factors such as the kinematic constraints, dynamic constraints, and environmental constraints of the drone.

[0114] The system first needs to geometrically represent and mathematically model the preset flight path and virtual anti-collision boundary. Common path representation methods include path point sequence, parametric curve, spline curve, etc. The virtual anti-collision boundary can be represented in the form of point cloud, mesh, implicit surface, etc. Then, the system defines an objective function for path optimization, such as minimizing path length, minimizing energy consumption, maximizing safety margin, etc., and uses the virtual anti-collision boundary as one of the optimization constraints.

[0115] Optimization can be solved by using classic numerical optimization methods, such as gradient descent, Newton's method, sequential quadratic programming, etc., to solve the minimum value of the objective function and obtain the optimal path parameters under the premise of satisfying the constraints. However, these methods are prone to local lag and are not suitable for dealing with complex non-convex constraints. Therefore, the system can also introduce intelligent optimization algorithms, such as genetic algorithms, particle swarm optimization algorithms, simulated annealing algorithms, etc., to conduct extensive searches and evolutions in the solution space through swarm intelligence, jump out of the local optimum, and find the global optimal solution.

[0116] In order to further improve the safety and smoothness of the path, the system can also introduce some heuristic rules in the optimization process. For example, by analyzing the geometric features of the virtual anti-collision boundary, high-risk areas such as narrow passages and sharp corners are extracted, and these areas are specially processed during path optimization, such as increasing path sampling points and improving safety margin coefficients. At the same time, the system can also smooth the optimized path according to the dynamic characteristics of the drone, such as B-spline fitting, minimum curvature spline, etc., so that the path is more in line with the actual flight characteristics of the drone.

[0117] Finally, the system sends the generated optimized flight path to the UAV's flight control system, controlling the UAV to perform the flight mission along this path. During the flight execution process, the system also needs to perform real-time monitoring and feedback control. On the one hand, the onboard sensors continuously perceive the environment around the UAV, compare the real-time environmental information with the virtual environment map, and promptly detect environmental changes and potential threats; on the other hand, the actual flight status of the UAV, such as position, speed, acceleration, etc., is monitored and compared with the planned path, and the trajectory is adjusted and corrected in real time according to the deviation, ensuring that the UAV can still accurately fly along the planned path while responding to environmental changes, and complete the inspection task safely and efficiently.

[0118] In the above embodiment, the position information, attitude information and panoramic image sequence of the UAV are obtained, and a three-dimensional terrain image is generated in combination with a low-polygon terrain model. Then, the panoramic image sequence is matched and registered with the three-dimensional terrain image to obtain a point cloud map, and then a virtual mining area environment map is generated, so that the system can accurately grasp the real environment of the UAV. The image quality parameter is calculated according to the virtual mining area environment map and the flight speed of the UAV. When the image quality parameter is lower than the preset threshold, a semi-real and semi-virtual fusion display screen is generated by reducing the proportion of texture images and increasing the proportion of three-dimensional model rendering. By detecting the brain wave data of the operator, fatigue characteristics are timely discovered, and when fatigue characteristics are detected, the preset autonomous flight mode is automatically switched to reduce the potential safety hazards caused by operator fatigue. During the autonomous flight process, a virtual anti-collision boundary is constructed based on visual odometer data and inertial navigation data, and the preset flight path is optimized accordingly, ensuring that the UAV can complete the inspection task safely and efficiently. By organically combining VR display technology, image quality adaptive adjustment, fatigue detection and autonomous obstacle avoidance, the continuity and reliability of the inspection task are improved, and the safety risk of the inspection task is reduced.

[0119] Furthermore, before step S108 of the above embodiment constructs a virtual anti-collision boundary based on the visual odometer data and inertial navigation data in combination with the virtual mining area environment map, the system can also collect meteorological parameters and lighting parameters of the environment around the drone; calculate the atmospheric turbulence intensity and visibility based on the meteorological parameters; determine the ambient light intensity and shadow distribution based on the lighting parameters; adjust the weight of the visual odometer data according to the atmospheric turbulence intensity and visibility; and compensate and calibrate the inertial navigation data based on the ambient light intensity and shadow distribution.

[0120] Specifically, the system first needs to use the meteorological sensors on the drone, such as thermometers, hygrometers, barometers, etc., to collect the meteorological parameters of the drone's surrounding environment in real time. The two most important indicators of meteorological parameters are atmospheric turbulence intensity and visibility. Atmospheric turbulence can cause the drone to shake and jitter, affecting the feature point extraction and matching of the visual mileage calculation method. The level of visibility directly determines the detection distance and image quality of the visual sensor. Therefore, the system needs to calculate the quantitative values ​​of atmospheric turbulence intensity and visibility in real time based on meteorological parameters.

[0121] For the intensity of atmospheric turbulence, the system can use empirical models, such as the Dryden turbulence model, to estimate the power spectral density and correlation time of turbulence based on the current wind speed, temperature, and air pressure data. For visibility, the system can use visibility sensors, such as lidar or image contrast method, to directly measure the visibility value of the current environment. After obtaining these two indicators, the system can adjust the weight of the visual odometer data in the fusion positioning algorithm accordingly. When the turbulence intensity is large or the visibility is low, the credibility of the visual odometer decreases, and the system needs to reduce its weight accordingly, and increase the weight of the inertial navigation data to ensure the stability of the drone state estimation.

[0122] In addition to meteorological parameters, the system also needs to use the lighting sensors on the drone, such as light intensity meters and color meters, to collect the lighting parameters of the environment in real time. The two most important indicators of lighting parameters are ambient light intensity and shadow distribution. The intensity of ambient light directly affects the exposure time and signal-to-noise ratio of the image sensor, which in turn affects the accuracy of the visual odometer. Shadow areas will cause underexposure of the image, reducing the visibility and matching rate of feature points. Therefore, the system needs to determine the value of ambient light intensity and the spatial distribution of shadow areas in real time based on the lighting parameters.

[0123] For ambient light intensity, the system can use illumination models, such as the Perez sky illumination model, to estimate global illumination and direct light intensity based on the current sun position, weather conditions, and geographic location. For shadow areas, the system can use shadow detection algorithms, such as color- and texture-based shadow detection, to segment pixel masks of shadow areas from images. After obtaining these two indicators, the system can compensate and calibrate the inertial navigation data accordingly. Since inertial navigation is based on inertial measurements of accelerometers and gyroscopes, their measurements will accumulate drift over time. The system can use the positioning results of the visual odometer in well-lit areas to perform periodic position and attitude calibration on the inertial navigation, and use the ambient light intensity and shadow distribution to adaptively adjust the timing and frequency of calibration.

[0124] In the above embodiment, meteorological parameters and lighting parameters of the environment around the drone are collected, and environmental characteristics such as atmospheric turbulence intensity, visibility, ambient light intensity and shadow distribution are calculated, thereby optimizing the environmental adaptability of visual odometer data and inertial navigation data. When atmospheric turbulence and visibility conditions are poor, the system reduces the impact of environmental factors on visual measurement accuracy by adjusting the weight of the visual odometer data. Compensation and calibration of inertial navigation data based on ambient light intensity and shadow distribution can reduce navigation errors caused by lighting changes, improve the reliability of visual odometers and inertial navigation, and make the subsequently constructed virtual anti-collision boundaries more accurate.

[0125] In the first embodiment, the overall process of the VR-based drone inspection method is introduced, including a complete technical solution from drone data collection, environmental modeling to autonomous flight control. In order to further improve the VR display effect and enhance the operator's experience, this application also proposes a display optimization method based on the visual characteristics of the human eye. Figure 2 , a display optimization method based on human visual characteristics in an embodiment of the present application is described:

[0126] See also Figure 2 , which is a flow chart of a display optimization method based on human visual characteristics in an embodiment of the present application.

[0127] S201, collecting motion parameters of the VR helmet and operator's sight focus data;

[0128] In this step, the system needs to collect the motion parameters of the VR helmet and the operator's eye focus data in real time. The motion parameters include the position, posture, speed, acceleration and other information of the helmet, which can be obtained through the IMU (inertial measurement unit) sensor on the helmet. The eye focus data reflects the operator's current gaze point position and can be collected through the eye tracking device on the helmet.

[0129] In order to obtain high-precision helmet motion parameters, the system can use multi-sensor fusion technology. In addition to IMU, other positioning technologies such as optical tracking and ultrasonic tracking can also be integrated to improve the robustness of positioning and tracking through filtering and fusion of multi-source perception data. At the same time, the system can also introduce motion prediction algorithms to predict the movement trend of the helmet based on historical motion data and reduce position drift caused by sensor delay.

[0130] Eye tracking technology is the key to realizing the collection of gaze focus data. The system can adopt a video-based eye tracking solution, using the built-in infrared camera in the helmet to capture the user's eye movement, and detect the pupil position through image processing algorithms to calculate the gaze focus coordinates. In order to improve the accuracy and real-time performance of eye tracking, the system can also combine eye tracking with head posture estimation, and convert the gaze direction in the eye coordinate system to the gaze point position in the world coordinate system through a three-dimensional gaze-gaze point mapping model.

[0131] S202, calculating the rate of change of the viewing angle and the acceleration of the head rotation according to the motion parameters;

[0132] In this step, the system needs to calculate two key parameters in real time: the rate of change of the field of view angle and the acceleration of the head rotation. The rate of change of the field of view angle indicates the speed at which the user's field of view changes, reflecting the amplitude of the movement of the screen content. The acceleration of the head rotation measures the intensity of the user's head rotation, reflecting the intensity of the user's own movement. Both parameters are closely related to the visual characteristics of the human eye and are important bases for subsequent display optimization.

[0133] In order to calculate the rate of change of the field of view angle, the system first needs to obtain the size of the user's field of view angle. Usually, the field of view angle is determined by the size of the display screen and the distance between the user and the screen. The system can calculate the user's horizontal and vertical field of view angles based on the screen parameters of the VR helmet and the optical model of the human eye. Then, combined with the posture change data of the helmet, the system can estimate the change in the field of view angle per unit time, that is, the rate of change of the field of view angle.

[0134] The calculation of the head rotation acceleration requires the angular velocity and angular acceleration data of the helmet. The system can obtain the angular velocity and angular acceleration of the head around the three coordinate axes by filtering and fusing the gyroscope and accelerometer data. Taking into account the physiological characteristics of human head movement, the system can also perform weighted processing on the angular acceleration data to highlight the rotation acceleration in the horizontal plane, because this kind of movement is more likely to cause visual discomfort.

[0135] S203, when the rate of change of the viewing angle exceeds the first dynamic threshold or the acceleration of the head rotation exceeds the second dynamic threshold, dynamic blur compensation is performed on the semi-real and semi-virtual fusion display picture;

[0136] When the rate of change of the field of view angle exceeds the first dynamic threshold or the acceleration of the head rotation exceeds the second dynamic threshold, dynamic blur compensation is performed on the semi-real and semi-virtual fused display image, specifically: the time resolution parameter and the spatial resolution parameter of the human eye retina are obtained; the maximum perceptible motion speed is calculated according to the time resolution parameter; the clarity attenuation coefficient within different viewing angles is determined based on the spatial resolution parameter; the maximum perceptible motion speed is compared with the current image motion speed; when the current image motion speed is greater than the maximum perceptible motion speed, the compensation coefficient is calculated according to the excess ratio, the excess ratio being the difference between the current image motion speed and the maximum perceptible motion speed and the ratio of the maximum perceptible motion speed; the image motion is compensated using the compensation coefficient and the clarity attenuation coefficient.

[0137] In this step, the system needs to determine whether dynamic blur compensation is needed for the semi-real and semi-virtual fusion display according to the field of view angle change rate and head rotation acceleration calculated in the previous step. Dynamic blur compensation is a display optimization technology based on the visual characteristics of the human eye. Its purpose is to reduce the user's dizziness and discomfort and improve the visual experience by appropriately blurring the picture when the picture moves at high speed or the user turns his head violently.

[0138] Specifically, the system first needs to set two dynamic thresholds: the first dynamic threshold is for the rate of change of the field of view angle, and the second dynamic threshold is for the acceleration of the head rotation. These two thresholds can be adjusted according to the physiological characteristics of the human eye and user feedback. They must be high enough to avoid excessive blur, but low enough to ensure the necessary compensation effect. When the rate of change of the field of view angle exceeds the first dynamic threshold or the acceleration of the head rotation exceeds the second dynamic threshold, the system will trigger the dynamic blur compensation mechanism.

[0139] The core of compensation is to perform motion blur processing on the picture. The system first needs to estimate the speed of the picture. It can calculate the motion vector of the object in the picture through algorithms such as optical flow method and block matching method. Then, the system calculates the maximum perceptible motion speed based on the spatiotemporal resolution characteristics of the human eye's retina, that is, the upper limit of the object's motion speed that the human eye can clearly perceive. If the current picture motion speed exceeds this upper limit, the system will calculate a compensation coefficient based on the excess ratio to control the degree of blur.

[0140] When performing blur processing, the system also needs to consider the spatial frequency characteristics of the image content. Generally, high-frequency details are more likely to cause dynamic blur, while low-frequency smooth parts are less affected. Therefore, the system can perform frequency domain transformation on the image, adopt different blur strategies for different frequency areas, focus on blurring high-frequency areas, and keep low-frequency areas relatively clear. At the same time, the system also combines the depth information of the displayed image to blur the distant background area to a greater extent, while keeping the nearby focus area clear, simulating the depth of field effect of the human eye.

[0141] S204, dividing the semi-real and semi-virtual fusion display screen into a focus area and a non-focus area according to the sight focus data;

[0142] In this step, the system needs to use the sight focus data collected in the previous step to divide the semi-real and semi-virtual fusion display screen into the focus area and the non-focus area. The focus area refers to the area of ​​the screen where the user's sight is focused, which usually contains important information that the user is interested in, and the display quality needs to be guaranteed. The non-focus area refers to the area of ​​the screen where the user's sight is not focused, which is relatively unimportant, and the display accuracy can be appropriately reduced to save computing resources.

[0143] In order to achieve accurate area division, the system first needs to perform cluster analysis on the gaze focus data. Since the human eye's gaze will constantly jump during observation, there may be random noise in a single gaze focus data. The system can use algorithms such as mean-shift clustering and DBSCAN clustering to aggregate spatially adjacent and temporally continuous gaze focuses into clusters, with each cluster corresponding to a potential area of ​​interest. During the clustering process, the system also needs to consider the gaze duration of the gaze focus, as longer gaze times often mean higher levels of attention.

[0144] After completing the cluster analysis, the system needs to generate the spatial boundary of the attention area based on the clustering results. A simple method is to calculate the convex hull of each visual focus cluster to obtain a set of irregular polygonal areas. However, this method does not take into account the spatial distribution characteristics of human vision. In fact, when the human eye is looking at a focus, it can also obtain a good visual experience within a certain range around it, and this range gradually decays as the focus deviates. Therefore, the system can use a Gaussian mixture model to establish a two-dimensional Gaussian distribution for each focus cluster, and generate a smooth and continuous attention attenuation curve based on the mean and variance of the distribution, forming a more realistic and natural attention area boundary.

[0145] S205 , performing high-precision rendering on the focus area and low-precision rendering on the non-focus area to generate a partitioned rendering display screen.

[0146] In this step, the system needs to perform partition rendering on the semi-real and semi-virtual fusion display screen according to the focus area and non-focus area divided in the previous step. The screen content in the focus area will be rendered with high precision to ensure visual quality; the screen content in the non-focus area will be rendered with low precision to save computing resources. This partition rendering strategy conforms to the foveal sensitivity of human vision and can achieve the best display effect with limited computing power.

[0147] In order to achieve high-precision rendering, the system can use higher rendering resolution, finer grid division, more lighting sampling points, etc. in the area of ​​interest to improve the clarity, sharpness and realism of the picture. At the same time, for virtual objects in the area of ​​interest, the system can also use physical based rendering (PBR) technology to finely simulate the reflection, refraction, scattering and other optical properties of the material to generate a more realistic and natural rendering effect. For real images in the area of ​​interest, the system can use super-resolution algorithms and use deep learning networks to intelligently interpolate and enhance the images to improve texture details and edge sharpness.

[0148] In non-areas of interest, the system uses a low-precision rendering strategy to reduce the rendering load. Specifically, the system can use lower rendering resolution, coarser grid division, fewer lighting sampling points, etc. to moderately reduce the fineness of the picture. For virtual objects in non-areas of interest, the system can use a simplified PBR model, ignoring minor optical properties and retaining only basic diffuse and specular reflection effects. For real images in non-areas of interest, the system can directly use the original resolution without additional enhancement processing.

[0149] In order to smoothly transition the display difference between the focus area and the non-focus area, the system also needs to take certain fusion measures on the boundary of the two areas. A common method is to establish a gradient transition zone at the boundary. Within the transition zone, the rendering parameters change continuously from high precision to low precision to achieve a smooth connection of the picture. However, this linear interpolation method sometimes produces unnatural edge effects. Therefore, the system can use an algorithm based on Poisson fusion to generate the optimal gradient field at the boundary by solving a Poisson equation, thereby achieving seamless fusion between the focus area and the non-focus area.

[0150] In the above embodiment, the motion parameters of the VR helmet and the operator's line of sight focus data are collected, and dynamic blur compensation is performed on the display screen when the field of view angle change rate or the head rotation acceleration exceeds the corresponding dynamic threshold, and the display screen is partitioned and rendered according to the line of sight focus data. This display optimization solution based on the visual characteristics of the human eye adopts differentiated rendering strategies for different areas, which not only improves the display quality of the focus area, but also reduces the occupation of computing resources. The dynamic blur compensation mechanism can reduce the operator's dizziness during rapid head turning and improve the viewing comfort of VR display. By combining motion compensation and partition rendering, the real-time performance of the system is improved while ensuring the display effect, allowing operators to monitor the flight status and surrounding environment of the drone more naturally and comfortably through the VR helmet.

[0151] The following describes the system in the embodiment of the present invention from the perspective of hardware processing. Figure 3, which is a schematic diagram of the physical device structure of a VR-based drone inspection system provided in an embodiment of the present application.

[0152] It should be noted that Figure 3 The structure of the system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0153] like Figure 3 As shown, the system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0154] The following components are connected to the I / O interface 305: an input section 306 including a camera, an infrared sensor, etc.; an output section 307 including a liquid crystal display (LCD) and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0155] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0156] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the system described in the above embodiment; or may exist independently without being assembled into the system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of a system, the system implements the method provided in the above embodiment.

[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0160] As used in the above embodiments, the term "when..." may be interpreted as "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted as "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0161] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0162] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A VR-based drone inspection method, characterized in that: include: Acquire the position information and attitude information of the drone and a panoramic image sequence taken in real time by a panoramic camera carried by the drone; Calculating a low polygon terrain model within the current viewing angle based on the position information and the posture information, and performing subdivision and smoothing processing on the low polygon terrain model to obtain a three-dimensional terrain image; Matching and registering the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map; Generate a virtual mining area environment map based on the point cloud map; Calculating image quality parameters according to the virtual mining area environment map and the flight speed of the UAV; When the image quality parameter is lower than a preset threshold, the weight of the texture image is reduced and the weight of the three-dimensional model rendering is increased to generate a semi-real and semi-virtual fusion display screen; Transmitting the semi-real and semi-virtual fusion display image to a VR helmet for display, and synchronizing the attitude information of the drone to the VR helmet; Acquiring brain wave data of an operator, and when it is determined according to the brain wave data that the operator has fatigue characteristics, controlling the drone to switch to a preset autonomous flight mode, wherein the preset autonomous flight mode is a mode with obstacle avoidance priority, and the drone flies according to a preset flight path; Collecting visual odometer data and inertial navigation data of the UAV; Constructing a virtual anti-collision boundary according to the visual odometer data and the inertial navigation data in combination with the virtual mining area environment map; The preset flight path is trajectory optimized according to the virtual anti-collision boundary to obtain an optimized flight path, and the UAV is controlled to perform inspections according to the optimized flight path.

2. The method according to claim 1, characterized in that The step of matching and registering the panoramic image sequence with the three-dimensional terrain image to obtain a point cloud map specifically includes: Extracting feature points of each frame of the panoramic image sequence to obtain a feature point set; Aligning the feature point set with vertex data in the three-dimensional terrain image to obtain an initial posture transformation matrix; Projecting the feature point set into the three-dimensional terrain image space according to the initial posture transformation matrix to form a three-dimensional feature point cloud; The three-dimensional feature point cloud is fused with the three-dimensional terrain image to obtain a point cloud map.

3. The method according to claim 1, characterized in that The calculating of the image quality parameter according to the virtual mining area environment map and the flight speed of the UAV specifically includes: Extracting texture density values, depth gradient values ​​and edge feature quantities in the virtual mining area environment map; Calculating an image motion blur coefficient according to the flight speed of the drone; Calculating an image clarity score based on the texture density value, the depth gradient value, and the number of edge features; The image clarity score is weightedly modified in combination with the image motion blur coefficient to obtain an image quality parameter.

4. The method according to claim 1, characterized in that: Before transmitting the semi-real and semi-virtual fusion display picture to the VR helmet for display, the method further includes: Collect the motion parameters of the VR helmet and the operator's sight focus data; Calculating the rate of change of the viewing angle and the acceleration of the head rotation according to the motion parameters; When the rate of change of the viewing angle exceeds a first dynamic threshold or the head rotation acceleration exceeds a second dynamic threshold, dynamic blur compensation is performed on the semi-real and semi-virtual fusion display image; Dividing the semi-real and semi-virtual fusion display image into a focus area and a non-focus area according to the sight focus data; The focus area is rendered with high precision, and the non-focus area is rendered with low precision to generate a partitioned rendering display screen.

5. The method according to claim 4, characterized in that The performing dynamic blur compensation on the semi-real and semi-virtual fusion display picture specifically includes: Obtaining the temporal resolution parameters and spatial resolution parameters of the human eye retina; Calculating the maximum perceptible motion speed according to the temporal resolution parameter; Determining the clarity attenuation coefficient within different viewing angle ranges based on the spatial resolution parameter; Comparing the maximum perceptible motion speed with the current picture motion speed; When the current picture motion speed is greater than the maximum perceptible motion speed, a compensation coefficient is calculated according to an excess ratio, wherein the excess ratio is a ratio of a difference between the current picture motion speed and the maximum perceptible motion speed and the maximum perceptible motion speed; The compensation coefficient and the definition attenuation coefficient are used to perform motion compensation on the picture.

6. The method according to claim 1, characterized in that Before constructing a virtual anti-collision boundary according to the visual odometer data and the inertial navigation data in combination with the virtual mining area environment map, the method further includes: Collecting meteorological parameters and lighting parameters of the environment surrounding the UAV; Calculating atmospheric turbulence intensity and visibility based on the meteorological parameters; determining ambient light intensity and shadow distribution based on the lighting parameters; adjusting the weight of the visual odometer data according to the atmospheric turbulence intensity and the visibility; The inertial navigation data is compensated and calibrated according to the ambient light intensity and the shadow distribution.

7. The method according to claim 1, characterized in that The performing trajectory optimization on the preset flight path according to the virtual anti-collision boundary to obtain an optimized flight path specifically includes: Extracting a contour feature point sequence of the virtual anti-collision boundary, and calculating a curvature value between adjacent feature points in the contour feature point sequence; Determining a turning area and a straight area of ​​the virtual anti-collision boundary according to the curvature value; Increasing the sampling point density of the preset flight path in the turning area and reducing the sampling point density in the straight area to obtain an initial waypoint sequence; Calculating a minimum spacing constraint between adjacent points in the initial waypoint sequence according to a minimum turning radius of the drone; Density optimization is performed on the initial waypoint sequence based on the minimum spacing constraint, and waypoints that do not meet the minimum spacing constraint are eliminated to obtain a sparse waypoint sequence; Smoothing the sparse waypoint sequence to obtain a continuous and smooth path curve; Calculating the tangential acceleration and the normal acceleration at any point on the path curve; The path curve in which the tangential acceleration and the normal acceleration satisfy the dynamic constraints of the UAV is used as the optimized flight path.

8. A VR-based drone inspection system, characterized in that: The system comprises: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the system to execute the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a system, the system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a system, the system is caused to execute the method according to any one of claims 1 to 7.

Images