Vehicle detection path planning and interaction method and system based on meta universe, and application of vehicle detection path planning and interaction method and system in Mars rover detection

Abstract

The invention discloses a vehicle detection path planning and interaction method and system based on the universe and application of the vehicle detection path planning and interaction method and system in Mars rover detection. The method comprises the following steps: constructing collaborative interaction: realizing voice, video and data transmission based on a WebRTC protocol, classifying gesture actions of a user by adopting a support vector machine algorithm, and mapping a classification result into a vehicle movement, data acquisition or path modification operation instruction in a meta-universe environment; a machine learning method is used for extracting geological data features, a terrain detection target is combined, a vehicle terrain detection path plan is generated and projected to the meta-universe environment in real time, and a user shares and visualizes data in the meta-universe environment through a collaborative interaction module and checks and modifies the path plan; and after the user confirms that the path planning is correct, the final optimized path planning and related operation instructions are fed back to the vehicle to guide the vehicle to execute the terrain detection task, so that the efficiency and scientificity of terrain detection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of planet exploration such as Mars, and particularly to a vehicle detection path planning and interaction method and system based on the metaverse, and its application in Mars rover detection. Background Art

[0002] In exploration missions such as those on Mars, terrain detection is crucial for revealing its geological evolution, climate change, etc. However, traditional vehicle path planning methods have many deficiencies when serving terrain detection tasks. Due to relying on limited geographical data, the planning accuracy and real-time performance are poor, and it is difficult to adapt to complex terrain environments, resulting in low efficiency and high risk when the vehicle reaches terrain areas with research value. At the same time, the lack of a multi-party real-time collaboration mechanism makes it impossible to fully integrate expert knowledge to optimize the path, and it is difficult to meet the requirements of terrain detection tasks for data comprehensiveness and accuracy.

[0003] With the development of advanced technologies such as the metaverse, VR, AR, and machine learning, new opportunities have emerged to solve these problems. The metaverse can build a realistic virtual environment to assist in decision-making, VR and AR enable natural interaction, and machine learning can process and analyze geological data to optimize the path. However, the application of these technologies in path planning for terrain detection tasks on planets such as Mars is still in its infancy, and a mature and efficient collaboration system has not yet been formed.

[0004] Therefore, there is an urgent need for an innovative solution that integrates advanced technologies. The present invention aims to provide a vehicle detection path planning and interaction method and system based on the metaverse through multi-technology integration to improve the efficiency and scientific nature of terrain detection. Summary of the Invention

[0005] In view of the above deficiencies or improvement requirements of the prior art, the present invention provides a vehicle detection path planning and interaction method and system based on the metaverse, and its application in Mars rover detection, to guide the vehicle to perform terrain detection tasks and improve the efficiency and scientific nature of terrain detection.

[0006] To achieve the above object, according to one aspect of the present invention, there is provided a vehicle detection path planning and interaction method based on the metaverse, including the following steps:

[0007] S1 Construct collaborative interaction: Implement voice, video, and data transmission based on the WebRTC protocol, classify user gesture actions using the support vector machine algorithm, and map the classification results to operation instructions for vehicle movement, data collection, or path modification in the metaverse environment;

[0008] S2 Plan and optimize the detection path: Based on step S1, use machine learning methods to extract geological data features, combine with the terrain detection target, generate a vehicle terrain detection path plan and project it onto the metaverse environment in real time. Users can share and visualize data in the metaverse environment through the collaborative interaction module, and review and modify the path plan.

[0009] S3 Execute the detection task: After the user confirms that the path plan obtained in step S2 is correct, feedback the final optimized path plan and related operation instructions to the vehicle to guide it to execute the terrain detection task.

[0010] Preferably, step S1 specifically includes the following steps:

[0011] S11: Based on step S1, use VR technology and multi-threaded rendering technology to build an immersive detection environment that supports multiple users to be online simultaneously.

[0012] S12: Combine the virtual environment in step S11 with the real scene through AR technology to support users to interact with the virtual environment in the real space.

[0013] S13: Based on step S12, adopt real-time communication technology based on the WebRTC protocol, and realize voice, video and data transmission between multiple participants through NAT penetration and media negotiation mechanisms; at the same time, with the help of gesture and action recognition technology, use the support vector machine algorithm to classify and recognize gesture and action data, and map the recognition results to operation instructions in the virtual environment to realize real-time interaction between people and vehicles.

[0014] Preferably, step S2 specifically includes the following steps:

[0015] S21: Preprocess the collected geological data, and the preprocessing includes data denoising, normalization and data augmentation.

[0016] S22: Input the data preprocessed in step S21 into a pre-trained ResNet-50 architecture deep learning model to extract terrain features, potential obstacle and resource distribution information.

[0017] S23: Based on the features extracted in step S22, combine with the terrain detection target and constraints, and adopt the A* algorithm combined with the dynamic weight adjustment strategy to generate a preliminary detection path.

[0018] The dynamic weight adjustment strategy satisfies:

[0019] h(n) = α·C + β·S + γ·D

[0020] Among them, C is the terrain complexity, S is the terrain feature richness, D is the resource density, and α, β, and γ are weight coefficients, which are dynamically adjusted according to the focus and requirements of the terrain detection task. And in terrain detection, the values of α and β are relatively large to highlight the leading role of terrain factors in path planning;

[0021] The evaluation function for each node of the A* algorithm is:

[0022] f(n) = g(n) + h(n)

[0023] In the formula, g(n) is the actual cost from the starting point to the current node, and h(n) is the estimated cost from the current node to the target node. The optimal path is searched by continuously expanding the node with the smallest f(n) value. The dynamic weight adjustment strategy adjusts the weight of h(n) in real time according to the terrain complexity, terrain feature richness, and resource density, that is:

[0024] In complex terrain areas, increase the weight of the terrain complexity factor in h(n) to make the path avoid complex terrain as much as possible; at the same time, for complex terrain areas with important scientific research value, appropriately reduce the avoidance weight so that the vehicle can approach for detailed detection;

[0025] S24: Project the preliminary terrain detection path plan generated in step S23 into the metaverse environment in real time for users to review and modify. Users can adjust and optimize the preliminary terrain detection path plan through the real-time discussion and modification functions in the collaborative interaction module. Users can adjust the path through gesture actions or the interaction interface;

[0026] S25: Update the path planning result optimized in step S24 into the metaverse environment in real time. Before updating, check the integrity and accuracy of the new path data, and then update it after ensuring it is correct for users to further confirm.

[0027] Preferably, in step S21, wavelet transform denoising method is used for data denoising: the signal is decomposed into sub-signals of different frequencies by wavelet transform, and the high-frequency sub-signals where the noise is located are removed through threshold processing. Finally, the denoised data is reconstructed by inverse wavelet transform to ensure the accuracy and reliability of the data;

[0028] Normalization uses the min-max normalization method to map the data to the [0,1] interval to ensure the consistency of different types of data in subsequent processing;

[0029] The data augmentation method is to perform operations such as rotation, flipping, and scaling on image data, and to use methods such as adding noise and linear transformation on numerical data to increase the diversity of data.

[0030] Preferably, the specific training process of the ResNet-50 architecture deep learning model in step S22 includes the following steps:

[0031] (1) Set the initial learning rate, momentum parameter, and weight decay coefficient, and use batch normalization technology to accelerate model convergence;

[0032] (2) During the training process, perform data augmentation operations such as random cropping, brightness adjustment, and contrast adjustment on the input images to further expand the training data;

[0033] (3) Use the Adam optimizer to dynamically adjust the learning rate according to the change of the loss function during the training process. When the decrease of the loss function in consecutive training epochs is less than a certain threshold, reduce the learning rate to a preset multiple.

[0034] Preferably, it also includes constructing a metaverse environment: obtaining surface geographical data, collecting data through a detector for real-scene modeling, generating a three-dimensional terrain model, and using AR technology to fuse the real-time data of the detector with the real-scene model to create a metaverse environment;

[0035] The obtained geographical data includes terrain elevation, landform features, geological structures, and mineral distribution information;

[0036] The detector collects data, performs image processing and feature extraction through computer vision technology, and generates real-scene textures of the surface.

[0037] Preferably, constructing the metaverse environment specifically includes the following steps:

[0038] S01: Based on the obtained geographical data and real-scene textures, adopt a grid division algorithm based on curvature threshold, adjust the model complexity according to the terrain complexity, increase details in complex terrain areas and reduce details in flat areas to generate a three-dimensional model of the terrain;

[0039] S02: Through texture mapping technology, accurately attach the real surface texture extracted from the images taken by the detector to the surface of the three-dimensional model generated in step S01;

[0040] S03: With the help of AR technology, adopt an image registration algorithm based on feature points to fuse the real-time data collected by the detector with the three-dimensional model generated in step S02 to achieve real-time update of the virtual environment.

[0041] Preferably, the specific steps of step S01 are as follows: Using terrain analysis software, calculate the curvature and slope characteristic parameters of the terrain. According to a large number of experimental and simulation results, set a complexity threshold. When the characteristic parameters are higher than the threshold, it is determined as a complex terrain area. At this time, a refined grid division method is adopted, increasing the number of vertices and polygons during model construction, and increasing the grid density to enhance details. When the characteristic parameters are lower than the threshold, a simplified grid division is adopted to reduce the data volume of the model and reduce the grid density to lower the computational load, thereby generating a high-precision three-dimensional terrain model;

[0042] The specific steps of step S02 are as follows: Using image segmentation technology, extract the corresponding texture features of different ground object types in the images captured by the detector, and then accurately attach the texture to the surface of the three-dimensional model through an accurate coordinate mapping algorithm;

[0043] The feature point-based image registration algorithm in step S03 includes the following steps:

[0044] (1) For the real-time image collected by the detector and the virtual image corresponding to the three-dimensional model, use the scale-invariant feature transform algorithm to extract feature points;

[0045] (2) Use a two-way feature matching method to find the pairs of mutually matching feature points in the two images in step (1). To screen out more accurate matching points, filtering will be performed according to information such as the distance and direction between the feature points to remove obviously incorrect matches;

[0046] (3) Using the accurately matched feature points in step (2), through homography matrix calculation, obtain the transformation parameters for fusing the real-time image collected by the detector with the three-dimensional model, thereby realizing the precise fusion of real-time data and the three-dimensional model, and ensuring that the virtual environment can be quickly updated according to the real-time data of the detector.

[0047] The method for fusing real-time data with a three-dimensional model includes the following steps:

[0048] In the metaverse environment, adopt a positioning algorithm based on virtual reality space coordinate mapping and combined with inertial measurement unit assistance to assign a unique virtual coordinate to each user; The system generates targeted detection instructions based on the user's real-time location information and the historical analysis results of geological data;

[0049] After receiving the detection instructions, the detector activates the corresponding detection equipment according to the instruction parameters to collect data on the user's location and surrounding areas;

[0050] The collected data undergoes preprocessing of denoising, normalization, and enhancement, and then is transmitted to the metaverse environment through the wireless communication technology of the collaborative interaction module. The new data is fused with the existing three-dimensional model using a feature point-based image registration algorithm.

[0051] The real-time data and 3D model fusion method includes the following steps:

[0052] In the metaverse environment, a positioning algorithm based on virtual reality space coordinate mapping and combined with an inertial measurement unit is used to assign a unique virtual coordinate to each user; the system generates targeted detection instructions based on the user's real-time position information and the historical analysis results of geological data.

[0053] After receiving the detection instructions, the detector activates the corresponding detection equipment according to the instruction parameters to collect data on the location and surrounding areas of the user.

[0054] The collected data is preprocessed by denoising, normalization, and enhancement, and then transmitted to the metaverse environment through the wireless communication technology of the collaborative interaction module. A feature point-based image registration algorithm is used to fuse the new data with the existing 3D model.

[0055] Preferably, the positioning algorithm based on virtual reality space coordinate mapping and combined with an inertial measurement unit includes the following steps:

[0056] The initial position and orientation information of the participant are obtained by using virtual reality equipment and mapped into the virtual space to assign a unique virtual coordinate to each user.

[0057] The inertial measurement unit in the user's wearable device obtains motion data in real time to update the user's position and attitude in real time.

[0058] The position information obtained by virtual reality space coordinate mapping and the inertial measurement unit data are fused through the Kalman filter algorithm to smooth the position information, remove noise interference, improve the positioning accuracy, and meet the high-precision interaction requirements in the metaverse environment.

[0059] The present invention combines an adaptive 3D modeling algorithm with AR technology to create a dynamic virtual environment for vehicle terrain detection. By collecting multi-dimensional geographical data and using a 3D modeling algorithm that can adapt to terrain complexity, the generated high-precision terrain model can accurately display various terrain features on the surface of planets such as Mars, providing accurate and dynamic geographical information support for the vehicle to plan terrain detection paths in the metaverse environment. With the help of AR technology to real-time fuse the data transmitted back by the detector, users can see the changes on the surface of planets such as Mars in real time, making the virtual scene and the real scene synchronized in real time, greatly improving the accuracy and timeliness of terrain detection, and helping the vehicle to more efficiently plan terrain detection paths.

[0060] To achieve the above object, according to one aspect of the present invention, there is provided a vehicle detection path planning and interaction system based on the metaverse, which adopts the above method and includes:

[0061] A collaborative interaction module that implements voice, video, and data transmission based on the WebRTC protocol, and uses a support vector machine algorithm to classify user gesture actions, and maps the classification results to operation instructions for vehicle movement, data collection, or path modification in the metaverse environment;

[0062] A path planning and optimization module that uses machine learning methods to extract geological data features, combines terrain exploration goals, generates a vehicle terrain exploration path plan and projects it onto the metaverse environment in real time. Users can share and visualize data in the metaverse environment through the collaborative interaction module to review and modify the path plan;

[0063] A feedback execution module that, after the user confirms that the path plan is correct, feeds back the final optimized path plan and related operation instructions to the vehicle to guide it to perform the exploration task;

[0064] It also includes a metaverse construction module, which includes a data collection module, a data processing module, and an environment simulation module. The data collection module obtains surface geographical data, and the data processing module performs preprocessing on the geographical data obtained by the data collection module and conducts model training; the environment simulation module collects data through detectors for real-scene modeling, generates a three-dimensional terrain model, and uses AR technology to fuse the real-time data of the detectors with the real-scene model to create a metaverse environment.

[0065] To achieve the above object, according to one aspect of the present invention, there is provided an application of a vehicle detection path planning and interaction method based on the metaverse in the exploration of a Mars rover. By using the above method, the Mars rover can perform exploration tasks.

[0066] Generally speaking, compared with the prior art through the above technical solutions conceived by the present invention, the following beneficial effects are obtained:

[0067] 1. The present invention innovatively integrates machine learning, user gesture action recognition, and intelligent path planning technologies, and optimizes the vehicle terrain exploration process. Machine learning preprocesses and extracts features from geological data of planets such as Mars, and can dig out key information of terrain and landforms, providing a scientific basis for terrain exploration. The support vector machine algorithm can accurately identify user gesture actions and convert them into vehicle operation instructions, allowing users to intuitively control the vehicle's movement. Combining the A* algorithm and the dynamic weight adjustment strategy, the weight of path planning is adjusted in real time according to the terrain complexity and landform feature distribution. This human-machine collaboration method realizes efficient path planning, significantly improves the interaction efficiency and accuracy of terrain exploration, and enables the vehicle to complete terrain exploration tasks more accurately.

[0068] 2. The present invention integrates VR, AR, real-time communication, and gesture recognition technologies, relying on the WebRTC protocol. It uses VR to create an immersive exploration environment for planets such as Mars, AR to achieve seamless interaction between virtual and real worlds, the WebRTC protocol to ensure multi-party real-time communication, and gesture recognition technology to convert user actions into operation instructions. In the metaverse environment, users can jointly formulate vehicle terrain exploration plans and interactively adjust them in real time, achieving efficient operation of collaborative path planning and interaction for vehicle terrain exploration, and opening up a new interaction mode for exploring planets such as Mars. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flowchart of the method for collaborative path planning and interaction of a Mars rover terrain exploration based on the metaverse in an embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of the detailed process for creating a virtual Mars environment in an embodiment of the present invention;

[0071] Figure 3 is a flowchart of the adaptive three-dimensional modeling algorithm in an embodiment of the present invention;

[0072] Figure 4 is a flowchart of the path planning algorithm in an embodiment of the present invention;

[0073] Figure 5 is a schematic diagram of the collaborative path planning effect of a Mars rover terrain exploration based on the metaverse provided by the present invention;

[0074] Figure 6 is an architecture diagram of the system for collaborative path planning and interaction of a Mars rover terrain exploration based on the metaverse in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0075] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0076] The present invention provides a flowchart of the method for vehicle detection path planning and interaction based on the metaverse. As Figure 1 shown, the planet to be detected in this embodiment is Mars, and the vehicle is a Mars rover, which includes the following steps:

[0077] S1 Build collaborative interaction: Implement voice, video, and data transmission based on the WebRTC protocol, classify user gesture actions using the support vector machine algorithm, and map the classification results to operation instructions for the rover's movement, data collection, or path modification in the metaverse environment of Mars;

[0078] Step S1 specifically includes the following steps:

[0079] S11: On the basis of step S1, use VR technology and multi-threaded rendering technology to build an immersive Mars exploration environment that supports multiple users to be online simultaneously;

[0080] The multi-threaded rendering technology divides the rendering tasks into multiple subtasks, which are respectively assigned to different threads for execution. Through synchronization and coordination among threads, the smoothness of rendering is ensured, and the number of threads is determined according to the core number, memory size of the hardware device, and the complexity of the rendering tasks.

[0081] Exemplarily, if the hardware device is an 8-core processor and the rendering scene includes complex terrain and high-resolution textures, set 3 scene rendering threads, 2 lighting calculation threads, 2 texture loading threads, and 1 auxiliary thread to achieve thread load balancing. At the same time, allocate larger computing resources to the scene rendering threads to ensure the efficiency of model construction; the lighting calculation threads and texture loading threads reasonably allocate the remaining resources according to the task characteristics to ensure the balance between rendering effects and speed.

[0082] S12: Combine the virtual Mars environment in step S11 with the real scene through AR technology to support users to interact with the virtual environment in the real space;

[0083] S13: On the basis of step S12, adopt real-time communication technology based on the WebRTC protocol to achieve voice, video, and data transmission among multiple participants through the NAT traversal and media negotiation mechanisms; at the same time, with the help of gesture and action recognition technology, use the support vector machine algorithm to classify and recognize gesture and action data, and map the recognition results to operation instructions in the virtual environment to achieve real-time interaction between humans and the Mars rover.

[0084] S2 Plan and optimize the detection path: On the basis of step S1, use machine learning methods to extract the characteristics of Mars geological data, combine with the terrain detection target, generate the terrain detection path planning of the Mars rover and project it onto the metaverse environment in real time. Users share and visualize the data in the Mars metaverse environment through the collaborative interaction module and review and modify the path planning;

[0085] Among them, it can be seen from Figure 4 , step S2 specifically includes the following steps:

[0086] S21: Preprocess the collected Martian geological data, where the preprocessing includes data denoising, normalization, and data augmentation;

[0087] The data denoising uses the wavelet transform denoising method: First, decompose the signal into sub-signals of different frequencies using wavelet transform; then, remove the high-frequency sub-signals where the noise is located through threshold processing; finally, reconstruct the denoised data through inverse wavelet transform to ensure the accuracy and reliability of the data;

[0088] The normalization uses the min-max normalization method, and the formula is:

[0089]

[0090] where, X norm is the data value obtained after normalization, X is the original collected data value, X min is the minimum data value collected, X max is the maximum data value collected.

[0091] This method maps the data to the [0, 1] interval to ensure the consistency of different types of data in subsequent processing;

[0092] The data augmentation includes: for image data, operations such as rotation, flipping, and scaling are adopted, and for numerical data, methods such as adding noise and linear transformation are used to increase the diversity of the data.

[0093] S22: Input the data preprocessed in step S21 into a pre-trained ResNet-50 architecture deep learning model to extract terrain features, potential obstacles, and resource distribution information;

[0094] Specifically, the specific training process of the ResNet-50 model includes the following steps:

[0095] (1) Set the initial learning rate to 0.001, the momentum parameter to 0.9, the weight decay coefficient to 0.0001, and adopt batch normalization technology to accelerate the convergence of the model;

[0096] (2) During the training process, perform data augmentation operations such as randomly cropping, adjusting brightness, and adjusting contrast on the input images to further expand the training data;

[0097] Exemplarily, randomly crop the image of Olympus Mons on Mars, allowing the model to learn the features of different parts of the mountain body, including the crater edge, slopes, etc. By adjusting the brightness and contrast, simulate the lighting changes under different times and weather conditions on Mars. During a Martian dust storm, the lighting weakens and the contrast decreases. By adjusting the brightness and contrast of the image, let the model learn the geological features under this special lighting condition, thereby improving the adaptability of the model to the complex environment on Mars.

[0098] (3) The Adam optimizer is adopted to dynamically adjust the learning rate according to the change of the loss function during the training process. When the decrease of the loss function in consecutive training epochs is less than a certain threshold, the learning rate is reduced to a preset multiple. In this embodiment, the learning rate is reduced to 0.1 times the original value.

[0099] S23: Based on the features extracted in step S22, combined with the terrain detection target and constraint conditions, the A* algorithm is adopted and combined with the dynamic weight adjustment strategy to generate a preliminary detection path;

[0100] The dynamic weight adjustment strategy satisfies:

[0101] h(n) = α·C + β·S + γ·D

[0102] where C is the terrain complexity, S is the terrain feature richness, D is the resource density, α, β, and γ are weight coefficients, which are dynamically adjusted according to the key points and requirements of the terrain detection task. And in terrain detection, the values of α and β are relatively large to highlight the dominant role of terrain factors in path planning;

[0103] The terrain complexity C is the average curvature of the area where the current node is located, and the calculation formula is:

[0104]

[0105] In the formula, N is the number of sampling points within a preset radius around the current node, and the curvature i is the curvature value of the i-th sampling point.

[0106] The calculation method of the terrain feature richness S is as follows: First, comprehensively consider the number of different terrain types in the current area, and each terrain type is given a basic richness value according to its occurrence frequency in the overall terrain of Mars.

[0107] Exemplarily, the basic richness value of the plain terrain with a higher occurrence frequency is 1, and the basic richness value of the hilly terrain with a lower occurrence frequency is 2. Add up the basic richness values of all terrain types in the current area to get S1.

[0108] For special landforms, a quantitative score (full score 10 points) is given according to their potential research value in aspects such as Martian geological evolution and climate research. If a certain type of special terrain is considered very likely to record Martian ancient geological activities and may preserve early Martian geological information, etc., its score can be set to 8 - 10 points; while for special terrains with relatively low research value, it is set to 3 - 5 points.

[0109] Calculate the total score of various special topographic and geomorphic features in the current area, and calculate it in combination with their area ratios. Suppose there are m types of special topographic and geomorphic features in the current area, and the quantified score of the j-th special topographic and geomorphic feature is v j , and the ratio of its area to the total area of the current area is p j , then the calculation formula for the contribution value S2 of the special topographic and geomorphic features to the richness of topographic features is:

[0110]

[0111] In the formula, the numerator is the sum of the products of the quantified scores of various special topographic and geomorphic features and their area ratios, representing the comprehensive value of the special topographic and geomorphic features considering the area factor. The denominator is the sum of the area ratios of all special topographic and geomorphic features, which is used for normalization to make the value of S2 more comparable;

[0112] Finally, the feature richness S = S1 + S2.

[0113] The resource density D is the matching degree between the spectral reflectance of the area where the current node is located and the preset mineral database, and the calculation formula is:

[0114]

[0115] In the formula, S k is the reflectance of the k-th spectral band, and R k is the weight value of the corresponding band in the preset mineral database, and M is the total number of spectral bands.

[0116] Among them, during the search process of the A* algorithm, the evaluation function of each node is:

[0117] f(n) = g(n) + h(n)

[0118] In the formula, g(n) is the actual cost from the starting point to the current node, and h(n) is the estimated cost from the current node to the target node. The optimal path is searched by continuously expanding the node with the smallest f(n) value, and the dynamic weight adjustment strategy adjusts the weight of h(n) in real time according to the complexity of the terrain, the richness of topographic features, and the resource density.

[0119] The dynamic weight adjustment strategy dynamically adjusts the weight of path planning according to the complexity of the terrain and the distribution of geomorphic features, that is:

[0120] In complex terrain areas, increase the weight of the terrain complexity factor in h(n) to make the path avoid complex terrain as much as possible; at the same time, for complex terrain areas with important scientific research value, appropriately reduce the avoidance weight so that the rover can approach for detailed detection;

[0121] S24: Project the preliminary terrain exploration path plan generated in step S23 onto the metaverse environment in real time for users to review and modify.

[0122] Specifically, as Figure 5 shown in the simulation diagram, using the real-time data transmission technology based on the UDP protocol, quickly transmit the path planning data to the Mars metaverse environment rendering server. The server adopts the ray tracing rendering algorithm to project the path onto the Mars metaverse environment in a three-dimensional form in real time, accurately superimposing it on the three-dimensional model of the Mars terrain. Participants can clearly view the path details through the high-resolution display screen of the VR device.

[0123] Users adjust and optimize the preliminary Mars rover exploration path plan through the real-time discussion and modification functions in the collaborative interaction module. Users can adjust the path through gesture actions or the interaction interface.

[0124] As an embodiment, the real-time discussion includes: adopting real-time caption and speech-to-text technologies to ensure that each user can accurately understand the discussion content; for conflicts that may occur when multiple users collaborate to develop a detection plan, adopting a conflict detection and resolution algorithm. When conflicts occur between the modification plans of different users, the system automatically detects the conflict points and prompts the users to negotiate and resolve them.

[0125] S25: Update the path planning result optimized in step S24 to the metaverse environment in real time. Before the update, perform integrity and accuracy verification on the new path data, and then perform the update after ensuring it is correct for users to further confirm.

[0126] Specifically, when the path optimization is completed, the system uses the UDP protocol again to send the optimized path planning data back to the rendering server. The server re-renders the path according to the new data and updates it to the metaverse environment in the latest form in real time. Before the update, the system will perform integrity and accuracy verification on the new path data, and then perform the update after ensuring it is correct for users to further confirm.

[0127] S3 Execute the detection task: After the user confirms that the path planning obtained in step S2 is correct, feedback the final optimized path planning and related operation instructions to the Mars rover to guide it to execute the terrain detection task.

[0128] Specifically, feedback the optimized path planning and operation instructions to the Mars rover through the collaborative interaction module to guide it to perform further data collection according to the new path planning, forming a closed-loop interaction system.

[0129] The present invention also includes S0 for constructing a Mars metaverse environment: obtaining geographical data on the Mars surface, collecting data through a detector for real-scene modeling, generating a three-dimensional model of the Mars terrain, and using AR technology to fuse the real-time data of the detector with the real-scene model to create a Mars metaverse environment;

[0130] Among them, reference can be made to Figure 2 , specifically, obtaining high-precision geographical data on the Mars surface, covering terrain elevation, landform features, geological structures, and mineral distribution information;

[0131] Collecting detector data, performing image processing and feature extraction through computer vision technology to generate real-scene textures on the Mars surface.

[0132] Furthermore, step S0 includes the following steps:

[0133] S01: Based on the obtained geographical data and real-scene textures, adopting a grid division algorithm based on curvature threshold, adjusting the model complexity according to the terrain complexity, increasing details in complex terrain areas and reducing details in flat areas to generate a three-dimensional model of the Mars terrain;

[0134] S02: Through texture mapping technology, accurately attaching the real surface textures extracted from the images taken by the Mars detector to the surface of the three-dimensional model generated in step S01;

[0135] Among them, reference can be made to Figure 3 , specifically, the implementation steps include: First, using terrain analysis software to calculate characteristic parameters such as the curvature and slope of the terrain, setting a complexity threshold according to a large number of experimental and simulation results. When the characteristic parameters are higher than the threshold, it is determined as a complex terrain area, and at this time, a refined grid division method is adopted to increase the number of vertices and polygons during model construction; when the characteristic parameters are lower than the threshold, a simplified grid division is adopted to reduce the data volume of the model; in terms of texture mapping, using image segmentation technology to extract the corresponding texture features of different ground object types in the images taken by the Mars detector, and then through an accurate coordinate mapping algorithm, accurately attaching the textures to the surface of the three-dimensional model.

[0136] S03: With the help of AR technology, adopting an image registration algorithm based on feature points to fuse the real-time data collected by the detector with the three-dimensional model generated in step S02 to achieve real-time update of the virtual Mars environment.

[0137] As an embodiment, the image registration algorithm based on feature points in step S03 includes the following steps:

[0138] First, for the real-time image collected by the detector and the virtual image corresponding to the 3D model, the Scale-Invariant Feature Transform (SIFT) algorithm is used to extract feature points. Then, a bidirectional feature matching method is employed to find the pairs of feature points that match each other in the two images. To screen out more accurate matching points, filtering is performed based on information such as the distance and direction between the feature points to remove obviously incorrect matches. Finally, using these accurately matched feature points, through the calculation of the homography matrix, the transformation parameters for fusing the real-time image collected by the detector with the 3D model are obtained, thereby achieving the precise fusion of real-time data and the 3D model and ensuring that the virtual Martian environment can be quickly updated according to the real-time data of the detector.

[0139] The method for fusing real-time data with the 3D model includes the following steps:

[0140] In the Martian metaverse environment, a positioning algorithm based on virtual reality space coordinate mapping combined with an inertial measurement unit (IMU) assistance is adopted to assign a unique virtual coordinate to each user. The system generates targeted detection instructions based on the user's real-time position information and the historical analysis results of Martian geological data. After receiving the detection instructions, the detector activates the corresponding detection equipment according to the instruction parameters to collect data on the location and surrounding areas where the user is located. The collected data undergoes preprocessing of denoising, normalization, and enhancement, and then is transmitted to the metaverse environment through the wireless communication technology of the collaborative interaction module. A feature-point-based image registration algorithm is used to fuse the new data with the existing 3D model.

[0141] Specifically, a virtual reality head-mounted display device with built-in high-sensitivity optical sensors and gyroscopes is used to obtain the initial position and orientation information of the user in the Cartesian coordinate system of the real world. Through a specific coordinate transformation function, it is accurately mapped into the 3D coordinate system of the Martian metaverse, assigning a virtual coordinate to each user and establishing an accurate correspondence between the real world and the Martian metaverse.

[0142] The user wears the inertial measurement unit in the device. This unit includes an accelerometer and a gyroscope that operate at a sampling frequency of hundreds of Hertz, and it can continuously obtain the motion data of the user in the Martian metaverse. The accelerometer measures the acceleration changes of the device in three axes, and the gyroscope monitors the angular velocity changes of the device. Through integral operations, these data are converted into the displacement and attitude change information of the user, and then the position and attitude of the user are updated in real time.

[0143] The Kalman filtering algorithm based on the state space model is used to fuse the position information obtained from the virtual reality coordinate mapping in the Martian metaverse with the data of the inertial measurement unit. In the prediction step, the state at the current moment is predicted based on the state and motion model at the previous moment. In the update step, the prediction result is corrected using the newly obtained measurement data, thereby smoothing the position information, removing noise interference, improving the positioning accuracy, and meeting the high-precision interaction requirements in the metaverse environment.

[0144] In addition, to expand the system to support more users or more complex detection tasks, a distributed architecture design is adopted to distribute the computing tasks and storage tasks of the system to multiple server nodes, and reasonable task allocation is achieved through load balancing technology. Among them, reference can be made to Figure 6 , specifically, a vehicle detection path planning and interaction system based on the metaverse includes:

[0145] A collaborative interaction module that implements voice, video, and data transmission based on the WebRTC protocol, and uses a support vector machine algorithm to classify user gesture actions, and maps the classification results to steps of movement, data collection, or path modification operation instructions of the vehicle in the metaverse environment;

[0146] A path planning and optimization module that uses machine learning methods to extract geological data features, combines terrain detection targets, generates a vehicle terrain detection path plan and projects it onto the metaverse environment in real time. Users share and visualize data in the metaverse environment through the collaborative interaction module, and review and modify the path plan;

[0147] A feedback execution module that, after the user confirms that the path plan is correct, feeds back the final optimized path plan and related operation instructions to the vehicle to guide it to execute the detection task;

[0148] It also includes a metaverse construction module, which includes a data collection module, a data processing module, and an environment simulation module. The data collection module obtains surface geographical data, and the data processing module performs preprocessing on the geographical data obtained by the data collection module and conducts model training; the environment simulation module collects data through detectors for real-scene modeling, generates a three-dimensional terrain model, and uses AR technology to fuse the real-time data of the detectors with the real-scene model to create a metaverse environment.

[0149] This embodiment also provides an application of the vehicle detection path planning and interaction method based on the metaverse in the detection of a Mars rover, to generate a three-dimensional terrain model of Mars, and use AR technology to fuse the real-time data of the detector with the real-scene model to create a Mars metaverse environment; generate a terrain detection path plan for the Mars rover and project it onto the metaverse environment in real time. Users share and visualize data in the Mars metaverse environment through the collaborative interaction module, and review and modify the path plan; after the user confirms that the path plan is correct, feed back the final optimized path plan and related operation instructions to the Mars rover to guide it to execute the detection task.

[0150] Each module has independent functions and interfaces, which is convenient for future function expansion.

[0151] Specifically, in adapting to different Mars exploration scenarios, the parameters and algorithm strategies of the machine learning model are dynamically adjusted according to different terrains and exploration targets. In terms of error handling and fault tolerance mechanisms, when error data appears in real-time data fusion, data verification and anomaly detection algorithms are adopted to promptly detect and discard the error data. At the same time, data recovery is performed from historical data. The system uses data backup and redundant computing technologies to regularly back up important data. When the main computing node fails, it automatically switches to the standby node for computing to ensure the stability and reliability of the system.

[0152] A specific embodiment will be given below to elaborate in detail on the role of the metaverse in the terrain exploration path planning of the Mars rover.

[0153] In this embodiment, taking the exploration mission of the Utopia Planitia on Mars as the background, combined with the geographical data of the detector and Mars, a high-precision three-dimensional real-scene model of the Utopia Planitia on Mars is constructed using a grid division algorithm based on curvature thresholds, and real surface textures are added.

[0154] Users enter the metaverse environment with the help of VR devices. In this environment, they can freely switch perspectives to observe. For example, when walking to a rock, they can observe it closely to analyze the rock characteristics through texture, and from an aerial perspective, they can grasp the overall landform and geological structure trend of the plain. This provides a comprehensive and intuitive information basis for the terrain exploration path planning of the Mars rover, enabling users to have a deeper understanding of the exploration area.

[0155] The system generates exploration instructions based on the position information of the user in the metaverse and combines the historical analysis results of the Mars geological data, and sends them to the detector. The detector collects data according to the real-time position of the user, and after preprocessing, it feeds back to the metaverse environment. The newly collected data is integrated into the three-dimensional real-scene model in real time with the help of AR technology, enabling users to grasp the changes on the Mars surface in real time.

[0156] Machine learning methods are used to extract features from the Mars geological data, and combined with real-time data analysis to realize the dynamic update of the Mars metaverse scenario. When planning the path, based on the extracted terrain features, potential obstacles, and resource distribution information, the A* algorithm combined with a dynamic weight adjustment strategy is adopted to generate the path planning. Users share data, visualize and jointly plan the terrain exploration path of the Mars rover in the metaverse environment through a collaborative interaction module built based on the WebRTC protocol. Users can communicate in real time through voice, gestures, etc., jointly optimize the path planning, and the planning results are projected into the metaverse environment in real time for review and modification. Finally, the optimized path planning and operation instructions are fed back to the Mars rover to guide its terrain exploration. The metaverse provides an immersive planning environment, dynamically updated geographical information, and a multi-party collaborative interaction platform for the path planning of the Mars rover, greatly improving the scientific nature of the path planning and the exploration efficiency of the rover.

[0157] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A vehicle detection path planning and interaction method based on the Metaverse, characterized in that: The following steps are involved: S1 builds collaborative interaction: It implements voice, video and data transmission based on the WebRTC protocol, uses the support vector machine algorithm to classify user gestures, and maps the classification results into vehicle movement, data collection or path modification operation instructions in the metaverse environment; S2 plans and optimizes the detection path: Based on step S1, the geological data features are extracted using machine learning methods, combined with terrain detection targets, to generate vehicle terrain detection path planning and project it to the Metaverse environment in real time. Users can share and visualize data in the Metaverse environment through the collaborative interaction module and review and modify the path planning; S3 performs the detection task: After the user confirms that the path planning obtained in step S2 is correct, the final optimized path planning and related operation instructions are fed back to the vehicle to guide it to perform the terrain detection task.

2. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Based on step S1, VR technology is used to adopt multi-threaded rendering technology to build an immersive detection environment that supports multiple users online at the same time; S12: combining the virtual environment in step S11 with the real scene through AR technology to support the user to interact with the virtual environment in the real space; S13: On the basis of step S12, the real-time communication technology based on the WebRTC protocol is used to realize voice, video and data transmission among multiple participants through NAT penetration and media negotiation mechanism; at the same time, with the help of gesture and action recognition technology, the support vector machine algorithm is used to classify and recognize gesture and action data, and the recognition results are mapped into operation instructions in the virtual environment to realize real-time interaction between people and vehicles.

3. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 1, characterized in that: Step S2 specifically includes the following steps: S21: preprocessing the collected geological data, wherein the preprocessing includes data denoising, normalization and data enhancement; S22: Input the data pre-processed in step S21 into a pre-trained ResNet-50 architecture deep learning model to extract terrain features, potential obstacles and resource distribution information; S23: Based on the features extracted in step S22, combined with terrain detection targets and constraints, an A* algorithm combined with a dynamic weight adjustment strategy is used to generate a preliminary detection path; The dynamic weight adjustment strategy satisfies: h(n)=α·C+β·S+γ·D Among them, C is the terrain complexity, S is the richness of terrain features, D is the resource density, α, β and γ are weight coefficients, which are dynamically adjusted according to the focus and needs of the terrain detection task. In terrain detection, the values ​​of α and β are relatively large to highlight the dominant role of terrain factors in path planning. The evaluation function of each node of the A* algorithm is: f(n)=g(n)+h(n) In the formula, g(n) is the actual cost from the starting point to the current node, and h(n) is the estimated cost from the current node to the target node. The optimal path is searched by continuously expanding the node with the smallest f(n) value. The dynamic weight adjustment strategy adjusts the weight of h(n) in real time according to the complexity of the terrain, the richness of terrain features, and the density of resources, that is: In areas with complex terrain, the weight of the terrain complexity factor in h(n) is increased to make the path avoid complex terrain as much as possible; at the same time, for complex terrain areas with important scientific research value, the avoidance weight is appropriately reduced so that the vehicle can get close for detailed detection; S24: The preliminary terrain detection path plan generated in step S23 is projected to the metaverse environment in real time for the user to review and modify. The user adjusts and optimizes the preliminary terrain detection path plan through the real-time discussion and modification functions in the collaborative interaction module. The user can adjust the path through gestures or the interactive interface; S25: Update the path planning results optimized in step S24 to the Metaverse environment in real time, and verify the integrity and accuracy of the new path data before updating. Update it only after ensuring that it is correct for further confirmation by the user.

4. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 3, characterized in that: In step S21, the data denoising adopts the wavelet transform denoising method: the signal is decomposed into sub-signals of different frequencies by wavelet transform, the high-frequency sub-signals where the noise is located are removed by threshold processing, and finally, the denoised data is reconstructed by inverse wavelet transform to ensure the accuracy and reliability of the data; Normalization uses the minimum-maximum normalization method to map the data to the [0,1] interval to ensure the consistency of different types of data in subsequent processing; The data enhancement method uses rotation, flipping, and scaling operations for image data, and adds noise and linear transformation for numerical data to increase data diversity.

5. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 3, characterized in that: The specific training process of the ResNet-50 architecture deep learning model in step S22 includes the following steps: (1) Set the initial learning rate, momentum parameter and weight decay coefficient, and use batch normalization technology to accelerate model convergence; (2) During the training process, random cropping, brightness adjustment, contrast adjustment and other data enhancement operations are performed on the input images to further expand the training data; (3) The Adam optimizer is used to dynamically adjust the learning rate according to the changes in the loss function during the training process. When the loss function drops below a certain threshold for multiple consecutive training cycles, the learning rate is reduced to a preset multiple.

6. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 1, characterized in that: It also includes building a metaverse environment: obtaining surface geographic data, performing real-scene modeling through detector data collection, generating a three-dimensional terrain model, and using AR technology to fuse the detector's real-time data with the real-scene model to create a metaverse environment; The geographic data acquired include terrain elevation, geomorphic features, geological structures, and mineral distribution information; The detector collects data, performs image processing and feature extraction through computer vision technology, and generates the real-life texture of the surface.

7. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 6, characterized in that: Building the metaverse environment specifically includes the following steps: S01: Based on the acquired geographic data and real-scene textures, a curvature threshold-based meshing algorithm is used to adjust the model complexity according to the complexity of the terrain, increase details in complex terrain areas, and reduce details in flat areas to generate a three-dimensional model of the terrain; S02: Using texture mapping technology, accurately fit the real surface texture extracted from the image taken by the detector to the surface of the three-dimensional model generated in step S01; S03: With the help of AR technology, an image registration algorithm based on feature points is adopted to fuse the real-time data collected by the detector with the three-dimensional model generated in step S02 to achieve real-time update of the virtual environment.

8. The vehicle detection path planning and interaction method based on the metaverse as claimed in claim 6, characterized in that: The specific steps of step S01 are: using terrain analysis software to calculate the curvature and slope characteristic parameters of the terrain, and setting a complexity threshold based on a large number of experiments and simulation results. When the characteristic parameters are higher than the threshold, it is determined to be a complex terrain area. At this time, a refined grid division method is used to increase the number of vertices and polygons when building the model, and increase the grid density to improve the details; when the characteristic parameters are lower than the threshold, a simplified grid division is used to reduce the data volume of the model, reduce the grid density to reduce the calculation load, and thus generate a high-precision three-dimensional terrain model; The image registration algorithm based on feature points in step S03 includes the following steps: (1) Using the scale-invariant feature transformation algorithm to extract feature points from the real-time image collected by the detector and the virtual image corresponding to the 3D model; (2) Using a bidirectional feature matching method, find the feature point pairs that match each other in the two images in step (1). In order to select more accurate matching points, the feature points are filtered based on information such as the distance and direction between them to remove obvious wrong matches. (3) Using the feature points accurately matched in step (2), the transformation parameters for fusing the real-time image collected by the detector with the three-dimensional model are obtained through homography matrix calculation, thereby achieving accurate fusion of real-time data and the three-dimensional model and ensuring that the virtual environment can be quickly updated according to the real-time data of the detector.

9. The vehicle detection path planning and interaction system based on Metaverse is characterized by: The method according to any one of claims 1 to 8, comprising: The collaborative interaction module implements voice, video and data transmission based on the WebRTC protocol, and uses the support vector machine algorithm to classify user gestures and map the classification results into the steps of vehicle movement, data collection or path modification operation instructions in the metaverse environment; The path planning and optimization module uses machine learning methods to extract geological data features, combines terrain detection targets, generates vehicle terrain detection path planning, and projects it to the Metaverse environment in real time. Users can share and visualize data in the Metaverse environment through the collaborative interaction module and review and modify the path planning; Feedback execution module: after the user confirms that the path planning is correct, the final optimized path planning and related operation instructions will be fed back to the vehicle to guide it to perform the detection task; It also includes a metaverse construction module, which includes a data acquisition module, a data processing module and an environmental simulation module. The data acquisition module acquires surface geographic data, and the data processing module preprocesses and performs model training based on the geographic data acquired by the data acquisition module; the environmental simulation module collects data through detectors to perform real-scene modeling, generates a three-dimensional terrain model, and uses AR technology to merge the real-time data of the detector with the real-scene model to create a metaverse environment.

10. The application of vehicle detection path planning and interaction method based on Metaverse in Mars rover detection is characterized by: The method described in any one of claims 1 to 8 is adopted to realize the exploration mission of the Mars rover.

Images