Virtual riding exercise system based on VR technology

Abstract

The application discloses a virtual riding exercise system based on VR technology and relates to the technical field of virtual reality.The system comprises a riding assembly, a VR head-mounted device, a motion control card, a four-degree-of-freedom motion platform, a stepless fan and a visual information processing device, wherein the riding assembly is located on the four-degree-of-freedom motion platform, the stepless fan is located in front of the riding assembly and faces the rider, and through the communication connection relationship and the function design of the riding assembly, the stepless fan and the visual information processing device, the system can not only provide a riding audio-visual experience through the VR head-mounted device, but also can realize the purpose that the posture of the riding vehicle is adaptively adjusted along with the change of the virtual scene road condition through the motion control card and the four-degree-of-freedom motion platform, and the stepless fan can provide the rider with the wind feeling caused by the riding, so that the riding experience can be greatly enriched, the riding experience immersion can be effectively improved, and the system is convenient for actual application and popularization.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality (VR) technology, specifically relating to a virtual cycling fitness system based on VR technology. Background Technology

[0002] Cycling is a healthy and natural form of exercise and tourism, allowing you to fully enjoy the beauty of the journey. All you need is a bicycle and a backpack—simple, environmentally friendly, and challenging. Cyclists can experience the thrill of overcoming obstacles and the satisfaction of reaching their destination. With the continuous development of technology, a new fitness method—virtual cycling—is gaining increasing attention. Virtual cycling uses virtual reality technology to simulate real-life cycling scenarios, allowing people to enjoy the pleasures of outdoor cycling indoors without worrying about external factors such as weather and traffic.

[0003] Currently, existing virtual cycling systems mainly consist of VR headsets and stationary bike frames. The cycling experience perceived by cyclists is primarily the visual and auditory experience provided by the VR headset. However, because the bike frame's posture does not adapt to changes in the virtual road conditions (for example, the bike frame should have a certain tilt in a virtual curve), the immersive experience is limited and needs further improvement. Summary of the Invention

[0004] The purpose of this invention is to provide a virtual cycling fitness system based on VR technology to solve the problem of limited immersion in the cycling experience of existing virtual cycling systems.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, a virtual cycling fitness system based on VR technology is provided, including a cycling component, a VR headset, a motion control card, a four-degree-of-freedom motion platform, a stepless fan, and a visual information processing device. The cycling component is located on the four-degree-of-freedom motion platform and includes a bicycle frame and pedal components, a front fork component, and a braking component mounted on the bicycle frame.

[0007] The cycling component is communicatively connected to the visual information processing device, and is used to generate cycling signals in response to the cyclist's operation, and transmit the cycling signals to the visual information processing device;

[0008] The visual information processing device is communicatively connected to the VR headset, the motion control card, and the stepless fan. It is used to determine the virtual image of the cyclist's vehicle, the vehicle's motion posture information, and the vehicle's speed information in the real-world 3D scene based on the cycling signal and a pre-constructed real-world 3D scene for the cycling area. The device then transmits the virtual image to the VR headset, the vehicle's motion posture information to the motion control card, and the vehicle's speed information to the stepless fan.

[0009] The VR headset is used to output and display the cycling simulation virtual image to the cyclist;

[0010] The motion control card is communicatively connected to the four-degree-of-freedom motion platform. It is used to calculate the motor pulse quantity based on the vehicle's motion posture information, and then control the four-degree-of-freedom motion platform and the riding component to perform riding simulation motion together based on the motor pulse quantity.

[0011] The stepless fan is located directly in front of the riding assembly and facing the rider, and is used to provide the rider with wind speed that is positively correlated with the vehicle speed, based on the vehicle's speed information.

[0012] Based on the above-mentioned invention, a novel virtual cycling solution is provided to enrich the cycling experience. This solution includes a cycling component, a VR headset, a motion control card, a four-degree-of-freedom (4DOF) motion platform, a stepless fan, and a visual information processing device. The cycling component is located on the 4DOF motion platform, and the stepless fan is positioned directly in front of the cycling component, facing the cyclist. Through their communication connection and functional design, in addition to providing a cycling audiovisual experience through the VR headset, the motion control card and 4DOF motion platform enable the cycling vehicle's posture to adaptively adjust to changes in virtual road conditions. Furthermore, the stepless fan provides the cyclist with a sense of wind due to cycling, thus greatly enriching the cycling experience, effectively enhancing immersion, and facilitating practical application and promotion.

[0013] In one possible design, the riding assembly also includes a wheel assembly, a freewheel assembly, and a chain assembly mounted on the riding frame, wherein the wheel assembly is suspended in the air.

[0014] In one possible design, the VR headset is an all-in-one head-mounted device that integrates VR glasses and a safety helmet.

[0015] In one possible design, the continuously variable fan is also located on the four-degree-of-freedom motion platform so that the posture of the continuously variable fan is kept consistent with that of the riding component in real time.

[0016] In one possible design, a spray device with adjustable spray direction is also included, wherein the spray device is located in the upper front area of ​​the riding assembly and is positioned facing the rider.

[0017] The visual information processing device is also communicatively connected to the sprinkler device, and is also used to transmit the vehicle speed information and the rainfall level information in the virtual rainy environment to the sprinkler device when the real-world three-dimensional scene is in a virtual rainy environment.

[0018] The spraying device is used to adjust the spraying direction according to the vehicle speed information and the spraying water volume according to the rainfall level information, wherein the angle between the spraying direction and the vertical direction is positively correlated with the vehicle speed and the spraying water volume is positively correlated with the rainfall level.

[0019] In one possible design, a temperature control device is also included, wherein the temperature control device is located around the riding assembly;

[0020] The visual information processing device is also communicatively connected to the temperature control device, and is also used to transmit temperature information in the virtual non-room temperature environment to the temperature control device when the real-world three-dimensional scene is in a virtual non-room temperature environment.

[0021] The temperature control device is used to adjust the actual ambient temperature based on the temperature information.

[0022] In one possible design, the real-world 3D scene is pre-constructed using the following steps:

[0023] Obtain oblique photography data collected by the UAV oblique photography equipment on the actual cycling area, as well as UAV attitude measurement data or image control measurement data recorded synchronously with the oblique photography data;

[0024] Based on the oblique photography data and the UAV attitude measurement data or the image control measurement data, a real-world 3D scene of the cycling area is constructed using UAV oblique photography real-world 3D modeling software. The real-world 3D scene contains 3D models of multiple objects on site.

[0025] In one possible design, based on the oblique photogrammetry data and the UAV attitude measurement data or the image control measurement data, a real-world 3D scene of the cycling area is constructed using UAV oblique photogrammetry real-world 3D modeling software, including:

[0026] Based on the oblique photography data and the UAV attitude measurement data or the image control measurement data, an initial real-world 3D scene of the cycling area is constructed using UAV oblique photography real-world 3D modeling software. The initial real-world 3D scene contains initial 3D models of multiple objects on site.

[0027] For each of the multiple field objects, an initial two-dimensional image of the corresponding model surface is obtained based on the corresponding initial three-dimensional model.

[0028] The initial two-dimensional images of the model surfaces of each of the on-site objects are processed to identify the modeling void regions, thereby obtaining the modeling void region identification results for each of the on-site objects.

[0029] For each of the on-site objects, if the corresponding modeling void region identification result indicates that there is at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the initial two-dimensional image of the model surface is processed by the generative adversarial neural network (GAN) to obtain the corresponding complete two-dimensional image of the model surface; otherwise, the initial two-dimensional image of the model surface is directly used as the corresponding complete two-dimensional image of the model surface.

[0030] For each of the on-site objects, the complete two-dimensional image of the corresponding model surface is rendered onto the surface of the corresponding initial three-dimensional model to obtain the corresponding final three-dimensional model.

[0031] In the initial real-world 3D scene, the initial 3D models of each on-site object are updated to the corresponding final 3D models to obtain the final real-world 3D scene of the cycling area.

[0032] In one possible design, for a given field object among the plurality of field objects, if the corresponding modeling void region identification result indicates the existence of at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then image inpainting processing is performed on the initial two-dimensional image of the model surface based on a generative adversarial neural network (GAN) to obtain a complete two-dimensional image of the corresponding model surface, including:

[0033] For a certain on-site object among the plurality of on-site objects, if the corresponding modeling void region identification result indicates that there is at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the at least one modeling void region is arranged in order of increasing region area to obtain a modeling void region sequence.

[0034] For the kth modeling void region in the modeling void region sequence, the model surface repair two-dimensional image corresponding to the (k-1)th modeling void region in the modeling void region sequence is processed by the generative adversarial neural network (GAN) to obtain the corresponding model surface repair two-dimensional image, where k represents a positive integer, and the initial two-dimensional image of the model surface of a certain on-site object is used as the model surface repair two-dimensional image corresponding to the zeroth modeling void region.

[0035] The two-dimensional image of the model surface repair corresponding to the last modeled void region in the modeled void region sequence is used as the complete final two-dimensional image of the model of the certain on-site object.

[0036] In one possible design, the initial two-dimensional images of the model surfaces of each of the field objects are processed to identify modeling void regions, resulting in the identification results of modeling void regions for each field object, including:

[0037] For a specific object among the multiple on-site objects, the initial two-dimensional image of the corresponding model surface is imported into a pre-trained modeling cavity region recognition model based on the YOLO object detection algorithm, and the corresponding modeling cavity region recognition result is output.

[0038] If the identification result of the modeling void region of a certain on-site object indicates that there is at least one modeling void region marker box in the initial two-dimensional image of the model surface of the certain on-site object, then according to the at least one modeling void region marker box, at least one modeling void region image corresponding to the at least one modeling void region marker box is extracted from the initial two-dimensional image of the model surface of the certain on-site object.

[0039] The at least one modeled hole region image is subjected to image denoising processing, grayscale conversion processing, and binarization processing based on a preset grayscale threshold in sequence to obtain at least one binarized image corresponding to the at least one modeled hole region image, wherein the preset grayscale threshold is preset according to the grayscale value of the modeled hole region.

[0040] For each binarized image in the at least one binarized image, the corresponding central connected component is extracted based on the Canny algorithm, and the central connected component is used as the modeling hole region within the corresponding modeling hole region marker box;

[0041] By summarizing all the modeled void regions, the final modeled void region identification result for a specific on-site object is obtained.

[0042] The beneficial effects of the above scheme are:

[0043] (1) This invention creatively provides a novel virtual cycling solution that can enrich the cycling experience, which includes a cycling component, a VR headset, a motion control card, a four-degree-of-freedom motion platform, a stepless fan, and a visual information processing device. The cycling component is located on the four-degree-of-freedom motion platform, and the stepless fan is located in front of the cycling component and facing the cyclist. Through their communication connection and functional design, in addition to providing a cycling audio-visual experience through the VR headset, the motion control card and the four-degree-of-freedom motion platform can also achieve the purpose of adaptively adjusting the posture of the cycling vehicle according to the changes in the virtual scene road conditions, and the stepless fan can provide the cyclist with the feeling of wind brought about by cycling, thereby greatly enriching the cycling experience, effectively improving the immersion of the cycling experience, and facilitating practical application and promotion.

[0044] (2) By configuring a spray device, cyclists can be provided with the feeling of being greeted by rain in a virtual rainy environment. By configuring a temperature control device, cyclists can also be provided with the cycling experience in a virtual high temperature / low temperature environment, making the cycling experience perceived by cyclists richer and further enhancing the immersiveness of the cycling experience.

[0045] (3) It can also reprocess the preliminary results of real-scene 3D modeling based on UAV oblique photography to repair the hollow areas of the 3D model, thereby obtaining a cloned real-scene 3D scene of the target site area, ensuring the audio-visual experience of cyclists. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of the structure of a VR-based virtual cycling fitness system provided in an embodiment of this application. Detailed Implementation

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0049] It should be understood that although the terms "first" and "second", etc., may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object may be referred to as the second object, and similarly, the second object may be referred to as the first object, without departing from the scope of the exemplary embodiments of the invention.

[0050] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Another example is A, B and / or C, which can mean that any one of A, B, and C or any combination thereof exists. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone or A and B exist simultaneously. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0051] Example:

[0052] like Figure 1 As shown, the virtual cycling fitness system based on VR technology provided in the first aspect of this embodiment includes, but is not limited to, a cycling component, a VR headset, a motion control card, a four-degree-of-freedom motion platform, a stepless fan, and a visual information processing device. The cycling component is located on the four-degree-of-freedom motion platform and includes a bicycle frame and pedal components, a fork component, and a braking component mounted on the bicycle frame.

[0053] The riding component is communicatively connected to the visual information processing device, used to generate riding signals in response to the rider's operations, and transmit the riding signals to the visual information processing device. The aforementioned riding signals specifically include, but are not limited to, riding acceleration signals generated by operating the pedal components, riding guidance signals generated by operating the front fork components, and riding deceleration signals generated by operating the braking components. These signals can be conventionally acquired using existing sensors configured on the corresponding components, for example, by measuring the rotational speed of the pedal components in real time using a speed sensor to obtain the riding acceleration signal. The bicycle frame, pedal components, front fork components, and braking components are all standard configurations on existing bicycles. Furthermore, to improve the rider's seamless riding experience with the riding component, the riding component also includes, but is not limited to, wheel components, freewheel components, and chain components mounted on the bicycle frame, wherein the wheel components are suspended in the air to allow the vehicle to remain stationary.

[0054] The visual information processing device is communicatively connected to the VR headset, the motion control card, and the stepless fan. It is used to determine, based on the cycling signal and a pre-constructed real-world 3D scene of the cycling area, a virtual image of the cyclist's vehicle, as well as the vehicle's motion posture and speed information within that real-world 3D scene. The virtual image is then transmitted to the VR headset, the vehicle's motion posture information to the motion control card, and the vehicle's speed information to the stepless fan. The specific technical details of determining the virtual image of the cyclist's vehicle and its motion posture and speed information based on the cycling signal and the real-world 3D scene can be implemented using relevant techniques found in existing driving training simulator systems.

[0055] The VR headset is used to output and display the virtual cycling simulation image to the cyclist. The aforementioned VR headset needs to be worn by the cyclist. To maintain the cyclist's safety awareness, preferably, the VR headset is an integrated head-mounted device combining VR glasses and a safety helmet.

[0056] The motion control card, communicatively connected to the four-degree-of-freedom motion platform, is used to calculate motor pulse quantities based on the vehicle's motion posture information, and then control the four-degree-of-freedom motion platform and the riding component to perform simulated riding motion based on the motor pulse quantities. The specific technical details of calculating motor pulse quantities based on the vehicle's motion posture information and then controlling the four-degree-of-freedom motion platform to perform simulated riding motion based on the motor pulse quantities can also be implemented with reference to relevant technical means in existing driving training simulator systems. Since the riding component is located on the four-degree-of-freedom motion platform, the posture of the riding component in degrees of freedom such as up / down, left / right, forward / backward, and pitch (rotation along the x-axis) can be adjusted in real time through the four-degree-of-freedom motion platform (for example, in a virtual right turn, the entire riding component can be tilted to the right at a certain angle to match the real riding experience), achieving the purpose of adaptively adjusting the posture of the riding vehicle according to changes in the virtual scene road conditions.

[0057] The continuously variable fan, located directly in front of the riding assembly and facing the cyclist, provides wind speed directly proportional to the vehicle's speed, based on the vehicle's speed information. This fan provides the cyclist with a headwind, enhancing their riding experience beyond just visual and auditory perception, thus enriching the overall experience and increasing immersion. Furthermore, to ensure the wind from the continuously variable fan is always a headwind, it is preferably also located on the four-degree-of-freedom motion platform, ensuring its orientation remains consistent with the riding assembly in real time.

[0058] Therefore, through the detailed structural description of the aforementioned virtual cycling fitness system, a novel virtual cycling solution that enriches the cycling experience is provided. This solution includes a cycling component, a VR headset, a motion control card, a four-degree-of-freedom (4DOF) motion platform, a stepless fan, and a visual information processing device. The cycling component is located on the 4DOF motion platform, and the stepless fan is positioned directly in front of the cycling component, facing the cyclist. Through their communication connection and functional design, in addition to providing a cycling audiovisual experience through the VR headset, the motion control card and 4DOF motion platform enable the cycling vehicle's posture to adaptively adjust to changes in virtual road conditions. Furthermore, the stepless fan provides the cyclist with a sense of wind due to cycling, thus greatly enriching the cycling experience, effectively enhancing immersion, and facilitating practical application and promotion.

[0059] Based on the aforementioned first aspect of the technical solution, this embodiment further provides a possible design for enhancing the immersive riding experience. Specifically, the virtual cycling fitness system includes a spray device with adjustable spray direction. This spray device is located in the upper front area of ​​the cycling component and faces the cyclist. The visual information processing device is also communicatively connected to the spray device and is used to transmit vehicle speed information and rainfall level information in a virtual rainy environment to the spray device when the real-world 3D scene is in such an environment. The spray device adjusts the spray direction according to the vehicle speed information and the spray volume according to the rainfall level information. The angle between the spray direction and the vertical direction is positively correlated with the vehicle speed, and the spray volume is positively correlated with the rainfall level. This spray device can provide cyclists with a sense of being greeted by rain in a virtual rainy environment, enriching the cyclist's perceived riding experience and further enhancing the immersive riding experience.

[0060] Based on the aforementioned possible design one, a sprinkler system can also be configured to provide cyclists with a sense of being greeted by rain in a virtual rainy environment, enriching the cyclist's perceived cycling experience and further enhancing the immersive feeling of the cycling experience. Furthermore, to provide cyclists with a cycling experience in virtual high / low temperature environments, preferably, the virtual cycling fitness system also includes a temperature control device, located around the cycling components; the visual information processing device is also communicatively connected to the temperature control device, and is also used to transmit temperature information of the virtual non-room temperature environment to the temperature control device when the real-world 3D scene is in a virtual non-room temperature environment; the temperature control device is used to adjust the real-world temperature according to the temperature information.

[0061] Based on the technical solution of the first aspect mentioned above, this embodiment also provides a possible design two for how to obtain a real-world three-dimensional scene in advance, that is, the real-world three-dimensional scene is constructed in advance using the following steps S1 to S2.

[0062] S1. Acquire oblique photography data collected by the UAV oblique photography equipment on the actual cycling area, as well as UAV attitude measurement data or image control measurement data recorded synchronously with the oblique photography data.

[0063] In step S1, specifically, the UAV oblique photography device preferably adopts a typical five-lens oblique gimbal. Since in its working state, the optical axis of the central camera is perpendicular to the horizontal plane, and four cameras are distributed in the four directions with their optical axes at a 45° angle to the horizontal plane, it can simultaneously complete the coverage of three or more images of the same ground object or feature point from different angles during a single flight of the UAV. (At the same time, since the higher the coverage and overlap of the images of the same ground object from different angles, the more refined the calculated model, the overlap of the images during flight will be increased as much as possible when acquiring real-scene 3D modeling data; however, considering that a higher overlap means an additional workload, considering efficiency and the tilt of the aircraft during flight, the flight path is generally set to have a forward overlap of more than 80% and a lateral overlap of more than 60%).

[0064] S2. Based on the oblique photography data and the UAV attitude measurement data or the image control measurement data, a real-world 3D scene of the cycling area is constructed using UAV oblique photography real-world 3D modeling software, wherein the real-world 3D scene contains 3D models of multiple objects on site.

[0065] In step S2, specifically, the UAV oblique photogrammetry real-scene 3D modeling software preferably uses ContextCapture software. While UAV oblique photogrammetry-based real-scene 3D modeling technology allows observation of the same ground feature from multiple angles, resulting in richer textures and more realistic effects, and is the mainstream direction for future 3D city modeling, it can also cause hollow areas in the automatically generated 3D model due to factors such as aerial photography blind spots and feature point matching errors. Therefore, to ensure the cyclist's audiovisual experience, it is preferable to construct a real-scene 3D scene of the cycling area using UAV oblique photogrammetry real-scene 3D modeling software based on the oblique photogrammetry data and the UAV attitude measurement data or the image control measurement data, including but not limited to the following steps S21 to S26.

[0066] S21. Based on the oblique photography data and the UAV attitude measurement data or the image control measurement data, an initial real-world 3D scene of the cycling area is constructed using UAV oblique photography real-world 3D modeling software, wherein the initial real-world 3D scene contains initial 3D models of multiple on-site objects.

[0067] S22. For each of the multiple field objects, based on the corresponding initial three-dimensional model, extract the corresponding initial two-dimensional image of the model surface.

[0068] S23. Perform modeling void region identification processing on the initial two-dimensional images of the model surfaces of each of the field objects to obtain the modeling void region identification results of each of the field objects.

[0069] In step S23, the specific steps include, but are not limited to: for each of the on-site objects, importing the corresponding initial two-dimensional image of the model surface into a pre-trained modeling hole region recognition model based on the YOLO object detection algorithm, and outputting the corresponding modeling hole region recognition result. The YOLO (You Only Look Once) object detection algorithm is an existing artificial intelligence recognition algorithm used to identify and mark the positions of objects in an image. The specific model structure of its YOLO V4 version consists of three parts: a backbone network, a neck network, and a head network. The backbone network can use a CSPDarknet53 (CSP stands for Cross Stage Partial) network for feature extraction. The neck network consists of an SPP (Spatial Pyramid Pooling block) and a PANet (Path Aggregation Network). The former is used to increase the receptive field and separate the most important features, while the latter ensures that semantic features are received from higher-level layers and fine-grained features are received from lower-level layers of the horizontal backbone network simultaneously. The head network described above performs detection based on anchor boxes and detects features of three different sizes (13×13, 26×26, and 52×52), which are used to detect targets from largest to smallest (here, larger feature maps contain more information; therefore, the 52×52 feature map is used to detect small targets, and vice versa). The aforementioned modeling hole region recognition model can be trained using conventional sample training methods so that, after inputting a test image, it can output the recognition results of whether or not modeling hole regions exist, as well as their confidence prediction values.

[0070] In step S23, in order to accurately determine the modeling void region in the initial two-dimensional image of the model surface, preferably, the initial two-dimensional image of the model surface of each on-site object is subjected to modeling void region identification processing to obtain the modeling void region identification result of each on-site object, including but not limited to the following steps S231 to S235.

[0071] S231. For a certain on-site object among the multiple on-site objects, import the corresponding initial two-dimensional image of the model surface into the pre-trained modeling cavity region recognition model based on the YOLO object detection algorithm, and output the corresponding modeling cavity region recognition result.

[0072] S232. If the identification result of the modeling void region of a certain on-site object indicates that there is at least one modeling void region marker box in the initial two-dimensional image of the model surface of the certain on-site object, then according to the at least one modeling void region marker box, at least one modeling void region image corresponding to the at least one modeling void region marker box is extracted from the initial two-dimensional image of the model surface of the certain on-site object.

[0073] S233. The at least one modeled hole region image is subjected to image denoising processing, grayscale conversion processing and binarization processing based on a preset grayscale threshold in sequence to obtain at least one binarized image corresponding to the at least one modeled hole region image, wherein the preset grayscale threshold is preset according to the grayscale value of the modeled hole region.

[0074] S234. For each binarized image in the at least one binarized image, the corresponding central connected component is extracted based on the Canny algorithm, and the central connected component is used as the modeling hole region within the corresponding modeling hole region marker box.

[0075] S235. Summarize all the modeled void regions to obtain the final modeled void region identification result for a certain on-site object.

[0076] S24. For each of the field objects, if the corresponding modeling void region identification result indicates that there is at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the initial two-dimensional image of the model surface is processed by the generative adversarial neural network (GAN) to obtain the corresponding complete two-dimensional image of the model surface; otherwise, the initial two-dimensional image of the model surface is directly used as the corresponding complete two-dimensional image of the model surface.

[0077] In step S24, the Generative Adversarial Network (GAN) is a novel framework for estimating generative models: two models are trained simultaneously, one for capturing the data distribution and the other for discriminating whether the data is real or generated (pseudo-data). Image processing tasks utilize neural networks such as CNNs to analyze and process input images to obtain information related to the content of the input images. In contrast to image processing tasks, in image generation tasks, the image generation model generates images based on the input information related to the content of the input images. For image generation tasks, the input to the image generation model is uncertain, depending on the scene and the specific model design, with style transfer being one such scenario. Generative Adversarial Networks (GANs) can be applied to style transfer. The generative adversarial neural network (GAN) has two core components: a generator (the aforementioned generative model) and a discriminator (the aforementioned discriminative model). Both the discriminator and the generator can be constructed using a multilayer perceptron (which can be viewed as a fully connected neural network, i.e., FC). During training, the key steps embodying the "adversarial" nature are: first, fixing the generator's parameters and training and optimizing the discriminator so that it can distinguish between "real images" and "fake images" as accurately as possible; then, fixing the discriminator's parameters and training and optimizing the generator so that it cannot accurately distinguish between "real images" and "fake images." After the model training is complete, the trained generator can be used to generate images. Furthermore, the discriminator can also use a convolutional neural network (Strided Convolution, a standard convolution operation; without padding, the number of channels will decrease), and the generator can also be implemented using transposed convolution (which can also be viewed as deconvolution).

[0078] In step S24, in order to complete the image restoration process quickly and effectively, the following restoration scheme of starting with the easier ones and then moving to the more difficult ones is preferred: for a certain object among the multiple objects in the field, if the corresponding modeling hole region identification result indicates that there is at least one modeling hole region in the initial two-dimensional image of the corresponding model surface, then the initial two-dimensional image of the model surface is restored based on the generative adversarial neural network (GAN) to obtain the corresponding complete two-dimensional image of the model surface, including but not limited to the following steps S241 to S243.

[0079] S241. For a certain field object among the plurality of field objects, if the corresponding modeling void region identification result indicates that there is at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the at least one modeling void region is arranged in ascending order of region area to obtain a modeling void region sequence.

[0080] S242. For the k-th modeling void region in the modeling void region sequence, perform image inpainting processing on the model surface repair two-dimensional image corresponding to the (k-1)-th modeling void region in the modeling void region sequence based on the generative adversarial neural network (GAN) to obtain the corresponding model surface repair two-dimensional image, where k represents a positive integer, and the initial two-dimensional image of the model surface of a certain on-site object is used as the model surface repair two-dimensional image corresponding to the zeroth modeling void region.

[0081] Step S242 includes, but is not limited to, the following steps S2421 to S2425.

[0082] S2421. The image generator applied to the pre-trained complete image generation model based on the generative adversarial neural network (GAN) generates a new image, and then step S2422 is performed.

[0083] In step S2421, the detailed training process of the complete image generation model includes, but is not limited to: first, acquiring multiple real two-dimensional images of object surfaces; then, using the multiple two-dimensional images of object surfaces to train a generative adversarial neural network (GAN) including an image generator and an image discriminator, to obtain the complete image generation model.

[0084] S2422. The image discriminator applied in the complete image generation model determines whether the new image is a complete image. If it is, then step S2423 is executed; otherwise, the image generator is applied again to generate a new image, and then step S2422 is executed.

[0085] S2423. Based on the new image and the two-dimensional image of the model surface repair corresponding to the (k-1)th modeling hole region, calculate the color difference of each pixel in the non-modeling hole region of the two images, and then execute step S2424, where k represents a positive integer, and the initial two-dimensional image of the model surface of a certain on-site object is used as the two-dimensional image of the model surface repair corresponding to the zeroth modeling hole region.

[0086] S2424. Determine whether the standard deviation of the color difference between the two images at each pixel point reaches a preset standard deviation threshold. If yes, use the new image as the two-dimensional image for model surface repair corresponding to the k-th modeling hole region in the modeling hole region sequence. Otherwise, proceed to step S2425.

[0087] S2425. The color difference between the two images at each pixel is imported into the image generator as content loss penalty data, and the image generator is applied again to generate a new image, and then step S2422 is executed.

[0088] S243. The model surface repair two-dimensional image corresponding to the last modeling void region in the modeling void region sequence is taken as the final two-dimensional image of the model surface of the certain on-site object.

[0089] S25. For each of the on-site objects, render the complete two-dimensional image of the corresponding model surface onto the surface of the corresponding initial three-dimensional model to obtain the corresponding final three-dimensional model.

[0090] S26. In the initial real-world 3D scene, update the initial 3D model of each on-site object to the corresponding final 3D model to obtain the final real-world 3D scene of the cycling area.

[0091] Based on the aforementioned second possible design, it is also possible to further process the preliminary results of the real-scene 3D modeling based on UAV oblique photography to repair the hollow areas of the 3D model, thereby obtaining a cloned real-scene 3D scene of the target site area and ensuring the audiovisual experience of cyclists.

[0092] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual cycling fitness system based on VR technology, characterized in that, The device includes a cycling component, a VR headset, a motion control card, a four-degree-of-freedom motion platform, a continuously variable fan, and a visual information processing device. The cycling component is located on the four-degree-of-freedom motion platform and includes a bicycle frame and pedal components, a front fork component, and a braking component mounted on the bicycle frame. The continuously variable fan is also located on the four-degree-of-freedom motion platform so that the posture of the continuously variable fan is kept consistent with that of the cycling component in real time. The cycling component is communicatively connected to the visual information processing device, and is used to generate cycling signals in response to the cyclist's operation, and transmit the cycling signals to the visual information processing device; The visual information processing device is communicatively connected to the VR headset, the motion control card, and the stepless fan. It is used to determine the virtual image of the cyclist's vehicle, the vehicle's motion posture information, and the vehicle's speed information in the real-world 3D scene based on the cycling signal and a pre-constructed real-world 3D scene for the cycling area. The device then transmits the virtual image to the VR headset, the vehicle's motion posture information to the motion control card, and the vehicle's speed information to the stepless fan. The VR headset is used to output and display the cycling simulation virtual image to the cyclist; The motion control card is communicatively connected to the four-degree-of-freedom motion platform. It is used to calculate the motor pulse quantity based on the vehicle's motion posture information, and then control the four-degree-of-freedom motion platform and the riding component to perform riding simulation motion based on the motor pulse quantity. The stepless fan is located directly in front of the riding assembly and facing the rider, and is used to provide the rider with wind speed that is positively correlated with the vehicle speed based on the vehicle's speed information. The real-world 3D scene is pre-constructed using the following steps: acquiring oblique photography data collected by a drone oblique photography device from the cycling area, and drone attitude measurement data or image control measurement data recorded synchronously with the oblique photography data; based on the oblique photography data and the drone attitude measurement data or image control measurement data, constructing an initial real-world 3D scene of the cycling area using drone oblique photography real-world 3D modeling software, wherein the initial real-world 3D scene includes initial 3D models of multiple on-site objects; for each on-site object, according to the corresponding initial 3D model, extracting the corresponding initial 2D image of the model surface; performing modeling void region identification processing on the initial 2D images of the model surfaces of each on-site object to obtain the... The modeling void region identification results for each on-site object; for each on-site object, if the corresponding modeling void region identification result indicates that there is at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the initial two-dimensional image of the model surface is inpainted based on the generative adversarial neural network (GAN) to obtain the corresponding complete two-dimensional image of the model surface; otherwise, the corresponding initial two-dimensional image of the model surface is directly used as the corresponding complete two-dimensional image of the model surface; for each on-site object, the corresponding complete two-dimensional image of the model surface is rendered onto the surface of the corresponding initial three-dimensional model to obtain the corresponding final three-dimensional model; in the initial real-world three-dimensional scene, the initial three-dimensional model of each on-site object is updated to the corresponding final three-dimensional model to obtain the final real-world three-dimensional scene of the cycling area; For a given object among the plurality of on-site objects, if the corresponding modeling void region identification result indicates the existence of at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then image inpainting processing is performed on the initial two-dimensional image of the model surface based on a generative adversarial neural network (GAN) to obtain a complete two-dimensional image of the corresponding model surface. This includes: for a given object among the plurality of on-site objects, if the corresponding modeling void region identification result indicates the existence of at least one modeling void region in the initial two-dimensional image of the corresponding model surface, then the at least one modeling void region is arranged in ascending order of area to obtain a modeling void. Region sequence; For the k-th modeling void region in the modeling void region sequence, based on the generative adversarial neural network (GAN), image inpainting processing is performed on the 2D image of the model surface repair corresponding to the (k-1)-th modeling void region in the modeling void region sequence to obtain the corresponding 2D image of the model surface repair, where k represents a positive integer, and the initial 2D image of the model surface of a certain on-site object is used as the 2D image of the model surface repair corresponding to the zeroth modeling void region; the 2D image of the model surface repair corresponding to the last modeling void region in the modeling void region sequence is used as the complete final 2D image of the model of the certain on-site object. For the k-th modeling void region in the modeling void region sequence, image inpainting processing is performed on the two-dimensional image of the model surface repair corresponding to the (k-1)-th modeling void region in the modeling void region sequence based on the generative adversarial neural network (GAN), to obtain the corresponding two-dimensional image of the model surface repair, including the following steps S2421 to S2425: S2421. The image generator applied to the pre-trained complete image generation model based on the generative adversarial neural network (GAN) generates a new image, and then step S2422 is executed. S2422. The image discriminator applied in the complete image generation model determines whether the new image is a complete image. If it is, then step S2423 is executed; otherwise, the image generator is applied again to generate a new image, and then step S2422 is executed. S2423. Based on the new image and the two-dimensional image of the model surface repair corresponding to the (k-1)th modeled hole region, calculate the color difference of each pixel in the non-modeled hole region of the two images, and then execute step S2424. S2424. Determine whether the standard deviation of the color difference between the two images at each pixel point reaches the preset standard deviation threshold. If so, use the new image as the two-dimensional image for model surface repair corresponding to the kth modeling hole region in the modeling hole region sequence. Otherwise, proceed to step S2425. S2425. The color difference between the two images at each pixel is imported into the image generator as content loss penalty data, and the image generator is applied again to generate a new image, and then step S2422 is executed.

2. The virtual cycling fitness system as described in claim 1, characterized in that, The cycling assembly also includes a wheel assembly, a freewheel assembly, and a chain assembly mounted on the cycling frame, wherein the wheel assembly is suspended in the air.

3. The virtual cycling fitness system as described in claim 1, characterized in that, The VR headset is an integrated head-mounted device that combines VR glasses and a safety helmet.

4. The virtual cycling fitness system as described in claim 1, characterized in that, It also includes a spray device with adjustable spray direction, wherein the spray device is located in the upper front area of ​​the riding component and is positioned facing the rider; The visual information processing device is also communicatively connected to the sprinkler device, and is also used to transmit the vehicle speed information and the rainfall level information in the virtual rainy environment to the sprinkler device when the real-world three-dimensional scene is in a virtual rainy environment. The spraying device is used to adjust the spraying direction according to the vehicle speed information and the spraying water volume according to the rainfall level information, wherein the angle between the spraying direction and the vertical direction is positively correlated with the vehicle speed and the spraying water volume is positively correlated with the rainfall level.

5. The virtual cycling fitness system as described in claim 1, characterized in that, It also includes a temperature control device, wherein the temperature control device is located around the riding assembly; The visual information processing device is also communicatively connected to the temperature control device, and is also used to transmit temperature information in the virtual non-room temperature environment to the temperature control device when the real-world three-dimensional scene is in a virtual non-room temperature environment. The temperature control device is used to adjust the actual ambient temperature based on the temperature information.

6. The virtual cycling fitness system according to claim 1, characterized in that, The initial two-dimensional images of the model surfaces of each of the aforementioned on-site objects are processed to identify modeling void regions, resulting in the identification results for the modeling void regions of each on-site object, including: For a specific object among the multiple on-site objects, the initial two-dimensional image of the corresponding model surface is imported into a pre-trained modeling cavity region recognition model based on the YOLO object detection algorithm, and the corresponding modeling cavity region recognition result is output. If the identification result of the modeling void region of a certain on-site object indicates that there is at least one modeling void region marker box in the initial two-dimensional image of the model surface of the certain on-site object, then according to the at least one modeling void region marker box, at least one modeling void region image corresponding to the at least one modeling void region marker box is extracted from the initial two-dimensional image of the model surface of the certain on-site object. The at least one modeled hole region image is subjected to image denoising processing, grayscale conversion processing, and binarization processing based on a preset grayscale threshold in sequence to obtain at least one binarized image corresponding to the at least one modeled hole region image, wherein the preset grayscale threshold is preset according to the grayscale value of the modeled hole region. For each binarized image in the at least one binarized image, the corresponding central connected component is extracted based on the Canny algorithm, and the central connected component is used as the modeling hole region within the corresponding modeling hole region marker box; By summarizing all the modeled void regions, the final modeled void region identification result for a specific on-site object is obtained.

Images