Virtual scene interaction method and device, medium, equipment and product
By generating adversarial network processing real scene models, high-quality virtual scenes are generated, and combined with user interaction intentions, solutions for high-quality virtual scenes and contactless interaction in virtual reality technology are realized, improving the quality and nature of the virtual reality experience.
Patent Information
- Application Number
- CN202411073473.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-19
AI Technical Summary
Existing virtual reality technology is difficult to generate high-quality virtual scenes and cannot achieve contactless interaction between users and the virtual environment.
By acquiring a real scene model using a scanning device, a high-quality virtual scene is generated based on the generation adversarial network processing feature information, and an interaction intention is determined through the target image of the user's eyes facing the virtual scene, so as to achieve contactless interaction between the user and the virtual scene.
It improves the quality and reality of the virtual scene, realizes natural and convenient interaction between users and the virtual environment, and enhances the immersive experience.
Smart Images

Figure CN118981252A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a method, device, medium, equipment and product for interacting with a virtual scene. Background Art
[0002] Virtual reality technology uses computer simulation to create a three-dimensional virtual world, providing users with simulations of vision and other senses, making users feel as if they are in the real world and can observe things in the three-dimensional space instantly and without restrictions. Virtual reality technology includes both hardware and software. Its system is usually composed of input devices, output devices, virtual world databases, professional graphics processing computers, and software systems.
[0003] The core of virtual reality technology is the modeling and drawing of virtual environments, and generating realistic virtual scenes according to user needs. However, the quality of virtual scenes generated in related technologies is not high, and it is impossible to achieve contactless interaction between users and virtual environments. Summary of the invention
[0004] The purpose of the present disclosure is to provide a virtual scene interaction method, device, medium, equipment and product, which are used to improve the quality of the generated virtual scene and enable contactless interaction between the user and the virtual environment.
[0005] In order to achieve the above objectives, in a first aspect, the present disclosure provides a virtual scene interaction method, comprising: Scanning the real scene using a scanning device to obtain a real scene model; Extracting feature information from the real scene model, and processing the feature information based on a generator in a trained generative adversarial network to determine a virtual scene; Using a virtual reality device to display the virtual scene to a user, and obtaining a target image of the user's eyes facing the virtual scene; Based on the target image, the user's interaction intention is determined, and based on the interaction intention, an interaction operation is performed on the interaction object in the virtual scene.
[0006] Optionally, the generative adversarial network further includes a discriminator, and the trained generative adversarial network is obtained based on the following method: Obtaining a sample real scene model that matches the sample real scene; Processing the sample real scene model using the generator to determine a sample virtual scene; Using the discriminator to process the sample real scene model and the sample virtual scene to determine a discrimination result, wherein the discrimination result is used to characterize the authenticity of the sample virtual scene; Based on the discrimination result, the parameters of the generative adversarial network are adjusted until preset conditions are met to obtain the trained generative adversarial network.
[0007] Optionally, determining the user's interaction intention based on the target image includes: Processing the target image based on a trained eye tracking model to obtain the eye direction and gaze position of the user; Based on the eye direction and the gaze position, the interaction intention is confirmed.
[0008] Optionally, scanning the real scene using a scanning device to obtain a real scene model includes: Scanning the real scene using the scanning device to obtain point cloud data; The point cloud data is subjected to meshing processing and texture mapping processing to obtain the real scene model.
[0009] Optionally, before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: Processing the real scene model using image segmentation technology to obtain multiple semantic regions; According to the adjustment operation of the user, the plurality of semantic areas in the real scene model are adjusted.
[0010] Optionally, before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The image style of the real scene model is adjusted using image style transfer technology.
[0011] Optionally, before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The image dynamic synthesis model is used to perform the following operations: Extracting motion features of static objects in the real scene model; The motion features are processed based on preset dynamic object generation rules to obtain a dynamic object.
[0012] Optionally, before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The resolution of the real scene model is adjusted to the target resolution using image super-resolution technology.
[0013] Optionally, the image dynamic synthesis model includes a variational autoencoder and a recurrent neural network.
[0014] In a second aspect, the present disclosure provides an interactive device for a virtual scene, comprising: A scanning module is configured to scan a real scene using a scanning device to obtain a real scene model; A processing module is configured to extract feature information from the real scene model, and process the feature information based on the generator in the trained generative adversarial network to determine a virtual scene; an acquisition module, configured to display the virtual scene to the user using a virtual reality device, and acquire a target image of the user's eyes facing the virtual scene; The interaction module is configured to determine the user's interaction intention based on the target image, and perform an interaction operation on the interactive object in the virtual scene based on the interaction intention.
[0015] In a third aspect, the present disclosure provides a non-temporary computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.
[0016] In a fourth aspect, the present disclosure provides an electronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of any one of the methods in the first aspect.
[0017] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods in the first aspect.
[0018] Through the above technical scheme, the feature information is processed by using the generator in the trained generative adversarial network to determine the virtual scene. Since the generative adversarial network has a powerful ability of deep learning, it can extract high-level feature information from the real scene model to generate a high-quality virtual scene; and through the user's eyes facing the target image of the virtual scene, the user's interaction intention is determined, and based on the interaction intention, interactive operations are performed on the interactive objects in the virtual scene, which can realize contactless interaction between the user and the virtual scene and improve the convenience and naturalness of the interaction.
[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 The figure is a flow chart of a virtual scene interaction method according to an exemplary embodiment.
[0021] Figure 2 The figure is a flowchart of training a generative adversarial network according to an exemplary embodiment.
[0022] Figure 3 The present invention is a block diagram of a virtual scene interaction device according to an exemplary embodiment.
[0023] Figure 4 It is a block diagram of an electronic device according to an exemplary embodiment.
[0024] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0025] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.
[0026] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.
[0027] Figure 1 is a flowchart of a virtual scene interaction method according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps 110-140.
[0028] Step 110, use a scanning device to scan the real scene to obtain a real scene model.
[0029] In some embodiments, the scanning device may be a camera device, a camera device or a scanner in a terminal device. For example, the terminal device may be a mobile phone, a laptop computer or a pad, etc., and the camera device may be a camera. In a possible implementation, the scanning may include multi-angle scanning, which can make the modeling of the real scene model more complete and comprehensive, thereby improving the accuracy of the real scene model.
[0030] In some embodiments, a real scene is scanned using a scanning device to obtain a real scene model, including: scanning the real scene using a scanning device to obtain point cloud data; and gridding and texture mapping the point cloud data to obtain a real scene model.
[0031] The real scene model can be three-dimensional information. By meshing and texture mapping the point cloud data, the real scene model can be obtained. The point cloud data scanned by any device can be processed to obtain the model data. No professional equipment and technology are required, and the three-dimensional information of the real scene can be quickly obtained.
[0032] Step 120, extracting feature information from the real scene model, and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene.
[0033] In a possible implementation, the feature information may include texture, lighting, and shadow. In some embodiments, the trained generative adversarial network includes a generator and a discriminator, wherein the generator is used to generate a realistic virtual scene based on the feature information. In a possible implementation, the generator may use a variational autoencoder and a pixel recurrent network. The training process of the generative adversarial network can be seen below. Figure 2 And the related description will not be repeated here.
[0034] The disclosed embodiment uses the generator in the trained generative adversarial network to process feature information and determine the virtual scene. Since the generative adversarial network has a powerful ability of deep learning, it can extract high-level feature information from the real scene model to generate a high-quality virtual scene. In addition, the generator can generate realistic virtual scenes, improve the realism of the virtual scenes, and bring a more immersive virtual experience to users.
[0035] In some embodiments, before extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method also includes: using image segmentation technology to process the real scene model to obtain multiple semantic regions; adjusting the multiple semantic regions in the real scene model according to the user's adjustment operation.
[0036] In some embodiments, the multiple semantic areas may include the sky, the ground, buildings, and people, etc. The user's adjustment operation may be replacement, color adjustment, zooming in, and zooming out, etc. The user may adjust all or part of the multiple semantic areas.
[0037] The disclosed embodiment adjusts multiple semantic areas in the real scene model according to the user's adjustment operation, so that a virtual scene can be generated according to the adjusted real scene model in the future, and the generated virtual scene (for example, the layout of the virtual scene) is more in line with the user's preferences, thereby achieving the customization of the virtual scene and increasing the user's creativity and customization. In addition, by first adjusting the real scene model and then using the generative adversarial network to generate the virtual scene, the real scene model is already a model that meets the user's requirements before inputting the generative adversarial network, which can avoid the user returning to modify the real scene model when he is dissatisfied with the virtual scene, thereby improving the user's satisfaction with the generated virtual scene.
[0038] In some embodiments, before extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method also includes: adjusting the image style of the real scene model using image style transfer technology.
[0039] It is worth noting that the real scene, the real scene model and the virtual scene are all displayed to the user in the form of images.
[0040] In some embodiments, the image style transfer technology may include a trained image style transfer model, which can adjust the image style of the real scene model according to the preference information input by the user. The preference information is used to reflect the image style preferred by the user, such as oil painting style, watercolor style, and sketch style.
[0041] The disclosed embodiment uses image style transfer technology to adjust the image style of the real scene model, so that a virtual scene can be subsequently generated based on the adjusted real scene model. The image style of the generated virtual scene is more in line with the user's preferences, providing the user with more visual effects, increasing the user's aesthetic experience, and increasing the artistic expression and emotional expression of the virtual scene.
[0042] In addition, by first adjusting the image style of the real scene model and then using the generative adversarial network to generate the virtual scene, the image style of the real scene model is already a model that meets the user's requirements before entering the generative adversarial network. This can avoid the user returning to modify the real scene model when they are not satisfied with the image style of the virtual scene, thereby improving the user's satisfaction with the generated virtual scene.
[0043] In some embodiments, before extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the step of virtual scene, the method also includes: using the image dynamic synthesis model to perform the following operations: extracting motion features of static objects in the real scene model; processing the motion features based on preset dynamic object generation rules to obtain dynamic objects.
[0044] The image dynamic synthesis model can convert static objects in the real scene model into dynamic objects. The objects may include people and / or objects. By converting static objects into dynamic objects, the generated virtual scene is more vivid, and the dynamic and interactive sense of the virtual scene is improved, thereby increasing the user's sense of participation and immersion.
[0045] In a possible implementation, the image dynamic synthesis model includes a variational autoencoder and a recurrent neural network. Since the image dynamic synthesis model composed of the variational autoencoder and the recurrent neural network has a powerful ability of deep learning, it can extract high-level motion features from the real scene model, thereby generating a high-quality dynamic virtual scene.
[0046] In some embodiments, before extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method also includes: using image super-resolution technology to adjust the resolution of the real scene model to the target resolution.
[0047] The target resolution may include a high-definition resolution or a 4K resolution. By adjusting the resolution of the real scene model to the target resolution, a clearer virtual scene can be generated, and the image quality and clarity of the virtual scene can be improved, thereby improving the realism and detail of the virtual scene and reducing the user's eye fatigue and discomfort.
[0048] In a possible implementation, before extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the step of the virtual scene, one or more of the aforementioned image segmentation, image style transfer, dynamic object generation, and resolution adjustment operations may be performed on the real scene model.
[0049] Step 130, using a virtual reality device to display a virtual scene to the user, and obtaining a target image of the virtual scene facing the user's eyes.
[0050] In some embodiments, the virtual reality device may include a virtual reality development platform, which may be a virtual reality development platform based on a preset engine, which may be Unity, Unreal, or CryEngine.
[0051] In a possible implementation, obtaining a target image of the user's eyes facing the virtual scene may include: obtaining a target image of the user's eyes facing the virtual scene through a camera in a virtual reality helmet worn by the user. For example, an eye image of the user facing the virtual scene may be used as the target image.
[0052] In some embodiments, the method further includes: using a virtual reality device to display game elements and interactive logic to the user, where the game elements may include music, sound effects, animations, and special effects, and the interactive logic may include menus, buttons, prompts, and feedback, etc. In possible implementations, both the game elements and the interactive logic may be selected according to user needs and preferences.
[0053] The disclosed embodiments can increase the fun and attractiveness of virtual scenes by displaying game elements, can enhance the operability and editability of virtual scenes by displaying interaction logic, and can achieve a variety of interaction effects by facilitating adjustment of virtual scenes by selecting game elements and interaction logic according to user needs and preferences.
[0054] Step 140: determining the user's interaction intention based on the target image, and performing an interaction operation on the interactive object in the virtual scene based on the interaction intention.
[0055] The disclosed embodiments determine the user's interaction intention by facing the user's eyes towards a target image of a virtual scene, and perform interactive operations on interactive objects in the virtual scene based on the interaction intention, thereby enabling contactless interaction between the user and the virtual scene and improving the convenience and naturalness of the interaction.
[0056] In some embodiments, determining the user's interaction intention based on the target image includes: processing the target image based on a trained eye tracking model to obtain the user's eye direction and gaze position; and determining the interaction intention based on the eye direction and gaze position. In a possible implementation, the eye tracking model may be a classification model.
[0057] In a possible implementation, the gaze position can reflect the object selected by the user, that is, the interactive object. For example, when the user gazes at object A in the virtual scene for more than 3 seconds, the object A is determined as the interactive object, and the eye direction can reflect the interactive operation that the user wants to perform. The interactive intention can reflect the interactive operation that the user wants to perform on the selected interactive object.
[0058] In some embodiments, the interaction operation can be determined based on the mapping relationship between the eye gaze direction and the interaction operation. For example, if the user's eye gaze direction is from left to right, the interaction operation is determined to be moving from left to right based on the mapping relationship, and the interaction object, such as object A, can be moved to the right until it finally stops at the right side of the eye gaze direction. The disclosed embodiment automatically calculates the eye gaze direction and gaze position through the eye tracking model, which can improve the efficiency and accuracy of determining the interaction intention.
[0059] Figure 2 is a flowchart of training a generative adversarial network according to an exemplary embodiment. Figure 2 As shown, the process includes the following steps 210-240.
[0060] Step 210: Acquire a sample real scene model that matches the sample real scene.
[0061] Step 220: Process the sample real scene model using the generator to determine the sample virtual scene.
[0062] Step 230, using a discriminator to process the sample real scene model and the sample virtual scene to determine a discrimination result, which is used to characterize the authenticity of the sample virtual scene.
[0063] Step 240, adjusting the parameters of the generative adversarial network based on the discrimination result until the preset conditions are met to obtain a trained generative adversarial network.
[0064] In a possible implementation, the discrimination result may include 0 or 1, 0 is used to indicate that the sample virtual scene is unrealistic, and 1 is used to indicate that the sample virtual scene is real. The authenticity of the sample virtual scene is obtained by comparing it with the sample real scene. Thus, the generative adversarial network trained by the discrimination result can generate a virtual scene that is very realistic with the real scene.
[0065] The preset condition may be that the loss function value of the generative adversarial network satisfies the preset condition. The loss function of the generative adversarial network may be specifically set according to actual needs, and the present disclosure does not impose any restrictions on this.
[0066] In the training process of the generative adversarial network, the parameters of the initial generative adversarial network can be iteratively updated based on the sample real scene model. Specifically, the parameters of the initial generative adversarial network can be continuously adjusted to reduce the loss function value corresponding to each sample real scene model so that the loss function value meets the preset conditions, for example, the loss function value converges, or the loss function value is less than the preset value. When the loss function meets the preset conditions, the model training is completed and a trained generative adversarial network is obtained. The trained generative adversarial network can process the real scene model to obtain a realistic virtual scene.
[0067] Figure 3 is a block diagram of an interactive device for a virtual scene according to an exemplary embodiment. Figure 3 As shown, the device 300 includes: The scanning module 310 is configured to scan the real scene using a scanning device to obtain a real scene model; The processing module 320 is configured to extract feature information from the real scene model, and process the feature information based on the generator in the trained generative adversarial network to determine the virtual scene; The acquisition module 330 is configured to display the virtual scene to the user using a virtual reality device, and acquire a target image of the user's eyes facing the virtual scene; The interaction module 340 is configured to determine the user's interaction intention based on the target image, and perform an interaction operation on the interactive object in the virtual scene based on the interaction intention.
[0068] Optionally, the generative adversarial network further includes a discriminator, and the apparatus 300 further includes a training module configured to perform the following operations: Obtaining a sample real scene model that matches the sample real scene; Processing the sample real scene model using the generator to determine a sample virtual scene; Using the discriminator to process the sample real scene model and the sample virtual scene to determine a discrimination result, wherein the discrimination result is used to characterize the authenticity of the sample virtual scene; Based on the discrimination result, the parameters of the generative adversarial network are adjusted until preset conditions are met to obtain the trained generative adversarial network.
[0069] Optionally, the interaction module 340 is further configured to: Processing the target image based on a trained eye tracking model to obtain the eye direction and gaze position of the user; Based on the eye direction and the gaze position, the interaction intention is confirmed.
[0070] Optionally, the scanning module 310 is further configured to: Scanning the real scene using the scanning device to obtain point cloud data; The point cloud data is subjected to meshing processing and texture mapping processing to obtain the real scene model.
[0071] Optionally, the device 300 further includes: A segmentation module is configured to process the real scene model using image segmentation technology to obtain multiple semantic regions; The first adjustment module is configured to adjust the multiple semantic areas in the real scene model according to the user's adjustment operation.
[0072] Optionally, the device 300 further includes: The second adjustment module is configured to adjust the image style of the real scene model by using image style transfer technology.
[0073] Optionally, the device 300 further includes: The synthesis module is configured to perform the following operations using the image dynamic synthesis model: Extracting motion features of static objects in the real scene model; The motion features are processed based on preset dynamic object generation rules to obtain a dynamic object.
[0074] Optionally, the device 300 further includes: The third adjustment module is configured to adjust the resolution of the real scene model to the target resolution by using image super-resolution technology.
[0075] Optionally, the image dynamic synthesis model includes a variational autoencoder and a recurrent neural network.
[0076] The device of the disclosed embodiment processes feature information by utilizing a generator in a trained generative adversarial network to determine a virtual scene. Since the generative adversarial network has a powerful ability of deep learning, it can extract high-level feature information from the real scene model to generate a high-quality virtual scene. It also determines the user's interaction intention by facing a target image of the virtual scene with the user's eyes, and performs interactive operations on interactive objects in the virtual scene based on the interactive intention, thereby realizing contactless interaction between the user and the virtual scene and improving the convenience and naturalness of the interaction.
[0077] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0078] Figure 4 FIG. 4 is a block diagram of an electronic device 400 according to an exemplary embodiment. Figure 4 As shown, the electronic device 400 may include: a processor 401 , a memory 402 . The electronic device 400 may also include one or more of a multimedia component 403 , an input / output (I / O) interface 404 , and a communication component 405 .
[0079] The processor 401 is used to control the overall operation of the electronic device 400 to complete all or part of the steps in the above-mentioned virtual scene interaction method. The memory 402 is used to store various types of data to support the operation of the electronic device 400, and these data may include instructions for any application or method used to operate on the electronic device 400, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 402 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 403 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 402 or sent through the communication component 405. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 404 provides an interface between the processor 401 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 405 is used for wired or wireless communication between the electronic device 400 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 405 may include: Wi-Fi module, Bluetooth module, NFC module, etc.
[0080] In an exemplary embodiment, the electronic device 400 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned virtual scene interaction method.
[0081] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned virtual scene interaction method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 402 including program instructions, and the above-mentioned program instructions can be executed by the processor 401 of the electronic device 400 to complete the above-mentioned virtual scene interaction method.
[0082] Figure 5 is a block diagram of an electronic device 500 according to an exemplary embodiment. For example, the electronic device 500 may be provided as a server. Figure 5 The electronic device 500 includes a processor 522, which may be one or more, and a memory 532 for storing a computer program executable by the processor 522. The computer program stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processor 522 may be configured to execute the computer program to perform the above-mentioned virtual scene interaction method.
[0083] In addition, the electronic device 500 may further include a power supply component 526 and a communication component 550, wherein the power supply component 526 may be configured to perform power management of the electronic device 500, and the communication component 550 may be configured to implement communication of the electronic device 500, for example, wired or wireless communication. In addition, the electronic device 500 may further include an input / output (I / O) interface 558. The electronic device 500 may operate based on an operating system stored in the memory 532.
[0084] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by the processor, the steps of the above-mentioned virtual scene interaction method are implemented. For example, the non-transitory computer-readable storage medium can be the above-mentioned memory 532 including program instructions, and the above-mentioned program instructions can be executed by the processor 522 of the electronic device 500 to complete the above-mentioned virtual scene interaction method.
[0085] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above-mentioned virtual scene interaction method when executed by the programmable device.
[0086] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.
[0087] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.
[0088] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.
Claims
1. A virtual scene interaction method, It is characterized in that include: Scanning the real scene using a scanning device to obtain a real scene model; Extracting feature information from the real scene model, and processing the feature information based on a generator in a trained generative adversarial network to determine a virtual scene; Using a virtual reality device to display the virtual scene to a user, and obtaining a target image of the user's eyes facing the virtual scene; Based on the target image, the user's interaction intention is determined, and based on the interaction intention, an interaction operation is performed on the interactive object in the virtual scene.
2. The method according to claim 1, It is characterized in that The generative adversarial network also includes a discriminator, and the trained generative adversarial network is obtained based on the following method: Obtaining a sample real scene model that matches the sample real scene; Processing the sample real scene model using the generator to determine a sample virtual scene; Using the discriminator to process the sample real scene model and the sample virtual scene to determine a discrimination result, wherein the discrimination result is used to characterize the authenticity of the sample virtual scene; Based on the discrimination result, the parameters of the generative adversarial network are adjusted until preset conditions are met to obtain the trained generative adversarial network.
3. The method according to claim 1, It is characterized in that The determining the user's interaction intention based on the target image includes: Processing the target image based on a trained eye tracking model to obtain the eye direction and gaze position of the user; Based on the eye direction and the gaze position, the interaction intention is confirmed.
4. The method according to claim 1, It is characterized in that The method of scanning the real scene using a scanning device to obtain a real scene model includes: Scanning the real scene using the scanning device to obtain point cloud data; The point cloud data is subjected to meshing processing and texture mapping processing to obtain the real scene model.
5. The method according to claim 1, It is characterized in that Before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: Processing the real scene model using image segmentation technology to obtain multiple semantic regions; According to the adjustment operation of the user, the plurality of semantic areas in the real scene model are adjusted.
6. The method according to claim 1 or 5, It is characterized in that Before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The image style of the real scene model is adjusted using image style transfer technology.
7. The method according to claim 6, It is characterized in that Before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The image dynamic synthesis model is used to perform the following operations: Extracting motion features of static objects in the real scene model; The motion features are processed based on preset dynamic object generation rules to obtain a dynamic object.
8. The method according to claim 7, It is characterized in that Before the step of extracting feature information from the real scene model and processing the feature information based on the generator in the trained generative adversarial network to determine the virtual scene, the method further includes: The resolution of the real scene model is adjusted to the target resolution using image super-resolution technology.
9. The method according to claim 8, It is characterized in that The image dynamic synthesis model includes a variational autoencoder and a recurrent neural network.
10. An interactive device for a virtual scene, It is characterized in that include: A scanning module is configured to scan a real scene using a scanning device to obtain a real scene model; A processing module is configured to extract feature information from the real scene model, and process the feature information based on the generator in the trained generative adversarial network to determine a virtual scene; an acquisition module, configured to display the virtual scene to the user using a virtual reality device, and acquire a target image of the user's eyes facing the virtual scene; The interaction module is configured to determine the user's interaction intention based on the target image, and perform an interaction operation on the interactive object in the virtual scene based on the interaction intention.
11. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
12. An electronic device, It is characterized in that include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 9.
13. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
