360-degree interface interaction method and device combining reality and virtuality and medium
Through the 360-degree interface interaction method without wearing a device, users can freely explore and interact in virtual reality, solving the problems of limited virtual reality interaction and fewer experience scenarios in the existing technology, and achieving a wider application of virtual reality technology and universal.
Patent Information
- Application Number
- CN202410983731.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing virtual reality interaction technology requires wearing VR devices, which limits users' movement and social interaction, has fewer experience scenarios and interaction methods, and high-quality VR devices are expensive, making it difficult for users to explore and interact with reality freely in the virtual world. .
Provide a 360-degree interface interaction method that combines reality and virtuality. By receiving the user's scenes and number of people to be experienced, performing 3D modeling and real-time rendering, planning projection areas, using 3D cameras to obtain user actions and map them to the virtual environment, achieving no need An immersive experience with wearing devices, where users can freely explore and interact with reality in the virtual world.
Without wearing heavy VR helmets, users can easily participate in the virtual reality experience, alleviate discomfort, improve the ease of use and popularity of virtual reality technology, reduce costs, and cover a wider range of fields such as education, promoting virtual reality technology in more Application and promotion in the field.
Smart Images

Figure CN118981247A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of virtual reality technology, and in particular to a 360-degree interface interaction method, device and medium combining reality and virtuality. Background Art
[0002] With the rapid development of virtual reality technology, its application areas are expanding to three-dimensional space at an unprecedented speed, bringing users an unprecedented immersive experience. This technology not only changes the way we interact with the digital world, but also has a profound impact on multiple industries such as education and entertainment.
[0003] However, currently most virtual reality interactions require wearing VR devices. When wearing VR devices, the user's head and body movements are subject to certain restrictions, especially large-scale movements or rapid turns, which increase the user's sense of uneasiness. When wearing VR devices, users are often isolated from the real environment and find it difficult to have normal social interactions with people around them. Although VR technology provides a variety of interaction methods, these methods are often not as natural and intuitive as interactions in the real world. High-quality VR devices are relatively expensive and not easily affordable for all users. Although the VR market continues to develop, compared with traditional games and entertainment platforms, VR content is still relatively scarce and of varying quality.
[0004] Through the above analysis, the problems and defects of the prior art are as follows:
[0005] Most of the existing virtual reality interactions require wearing VR devices, and there are relatively few experience scenes and interaction methods. Users wearing VR devices cannot interact normally with people around them. Summary of the invention
[0006] The embodiments of the present application provide a 360-degree interface interaction method, device and medium that combine reality and virtuality, which can solve the problem in the prior art that most virtual reality interactions require wearing VR devices, and there are relatively few experience scenes and interaction methods, and users wearing VR devices cannot interact normally with people around them.
[0007] In the first aspect, an embodiment of the present application provides a 360-degree interface interaction method that combines reality and virtuality, which is applied to a device to be experienced, and the method includes: receiving a scene to be experienced by a user, as well as the number and height of users; based on the height, 3D modeling of the scene to be experienced, and rendering through a real-time rendering engine to obtain a first 3D model; creating a 360-degree basic model based on the device to be experienced, and importing the first 3D model into the 360-degree basic model to obtain a second 3D model; planning a projection area based on the second 3D model and the number of users; after the user enters the experience device, obtaining the user's movements through a 3D camera, and mapping the movements to a virtual environment, so that the virtual environment provides feedback based on the movements.
[0008] In one implementation of the present application, the projection area is planned according to the second 3D model and the number of users, specifically including: dividing the projection area into multiple sub-areas through a virtual grid; predicting the movement path according to the second 3D model and the number of users to obtain mobile sub-areas and non-mobile sub-areas; and placing projectors at multiple angles in the non-mobile sub-areas.
[0009] In one implementation of the present application, the method also includes: when the projector is placed, using a virtual user to conduct a simulation experiment to obtain an occlusion area; obtaining a projection angle blind area of the projector based on the occlusion area, and adjusting the angle and position of the projector based on the projection angle blind area.
[0010] In one implementation of the present application, the user's movements are obtained through a 3D camera, specifically including: after the user enters the experience device, a front image of the user is obtained through the 3D camera; joint points are identified in the front image, and the joint points include the head, neck, spine shoulder, left / right shoulder, left / right elbow, left / right wrist, sacrum, left / right hip, left / right knee, and left / right ankle; and the user's initial limb length and initial joint angle are calculated based on the joint points.
[0011] In one implementation of the present application, the method also includes: when the user performs an action, tracking the movement trajectory of the joint point and calculating the movement curve; matching the initial movement trajectory and movement curve with the preset basic action; if no match is successful, classifying and identifying the movement trajectory and movement curve through a machine learning algorithm, and adding basic actions.
[0012] In one implementation of the present application, the method also includes: equipping the user with a safety body sensing device to collect the user's reaction time and heart rate changes during the experience; when the heart rate changes exceed a preset safety range, the experience is automatically paused.
[0013] In one implementation of the present application, before receiving the user's scene to be experienced, the method also includes: presetting basic actions, the basic actions include the starting state, intermediate process and end state of the action, as well as the speed curve and joint angle; configuring voice commands and integrating a voice recognition engine to recognize the user's voice instructions; importing sound effect files, and setting playback parameters according to the scene to be experienced, the playback parameters including playback volume, number of playbacks and playback time; when matching a preset action template with accompanying sound effects, playing the corresponding sound effects.
[0014] In one implementation of the present application, before receiving the scene to be experienced by the user, the method also includes: placing multiple 3D cameras in the experience device; calibrating the focal length, viewing angle and depth of field of the 3D cameras, and calibrating the brightness, contrast and color of the projector.
[0015] In a second aspect, an embodiment of the present application further provides a 360-degree interface interaction device that combines reality and virtuality, the device comprising at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: receive a scene to be experienced by a user, as well as the number and height of the users; perform 3D modeling on the scene to be experienced based on the height, and render it through a real-time rendering engine to obtain a first 3D model; create a 360-degree basic model based on the device to be experienced, and import the first 3D model into the 360-degree basic model to obtain a second 3D model; plan a projection area based on the second 3D model and the number of users; after the user enters the experience device, obtain the user's movements through a 3D camera, and map the movements to the virtual environment, so that the virtual environment provides feedback based on the movements.
[0016] On the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for a 360-degree interface interaction that combines reality and virtuality, storing computer executable instructions, wherein the computer executable instructions are configured to: receive a scene to be experienced by a user, as well as the number and height of users; perform 3D modeling on the scene to be experienced based on the height, and render it through a real-time rendering engine to obtain a first 3D model; create a 360-degree basic model based on the device to be experienced, and import the first 3D model into the 360-degree basic model to obtain a second 3D model; plan a projection area based on the second 3D model and the number of users; after the user enters the experience device, obtain the user's movements through a 3D camera, and map the movements into the virtual environment, so that the virtual environment provides feedback based on the movements.
[0017] The embodiments of the present application provide a 360-degree interface interaction method, device and medium that combine reality and virtuality. Without wearing heavy VR helmets or devices, users can participate in the virtual reality experience more easily and reduce the discomfort caused by long-term use. Users can not only explore freely in the virtual world, but also communicate and interact naturally with people around them. The improved virtual reality technology will be easier to be accepted and used by the public, reducing technical barriers and costs. It covers a wider range of fields, such as education, and special immersive functions can be developed. Users can study and read in experience halls, which will promote the application and promotion of virtual reality technology in more fields and bring innovation and development opportunities to all walks of life. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 A flow chart of a 360-degree interface interaction method combining reality and virtuality provided in an embodiment of the present application;
[0020] Figure 2 A schematic diagram of joint marking points of a 360-degree interface interaction method combining reality and virtuality provided in an embodiment of the present application;
[0021] Figure 3 A schematic diagram of the internal structure of a 360-degree interface interaction device that combines reality and virtuality provided in an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0023] The embodiments of the present application provide a 360-degree interface interaction method, device and medium that combine reality and virtuality, which solves the problem in the prior art that most virtual reality interactions require wearing VR devices, and there are relatively few experience scenes and interaction methods, and users wearing VR devices cannot interact normally with people around them.
[0024] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.
[0025] Figure 1A flowchart of a 360-degree interface interaction method combining reality and virtuality provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides a 360-degree interface interaction method combining reality and virtuality, which specifically includes the following steps:
[0026] Step 10: Receive the user's experience scenario, as well as the number and height of the user.
[0027] In the embodiments of the present application, the devices to be experienced can be understood as small experience halls, which can be set up inside science and technology museums and shopping malls. The specific implementation depends on the actual situation of the users. In this way, users can experience and interact with the virtual without wearing the device, making the virtual scenes that users want to experience as realistic as possible.
[0028] In this step, for example, if the user wants to experience an immersive self-study activity, or wants to experience the room to be lived in, the user first needs to provide the scene to be experienced, which can be done by using high-precision scanning technology or based on user-input data, such as CAD drawings; further, it may be that friends travel together, and then the number and height of users are obtained to perform personalized 3D modeling.
[0029] Step 20: Perform 3D modeling on the scene to be experienced based on the height, and render it through a real-time rendering engine to obtain a first 3D model.
[0030] In this step, when performing 3D modeling, first create basic geometric shapes, and then adjust the height of the geometric shapes according to the user's height data, for example, the height of tables and chairs can be adapted to the user (if the user needs real tables and chairs for real self-study and reading activities, the experience venue can also provide them); or, restore the real scene according to the data entered by the user, for example, the user wants to actually feel the size of the house; after the basic modeling is completed, use the rendering engine Unreal Engine to optimize the model in detail, including texture mapping, lighting effects, shadow processing, etc., to enhance the realism and immersion of the model.
[0031] Step 30: Create a 360-degree basic model according to the device to be experienced, and import the first 3D model into the 360-degree basic model to obtain a second 3D model;
[0032] In this step, first determine the scope of the 360-degree basic model according to the actual size and layout of the device to be experienced (i.e., the small experience hall). This includes determining the horizontal and vertical viewing angles of the model to ensure that after entering the experience hall, the user can freely rotate the viewing angle in the virtual environment and see all parts of the entire scene.
[0033] Step 40: planning a projection area according to the second 3D model and the number of users;
[0034] As an optional embodiment, planning the projection area according to the second 3D model and the number of users may specifically include: step 401: dividing the projection area into multiple sub-areas through a virtual grid; step 402: predicting the movement path according to the second 3D model and the number of users to obtain mobile sub-areas and non-mobile sub-areas; step 403: placing projectors at multiple angles in the non-mobile sub-areas.
[0035] In this step, ensure that the projection range of the projector can cover the entire area and try to avoid projection overlap or blank areas. During the specific implementation, the staff can simulate in advance to adjust the projector position.
[0036] As an optional embodiment, the method may also include: step 404: when the projector is placed, use a virtual user to conduct a simulation experiment to obtain the occlusion area; step 405: obtain the projection angle blind area of the projector according to the occlusion area, and adjust the angle and position of the projector according to the projection angle blind area.
[0037] For example, two virtual users can be created to conduct simulation experiments, or multiple staff members can enter the experience center to conduct simulation experiments.
[0038] Step 50: After the user enters the experience device, the user's movements are acquired through a 3D camera, and the movements are mapped to the virtual environment so that the virtual environment gives feedback according to the movements.
[0039] As an optional embodiment, obtaining the user's action through a 3D camera may specifically include: Step 501: after the user enters the experience device, obtaining a front image of the user through a 3D camera;
[0040] In this step, when the user enters the experience device for the first time, the user may be guided to face one of the 3D cameras.
[0041] Step 502: Identify joints in the frontal image, where the joints include the head, neck, spine, shoulder, left / right shoulder, left / right elbow, left / right wrist, sacrum, left / right hip, left / right knee, and left / right ankle;
[0042] In this step, if Figure 2 As shown in Table 1, the current state of the human body can be estimated by calculating the relative positions of the human body joints in three-dimensional space.
[0043] Table 1 Joint marking points and numbers
[0044] serial number Marking Points serial number Marking Points 0 head 8 right wrist 1 neck 9 Sacral 2 Spine and Shoulder 10 Left hip 3 Left shoulder 11 Left knee 4 left elbow 12 Left ankle 5 left wrist 13 Right hip 6 Right shoulder 14 right knee 7 right elbow 15 Right ankle
[0045] Step 503: Calculate the user's initial limb length and initial joint angle according to the joint points.
[0046] In this step, joint marker points can be identified, for example, the coordinate sequence of the shoulder, elbow and wrist can be obtained, and based on the Y-axis coordinates Ys, Ye, Yw of the shoulder, elbow and wrist, the vertical length of the upper arm is Ye-Ys, and the vertical length of the forearm is Sl=Yw-Ye; then the inverse trigonometric function operation (specifically inverse sine) can be performed on the tester's upper arm and forearm from the initial state to the completed state to obtain the joint angle data.
[0047] As an optional embodiment, the method may also include: step 504: when the user performs an action, tracking the movement trajectory of the joint point and calculating the movement curve; step 505: matching the initial movement trajectory and movement curve with the preset basic action; step 506: if no match is successful, classifying and identifying the movement trajectory and movement curve through a machine learning algorithm, and adding a basic action.
[0048] In this step, a series of joint marker points, P0, P1, ..., Pn, can be first determined, and a parameter t is selected, usually t∈[0,1], which is used to describe the change of points on the curve over time or other variables. According to the marker points and parameter t, a hybrid function (such as Bernstein polynomial) is used to calculate the points on the curve. The points on the curve corresponding to parameter t are calculated by the hybrid function and the coordinates of the marker points, and the calculated curve points are connected to form a smooth curve. According to the motion curve matching the curve of the preset basic action, the machine learning model may distinguish different actions based on multiple features, such as the speed, acceleration, shape of the trajectory, etc. of the movement of the joint point. By continuously training and optimizing the model, the accuracy and robustness of action recognition can be improved. If it cannot be matched, the new action can be identified and added to the preset basic action.
[0049] As an optional embodiment, the method may also include: remotely monitoring the experience process, and equipping the user with a safety body sensing device to collect the user's reaction time and heart rate changes during the experience; when the heart rate changes exceed a preset safety range, automatically pausing the experience.
[0050] In this step, a remote monitoring system can be installed during the user's experience. This system can include cameras, microphones, sensors and other equipment to capture the user's movements, sounds and environmental information in real time, and preset a reasonable heart rate safety range based on the user's age, health status and experience content.
[0051] It is understandable that, before receiving the scene to be experienced by the user, the method may further include:
[0052] Step 01: Preset basic actions, which include the starting state, intermediate process and ending state of the action, as well as the speed curve and joint angle; Step 02: Configure voice commands and integrate the voice recognition engine to recognize the user's voice commands; Step 03: Import sound effect files and set the playback parameters according to the scene to be experienced. The playback parameters include playback volume, number of playbacks and playback time; Step 04: When matching the preset action template with attached sound effects, play the corresponding sound effects.
[0053] For example, if a user wants to experience an immersive study room, the sound of turning pages can be played in an infinite loop; if a user wants to observe the house to be rented, there can be the sound of a door being pushed open; in this way, when presetting basic actions, specific sound effects are associated with corresponding action templates. When the system recognizes that the user has performed an action that matches a preset action template with an accompanying sound effect, the corresponding sound effect playback mechanism is automatically triggered.
[0054] As an optional embodiment, before receiving the scene to be experienced by the user, the method may also include: Step 05: placing multiple 3D cameras in the experience device; Step 06: calibrating the focal length, viewing angle and depth of field of the 3D camera, and calibrating the brightness, contrast and color of the projector.
[0055] During specific implementation, the number of 3D cameras required is determined based on the scale and complexity of the experience device. More cameras usually mean higher capture accuracy and fewer blind spots.
[0056] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a 360-degree interface interaction device combining reality and virtuality, and its structure is as follows: Figure 3 shown.
[0057] Figure 3 This is a schematic diagram of the internal structure of a 360-degree interface interaction device that combines reality and virtuality provided in an embodiment of the present application. Figure 3 As shown, the device includes:
[0058] at least one processor 301;
[0059] and, a memory 302 communicatively connected to at least one processor;
[0060] Among them, the memory 302 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 301 so that the at least one processor 301 can: receive the scene to be experienced by the user, as well as the number and height of the users; perform 3D modeling on the scene to be experienced according to the height, and render it through a real-time rendering engine to obtain a first 3D model; create a 360-degree basic model according to the device to be experienced, and import the first 3D model into the 360-degree basic model to obtain a second 3D model; plan the projection area according to the second 3D model and the number of users; after the user enters the experience device, obtain the user's movements through a 3D camera, and map the movements to the virtual environment, so that the virtual environment provides feedback based on the movements.
[0061] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for 360-degree interface interaction combining reality and virtuality, storing computer executable instructions, wherein the computer executable instructions are configured to: receive a scene to be experienced by a user, as well as the number and height of the users; perform 3D modeling of the scene to be experienced according to the height, and render it through a real-time rendering engine to obtain a first 3D model; create a 360-degree basic model according to a device to be experienced, and import the first 3D model into the 360-degree basic model to obtain a second 3D model; plan a projection area according to the second 3D model and the number of users; after the user enters the experience device, obtain the user's actions through a 3D camera, and map the actions to a virtual environment, so that the virtual environment gives feedback according to the actions.
[0062] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0063] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.
[0064] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0066] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0069] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0070] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0071] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0072] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A 360-degree interface interaction method combining reality and virtuality, characterized in that: Applied to the device to be experienced, the method comprises: Receive the user's experience scenario, as well as the number and height of the user; According to the height, 3D modeling is performed on the scene to be experienced, and rendering is performed through a real-time rendering engine to obtain a first 3D model; Creating a 360-degree basic model according to the device to be experienced, and importing the first 3D model into the 360-degree basic model to obtain a second 3D model; Planning a projection area according to the second 3D model and the number of users; After the user enters the experience device, the user's movements are acquired through a 3D camera, and the movements are mapped to the virtual environment, so that the virtual environment gives feedback according to the movements.
2. A 360-degree interface interaction method combining reality and virtuality according to claim 1, characterized in that: Planning the projection area according to the second 3D model and the number of users, specifically including: Divide the projection area into multiple sub-areas through a virtual grid; Predicting a moving path according to the second 3D model and the number of users, and obtaining a moving sub-area and a non-moving sub-area; Projectors are placed at multiple angles in the non-moving sub-area.
3. The 360-degree interface interaction method combining reality and virtuality according to claim 2, characterized in that: The method further comprises: When the projector is placed, a simulation experiment is performed using a virtual user to obtain a blocked area; A projection angle blind area of the projector is obtained according to the shielding area, and the angle and position of the projector are adjusted according to the projection angle blind area.
4. The 360-degree interface interaction method combining reality and virtuality according to claim 1, characterized in that: The obtaining of the user's action through the 3D camera specifically includes: After the user enters the experience device, a frontal image of the user is obtained through a 3D camera; identifying joint points in the frontal image, the joint points comprising head, neck, spine shoulder, left / right shoulder, left / right elbow, left / right wrist, sacrum, left / right hip, left / right knee, and left / right ankle; The user's initial limb length and initial joint angle are calculated according to the joint points.
5. The 360-degree interface interaction method combining reality and virtuality according to claim 4, characterized in that: The method further comprises: When the user performs an action, the movement trajectory of the joint point is tracked and the movement curve is calculated; Matching the initial movement trajectory and movement curve with the preset basic action; In the case of no successful matching, the movement trajectory and movement curve are classified and identified through a machine learning algorithm, and basic actions are added.
6. The 360-degree interface interaction method combining reality and virtuality according to claim 1, characterized in that: The method further comprises: Equip users with safety somatosensory devices to collect information about their reaction time and heart rate changes during the experience; When the heart rate changes beyond the preset safety range, the experience will be automatically paused.
7. The 360-degree interface interaction method combining reality and virtuality according to claim 1, characterized in that: Before receiving the scene to be experienced by the user, the method further includes: Preset basic movements, wherein the basic movements include a starting state, an intermediate process, and an ending state of the movement, as well as a speed curve and a joint angle; Configure voice commands and integrate a voice recognition engine to recognize user voice commands; Import the sound effect file and set the playback parameters according to the scene to be experienced, wherein the playback parameters include the playback volume, the number of playbacks and the playback time; When a preset action template with accompanying sound effects is matched, the corresponding sound effects are played.
8. The 360-degree interface interaction method combining reality and virtuality according to claim 1, characterized in that: Before receiving the scene to be experienced by the user, the method further includes: Placing a plurality of 3D cameras in the experience device; The 3D camera is calibrated for focal length, viewing angle and depth of field, and the projector is calibrated for brightness, contrast and color.
9. A 360-degree interface interactive device combining reality and virtuality, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Receive the user's experience scenario, as well as the number and height of the user; According to the height, 3D modeling is performed on the scene to be experienced, and rendering is performed through a real-time rendering engine to obtain a first 3D model; Creating a 360-degree basic model according to the device to be experienced, and importing the first 3D model into the 360-degree basic model to obtain a second 3D model; Planning a projection area according to the second 3D model and the number of users; After the user enters the experience device, the user's movements are acquired through a 3D camera, and the movements are mapped to the virtual environment, so that the virtual environment gives feedback according to the movements.
10. A non-volatile computer storage medium for 360-degree interface interaction combining reality and virtuality, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Receive the user's experience scenario, as well as the number and height of the user; According to the height, 3D modeling is performed on the scene to be experienced, and rendering is performed through a real-time rendering engine to obtain a first 3D model; Creating a 360-degree basic model according to the device to be experienced, and importing the first 3D model into the 360-degree basic model to obtain a second 3D model; Planning a projection area according to the second 3D model and the number of users; After the user enters the experience device, the user's movements are acquired through a 3D camera, and the movements are mapped to the virtual environment, so that the virtual environment gives feedback according to the movements.
Citation Information
Patent Citations
Virtual reality museum construction method, electronic equipment and storage medium
CN115187751A
Immersive interactive large screen implementation method
CN115525151A
Intangible cultural heritage display system based on virtual reality technology
CN116166123A
Organizing projections on a surface
US20120236038A1
