Data processing method and device for augmented reality environment perception and electronic equipment

Abstract

The invention relates to the technical field of computers, and discloses a data processing method and device for augmented reality environment perception. When the user is in the virtual reality immersive scene, real scene information currently corresponding to the user is obtained, the real scene information is analyzed to obtain environment information corresponding to the real scene information, reminding information is generated according to the environment information, and the reminding information is prompted in the virtual reality immersive scene of the user. According to the method and the device, the environment and situation of the user can be better understood, more personalized and targeted reminding information can be provided, the user can obtain more real and more appropriate experience in a virtual reality immersive scene, and the safety of the user can be monitored and reminded by using real scene information. Besides, the user can obtain the key information without interrupting the current experience, which is helpful for reducing the demand of switching tasks or devices, so that the user can concentrate more attention and better complete the virtual reality immersive experience.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device and electronic device for augmented reality environment perception. Background Art

[0002] In current VR (Virtual Reality) immersive scenarios, when users wear VR devices, their visual and auditory senses mainly interact with the virtual environment and are isolated from the real environment. If users want to obtain this environmental information, they usually need to take off the VR device, which interrupts their virtual experience. For example, users are playing a virtual reality game, they wear a head-mounted display and are completely immersed in the virtual world of the game. However, when someone enters the room where the user is or the phone rings, the user cannot perceive these changes in the real environment. If users want to understand what is happening outside, they need to take off the VR device, which interrupts their experience in the game. This frequent wearing and taking off process destroys the sense of immersion and interactivity, making it difficult for users to truly immerse themselves in the virtual reality experience. Summary of the invention

[0003] The main technical problem solved by the implementation methods of the present application is how to improve the user's immersion and interactivity in a virtual reality scenario.

[0004] To solve the above technical problems, a technical solution adopted in the implementation mode of the present application is: to provide a data processing method for augmented reality environment perception, including: when a user is in a virtual reality immersive scene, obtaining the real scene information currently corresponding to the user; analyzing the real scene information to obtain the environmental information corresponding to the real scene information; generating reminder information according to the environmental information; and prompting the reminder information in the user's virtual reality immersive scene.

[0005] To solve the above technical problems, another technical solution adopted in the implementation mode of the present application is: to provide a data processing device for augmented reality environment perception, including: a real scene information acquisition module, which is used to obtain the real scene information currently corresponding to the user when the user is in a virtual reality immersive scene; an environmental information acquisition module, which analyzes the real scene information to obtain environmental information corresponding to the real scene information; a reminder information generation module, which is used to generate reminder information according to the environmental information; and a reminder module, which is used to prompt the reminder information in the user's virtual reality immersive scene.

[0006] To solve the above technical problems, another technical solution adopted in the implementation mode of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0007] In order to solve the above technical problems, another technical solution adopted in the implementation mode of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device executes the method as described above.

[0008] To solve the above technical problems, another technical solution adopted in the implementation mode of the present application is: to provide a computer program product, wherein the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes the method as described above.

[0009] Different from the related art, the embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium and computer program product for augmented reality environment perception, which mainly obtains the real scene information corresponding to the user when the user is in a virtual reality immersive scene, analyzes the real scene information to obtain the environmental information corresponding to the real scene information, and then generates reminder information based on the environmental information, and finally prompts the reminder information in the user's virtual reality immersive scene. The implementation method provided by the present application, on the one hand, by obtaining the real scene information corresponding to the user, enables the system to better understand the environment and situation of the user, helps to provide more personalized and targeted reminder information, and enables the user to obtain a more real and more appropriate actual experience in the virtual reality immersive scene. In addition, the system can use the real scene information to monitor and remind the user's safety. For example, the system can detect that the user is too close to the edge, fatigued or there are other safety risks in the virtual reality scene and send corresponding reminder information to ensure the safety of the user. On the other hand, by providing reminder information within the user's virtual reality immersive scene, the user can obtain key information without interrupting the current experience, which helps to improve the efficiency of the user's virtual reality immersive experience, reduce the need to switch tasks or devices, and enable the user to concentrate more and better complete the virtual reality immersive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.

[0011] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0012] Figure 2 is a schematic diagram of a user perspective in a virtual reality immersive scene in the application scenario provided in an embodiment of the present application;

[0013] Figure 3 is a flow chart of a data processing method for augmented reality environment perception provided by an embodiment of the present application;

[0014] Figure 4 is a structural schematic diagram of a data processing device for augmented reality environment perception provided by an embodiment of the present application;

[0015] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the protection scope of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device schematic diagram or the flow chart. Unless otherwise defined, all technical and scientific terms used in this specification are the same as those generally understood by technicians in the technical field of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not used to limit the present application.

[0018] Virtual Reality (VR) is a simulated environment created by computer technology, which enables users to interact and experience in this virtual environment. VR technology usually immerses users completely in the virtual world by wearing special equipment, such as head-mounted displays (HMD), handles, gloves, etc. VR technology uses computer-generated three-dimensional images and sounds to create an immersive feeling for users. Users can interact with virtual objects in the virtual reality environment, such as touching, moving, grabbing, etc. At the same time, VR technology can also adjust the presentation of the virtual environment in real time according to the user's head and body movements to enhance the user's sense of immersion. VR technology has a wide range of applications in many fields. In the field of entertainment and games, users can enter the virtual game world through VR devices, interact with game characters, and get a more immersive gaming experience. In the field of education, VR can be used to create virtual laboratories, virtual scenes, etc., to help students conduct interactive learning and practical training. In the medical field, VR can be used for surgical simulation, psychotherapy, etc., to provide safer and more effective treatment methods. Virtual reality technology is also used in other fields such as architectural design, engineering training, and military simulation.

[0019] For example, when applying VR immersive teaching to geography classes, virtual reality technology can be used to simulate and present learning scenarios of real geographical environments. Both teachers and students can enter the deep sea, desert, rainforest and other geographical environments in the virtual world by wearing VR helmets and handles. In this virtual geographical exploration scene, students can observe and learn various geographical phenomena in an immersive way. For example, students can dive into the virtual deep-sea environment to observe marine life, coral reefs and submarine landforms; students can enter the virtual desert area to explore sand dunes, desert vegetation and desert animals. In this process, the teacher can play the role of a commentator to explain to students the phenomena and things observed in the virtual geographical environment. This virtual geographical exploration scene allows students to experience the geographical environment in an immersive way, obtain a more intuitive and in-depth learning experience, and help stimulate students' interest and curiosity, and improve their understanding and memory of geographical subjects.

[0020] In the above virtual geographical exploration, since both students and teachers are focused on things in the virtual environment, they may not be able to pay attention to what is happening in the real scene at the same time. For example, the teacher cannot always pay attention to the situation of the students in the classroom, which may cause safety hazards or the teacher may have to take off the VR glasses frequently to check the status of the students, which makes the teacher unable to teach at ease. This is caused by the limitations of virtual reality technology. Therefore, how to make users pay attention to what is happening in the real scene during the virtual reality immersive experience is a problem that needs to be solved in the embodiments of this application.

[0021] See also Figure 1 , Figure 1 : is a schematic diagram of an application scenario provided by an embodiment of the present application. The data processing method and device for augmented reality environment perception of the embodiment of the present application can be applied to this application scenario. The application scenario includes: a computing device 10, a VR device 20, an intelligent acquisition terminal device 30, and an interactive intelligent all-in-one machine 40. The computing device 10 is respectively connected to the VR device 20, the intelligent acquisition terminal device 30, and the interactive intelligent all-in-one machine 40 in a wired or wireless manner.

[0022] The computing device 10 can be a computer, a server, a cloud computing platform or other device with computing and processing capabilities. The main function of the computing device 10 is to process and manage data and tasks related to virtual reality, intelligent acquisition terminal 30 and interactive intelligent all-in-one machine 40. Specifically, for example, the computing device 10 is responsible for processing graphics, sounds and interactive information in the virtual reality environment. It may run virtual reality applications and algorithms to generate realistic virtual experiences and transmit the processed data to the VR device 20 for display and interaction. The computing device 10 communicates with other devices through wired or wireless means. It can be responsible for transmitting data to the VR device 20, the intelligent acquisition terminal device 30 and the interactive intelligent all-in-one machine 40 to achieve data exchange and collaborative work between devices. The computing device 10 can manage and control the entire system. It can monitor the status and performance of the device, and schedule the allocation and priority of tasks to ensure the stable operation and efficient operation of the system. The computing device 10 can also process and analyze the data obtained from the intelligent acquisition terminal device 30, and can run machine learning algorithms, image processing algorithms or other data processing technologies to extract useful information and insights, and support corresponding decisions and applications. The computing device 10 in this application scenario is responsible for data processing, communication connection, management and control, etc. The specific computing device type and configuration will depend on the application requirements, and may be a computer, server or other dedicated computing platform.

[0023] Optionally, the computing device 10 is an Internet of Things (IoT) middle station, which is a core platform for connecting and managing various IoT devices, and is used to provide data processing, storage, analysis and management functions to support real-time monitoring, remote control and data exchange of IoT applications. In an embodiment of the present application, the computing device 10 is responsible for connecting and communicating with various IoT devices, including VR devices 20, intelligent acquisition terminal devices 30 and interactive intelligent all-in-one machines 40. It can support a variety of communication protocols and technologies, such as Wi-Fi, Bluetooth, Zigbee, etc., to achieve data exchange and transmission between devices. The computing device 10 receives data from IoT devices, processes, stores and manages them, and can run specific IoT platform software or applications for data parsing, filtering, aggregation and storage for subsequent data analysis and application. For example, the computing device 10 is used to obtain the real scene information currently corresponding to the user through the intelligent acquisition terminal device 30 when the user wears the VR device 20 in the virtual reality immersive scene; analyze the real scene information to obtain the environmental information corresponding to the real scene information; generate reminder information according to the environmental information; and then send the reminder information to the VR device 20 and the interactive intelligent all-in-one machine 40, so as to prompt the reminder information in the user's virtual reality immersive scene.

[0024] Among them, the VR device 20 is a technical tool or device for creating a virtual environment generated by a computer, so that the user can experience and interact with it in an immersive way. The VR device 20 can be composed of a head-mounted display (HMD) and a tracking system. The head-mounted display is one of the important components of the VR device 20. It usually consists of one or more display screens and is placed in front of the user's eyes to provide a three-dimensional, panoramic visual effect. By wearing a head-mounted display, the user can experience an immersive virtual reality experience.

[0025] The intelligent acquisition terminal device 30 is used to collect data such as audio, video, and text in the VR immersive scene. It can specifically be a camera, a stereo microphone, a scanner, etc. Its number and installation location can be set according to the area and space of the specific application scene. In the embodiment of the present application, the real scene information in the user's current scene can be obtained through the intelligent acquisition terminal device 30.

[0026] The interactive intelligent all-in-one machine 40 is a comprehensive device that integrates computing, display, perception and interaction functions. It usually includes components such as a display screen, a computing processing unit, a camera, a microphone, a speaker and a touch interface to realize human-computer interaction and multimedia applications. The interactive intelligent all-in-one machine 40 can also be used to collect real scene information in the user's current scene. The interactive intelligent all-in-one machine 40 can be applied to multimedia teaching. For example, it can display teaching resources, including multimedia content such as pictures, videos, animations and audio. Teachers can use the multimedia functions of the all-in-one machine to vividly present knowledge and increase students' participation and understanding.

[0027] See also Figure 1 , Figure 1 Specifically, it can be a scene applied to VR immersive teaching. VR devices 20 are worn by teachers and students respectively, the intelligent acquisition terminal device 30 is specifically a camera set in the classroom, and the interactive intelligent all-in-one machine 40 is an all-in-one machine with a large screen. When the teacher starts VR immersive teaching, the teacher and the student wear VR devices 20 respectively, and the interactive intelligent all-in-one machine 40 is used as a teaching auxiliary device to display the courseware content. When the student and the teacher do not wear the VR device 20, the courseware content on the interactive intelligent all-in-one machine 40 is presented in a two-dimensional form. When the student and the teacher wear the VR device 20, the courseware content can be presented in a three-dimensional form within the field of vision of the student and the teacher. During the teacher's teaching process, the real scene information in the classroom is collected through the camera set in the classroom and the interactive intelligent all-in-one machine 40. The real scene information includes the body movements of students and teachers (such as getting up, walking, fighting, etc.), sounds (such as talking, discussing, etc.), expressions (such as students' facial expressions), etc.; it also includes environmental information in the scene, such as the temperature of the classroom, whether there is a sudden entry of outsiders in the classroom or the ringing of mobile phones, etc. The collected real scene information can be sent to the background (such as the i-IOT Internet of Things middle platform), and the background analyzes and processes the real scene information to obtain the environmental information corresponding to the real scene information. Specifically, the behavior information of users in the scene (such as students and teachers' body movements, sounds, expressions, etc.) and interference information in the scene (such as outsiders suddenly entering the classroom or mobile phones ringing in the classroom) can be obtained based on the real scene information. Then, corresponding reminder information is generated based on the obtained environmental information, and the reminder information is used to be displayed in the user's virtual reality immersive scene. For example, when it is determined based on the environmental information that a student has violated classroom discipline, a warning message for the student is generated, and the warning message is displayed in the teacher's virtual reality immersive scene. For example, Figure 2As shown, from the perspective of the teacher in the virtual reality immersive scene, a warning message of a certain classmate leaving his seat can be seen. As a result, the teacher can clearly understand some of the situations happening in the classroom without taking off the head-mounted display. Among them, the background can set the user's permissions, and only users with this permission can view the reminder information from the perspective of the virtual reality immersive scene. The user's permissions can be modified and updated according to the specific application scenario. In this embodiment, a richer, interactive and personalized teaching experience is provided to enhance student participation and learning effects, while providing real-time feedback and monitoring for teachers to promote teaching management and personalized teaching.

[0028] It should be noted that Figure 1 As only an example of an application scenario, the data processing method and device for augmented reality environment perception provided in the embodiments of the present application can also be applied to other application scenarios. The data processing method for augmented reality environment perception is described below in conjunction with specific embodiments.

[0029] See also Figure 3 , Figure 3 1 is a flowchart of a data processing method for augmented reality environment perception provided by an embodiment of the present application. The method can be applied to the above-mentioned computing device 10. The method comprises the following steps:

[0030] S11. When a user is in a virtual reality immersive scene, obtain real scene information currently corresponding to the user.

[0031] Among them, virtual reality immersive scene refers to the use of virtual reality technology to completely immerse users in a virtual environment, making users feel as if they are in a virtual real world. Users can wear head-mounted displays or other virtual reality devices to interact with the virtual environment.

[0032] Among them, real scene information refers to the relevant information of the real environment where the user is currently located. For example, location information, including the user's current geographical location coordinates, the identification of the area or building, etc., can be obtained through the global positioning system or other positioning technologies. Environmental sensor data, including environmental parameters such as temperature, humidity, light intensity, air quality, etc., can be obtained through sensor devices. User behavior and actions, including the user's posture, gestures, movement status, etc., can be obtained through the above-mentioned sensors (such as gyroscopes, accelerometers, etc.) of virtual reality devices or other sensor devices. Identified or tracked object information, including objects, people or other entities around the user, and tracking their positions and dynamics, can be achieved through technologies such as image recognition and depth sensors. Time and date information, including the user's current time and date, as well as related calendar, season and other information. User health and physiological status information, including the user's heart rate, body temperature, blood pressure, exercise volume and other physiological parameters of the user, which can be obtained through biosensors or wearable devices.

[0033] In some embodiments, an intelligent acquisition terminal device is set in the scene where the user is currently located, and the acquisition of the real scene information currently corresponding to the user includes: collecting audio, video, and text in the real environment through the intelligent acquisition terminal device to obtain the real scene information. Among them, the intelligent acquisition terminal device can capture the sound in the environment where the user is located, and the audio can be collected through the built-in microphone of the intelligent acquisition terminal device or an external audio device. The intelligent acquisition terminal device is also equipped with a camera, which can collect video data of the user's environment and text information in the shooting scene in real time. When collecting audio, video and text data in the real environment, the position and number of specific acquisition devices can be set differently to meet specific needs and application scenarios. For example, multiple microphones or cameras are placed in a room to capture sound or video from different angles. The specific position of the microphone or camera can be set in combination with the position of the user in the scene, such as setting a microphone or camera near the user and at different angles. For example, placing a specific acquisition device on a movable device, such as a robot or drone, to achieve mobile data collection, is suitable for applications that need to cover a wide area or mobile scenes. The location and number of acquisition devices depend on the specific application requirements and acquisition goals. According to the requirements of the scene, you can choose the appropriate device layout to obtain the required audio, video and text data.

[0034] In some embodiments, the obtaining of the real scene information corresponding to the user in the virtual reality immersive scene further includes: collecting the user's physiological information through the sensor device worn by the user; obtaining the user's location information; obtaining the real scene information according to the physiological information, the location information, and the audio, video, and text. The physiological information includes the user's heart rate, blood pressure, body temperature, etc., and also includes the user's motion state, such as whether the user is sitting still or walking or standing up. The location information includes the geographical information of the user's current location, and also includes the building information or identification information of the geographical location. The user's health status and physiological changes can be understood through physiological information such as heart rate, blood pressure, body temperature, etc. For example, the user's heart rate changes can be monitored to evaluate their emotional state or physical activity level, so as to obtain the user's state in the real environment, that is, the real scene information. The user's activity state can be judged through motion sensors and location information, such as sitting still, walking, standing up, etc., and this information can be used for applications such as user motion monitoring and posture recognition in the real environment. Through audio, video and text data, the sound, image and text information of the user's surrounding environment can be analyzed to obtain the context of the user's environment, such as identifying the noise level in the environment, detecting objects and scenes in images, parsing logos and keywords in text, etc. The user's current geographical location and information about the building in which they are located can be determined through location information and building data, which can be used to obtain location-related information, such as the weather at the current location and other upcoming activities at the current location. Combining the above information, the user's situation and behavior can be analyzed.

[0035] The real scene information obtained above can provide users with a personalized virtual reality experience through the analysis below.

[0036] S12. Analyze the real scene information to obtain environment information corresponding to the real scene information.

[0037] Wherein, the analysis of the real scene information to obtain the environmental information corresponding to the real scene information includes: analyzing the audio, the video and the text to obtain the user's behavior information; analyzing the audio, the video and the text to obtain the interference information appearing in the current scene; the behavior information and the interference information constitute the environmental information. Wherein, the audio data can be used for speech recognition and sentiment analysis to understand the user's language expression and emotional state. Video data can be used for object recognition, action recognition and posture recognition to obtain the user's action and posture information. Text data can be processed by natural language to extract keywords, themes or emotions and other information to further understand the user's behavior and intention. By analyzing the audio, video and text data, interference information in the current scene can also be detected and identified. For example, audio data can be used for noise detection and sound classification to determine the type and intensity of noise in the environment; video data can be used for image analysis and object recognition to identify objects, people or other interference factors in the scene; text data can be used for sentiment analysis and keyword extraction to understand the interference information or irrelevant content contained in the text. The behavior information and the interference information are combined to form corresponding environmental information, which can provide a comprehensive understanding of the user's real environment, including the user's behavior habits, emotional state, interference factors in the environment, etc.

[0038] In some embodiments, the user's behavior information can also be obtained based on the collected physiological information of the user and the user's current location information, as well as the interference information appearing in the current scene. Physiological information and location information can provide important clues about the user's behavior. By analyzing the user's physiological information, such as heart rate, blood pressure, body temperature, etc., the user's physical state and activity level can be understood. For example, the change in heart rate can be used to infer whether the user is doing strenuous exercise, and the change in blood pressure can be used to infer whether the user is in a tense or relaxed state. In addition, combined with location information, the user's environment can be understood, such as whether the user is in a classroom or outdoors, etc. The comprehensive analysis of this information can help infer the user's specific behavior, such as learning, working, exercising, resting, etc. Among them, physiological information and location information can also be used to detect interference information in the current scene. Through the change of physiological information, it can be inferred whether there are factors in the scene that cause the user's emotional fluctuations, such as noise, light intensity, temperature, etc. Location information can provide the context of the scene, such as the type of building the user is in, the density of the surrounding crowd, etc., combined with this information, interference factors in the scene can be identified and analyzed, such as noisy background sounds, crowded crowds, etc.

[0039] In the embodiment of the present application, it is of great significance to remind the user who is experiencing the virtual reality through interference information and behavior information, which may include management reminders, safety reminders, personalized reminders, etc., as detailed in the following steps.

[0040] S13: Generate reminder information according to the environmental information.

[0041] The reminder information may be management reminder information. Based on the user's behavior information, the manager can learn whether other users participating in the virtual reality immersive experience have not invested in the virtual reality immersive experience during this process, and generate corresponding reminder information for the users who have not invested. The manager can analyze the reasons why the user has not invested in the virtual reality immersive experience based on the reminder information, which is also helpful for improving the virtual reality immersive experience. Among them, the manager is also participating in the virtual reality immersive experience, and the reminder information is presented in the manager's virtual reality world.

[0042] The reminder information can be a safety reminder information. Interference information can help remind users of safety issues in the virtual reality experience. For example, if it is analyzed that there is noisy background sound or other noise interference in the current scene, the system can remind the user to pay attention to the surrounding situation through audio or other means in the user's virtual reality world to avoid accidental collisions or other safety risks. Similarly, if the user's physiological indicators are analyzed to indicate that they are in a state of tension or fatigue, the system can remind the user to take a proper rest in the user's virtual reality world through text or other means to prevent excessive fatigue or discomfort.

[0043] The reminder information can be personalized reminder information. For example, in a VR immersive game scene, if the analysis of the user's behavior pattern shows that the user often ignores or forgets specific actions, the system can remind the user to pay attention to the need to perform a certain action at a certain time point based on the user's personal preferences and schedule, thereby ensuring the completion of the game task or participation in the activity in the virtual world. For example, in a VR immersive classroom, the system can provide personalized feedback and guidance based on the user's performance and reaction in the VR teaching scene. For example, based on the user's action and posture recognition, the system can detect whether the user performs a specific learning task correctly, and then provide real-time feedback to the teacher's virtual reality world, so that the teacher can guide the user to make appropriate adjustments and improvements.

[0044] In some embodiments, the generating of reminder information according to the environmental information includes: analyzing the environmental information to extract the user's behavior characteristics and the scene characteristics of the current scene; judging whether the reminder condition is triggered according to the behavior characteristics and / or the scene characteristics; if so, generating reminder information according to the behavior characteristics and / or the scene characteristics. Among them, it can be judged whether the conditions for triggering the reminder are met according to the user's behavior characteristics and the scene characteristics, and whether the conditions for triggering the reminder are met can be determined by pre-set rules, thresholds or machine learning models. For example, if the user frequently makes mistakes in a specific task or the duration exceeds a certain threshold, the relevant reminder conditions can be triggered. It is judged that the reminder condition is triggered, and the system can generate corresponding reminder information based on the user's behavior characteristics and the scene characteristics. The reminder information can be conveyed to the user's virtual reality immersive scene through visual, auditory and other means. For example, the system can display text prompts, provide voice guidance or transmit reminder information in the form of vibration in the virtual reality scene.

[0045] S14: Prompt the reminder information in the user's virtual reality immersive scene.

[0046] Prompting the reminder information in the user's virtual reality immersive scene includes: obtaining the type of the virtual reality immersive scene and obtaining the authority of the user; obtaining a reminder method according to the type of the virtual reality immersive scene; and based on the authority of the user, prompting the reminder information corresponding to the reminder method in the virtual reality immersive scene of the user with the reminder authority in the reminder method.

[0047] Among them, first obtain the type of virtual reality immersive scene, such as immersive teaching scene, or immersive conference scene, or immersive movie scene, or immersive game scene, etc., because the reminder information that needs to be prompted in different virtual reality immersive scenes is different. For example, in the immersive teaching scene, some body movements of students (such as drinking water, eating, etc.) need to be sent to the teacher as reminder information, but in the immersive conference scene, some body movements of participants (such as drinking water, etc.) do not need to be reminded to other participants in the virtual reality immersive scene. At the same time, the permissions of users in the virtual reality immersive scene can also be obtained, because users with different permissions can receive different reminder information. For example, the permissions of teachers and students in immersive teaching scenes are different. Teachers can receive classroom discipline reminder information related to teaching classroom discipline (for example, students have body movements unrelated to teaching), but students do not have the permission to receive the classroom discipline reminder information. For example, in the virtual reality immersive psychological medical scene, psychologists have the authority to receive facial expression reminder information of patients during immersive psychological medical treatment, while patients do not have the authority to receive their own facial expression reminder information during immersive psychological medical treatment. Among them, the reminder information can also be graded according to the importance of the information to the virtual reality immersive scene and the users in the virtual reality immersive scene, so that users with different permissions receive reminder information of different importance levels, or important level information needs to be reminded to all users. For example, when a fire, earthquake, etc. threatens the users in the virtual reality immersive scene, the user's permissions can be ignored and a global broadcast reminder can be made to the users in the virtual reality immersive scene.

[0048] Optionally, after obtaining the type of virtual reality immersive scene and the user's authority, the reminder method can also be obtained according to the type of virtual reality immersive scene and reminder information. After wearing the VR device, in addition to being able to add a layer of different transparency (or an opaque layer) on the screen of the VR device to display the reminder information (or a reminder video or a reminder screen), the earpiece and micro-vibration device on the VR device can also be used to set different reminder methods. For example, in an immersive teaching scene, the teacher's head can be vibrated by the micro-vibration device on the VR device to prompt that there are students who raise their hands to answer questions during class. For example, in an immersive meeting scene, the earpiece device on the VR device plays the reminder information such as the duration of the meeting and the remaining time of the meeting to the conference host by voice. At the same time, it should be specially noted that the reminder method can also be a combination. For example, during the immersive teaching process, after the teacher asks a question, the reminder information that a student has raised his hand can be reminded by the vibration of the VR device or the earpiece on the VR device plays a voice to remind the teacher, and at the same time, the video screen of the student who raised his hand captured by the surveillance camera in the classroom is displayed on the top screen of the teacher's VR device.

[0049] The data processing method for augmented reality environment perception provided in the embodiment of the present application can enable the system to better understand the environment and situation of the user by obtaining the real scene information currently corresponding to the user, and help provide more personalized and targeted reminder information, so that the user can obtain a more real and more appropriate actual experience in the virtual reality immersive scene. In addition, the system can use the real scene information to monitor and remind the user's safety. For example, the system can send corresponding reminder information when it detects that the user is too close to the edge, fatigued, or there are other safety risks in the virtual reality scene to ensure the safety of the user. By prompting reminder information in the user's virtual reality immersive scene, the user can obtain key information without interrupting the current experience, which helps to improve the efficiency of the user's virtual reality immersive experience, reduce the need to switch tasks or devices, and enable the user to concentrate more and better complete the virtual reality immersive experience.

[0050] See also Figure 4 , Figure 4 Schematic diagram of a data processing device for augmented reality environment perception provided by an embodiment of the present application. Figure 4 As shown, the data processing device 20 for augmented reality environment perception includes a real scene information acquisition module 21, an environment information acquisition module 22, a reminder information generation module 23 and a reminder module 24.

[0051] The real scene information acquisition module 21 is used to obtain the real scene information currently corresponding to the user when the user is in a virtual reality immersive scene; the environmental information acquisition module 22 analyzes the real scene information to obtain the environmental information corresponding to the real scene information; the reminder information generation module 23 is used to generate reminder information based on the environmental information; the reminder module 24 is used to prompt the reminder information in the user's virtual reality immersive scene.

[0052] Among them, an intelligent collection terminal device is set in the scene where the user is currently located, and the real scene information acquisition module 21 is specifically used to collect audio, video, and text in the real environment through the intelligent collection terminal device to obtain the real scene information.

[0053] Among them, the environmental information acquisition module 22 is specifically used to analyze the audio, the video and the text to obtain the user's behavior information; analyze the audio, the video and the text to obtain the interference information appearing in the current scene; the behavior information and the interference information constitute the environmental information.

[0054] Among them, the reminder information generation module 23 is specifically used to analyze the environmental information to extract the user's behavioral characteristics and the scene characteristics of the current scene; based on the behavioral characteristics and / or the scene characteristics, determine whether the reminder condition is triggered; if so, generate reminder information based on the behavioral characteristics and / or the scene characteristics.

[0055] Among them, the reminder module 24 is specifically used to obtain the type of the virtual reality immersive scene and obtain the authority of the user; obtain the reminder method according to the type of the virtual reality immersive scene; based on the authority of the user, prompt the reminder information corresponding to the reminder method in the virtual reality immersive scene of the user with the reminder authority in the reminder method.

[0056] It should be noted that the above-mentioned data processing device for augmented reality environment perception can execute the data processing method for augmented reality environment perception provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the embodiment of the data processing device for augmented reality environment perception, please refer to the data processing method for augmented reality environment perception provided in the embodiment of the present application.

[0057] See also Figure 5 , Figure 5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device 30 includes: one or more processors 31 and a memory 32, Figure 5 A processor 31 is taken as an example.

[0058] The processor 31 and the memory 32 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0059] The memory 32 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the data processing method for augmented reality environment perception in the embodiment of the present application. The processor 31 executes various functional applications and data processing of the electronic device by running the non-volatile software programs, instructions and modules stored in the memory 32, that is, the data processing method for augmented reality environment perception in the above method embodiment is implemented.

[0060] The memory 32 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of a data processing device for augmented reality environment perception, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 32 may optionally include a memory remotely arranged relative to the processor 31, and these remote memories may be connected to the data processing device for augmented reality environment perception via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0061] The one or more modules are stored in the memory 32, and when executed by the one or more processors 31, the data processing method for augmented reality environment perception in any of the above method embodiments is executed.

[0062] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0063] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, servers, and other electronic devices with data interaction functions.

[0064] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Figure 5 A processor 31 in the embodiment may enable the one or more processors to execute the data processing method for augmented reality environment perception in any of the above method embodiments.

[0065] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device is enabled to execute the data processing method for augmented reality environment perception in any of the above-mentioned method embodiments.

[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method for augmented reality environment perception, characterized in that: include: When a user is in a virtual reality immersive scene, obtaining real scene information currently corresponding to the user; Analyzing the real scene information to obtain environment information corresponding to the real scene information; generating reminder information according to the environmental information; The reminder information is prompted in the user's virtual reality immersive scene.

2. The method according to claim 1, characterized in that: The scene currently located by the user is provided with an intelligent acquisition terminal device, and the obtaining of the real scene information currently corresponding to the user includes: The audio, video and text in the real environment are collected by the intelligent collection terminal device to obtain the real scene information.

3. The method according to claim 2, characterized in that The analyzing the real scene information to obtain environment information corresponding to the real scene information includes: Analyze the audio, video and text to obtain user behavior information; Analyze the audio, the video, and the text to obtain interference information appearing in the current scene; The behavior information and the interference information constitute the environmental information.

4. The method according to claim 1, characterized in that: The generating reminder information according to the environmental information includes: Analyzing the environmental information to extract the user's behavior characteristics and the scene characteristics of the current scene; Determining whether a reminder condition is triggered according to the behavior characteristics and / or the scene characteristics; If so, a reminder message is generated according to the behavior feature and / or the scene feature.

5. The method according to any one of claims 1 to 4, characterized in that: The prompting of the reminder information in the user's virtual reality immersive scene includes: Obtaining the type of the virtual reality immersive scene and obtaining the authority of the user; Obtaining a reminder method according to the type of the virtual reality immersive scene; Based on the authority of the user, the reminder information corresponding to the reminder method is prompted in the virtual reality immersive scene of the user with the reminder authority in the reminder method.

6. The method according to claim 2, characterized in that The obtaining of the real scene information corresponding to the user in the virtual reality immersive scene also includes: Collecting physiological information of the user through a sensor device worn by the user; Obtaining location information of the user; The real scene information is acquired according to the physiological information, the position information, and the audio, video, and text.

7. A data processing device for augmented reality environment perception, characterized in that: include: A real scene information acquisition module, used to acquire the real scene information currently corresponding to the user when the user is in a virtual reality immersive scene; An environmental information acquisition module analyzes the real scene information to obtain environmental information corresponding to the real scene information; A reminder information generating module, used to generate reminder information according to the environmental information; The reminder module is used to prompt the reminder information in the user's virtual reality immersive scene.

8. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

Images