Scene rendering method, device and electronic device based on extended reality device

Abstract

The present invention discloses a scene rendering method, device and electronic device based on an extended reality device. Involving the field of metaverse, the method comprises: receiving a connection request of a target extended reality device, identifying the target device information of the target extended reality device; based on an application program interface, calling a target software development kit matching the target device information from multiple software development kits; based on the device interface provided by the target software development kit, receiving and processing interactive data from the target extended reality device; based on the interactive data, constructing a virtual scene, and rendering the virtual scene through a graphics engine to obtain a rendering result. The present invention solves the technical problems of high difficulty and high time cost in developing scene rendering due to the wide variety of extended reality devices, each of which has its own exclusive software development kit application programming interface.

Description

Technical Field

[0001] The present invention relates to the field of metaverse, and in particular to a scene rendering method, device and electronic device based on an extended reality device. Background Art

[0002] There are currently multiple frameworks and engines for developing extended reality devices (such as augmented reality (AR) / virtual reality (VR)) applications in related technologies, and there are many types of AR / VR devices in related technologies. Each device has its own exclusive software development kit (SDK) and application programming interface (API). Developers need to learn and integrate these software development kits separately, which increases the difficulty and time cost of development. In addition, different devices have different data formats and interaction methods, and it is also challenging to uniformly manage and process the data of these devices.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a scene rendering method, device and electronic device based on an extended reality device, so as to at least solve the technical problems of high difficulty and time cost in developing scene rendering due to the wide variety of extended reality devices, each device having its own exclusive software development tool kit application programming interface.

[0005] According to one aspect of an embodiment of the present invention, there is provided a scene rendering method based on an extended reality device, comprising: receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device; based on an application program interface, calling a target software development kit that matches the target device information from multiple software development kits, wherein one application program interface is used to interact with multiple software development kits; based on a device interface provided by the target software development kit, receiving and processing interaction data from the target extended reality device, wherein the interaction data is data generated when an operation object of the target extended reality device interacts with the target extended reality device; based on the interaction data, constructing a virtual scene, and rendering the virtual scene through a graphics engine to obtain a rendering result.

[0006] According to another aspect of an embodiment of the present invention, a scene rendering device based on an extended reality device is also provided, including: an identification module, used to receive a connection request from a target extended reality device and identify target device information of the target extended reality device; a calling module, used to call a target software development kit matching the target device information from multiple software development kits based on an application program interface, wherein an application program interface is used to interact with multiple software development kits; a processing module, used to receive and process interaction data from the target extended reality device based on a device interface provided by the target software development kit, wherein the interaction data is data generated when an operating object of the target extended reality device interacts with the target extended reality device; a rendering module, used to construct a virtual scene based on the interaction data, and render the virtual scene through a graphics engine to obtain a rendering result.

[0007] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is further provided, wherein the non-volatile storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any one of the scene rendering methods based on an extended reality device.

[0008] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the scene rendering methods based on an extended reality device.

[0009] In an embodiment of the present invention, by receiving a connection request from a target extended reality device, target device information of the target extended reality device is identified; based on an application program interface, a target software development kit matching the target device information is called from multiple software development kits, wherein one application program interface is used to interact with multiple software development kits; based on a device interface provided by the target software development kit, interaction data from the target extended reality device is received and processed, wherein the interaction data is data generated when an operation object of the target extended reality device interacts with the target extended reality device; based on the interaction data, a virtual scene is constructed, and the virtual scene is rendered by a graphics engine to obtain a rendering result, thereby achieving the purpose of quickly calling a software development kit matching the extended reality device through a universal application program interface to efficiently develop a virtual scene, thereby achieving the technical effect of improving the efficiency of scene rendering development based on extended reality devices, and further solving the technical problems of high difficulty and high time cost in scene rendering development caused by the wide variety of extended reality devices, each of which has its own exclusive software development kit application programming interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a flow chart of a scene rendering method based on an extended reality device according to an embodiment of the present invention;

[0012] Figure 2 is an optional device integration schematic diagram according to an embodiment of the present invention;

[0013] Figure 3 is an optional system framework diagram according to an embodiment of the present invention;

[0014] Figure 4 is a flowchart of an optional scene rendering method based on an extended reality device according to an embodiment of the present invention;

[0015] Figure 5 is a schematic diagram of a scene rendering apparatus based on an extended reality device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0018] First, to facilitate understanding of the embodiments of the present invention, some terms or nouns involved in the present invention are explained below:

[0019] Augmented Reality (AR) refers to a technology that cleverly integrates virtual information with the real world and is applied to interactive experience.

[0020] Virtual Reality (VR) refers to a virtual world in which a three-dimensional space is created through computer simulation. Users interact with the virtual world through devices such as head-mounted displays.

[0021] Application Programming Interface (API) refers to a set of function definitions and protocols that allow different software programs to communicate with each other and exchange data.

[0022] A software development kit (SDK) is a collection of tools used to develop application software, usually including APIs, sample codes, and documentation.

[0023] There are currently multiple frameworks and engines for developing extended reality devices (such as augmented reality (AR) / virtual reality (VR)) applications in related technologies, such as Unity, Unreal Engine, ARKit, ARCore, etc. These frameworks and engines are powerful and provide a wealth of tools and libraries to help developers create high-quality AR / VR content. However, there are some limitations and shortcomings in the solutions in these related technologies, including:

[0024] Unity and Unreal Engine: These two engines are currently the most widely used AR / VR development tools, supporting multiple platforms and multiple devices. However, due to their complex architecture and high learning curve, developers need to invest a lot of time and effort to master them. At the same time, although the cross-platform development of these engines is convenient, they often require additional optimization and adjustments for different platforms.

[0025] ARKit and ARCore: are specifically designed for developing AR applications on specific operating system platforms. Although these SDKs perform well on their platforms, their cross-platform capabilities are limited, and developers need to write different codes for different platforms, which increases the development cost and maintenance difficulty of AR / VR.

[0026] Cross-platform development is complex: Although Unity and Unreal Engine support cross-platform development, due to differences in hardware and software on different platforms, developers need to optimize for each platform, making it difficult to ensure a consistent user experience. At the same time, the differences in SDKs and APIs on different platforms increase the complexity of development and maintenance.

[0027] Difficulty in integrating multiple devices: There are many types of AR / VR devices in related technologies. Each device has its own SDK and API, and developers need to learn and integrate these SDKs separately, which increases the difficulty and time cost of development. In addition, different devices have different data formats and interaction methods, and it is also challenging to uniformly manage and process the data of these devices.

[0028] High-performance rendering and real-time interaction are difficult: AR / VR applications have high requirements for rendering performance and real-time interaction, especially at high resolution and high frame rate. Although the framework in related technologies provides certain optimization methods, in order to achieve high-performance rendering and smooth interaction, developers still need to have in-depth graphics programming knowledge and optimization skills. This not only increases the difficulty of development, but also leads to an increase in development costs.

[0029] Steep learning curve: The AR / VR development frameworks and engines in related technologies are powerful, but their complex functions and architectures also make the learning curve steep. Developers need to spend a lot of time learning how to use these tools and need to deeply understand the underlying implementation details, which is a big challenge for beginners and small development teams.

[0030] In response to the above problems, an embodiment of the present invention provides a method embodiment of scene rendering based on an extended reality device. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Figure 1 is a flow chart of a scene rendering method based on an extended reality device according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0032] Step S102, receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device;

[0033] Optionally, when the extended reality device attempts to establish a connection with the application framework, by identifying device information such as the type, model, and features of the device, it can ensure that it can respond appropriately to different devices, load the corresponding software development kit (SDK), and optimize the specific hardware and software features of the device to provide the best performance and user experience. After identifying the device information, it can interact with multiple SDKs through a unified application program interface (API), which can simplify the difficulty of device integration and operation and improve development efficiency. By identifying the target device information, dynamic device management can be achieved, such as monitoring the device connection status and real-time processing of device data. During the operation of the application, it can also respond to device changes in a timely manner, such as device disconnection or the addition of new devices.

[0034] In an optional embodiment, there are multiple target extended reality devices, and the connection requests corresponding to the multiple target extended reality devices are transmitted based on the real-time communication protocol. The device types of the multiple target extended reality devices are the same or different.

[0035] Optionally, this embodiment can support processing connection requests from multiple target extended reality devices and accessing multiple AR / VR devices at the same time, regardless of whether the types of these devices are the same. For example, in a VR application, multiple users can use different head displays at the same time, or in an AR meeting, participants can use different types of AR devices, that is, through the network communication module, multiple people can enter the same virtual environment at the same time to interact. This embodiment can also support multi-person collaboration, using real-time communication protocols to achieve low-latency voice and video communications; support multi-person collaborative virtual meetings, teaching and entertainment applications; support multiple different types of extended reality devices to access simultaneously.

[0036] Among them, extended reality devices can include AR devices, VR devices, etc.; real-time communication protocols include Web Real-Time Communication (WebRTC), which is an open source project that provides a set of APIs for implementing real-time audio, video communication and data transmission. WebRTC allows web browsers to communicate directly without relying on any plug-ins or additional software. It supports peer-to-peer communication (P2P) and data channels, and can be used to build real-time interactive applications such as video chat, online meetings, file transfers, and multiplayer games.

[0037] Step S104, based on an application program interface, calling a target software development kit matching the target device information from multiple software development kits, wherein one application program interface is used to interact with multiple software development kits;

[0038] Optionally, the application programming interface (API) serves as a unified interface between developers and various software development kits (SDKs), so that developers do not need to interact directly with the SDK of each device. The complexity of development can be greatly reduced. Developers only need to learn and use this unified API instead of the API unique to each device. After identifying the device information of the target extended reality device, the specific SDK that matches the device information can be automatically selected from multiple available SDKs to call. Ensure that the application can fully utilize the specific functions and optimizations of the device to provide the best performance and user experience. The above method can not only support a single type of device, but also seamlessly support multiple different types and brands of extended reality devices. The ability to support multiple devices can expand the audience range of the application and improve its market competitiveness. Interact with multiple SDKs through the application programming interface to simplify the device integration process. Developers do not need to write additional integration code for each device, reducing development and testing time.

[0039] In an optional embodiment, based on an application program interface, a target software development kit matching the target device information is called from multiple software development kits, including: identifying a device identifier and / or a device type of a target extended reality device from the target device information; and calling the target software development kit from multiple software development kits based on the device identifier and / or the device type of the target extended reality device, wherein the multiple software development kits correspond to different device identifiers and / or device types.

[0040] Optionally, this embodiment can accurately determine which specific device the connection request comes from and which type of device it is, whether it is a VR headset or AR glasses. The device identification can be used to track and manage the status of each device; and the device type is used to determine which SDK to call. Based on the identified device identification and type, the matching SDK can be automatically selected and called from multiple preset software development kits. Different devices will interact using the SDK that best suits them, without the need for developers to manually specify or configure, simplifying the development process. By using an API that can identify device identification and type, a wider range of devices can be supported, not just the same type or brand, thereby improving device compatibility and application portability.

[0041] Step S106, receiving and processing interaction data from the target extended reality device based on the device interface provided by the target software development kit, wherein the interaction data is data generated when the operation object of the target extended reality device interacts with the target extended reality device;

[0042] Optionally, the target software development kit can provide a standardized device interface for the application, so that developers can access and control different extended reality devices through a unified interface, regardless of which manufacturer these devices come from. It can simplify the developer's work, without the need to write specific code for each device, reducing the complexity of development and maintenance. Through the device interface, you can receive interaction data from the target extended reality device. The data may include but is not limited to user gestures, head movements, voice commands, etc. By processing the data and parsing the user's intent, it can ensure that the application can respond to the user's actions in a timely manner and provide a smooth interactive experience. Different extended reality devices send interaction data in different formats. The SDK receives this data through the device interface and converts it into a unified format, ensuring that the application can process this data consistently without considering the differences between devices.

[0043] In an optional embodiment, when the interaction data includes at least one of visual data, sound data, tactile data, position data, motion data, physiological data, and environmental perception data, receiving and processing the interaction data from the target extended reality device based on the device interface provided by the target software development kit includes at least one of the following:

[0044] Receive and process visual data from a target extended reality device based on a visual API provided by a target software development kit, wherein the visual API includes at least one of the following: a gesture recognition API, a facial expression recognition API, and an eye tracking API;

[0045] Receive and process sound data from a target extended reality device based on a sound recognition API and / or a spatial audio API provided by a target software development kit;

[0046] Receive and process tactile data from the target extended reality device based on the tactile feedback API provided by the target SDK;

[0047] Receive and process location data from a target XR device based on a location tracking API provided by the target SDK;

[0048] Receive and process motion data from the target extended reality device based on the motion capture API provided by the target software development kit;

[0049] Receive and process physiological data from the target extended reality device based on the physiological signal API provided by the target software development kit;

[0050] Receive and process environmental perception data from a target extended reality device based on an environmental recognition API and / or a light detection API provided by a target software development kit;

[0051] Among them, the device interface includes at least one of the following: visual API, sound recognition API, spatial audio API, tactile feedback API, position tracking API, motion capture API, physiological signal API, environment recognition API, and light detection API.

[0052] Optionally, in an AR / VR application framework that processes multimodal user input information, different data types usually correspond to different API interfaces in order to more effectively receive, parse, and process this information. The following are the API interfaces that can correspond to visual data, sound data, tactile data, position data, motion data, physiological data, and environmental perception data:

[0053] Visual data refers to data obtained from the device's camera or sensor, including gesture recognition, facial expressions, eye tracking, etc.; the corresponding API interfaces may include but are not limited to: gesture recognition API, facial expression recognition API, eye tracking API, among which the gesture recognition API is used to provide gesture detection and recognition functions, such as recognizing waving, grabbing and other actions; the facial expression recognition API is used to capture and interpret the user's facial expressions, such as smiling, frowning, etc.; the eye tracking API tracks the user's eye movements to determine the direction of sight or identify specific eye movements.

[0054] Sound data involves functions such as speech recognition and ambient sound detection; tactile data involves physical interaction information captured by devices such as handles and tactile gloves; the corresponding API interfaces may include but are not limited to: speech recognition API, spatial audio API, among which the speech recognition API is used to receive audio from the microphone, recognize the user's voice commands, and convert them into text or control signals; the spatial audio API is used to process sounds from different directions to provide 3D sound effects and immersive audio experience.

[0055] The location data is provided by the location tracking system of the AR / VR device, including the user's current location and direction; the corresponding API interface may include but is not limited to: a location tracking API, wherein the location tracking API is used to provide the user's precise location and direction data for locating the user in the virtual environment.

[0056] The motion data involves the user's body movements, such as head rotation, hand movement, etc.; the corresponding API interface may include but is not limited to: motion capture API, where the motion capture API is used to receive sensor data, track and interpret the user's movements, such as head posture and hand posture.

[0057] Physiological data involves physiological signals such as heart rate and brain waves obtained through sensors; the corresponding API interfaces may include but are not limited to: physiological signal AP, which is used to receive data from physiological sensors and analyze the user's physiological state, such as tension, emotions, etc., to personalize the interactive experience.

[0058] Environmental perception data obtains data about the surrounding environment through the device's sensors or cameras, and supports information overlay in AR applications. The corresponding API interfaces may include but are not limited to: environmental recognition API and light detection API. The environmental recognition API is used to analyze the camera's video stream and identify objects in the environment, such as planes and object boundaries, for precise positioning and overlay of AR content. The light detection API is used to detect ambient light and adjust the lighting effect of the virtual scene to better integrate it with the real environment.

[0059] Optionally, the above API interface can not only improve the efficiency and accuracy of data processing by specifically processing specific types of data, but also enhance the multimodal input processing capabilities of the application. This enables AR / VR applications to interact with users and the real environment more naturally, providing a richer, immersive and personalized experience. At the same time, the design of the API interface can simplify the work of developers, allowing them to focus more on the core functions of the application and user experience design.

[0060] Step S108, constructing a virtual scene based on the interactive data, and rendering the virtual scene through a graphics engine to obtain a rendering result.

[0061] Optionally, the application can adjust the content or behavior in the virtual scene in real time based on the received interaction data, such as gestures, voice, position movement, etc. By closely linking the interaction data with the virtual scene construction, the application can create a virtual environment that interacts with the user more naturally and smoothly. The construction of the virtual scene not only relies on the preset 3D model and environment, but can also change dynamically according to the user's real-time input to provide a more personalized and realistic experience. The graphics engine is responsible for efficiently rendering the constructed virtual scene, ensuring high resolution and high frame rate on different devices, thereby achieving smooth and delicate visual effects.

[0062] Optionally, the graphics engine can be built based on the underlying architecture of Java and Open Graphics Library (OpenGL) / Vulkan to ensure efficient graphics rendering performance and meet the requirements of AR / VR applications for high resolution and high frame rate.

[0063] It should be noted that Java's cross-platform features combined with the broad hardware support of OpenGL or Vulkan can build a graphics engine that can run efficiently on multiple operating systems and devices. This combination can simplify the cross-platform development of AR / VR applications and reduce the work of rewriting code, thus becoming a technological innovation.

[0064] In an optional embodiment, after constructing a virtual scene based on the interaction data and rendering the virtual scene through a graphics engine to obtain a rendering result, the method further includes: determining current behavior data of the operation object in the current rendering stage based on the interaction data; obtaining predicted action data of the operation object based on the current behavior data, wherein the predicted action data is used to indicate the action performed by the operation object in the next rendering stage, and the next rendering stage is the rendering stage after the current rendering stage; and adjusting the virtual scene based on the action data to obtain an updated rendering result.

[0065] Optionally, the application analyzes the behavior of the user's operating object based on the current interaction data and predicts the actions that the user may perform in the next rendering stage. This prediction capability enables the application to prepare in advance to ensure the smoothness of the rendering process. By predicting the action data, the application can dynamically adjust the content and layout in the virtual scene to adapt to the user's upcoming actions. Predicted action data can help the application optimize the rendering strategy of the graphics engine, such as pre-allocating resources, optimizing lighting calculations, etc., thereby improving rendering efficiency and reducing unnecessary calculations and resource waste.

[0066] It should be noted that behavioral data may include but is not limited to gestures, voice commands, and head and eye movement trajectories; action data may refer to the action or requirement to be performed next.

[0067] In an optional embodiment, based on current behavior data, predicted action data of the operation object is obtained, including: performing feature extraction on the current behavior data to obtain current action features of the operation object in the current rendering stage; obtaining historical action features of the operation object in the historical rendering stage, wherein the historical rendering stage is the rendering stage before the current rendering stage; based on the current action features and the historical action features, an action prediction model is used to obtain predicted action data, wherein the action prediction model is based on multiple sets of action features and is obtained through machine learning.

[0068] Optionally, the application first extracts the action features of the operation object from the current interaction data, and then combines them with the historical action features to provide more comprehensive information for the prediction model. By collecting and analyzing the action features of the operation object in the historical rendering phase, the application can learn the user's action patterns and habits. Based on the feature extraction and historical data, the application uses a pre-trained machine learning model to predict the action that the operation object will perform in the next rendering phase. This prediction enables the application to respond to user actions more intelligently by considering the continuity of the action and the user's intention.

[0069] Among them, action features refer to the cleaning, formatting and preprocessing of the original behavioral data, such as noise removal, smoothing gesture trajectories, speech recognition, etc., for subsequent analysis, and extracting key features from the preprocessed data, such as the shape, speed, direction of the gesture, the tilt angle of the head, the gaze point of the eyes, etc.

[0070] It should be noted that, for multiple groups of action features, each group of action features includes action features corresponding to N sampling moments. The action features of the first N-1 sampling moments are used as model input, and the action features of the left and last sampling moment are used as output. The initial model is trained to obtain the action prediction model.

[0071] In an optional embodiment, the graphics engine is based on Java language and graphics application program interface encapsulation.

[0072] Optionally, the combination of the cross-platform characteristics of the Java language and the graphics API enables the graphics engine to run on different operating systems and hardware platforms without rewriting a large amount of platform-related code. This can solve the problem of cross-platform development in current AR / VR development, reduce the workload of developers, and improve development efficiency. By calling high-performance graphics APIs such as OpenGL or Vulkan through Java, consistent and efficient graphics rendering performance can be achieved on multiple platforms. This encapsulation method can utilize the advanced features of Java while maintaining high performance of graphics rendering, meeting the high requirements of AR / VR applications for rendering quality and frame rate. In the architectural design of the graphics engine, the introduction of the device driver layer and the device management interface can simplify the access and operation of AR / VR devices and reduce the complexity and cost of device integration.

[0073] Through the above steps S102 to S108, consistency and high performance can be achieved on multiple platforms; support for multiple AR / VR devices is provided to simplify device access and operation; advanced graphics rendering technology is used to ensure high-quality graphics output and smooth user experience; through modular design, developers can flexibly select and expand functional modules as needed.

[0074] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation of a scene rendering method based on an extended reality device. Figure 2 is a schematic diagram of an optional device integration provided according to an embodiment of the present invention, Figure 3 is an optional system framework diagram provided according to an embodiment of the present invention. The method can be applied to Figure 2 and Figure 3 The system architecture shown, wherein the usage scenarios of this embodiment may include but are not limited to AR / VR application development in the fields of game development, industrial design, medical training, etc., and the devices involved include but are not limited to AR / VR head-mounted devices.

[0075] like Figure 3 As shown in the figure, the overall framework of the system architecture includes:

[0076] Underlying architecture: The underlying rendering architecture based on Java and OpenGL / Vulkan ensures high-performance graphics rendering across platforms.

[0077] Device driver layer: Integrates software development kits (SDKs) for various AR / VR devices, provides a unified device interface layer, and simplifies device access and operation.

[0078] Middleware layer: provides commonly used AR / VR functional modules, such as gesture recognition, voice control, 3D sound effects, physics engine, etc. Developers can choose according to their needs.

[0079] High-level API: Provides an easy-to-use high-level API to support rapid development and prototyping while taking into account flexibility and scalability.

[0080] It should be noted that the device management module is responsible for identifying and managing various extended reality devices connected to the system, which may include but are not limited to AR / VR head-mounted devices, handles, gloves, cameras, etc. By detecting the connection status of the device, the corresponding device driver and SDK are automatically loaded, and a unified device interface is provided to simplify the access and operation process of the device and process the interactive data from the device; the core engine is responsible for basic functions such as graphics rendering, physical simulation, and sound processing. Based on high-performance graphics libraries (such as OpenGL or Vulkan), high-quality 3D graphics rendering is achieved to ensure the presentation effect of virtual scenes on various devices; the application layer interface allows developers to interact with the core engine and device management module in a simple and easy-to-understand way by providing a set of high-level APIs; the input and output module is responsible for interacting with the user, including receiving the user's input data and outputting visual and sound feedback to the user. The input data can come from a variety of devices and sensors, such as gesture recognition, voice input, location tracking, etc. The output module presents the rendered virtual scene and sound effects to the user through the device's display and speakers, providing an immersive visual and auditory experience. The input and output module is a bridge connecting the user and the virtual world, ensuring that the user can interact with the virtual environment intuitively and naturally.

[0081] Figure 4 is a flowchart of an optional scene rendering method based on an extended reality device according to an embodiment of the present invention. Figure 4 As shown, the method includes:

[0082] S1, initialization framework, including:

[0083] S11, load the framework core modules, including the graphics engine, device manager, input processing module, etc.

[0084] S12, initializing the graphics engine, setting rendering parameters and rendering environment.

[0085] S13, initialize the device interface, detect and connect to supported AR / VR devices.

[0086] S2, device access, including:

[0087] S21, automatically detects connected AR / VR devices and obtains device model and specification information.

[0088] S22, according to the device type, load the corresponding driver and SDK to initialize the device.

[0089] S23, obtaining the interaction data of the device (such as sensor data and user input information, etc.) through a unified device interface.

[0090] S3, scene construction, including:

[0091] S31, use the SDK API to build a virtual scene, including loading 3D models, materials, lights and sound effects, etc.

[0092] S32 supports dynamic loading and switching of scenes, making it easier for developers to manage scenes.

[0093] S4, interaction design, specifically includes:

[0094] S41, design user interaction methods, including gesture recognition, voice control and motion capture.

[0095] S42, using sensor data to achieve interaction with virtual objects, such as grabbing, rotating, scaling and other operations.

[0096] S43 supports multiple input devices, such as handles, gloves, cameras, etc., providing a diverse interactive experience.

[0097] S5, rendering and display, including:

[0098] S51, calls the graphics engine to render the scene, supporting real-time lighting and shadow effects.

[0099] S52, optimizes rendering performance to ensure smooth operation on different devices.

[0100] S53 outputs the rendering result to the display screen of the AR / VR device to provide an immersive visual experience.

[0101] S6, real-time adjustment, including:

[0102] S61, adjusts the virtual scene and interaction mode in real time according to user input and device feedback.

[0103] S62, dynamically adjusts rendering parameters to optimize graphics quality and performance.

[0104] S63 supports real-time editing and adjustment of scenes, making it easier for developers to debug and optimize.

[0105] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0106] In this embodiment, a scene rendering device based on an extended reality device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the terms "module" and "device" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the above-mentioned scene rendering method based on an extended reality device. Figure 5 is a structural schematic diagram of a scene rendering device based on an extended reality device according to an embodiment of the present invention. Figure 5 As shown, the above-mentioned scene rendering device based on the extended reality device includes: an identification module 200, which is used to receive a connection request from a target extended reality device and identify the target device information of the target extended reality device; a calling module 202, which is connected to the identification module 200, and is used to call a target software development kit matching the target device information from multiple software development kits based on an application interface, wherein an application interface is used to interact with multiple software development kits; a processing module 204, which is connected to the calling module 202, and is used to receive and process interaction data from the target extended reality device based on a device interface provided by the target software development kit, wherein the interaction data is data generated when the operation object of the target extended reality device interacts with the target extended reality device; a rendering module 206, which is connected to the processing module 204, and is used to construct a virtual scene based on the interaction data, and render the virtual scene through a graphics engine to obtain a rendering result.

[0108] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0109] It should be noted that the identification module 200, the calling module 202, the processing module 204, and the rendering module 206 correspond to steps S102 to S108 in the embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the modules as part of the device can be run in a computer terminal.

[0110] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in the embodiment, which will not be repeated here.

[0111] The above-mentioned scene rendering device based on the extended reality device may also include a processor and a memory. The above-mentioned identification module 200, calling module 202, processing module 204, rendering module 206, etc. are all stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to realize the corresponding functions.

[0112] The processor includes a kernel, which retrieves the corresponding program module from the memory. The kernel may be one or more. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0113] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any of the above-mentioned scene rendering methods based on an extended reality device.

[0114] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group, and the non-volatile storage medium includes a stored program.

[0115] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device; based on an application program interface, calling a target software development kit that matches the target device information from multiple software development kits, wherein one application program interface is used to interact with multiple software development kits; based on a device interface provided by the target software development kit, receiving and processing interaction data from the target extended reality device, wherein the interaction data is data generated when an operating object of the target extended reality device interacts with the target extended reality device; based on the interaction data, constructing a virtual scene, and rendering the virtual scene through a graphics engine to obtain a rendering result.

[0116] According to an embodiment of the present application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein when the program is run, any of the above-mentioned scene rendering methods based on an extended reality device is executed.

[0117] According to an embodiment of the present application, an embodiment of a computer program product is also provided. When executed on a data processing device, the program is suitable for executing the steps of initializing any one of the above-mentioned scene rendering methods based on an extended reality device.

[0118] Optionally, the above-mentioned computer program product, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device; based on an application program interface, calling a target software development kit that matches the target device information from multiple software development kits, wherein an application program interface is used to interact with multiple software development kits; based on a device interface provided by the target software development kit, receiving and processing interaction data from the target extended reality device, wherein the interaction data is data generated when an operating object of the target extended reality device interacts with the target extended reality device; based on the interaction data, constructing a virtual scene, and rendering the virtual scene through a graphics engine to obtain a rendering result.

[0119] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device; based on an application program interface, calling a target software development kit that matches the target device information from multiple software development kits, wherein an application program interface is used to interact with multiple software development kits; based on a device interface provided by the target software development kit, receiving and processing interaction data from the target extended reality device, wherein the interaction data is data generated when an operation object of the target extended reality device interacts with the target extended reality device; based on the interaction data, constructing a virtual scene, and rendering the virtual scene through a graphics engine to obtain a rendering result.

[0120] The above sequence of the embodiments of the present invention is for description only and does not represent the superiority or inferiority of the embodiments.

[0121] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above modules can be a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, modules or indirect coupling or communication connection of modules, which can be electrical or other forms.

[0123] The modules described above as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0125] If the above-mentioned integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0126] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A scene rendering method based on an extended reality device, characterized in that: include: Receiving a connection request from a target extended reality device, and identifying target device information of the target extended reality device; Based on an application program interface, calling a target software development kit matching the target device information from a plurality of software development kits, wherein the one application program interface is used to interact with the plurality of software development kits; Based on the device interface provided by the target software development kit, receiving and processing interaction data from the target extended reality device, wherein the interaction data is data generated when an operation object of the target extended reality device interacts with the target extended reality device; Based on the interactive data, a virtual scene is constructed, and the virtual scene is rendered by a graphics engine to obtain a rendering result.

2. The method according to claim 1, characterized in that The method of calling a target software development kit matching the target device information from a plurality of software development kits based on an application program interface includes: Identify a device identifier and / or a device type of the target extended reality device from the target device information; Based on the device identification and / or device type of the target extended reality device, the target software development kit is called from the multiple software development kits, wherein the multiple software development kits correspond to different device identifications and / or device types.

3. The method according to claim 1, characterized in that In the case where the interaction data includes at least one of visual data, sound data, tactile data, position data, motion data, physiological data, and environmental perception data, the receiving and processing of the interaction data from the target extended reality device based on the device interface provided by the target software development kit includes at least one of the following: Based on a visual API provided by the target software development kit, receiving and processing the visual data from the target extended reality device, wherein the visual API includes at least one of the following: a gesture recognition API, a facial expression recognition API, and an eye tracking API; Based on a sound recognition API and / or a spatial audio API provided by the target software development kit, receiving and processing the sound data from the target extended reality device; Based on a haptic feedback API provided by the target software development kit, receiving and processing the haptic data from the target extended reality device; Receiving and processing the location data from the target extended reality device based on a location tracking API provided by the target software development kit; Based on the motion capture API provided by the target software development kit, receiving and processing the motion data from the target extended reality device; Based on the physiological signal API provided by the target software development kit, receiving and processing the physiological data from the target extended reality device; Based on the environment recognition API and / or light detection API provided by the target software development kit, receiving and processing the environment perception data from the target extended reality device; Among them, the device interface includes at least one of the following: the visual API, the sound recognition API, the spatial audio API, the tactile feedback API, the position tracking API, the motion capture API, the physiological signal API, the environment recognition API, and the light detection API.

4. The method according to claim 1, characterized in that: After constructing a virtual scene based on the interactive data and rendering the virtual scene by a graphics engine to obtain a rendering result, the method further includes: Based on the interaction data, determining current behavior data of the operation object in the current rendering phase; Based on the current behavior data, obtaining predicted action data of the operation object, wherein the predicted action data is used to indicate an action performed by the operation object in a next rendering stage, and the next rendering stage is a rendering stage after the current rendering stage; Based on the action data, the virtual scene is adjusted to obtain an updated rendering result.

5. The method according to claim 4, characterized in that The obtaining predicted action data of the operation object based on the current behavior data includes: Performing feature extraction on the current behavior data to obtain current action features of the operation object at the current rendering stage; Acquire historical action features of the operation object in a historical rendering stage, wherein the historical rendering stage is a rendering stage before the current rendering stage; Based on the current action features and the historical action features, an action prediction model is used to obtain the predicted action data, wherein the action prediction model is obtained through machine learning based on multiple groups of action features.

6. The method according to any one of claims 1 to 5, characterized in that There are multiple target extended reality devices, and the connection requests corresponding to the multiple target extended reality devices are transmitted based on a real-time communication protocol. The device types of the multiple target extended reality devices are the same or different.

7. The method according to any one of claims 1 to 5, characterized in that The graphic engine is based on Java language and graphic application program interface encapsulation.

8. A scene rendering device based on an extended reality device, characterized in that: include: An identification module, configured to receive a connection request from a target extended reality device and identify target device information of the target extended reality device; A calling module, used for calling a target software development kit matching the target device information from a plurality of software development kits based on an application program interface, wherein the one application program interface is used for interacting with the plurality of software development kits; A processing module, configured to receive and process interaction data from a target extended reality device based on a device interface provided by the target software development kit, wherein the interaction data is data generated when an operation object of the target extended reality device interacts with the target extended reality device; The rendering module is used to construct a virtual scene based on the interactive data, and render the virtual scene through a graphics engine to obtain a rendering result.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the scene rendering method based on an extended reality device as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the scene rendering method based on the extended reality device as described in any one of claims 1 to 7.

Images