Adaptive sizing of operatable and readable objects

Abstract

The invention relates to image rendering. The electronic device generates information of the virtual object for display with contextual content from the perspective of the camera. The virtual object has an object position and an object size, and the camera has a camera position that enables a viewing angle of the camera. And the electronic equipment determines the distance between the virtual object and the camera according to the object position and the camera position, and adjusts the object size of the virtual object according to the distance between the virtual object and the camera. The contextual content and a virtual object are rendered, the virtual object being displayed at the object location in a scene associated with the contextual content with the adjusted object size.

Description

Technical Field

[0001] The present application relates to image processing technology, including but not limited to, a method, system, device and non-transitory computer-readable medium for selectively and adaptively rendering objects in a mixed, virtual or augmented reality environment. Background Art

[0002] Detect, track, or render objects in various mixed, virtual, or augmented reality applications. For example, in augmented reality (AR) applications executed on mobile phones, objects are rendered and integrated in real time based on natural features. Virtual object rendering, tracking, and manipulation have become essential functions for many video applications implemented in many technology fields, such as human-computer interaction, virtual reality (VR), games, and video surveillance. Most visual content applications follow the basic principle of perspective drawing to render objects, and the rendered size of the object decreases as the distance from the camera increases. If the object is too close or too far from the camera, the object rendered by perspective drawing may occupy too large or too small a portion of the screen. A more efficient object rendering mechanism than current practices would be beneficial. Summary of the invention

[0003] Various embodiments of the present application are intended to render a virtual object with a display object size, and to adaptively control the display object size according to the distance between the virtual object and the camera in the extended reality environment. The virtual object is associated with a bounding box, which includes an interaction box surrounding the virtual object, and the user can operate the virtual object through the interaction box of the bounding box. The interaction box includes multiple bounding box elements to facilitate the user to operate the virtual object. According to the distance between the virtual object and the camera position, the bounding box element is adaptively expanded or contracted within a predefined size range. Through these methods, when the user operates the virtual object through the bounding box element, the bounding box element is guaranteed to be displayed in the field of view at a reasonable object size, which helps to provide users with a reliable experience when viewing and selecting virtual objects.

[0004] According to one aspect, a method for rendering visual content is performed at an electronic device. The method includes generating information of a virtual object to be displayed together with contextual content from a camera's perspective (i.e., from the camera's viewing point). The virtual object has an object position and an object size, and the camera has a camera position that enables the camera's perspective. The method also includes determining a distance of the virtual object from the camera based on the object position and the camera position, and adjusting the object size of the virtual object based on the distance of the virtual object from the camera. The method also includes rendering the contextual content and the virtual object, and the virtual object is displayed at the object position with the adjusted object size in a scene associated with the contextual content.

[0005] In some embodiments, adjusting the object size of the virtual object includes: determining the angular size of the virtual object according to the object size and the distance of the virtual object from the camera, and controlling the angular size of the virtual object on the display within an angular size range. In some embodiments, the virtual object is displayed on the display of the electronic device at a display object size, and the object size of the virtual object is adjusted so that the ratio of the display object size to the display size of the display is controlled within a display ratio range.

[0006] According to another aspect, some embodiments include an electronic device comprising one or more processors and a memory, wherein the memory stores instructions, and the instructions are executed by the one or more processors to cause the one or more processors to perform any of the methods described above.

[0007] According to yet another aspect, some embodiments include a non-transitory computer-readable medium having instructions stored therein, the instructions being executed by one or more processors to cause the one or more processors to perform any of the methods described above.

[0008] These illustrative embodiments and implementations are mentioned not to limit or define the present disclosure, but to provide examples to aid understanding. Other embodiments are discussed in the detailed description, and further description is provided there. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a better understanding of the various described embodiments, please refer to the following detailed description in conjunction with the following drawings, wherein like reference numerals refer to like parts throughout the figures.

[0010] Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices in accordance with some embodiments.

[0011] Figure 2 is a schematic diagram illustrating an electronic system according to some embodiments.

[0012] Figure 3 is a flow chart of a process for processing inertial sensor data and image data of an electronic system using a simultaneous localization and mapping (SLAM) module according to some embodiments.

[0013] Figure 4A is an image of an object (eg, a moving vehicle) rendered at different distances according to a one-point perspective in some embodiments, Figure 4B is another image of an object displayed with the display object size controlled within a display size range according to some embodiments.

[0014] Figure 5 is a perspective view of a bounding box of an object according to some embodiments.

[0015] Figure 6 It is the process of adaptively adjusting the angular size or display size of an object according to some embodiments.

[0016] Figure 7 is a flow chart of a method of rendering visual content according to some embodiments.

[0017] Like reference numerals refer to corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0018] Specific embodiments will now be described in detail, examples of which are shown in the accompanying drawings. In the following detailed description, many non-limiting specific details are listed to assist in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter presented herein may be implemented without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented at a variety of electronic systems having digital video capabilities.

[0019] Extended reality includes augmented reality (AR) in which virtual objects are superimposed on a view of the real physical world, virtual reality (VR) that contains only virtual content, and mixed reality (MR) that combines AR and VR and allows users to interact with real-world objects and virtual objects. More specifically, AR is an interactive experience of a real-world environment in which objects residing in the real world are enhanced by computer-generated perceptual information, such as visual, auditory, tactile, somatosensory, and olfactory perceptions across multiple perceptual modalities. In various embodiments of the present application, a virtual object is rendered in AR and associated with a bounding box, which includes an interactive box surrounding the virtual object, and the user can operate the virtual object through the interactive box of the bounding box. The interactive box includes multiple bounding box elements to facilitate the user to operate the virtual object. Depending on the distance between the virtual object and the camera position, the bounding box elements are adaptively expanded or contracted within a predefined object size range. Through these methods, it is ensured that the bounding box elements are displayed in the field of view at a reasonable size, so that the user can view and select the corresponding object.

[0020] Figure 1 1 is an example data processing environment 100 according to some embodiments, having one or more servers 102 communicatively coupled to one or more client devices 104. One or more client devices 104 may be, for example, a laptop 104A, a tablet 104B, a mobile phone 104C, or an intelligent, multi-sensing and networked home device (e.g., a surveillance camera 104E, a smart TV device, a drone). In some embodiments, one or more client devices 104 include a head-mounted display 104D for rendering extended reality content. Each client device 104 may collect data or user input, execute a user application, and display output on its user interface. The collected data or user input may be processed locally on the client device 104 and / or remotely by the server 102. One or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client device 104, and in some embodiments, process data and user input received from the client device 104 when the user application is executed on the client device 104. In some embodiments, data processing environment 100 also includes storage 106 for storing data related to server 102, client device 104, and applications executing on client device 104. For example, storage 106 may store video content (including visual content and audio content), static visual content, and / or inertial sensor data.

[0021] One or more servers 102 can implement real-time data communication with client devices 104 that are remote from each other or from one or more servers 102. In addition, in some embodiments, one or more servers 102 can perform data processing tasks that the client device 104 cannot or does not tend to complete locally. For example, the client device 104 includes a game console (e.g., composed of a head-mounted display 104D) that executes an interactive online game application. The game console receives user instructions and sends the user instructions together with user data to the game server 102. The game server 102 generates a video data stream based on the user instructions and user data, and provides the video data stream to the game console and other client devices in the same game session with the game console for display. For another example, the client device 104 includes a network surveillance camera 104E and a mobile phone 104C. The network surveillance camera 104E collects video data and streams the video data to the surveillance camera server 102 in real time. When the video data is optionally pre-processed on the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion events or audio events in the video data, and shares information of these events to the mobile phone 104C, thereby enabling the user of the mobile phone 104C to remotely monitor events occurring near the network surveillance camera 104E in real time.

[0022] The one or more servers 102, the one or more client devices 104, and the storage 106 are communicatively coupled to each other via one or more communication networks 108, which are the medium used to provide communication links between these devices and computers connected together in the data processing environment 100. The one or more communication networks 108 may include connections such as wired, wireless communication links, or fiber optic cables. Examples of the one or more communication networks 108 include a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. Optionally, one or more communication networks 108 are implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, universal serial bus (USB), FireWire, long term evolution (LTE), global system for mobile communications (GSM), enhanced data GSM environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over internet protocol (VoIP), world interoperability for microwave access (Wi-MAX), or any other applicable communication protocol. The connection to one or more communication networks 108 can be established directly (e.g., using 3G / 4G to connect to a wireless bearer), or through a network interface 110 (e.g., a router, switch, gateway, hub, or smart dedicated whole home control node), or through any combination of the above. Thus, one or more communication networks 108 may represent the Internet, a global collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite to communicate with each other. At its core, the Internet is a backbone of high-speed data communication lines between major nodes or host computers, including thousands of commercial, government, educational, and other electronic systems that route data and messages.

[0023] The head-mounted display 104D (also known as AR glasses 104D) includes one or more cameras (e.g., visible light cameras, depth cameras), microphones, speakers, one or more inertial sensors (e.g., gyroscopes, accelerometers) and displays. The camera and microphone are used to obtain video and audio data from the scene of the AR glasses 104D, while the one or more inertial sensors are used to obtain inertial sensor data. In some cases, the camera obtains the gestures of the user wearing the AR glasses 104D. In some cases, the microphone records the ambient sound including the user's voice commands. In some cases, the video or static visual data obtained by the visible light camera and the inertial sensor data measured by one or more inertial sensors are used to determine and predict the device posture (i.e., the device position and orientation). The video, static image, audio or inertial sensor data obtained by the AR glasses 104D can be processed by at least one of the AR glasses 104D and the server 102 to identify the device posture. Optionally, in some embodiments, the depth data (e.g., depth map and confidence map) obtained by the depth camera and the inertial sensor data measured by one or more inertial sensors are used to determine and predict the device posture. The depth data and inertial sensor data acquired by the AR glasses 104D may be processed by at least one of the AR glasses 104D and the server 102 to identify the device posture. The device posture is used to control the AR glasses 104D itself or to interact with an application (e.g., a game application) executed by the AR glasses 104D. In some embodiments, the display of the AR glasses 104D displays a user interface, and the recognized or predicted device posture is used to render virtual objects with high fidelity or to interact with user-selectable display items on the user interface.

[0024] In some embodiments, SLAM technology is applied to the data processing environment 100 to process the video data, static image data, or depth data acquired by the AR glasses 104D together with the inertial sensor data. Identify and predict the device posture, and draw and update the scene where the AR glasses 104D are located. Optionally, the SLAM technology is implemented by the AR glasses 104D alone, or by the server 102 and the AR glasses 104D together.

[0025] Figure 2 1 is a schematic diagram of an electronic system 200 for processing content data (eg, image data) according to some embodiments. The electronic system 200 includes a server 102, a client device 104 (eg, Figure 1AR glasses 104D in), storage 106, or a combination thereof. The electronic system 200 typically includes one or more processing units (central processing unit, CPU) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes referred to as chipsets). The electronic system 200 includes one or more input devices 210 for facilitating user input, such as a keyboard, a mouse, a voice command input unit or a microphone, a touch screen display, a touch-sensitive input pad, a gesture-capturing camera, or other input buttons or controls. In addition, in some embodiments, the client device 104 of the electronic system 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard. In some embodiments, the client device 104 includes one or more optical cameras 260 (e.g., RGB cameras), scanners, or picture sensor units for capturing images such as graphic serial codes printed on electronic devices. The electronic system 200 also includes one or more output devices 212 for presenting a user interface and displaying content, including one or more speakers and / or one or more visual displays. Optionally, the client device 104 includes a location detection device, such as a global positioning system (GPS) or other geographic location receiver, for determining the location of the client device 104 .

[0026] Optionally, the client device 104 includes an inertial measurement unit (IMU) 280 that integrates sensor data acquired by multi-axis inertial sensors to provide an estimate of the position and orientation in space of the client device 104. Examples of one or more inertial measurement units 280 include, but are not limited to, gyroscopes, accelerometers, magnetometers, and inclinometers.

[0027] The memory 206 may include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid-state storage devices; optionally, it may also include a non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Optionally, the memory 206 may include one or more storage devices remotely arranged from one or more processing units 202. The memory 206 or the non-volatile memory in the memory 206 includes a non-temporary computer-readable storage medium. In some embodiments, the memory 206 or the non-temporary computer-readable storage medium of the memory 206 stores the following programs, modules, and data structures, or a subset or superset thereof:

[0028] Operating system 214, including programs for handling various basic system services and performing hardware-dependent tasks;

[0029] The network communication module 216 is used to connect each server 102 or client 102 to the network interface 204 (wired or wireless) through one or more network interfaces 204 and one or more communication networks 108, such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.

[0030] Device 104 is connected to other devices (e.g., server 102, client device 104, or storage 106);

[0031] A user interface module 218 for displaying the content of each

[0032] The information is displayed on each client device 104 (e.g., a graphical user interface of an application 224, a widget,

[0033] Website and its pages, and / or games, audio and / or video content, texts, etc.);

[0034] An input processing module 220 for detecting one or more user inputs or interactions from one or more input devices 210,

[0035] and decodes the detected input or interaction;

[0036] Web browser module 222, for navigating, requesting (e.g., via Hypertext Transfer Protocol (HTTP)), and displaying websites and web pages thereof, including web interfaces for logging into a user account associated with client device 104 or other electronic device, and controlling a client or electronic device associated with a user account;

[0037] and edit and view settings and data associated with a user account;

[0038] One or more user applications 224 (e.g., games, social networking applications, smart home applications, and / or other web-based or non-web-based applications for controlling other electronic devices and

[0039] View data acquired by this device);

[0040] Model training module 226, for receiving training data and establishing a data processing model to process the data collected by client device 104

[0041] or acquired content data (e.g., video, image, audio, or text data);

[0042] Data processing module 228, used to process content data using data processing model 250, thereby identifying information contained in the content data, matching the content data with other data, classifying the content data, or synthesizing related content data. In some embodiments, data processing module 228 is coupled to one of the plurality of user applications 224.

[0043] associated to process content data in response to user instructions received from a user application 224;

[0044] a posture determination prediction module 230 for determining and predicting the posture of the client device 104 (e.g., the AR glasses 104D),

[0045] In some embodiments, the posture is determined and predicted by the posture determination prediction module 230 and the data processing module 228. The posture determination prediction module 230 also includes a SLAM module 232 for mapping the scene where the client device 104 is located and identifying the posture of the client device 104 in the scene using the image data and the IMU sensor data;

[0046] a gesture-based rendering module 238 for rendering virtual objects in the field of view of the camera 260 of the client device 104,

[0047] or creating mixed, virtual, or augmented reality content using an image acquired by the camera 260, wherein a virtual object is rendered from the perspective of the camera 260 (i.e., from the viewing point of the camera 260) based on the camera pose of the camera 260, and the mixed, virtual, or augmented reality content is created; and

[0048] One or more databases 240 for storing at least data, including one or more of the following:

[0049] o Device settings 242, including common device settings for one or more servers 102 or client devices 104 (e.g.,

[0050] service tier, device model, storage capacity, processing power, communication capabilities, etc.);

[0051] o User account information 244 for one or more user applications 224, such as user name, security questions, account history data, user preferences, and predefined account settings;

[0052] o Network parameters 246 of one or more communication networks 108, such as IP address, subnet mask, default gateway, DNS server, and host name;

[0053] o Training data 248, used to train one or more data processing models 250;

[0054] o Data processing model 250 for processing content data (e.g., video, image, audio, or text data) using deep learning techniques;

[0055] o a posture data database 252 for storing posture data of the camera 260; and

[0056] ○ Content data and results 254 are respectively acquired by and output to the client device 104 of the electronic system 200, wherein the content data is processed locally on the client device 104 or remotely on the server 102 by the data processing model 250 to provide relevant results to be presented on the client device 104 and include candidate images.

[0057] Optionally, one or more databases 240 are stored in one of the server 102, the client device 104, and the storage 106 of the electronic system 200. Optionally, one or more databases 240 are distributed in multiple of the server 102, the client device 104, and the storage 106 of the electronic system 200. In some embodiments, multiple copies of the above data are stored in different devices, for example, two copies of the data processing model 250 are stored in the server 102 and the storage 106 respectively.

[0058] Each of the above-mentioned determined elements can be stored in one or more storage devices mentioned above and corresponds to the instruction set that performs the above-mentioned functions. The above-mentioned determined modules or programs (i.e., instruction sets) do not have to be executed separately as software programs, programs, modules or data structures, so in various embodiments, various subsets of these modules can be combined or otherwise rearranged. In some embodiments, optionally, the memory 206 stores a subset of the above-mentioned determined modules and data structures. In addition, optionally, the memory 206 stores other modules and data structures not mentioned above.

[0059] Figure 3 2 is a flow chart of a process 300 for processing inertial sensor data and image data of an electronic system (e.g., a server 102, a client device 104, or a combination thereof) using a visual-inertial SLAM module 232 according to some embodiments. The process 300 includes measurement preprocessing 302, initialization 304, local visual-inertial odometry (VIO) with relocalization 306, and global pose graph optimization 308. In measurement preprocessing 302, the RGB camera 260 captures image data of a scene at an image rate (e.g., 30 FPS), and detects and tracks (310) features from the image data. The IMU 280 measures inertial sensor data at a sampling frequency (e.g., 1000 Hz) while the RGB camera 260 captures image data, and the inertial sensor data is pre-integrated (312) to provide data of device pose change 340. In initialization 304, the image data captured by the RGB camera 260 and the inertial sensor data measured by the IMU 280 are aligned in time (314). A purely vision-based structure from motion (SfM) technique 314 is applied ( 316 ) to couple the image data and the inertial sensor data, estimate the 3D structure, and render a scene graph for the RGB camera 260 .

[0060] After initialization 304, and during relocalization 306, the VIO is optimized (322) using a sliding window 318 and associated states 320 from loop closures. When the VIO corresponds to a keyframe (324) for a smooth video transition and a corresponding loop is detected (326), features are extracted (328) and used to generate associated states 320 from the loop closures. In a global pose graph optimization 308, a multi-degree-of-freedom (multi-DOF) pose graph is optimized (330) based on the states from the loop closures 320, and a keyframe database 332 is updated using the keyframes associated with the VIO.

[0061] In addition, the detected and tracked (310) features are used to monitor (334) the motion of the object in the image data and, for example, estimate an image-based pose 336 based on the image rate. In some embodiments, the pre-integrated (234) inertial sensor data can be propagated (338) based on the motion of the object and used to estimate an inertial-based pose 340, for example, based on the sampling frequency of the IMU 280. The image-based pose 336 and the inertial-based pose 340 are stored in the database 240 and used by the module 230 to estimate and predict poses used by the real-time video rendering system 234. Optionally, in some embodiments, the module 232 receives the inertial sensor data measured by the IMU 280 and obtains the image-based pose 336 to estimate and predict more poses 340, which are further used by the real-time video rendering system 234.

[0062] In SLAM, high-frequency pose estimation is achieved through sensor fusion and relies on data synchronization between image sensors and IMU 280. Image sensors (e.g., RGB camera 260, lidar scanner) provide image data required for pose estimation and typically operate at a lower frequency (e.g., 30 frames per second) and with greater latency (e.g., 30 milliseconds) than IMU 280. In contrast, IMU 280 can measure inertial sensor data and operate at a very high frequency (e.g., 1000 samples per second) and negligible latency (e.g., less than 0.1 milliseconds). Asynchronous time warping (ATW) is often applied in AR systems to warp images before sending them to the display to correct for head movement and pose changes that occur after the image is rendered. ATW algorithms can reduce image latency, increase or maintain frame rate, or reduce jitter caused by missing images. In SLAM and ATW, relevant image data and inertial sensor data are stored locally to be synchronized and used for pose estimation / prediction. In some embodiments, the image data and the inertial sensor data are stored in one of a plurality of standard tessellation language (STL) containers, such as std::vector, std::queue, std::list, etc., or other custom containers. These containers are generally convenient to use. The image data and the inertial sensor data are stored in the STL container together with their timestamps, which are used for data search, data insertion, and data organization.

[0063] Figure 4AIt is an image 400 rendered at different distances according to a one-point perspective of a virtual object 402 (e.g., a moving vehicle) in some embodiments. In some embodiments, the image 400 is captured by a camera 260 and displayed on a screen of an electronic device including the camera 260. Examples of electronic devices include a mobile phone 104C, a vehicle, AR glasses 104D, and video perspective VR glasses. Optionally, in some embodiments, the image 400 is captured by a camera 260 (e.g., a surveillance camera 104E) of a first electronic device and displayed on a screen of a second electronic device (e.g., a mobile phone 104C, a laptop computer 104A). The virtual object 402 is rendered on the image 400 based on the basic principle of perspective drawing, that is, it is rendered at different rendering sizes according to the distance of the virtual object 402 measured from the camera position of the camera 260.

[0064] Image 400 is captured by camera 260 at a conventional viewing angle, showing the field of view of camera 260 under one-point perspective. Image 400 has a horizon 404 and a vanishing point 406. Horizon 404 passes through vanishing point 406, and parallel lines in real life (e.g., road edges, treetops) converge at vanishing point 406. As virtual object 402 moves along road 408 and approaches camera position 510, the size of virtual object 402 on image 400 increases. In some cases, virtual object 402A is far away in the field of view and the size of the rendering is too small to be an affordable option. Conversely, in some cases, virtual object 402 is immediately in front of camera 260 and the size of the rendering is so large that it blocks the view of other objects behind the object, limiting the visibility of the field of view of camera 260.

[0065] Figure 4B Another image 420 is a virtual object 402 displayed at a display object size controlled within a display size range according to some embodiments. The field of view of the camera 260 is divided into three parts according to the distance of the virtual object 402 measured from the camera position of the camera 260 that captured the image 420. In a first part of the field of view (also referred to as a mid-distance field) 422, the distance of the virtual object 402 is between a first object distance 424 and a second object distance 426 greater than the first object distance 424, and the virtual object 402 is rendered in the image 420 based on the basic principles of perspective drawing. The display object size of the virtual object 402 decreases as the distance of the virtual object 402 increases, for example, at a first rate determined by the basic principles of perspective drawing. Specifically, on the image 420, the virtual object 402 has a first display size at the first object distance 424 and a second display size at the second object distance 426. The second display size is smaller than the first display size.

[0066] In a second portion (also referred to as the far field) 428 of the field of view, the distance of the virtual object 402 is greater than the second object distance 426, and the virtual object 402 is rendered in the image 420 without following the basic principles of perspective drawing. In some embodiments, the size of the virtual object 402 remains unchanged at the second display size or the third display size in the second portion 428 of the field of view. The third display size is different from (e.g., greater than or less than) the second display size. Optionally, in the far field 428, the size of the virtual object 402 decreases as the distance of the virtual object 402 increases, for example, at a second rate that is less than the first rate, so that the size of the virtual object 402 can maintain a reasonably controllable size to ensure the visibility and selectability of the virtual object 402. In addition, in some embodiments, the size of the virtual object 402 decreases at the second rate until the display size lower limit is reached, so that the size of the virtual object 402 remains above or equal to the display size lower limit.

[0067] In a third portion (also referred to as the near field) 430 of the field of view, the distance of the virtual object 402 is less than the first object distance 424, and the virtual object 402 is also rendered in the image 420 without following the basic principle of perspective drawing. In some embodiments, the size of the virtual object 402 remains unchanged at the first display size or the fourth display size in the near field 430. The fourth display size is different from (e.g., greater than or less than) the first display size. Optionally, in the near field 430, the size of the virtual object 402 increases as the distance of the virtual object 402 decreases, for example, at a third rate that is less than the first rate, so that the size of the virtual object 402 can maintain a reasonably controllable size to ensure the visibility of the virtual object 402 and other objects behind the virtual object 402 in the field of view. In addition, in some embodiments, the size of the virtual object 402 increases at a third rate until the display size upper limit is reached, so that the size of the virtual object 402 remains less than or equal to the display size upper limit.

[0068] The display object size of the virtual object 402 is controlled within a display size range. Optionally, the display size range is defined by a first display size and a second display size corresponding to a first object distance 424 and a second object distance 426, respectively. In some embodiments, the first object distance 424 and the second object distance 426 are not measured. Instead, the display size of the virtual object 402 increases at a first rate as the distance of the virtual object 402 decreases. If it is determined that the display object size of the virtual object 402 is greater than the first display size, the display object size of the virtual object 402 is controlled so that the display object size is equal to the first display size. If it is determined that the display object size of the virtual object 402 is less than the second display size, the display object size of the virtual object 402 is controlled so that the display object size is equal to the second display size. Further, in some embodiments, the display size range is modified by a third display size, a fourth display size, a display size lower limit, or a display size upper limit. In some embodiments, if it is determined that the display object size of the virtual object 402 is greater than the display size upper limit, the display object size of the virtual object 402 is reduced so that the display object size is equal to the display size upper limit.

[0069] Image 420 is displayed on a screen having a display size. In some embodiments, image 420 is captured by a camera and scaled to be displayed on a screen having a display size. Optionally, in some embodiments, the electronic device includes a perspective screen that is at least partially transparent, through which the physical world can be observed. Virtual object 402 is rendered on the screen and superimposed on the physical world. In some embodiments, the ratio between the display size of virtual object 402 and the display size of the screen is controlled within a display ratio range. When the display size of virtual object 402 is between the first display size and the second display size in the mid-distance field 422, the display ratio range is between the first display ratio threshold and the second display ratio threshold that is less than the first display ratio threshold. The first display ratio threshold is the ratio of the first display size to the display size, and the second display ratio threshold is the ratio of the second display size to the display size. In some embodiments, the display ratio range is less than the upper limit of the display ratio, and the display object size is less than the upper limit of the display size. In some embodiments, the display ratio range is greater than the lower limit of the display ratio, and the display object size is greater than or equal to the lower limit of the display size.

[0070] Figure 54 is a perspective view 500 of a bounding box 502 of a virtual object 402 according to some embodiments. The virtual object 402 is rendered on an image 420 captured by the camera 260. The virtual object 402 has an object size and is displayed on the image 420 at a display object size. The object size remains constant, while the display object size changes with the distance of the virtual object 402 and the camera position 510. The display object size of the virtual object 402 is associated with an angular size 504 (i.e., degrees of viewing angle) of the virtual object 402 relative to the camera position 510 and the camera field of view (FOV). In the example, the unit of the angular size 504 is radians, and the size of the virtual object 402 displayed on the image 420 is also measured in degrees. In some embodiments, the size of the virtual object 402 on the image 420 is determined based on the projection of the object in the user's view. The projected size of the virtual object 402 appears too large in the near field 430 and too small in the far field 428. Therefore, when the virtual object 402 is rendered in the near field 430, the far field 428 and the mid-distance field 422 of the field of view of the camera 260, the projection size of the virtual object 402 is monitored, and the angle size 504 is controlled within the angle size range, and the angle size 504 is adaptively changed according to the position of the virtual object 402 relative to the camera position 510 and the projection size of the virtual object 402.

[0071] In some embodiments, the virtual object 402 is rendered in a bounding box 502. The bounding box 502 includes a plurality of bounding box elements, such as a box corner 502C and a box edge 502E. If the virtual object 402 is selectable, the bounding box 502 defines an area corresponding to a selectable affordable item, and a user action on the selectable affordable item will initiate the selection of the virtual object 402, optionally followed by more user actions. If the size of the virtual object 402 can be manually adjusted, a user action on any bounding box element will initiate an operation to enlarge or reduce the virtual object 402.

[0072] In some embodiments, the angular size of the virtual object 402 is measured by a horizontal angular size 504H and a vertical angular size 504V. The horizontal line 506H and the vertical line 506V are orthogonal to each other and intersect at the camera position 510. The horizontal line 506H and the vertical line 506V define a plane parallel to the image sensor array of the camera 260. The center of the virtual object 402 is projected to a first node 510H on the horizontal line 506H and a second node 510V on the vertical line 506V, respectively. The nodes on the surface of the virtual object 402 are connected to the first node 510H, and the line connecting the object 402 to the first node 510H forms a horizontal angle with a horizontal angular size 504H. The nodes on the surface of the virtual object 402 are also connected to the second node 510V, and the line connecting the object 402 to the second node 510V forms a vertical angle with a vertical angular size 504V.

[0073] Figure 6 The process 600 of adaptively adjusting the object size of a virtual object 402 according to some embodiments. The virtual object 402 is rendered on an image 420 captured by the camera 260. The actual size of the virtual object 402, if any, is fixed and is independent of the position of the virtual object 402 in the physical world. However, when the virtual object 402 is projected onto the image 420 through one-point perspective, the rendered size of the virtual object 402 varies according to the distance of the virtual object 402 from the camera 260. The virtual object 402 is perceived by the light rays 602 entering our eyes. For example, the virtual object 402A is closer to the user or the camera 260 than the virtual object 402C. The angle formed by the light rays 602A corresponding to the virtual object 402A is greater than the angle formed by the light rays 602C corresponding to the virtual object 402C. When the light rays 602 reach the retina or the image sensor array to form the image 420, the displayed object size of the virtual object 402A on the image 420 is greater than the displayed object size of the virtual object 402C.

[0074] The virtual object 402 includes a bounding box 604 having a plurality of bounding box elements 606. Optionally, the bounding box 604 and the bounding box elements 606 are not rendered together with the virtual object 402, i.e., are not visible on the image 420. Optionally, the bounding box 604 and the bounding box elements 606 are rendered together with the virtual object 402, i.e., are visible on the image 420. The object size of the virtual object 402 includes a first size S1 of the virtual object 402, a second size S2 of the bounding box 604, and a third size S3 of a subset of the plurality of bounding box elements 606. In the three-dimensional virtual world, the third size S3 of the subset of the plurality of bounding box elements 606 is adjusted according to the distance of the virtual object 402 from the camera 510, while the first size S1 of the virtual object 402 and the second size S2 of the bounding box 604 remain unchanged and are consistent with the distance of the virtual object 402 from the camera 510. Through these methods, the virtual object 402 and the bounding box 604 appear normal to the observer, in accordance with the principle of perspective drawing, while the third size S3 of the bounding box element is still controlled to facilitate user interaction.

[0075] According to the basic principles of perspective drawing, as the virtual object 402 approaches the user, the object 402 and the bounding box 604 (both of fixed size) appear to be very close to the user and can occupy an increasingly larger display space in the field of view of the image 420. The basic principles of perspective drawing apply to the bounding box elements 606 in the mid-distance field 422, but not to the near field 430 and the far field 428. In the mid-distance field 422, as the distance between the object 402 and the camera position 510 decreases, the displayed third size S3' of these bounding box elements 606 will be enlarged. When the virtual object 402 moves away from the user in the mid-distance field 422, the displayed third size S3' of the bounding box elements 606 on the image 420 will be reduced as the object 402 moves away from the camera 260.

[0076] When the virtual object 402 enters the near field 430 of the camera 260, the distance between the user and the virtual object 402 drops below the first object distance 424. The displayed third size S3' of the bounding box element 606 will stop expanding and remain unchanged so that the bounding box element 606 does not become too large. On the contrary, if the object 402 is too far away from the user in the far field 428, reducing the displayed third size S3' of the bounding box element 606 may cause the bounding box element 606 to be too small to be selected and operated. In some cases, when the virtual object 402 enters the far field 428 of the camera 260, the distance between the user and the virtual object 402 will exceed the second object distance 426, and the displayed third size S3' of the bounding box element 606 will stop shrinking. This maintains the visibility of the bounding box element 606 in the image 420 and improves the usability of the selectable bounding box element 606. By these methods, the bounding box elements 606 are adaptively adjusted relative to the image 420, ensuring that they can be easily selected and manipulated regardless of the distance between the target object and the viewer. This ensures the usability of the bounding box elements 606 and the stability of all interactions based on the bounding box elements 606.

[0077] The angular size (AS) of the virtual object 402 rendered on the image 420 is defined as:

[0078]

[0079] Among them, AOS represents the fixed actual object size, DOS represents the display object size, and D OBJ represents the distance between the object 402 and the camera position 510, D OL represents the fixed distance between the lens and the image sensor array inside the camera 260. The angular size AS is proportional to the display object size DOS. Both the angular size AS and the display object size DOS are proportional to the distance D between the object 402 and the camera position 510. OBJ Inversely proportional to. Further, in some embodiments, in the middle distance field 422, the distance of the virtual object 402 is between the first object distance 424 and the second object distance 426 greater than the first object distance 424, and the virtual object 402 is rendered in the image 420 based on the basic principles of perspective drawing. The angular size of the virtual object 402 decreases as the distance of the virtual object 402 increases, for example, at a first rate determined by the basic principles of perspective drawing. Specifically, on the image 420, the virtual object 402 has a first angular size at the first object distance 424 and a second angular size at the second object distance 426. The second angular size is smaller than the first angular size.

[0080] In the far field 428, the distance of the virtual object 402 from the camera position 510 is greater than the second object distance 426, and the virtual object 402 is rendered in the image 420 based on the basic principle of perspective drawing. In some embodiments, the third size S3 of the bounding box element 606 remains unchanged in the far field 428. Optionally, in the far field 428, the angular size of the bounding box element 606 decreases as the distance of the virtual object 402 increases, for example, at a second rate that is less than the first rate, so that the bounding box element 606 can maintain a reasonably controllable size to ensure the selectability of the bounding box element 606. In addition, in some embodiments, the angular size of the third size S3 decreases at the second rate until the lower angular limit is reached, so that the angular size of the bounding box element 606 remains above or equal to the lower angular limit.

[0081] In the near field 430, the distance of the virtual object 402 is less than the first object distance 424, and the virtual object 402 is rendered in the image 420 according to the basic principle of perspective drawing. In some embodiments, the third size S3 of the bounding box element 606 remains unchanged in the near field 430. Optionally, in the near field 430, the angular size of the bounding box element 606 increases as the distance of the virtual object 402 decreases, for example, at a third rate that is less than the first rate, so that the bounding box element 606 can maintain a reasonable and controllable size to ensure the visibility of the virtual object 402 and other objects behind the virtual object 402 in the field of view. In addition, in some embodiments, the angular size of the bounding box element 606 increases at the third rate until the upper angle limit is reached, so that the angular size of the bounding box element 606 remains less than or equal to the upper angle limit.

[0082] In some embodiments, the angular size of the virtual object 402 is controlled within an angular size range. Optionally, the angular size range is defined by a first angular size and a second angular size corresponding to a first object distance 424 and a second object distance 426, respectively. In some embodiments, the first object distance 424 and the second object distance 426 are not measured. Instead, the angular size of the virtual object 402 increases at a first rate as the distance of the virtual object 402 decreases. If it is determined that the angular size of the virtual object is greater than the first angular size, the angular size of the virtual object is controlled so that the angular size is equal to the second angular size. If it is determined that the angular size of the virtual object is less than the second angular size, the angular size of the virtual object is controlled so that the angular size is equal to the first angular size. Further, in some embodiments, the angular size range is modified by a third angular size, a fourth angular size, an angular lower limit, or an angular upper limit. In some embodiments, if it is determined that the angular size of the virtual object is greater than the angular upper limit, the angular size of the virtual object is reduced so that the angular size is equal to the angular upper limit.

[0083] In contrast, in some embodiments, the angular size of virtual object 402 and the angular size of bounding box 604 are not restricted and follow the principle of perspective drawing. In contrast, the angular size of bounding box element 606 is controlled according to two angular sizes corresponding to first object distance 424 and second object distance 426. In some embodiments, the angular size of bounding box element 606 is restricted between two angular sizes. Optionally, in some embodiments, the rate of change of the angular size of bounding box element 606 is different in fields 422, 428, and 430. In mid-distance field 422, the first rate of change of the angular size of bounding box element 606 is greater than the rate of change of the angular size of bounding box element 606 in near field 430 and far field 428.

[0084] In various embodiments of the present application, different methods can be used to measure the size of the virtual object 402 rendered on the image. For example, the size of the virtual object 402 is measured by an angular size that is equal to the size of the angle formed by the line connecting the camera position 510 and the surface of the virtual object 402. In some embodiments, the angular size corresponding to the first size S1 of the virtual object 402 increases as the virtual object 402 approaches the camera 260 in the mid-distance field 422, and remains unchanged or changes at a slower rate as the virtual object 402 moves in the near field 430 and the far field 428. In some embodiments, the object size of the virtual object 402 (e.g., the first size S1) is measured by the display object size (e.g., the number of pixels) on the image 420. Refer to Figure 4B , the displayed object size increases as the virtual object 402 approaches the camera 260 in the medium distance field 422. The displayed object size increases from the first display size to the second display size when the virtual object 402 moves from the second object distance 426 to the first object distance 424. The displayed object size remains unchanged at the first display size in the near field 430 and remains unchanged at the second display size in the far field 428. The second display size is smaller than the first display size.

[0085] In addition, refer to Figure 6In some embodiments, the object size of the virtual object 402 includes a first size S1 of the virtual object 402, a second size S2 of the bounding box 604, and a third size S3 of a subset of the plurality of bounding box elements 606. In the virtual space, the third size S3 of the subset of the plurality of bounding box elements 606 is adjusted according to the distance of the virtual object 402 from the camera 510, while the first size S1 of the virtual object 402 and the second size S2 of the bounding box 604 remain unchanged and are consistent with the distance of the virtual object 402 from the camera 510. According to the principle of perspective drawing, each of the displayed first size S1′ and the displayed second size S2′ rendered on the image 420 increases as the distance from the camera position 510 decreases. However, when the virtual object 402 moves from the second object distance 426 to the first object distance 424, the displayed third size S3′ on the image 420 increases from the size S1′ to the second size S2′. 31 ' Increase to another size S 32 '(larger than size S 31 Optionally, when the object 402 moves in the near field 430, the third size S3' is displayed with the size S 31 'Remains unchanged, when the object 402 moves in the far field 428, a third size S3' is displayed with a size S 32 'remains unchanged. Optionally, when the object 402 moves in the near field 430 and the far field 428, the displayed third size S3' in the image 420 increases at a smaller rate as the distance from the camera position 510 decreases compared to when the object moves in the medium distance field 422. The virtual object 402 appears normal to the observer, in accordance with the principle of perspective drawing, while the third size S3 is still controlled to facilitate user interaction.

[0086] Figure 7 is a flowchart of a method 700 for rendering visual content according to some embodiments. In some embodiments, the method is applied to AR glasses 104D, a robotic system, a vehicle, or a mobile phone. For convenience, method 700 is described as being implemented by an electronic device (e.g., client device 104). In the example, method 700 is used to determine and predict gestures, draw scenes, and simultaneously render virtual and real content in extended reality (e.g., VR, AR). Optionally, method 700 is controlled by instructions stored in a non-temporary computer-readable storage medium, which are executed by one or more processors of the electronic system. Figure 7 Each of the operations shown in can be stored in a computer memory or a non-transitory computer-readable storage medium (e.g., Figure 2The computer readable storage medium may correspond to the instructions in the memory 206 of the electronic system 200. The computer readable storage medium may include a magnetic disk or optical disk storage device, a solid state storage device such as a flash memory, or other non-volatile storage device. The instructions stored on the computer readable storage medium may include one or more of the following: source code, assembly language code, object code, or other instruction formats interpreted by one or more processors. Some operations in method 700 may be combined and / or the order of some operations may be changed.

[0087] The electronic device generates (702) information of a virtual object 402 to be displayed together with contextual content from the perspective of the camera 260 (i.e., the viewing point of the camera 260). The virtual object 402 has (704) an object position and an object size, and the camera 260 has (706) a camera position that can realize the perspective of the camera 260. The electronic device determines (708) the distance between the virtual object 402 and the camera 260 based on the object position and the camera position, and adjusts (710) the object size of the virtual object 402 based on the distance between the virtual object 402 and the camera 260. The contextual content and the virtual object 402 are rendered (712) and displayed at the object position with the adjusted object size in a scene associated with the contextual content (714). Therefore, the virtual object 402 is displayed together with the contextual content in the image 420.

[0088] In some embodiments, the object size of the virtual object 402 is adjusted by determining (716) the angular size of the virtual object 402 based on the object size and the distance of the virtual object 402 from the camera 260, and controlling (718) the angular size of the virtual object 402 within an angular size range. In addition, in some embodiments, the angular size range is between a first angular size and a second angular size that is smaller than the first angular size. In addition, in some embodiments, the angular size of the virtual object 402 is controlled within the angular size range by: (1) according to a determination that the angular size of the virtual object 402 is larger than the second angular size, reducing the object size of the virtual object 402 so that the angular size is equal to the second angular size; (2) according to a determination that the angular size of the virtual object 402 is smaller than the first angular size, increasing the object size of the virtual object 402 so that the angular size is equal to the second angular size.

[0089] In addition, in some embodiments, the angle size range is smaller than the angle upper limit. By reducing the object size of the virtual object 402 to make the angle size equal to the angle upper limit according to the determination that the angle size of the virtual object 402 is larger than the angle upper limit, the angle size of the virtual object 402 is controlled within the angle size range.

[0090] Optionally, in some embodiments, the angle size range is greater than the angle lower limit. The angle size of the virtual object is controlled within the angle size range by increasing the object size of the virtual object so that the angle size is equal to the angle lower limit according to the determination that the angle size of the virtual object on the display is less than the angle lower limit.

[0091] In some embodiments, the virtual object 402 is displayed (720) on the display of the electronic device at the display object size, and the object size of the virtual object 402 is adjusted (722) so that the ratio between the display object size and the display size of the display is controlled within the display ratio range. Further, in some embodiments, the display ratio range (724) is between a first display ratio threshold and a second display ratio threshold that is less than the first display ratio threshold, and the display object size is within (726) a display size range between the first display size and a second display size that is less than the first display size. In addition, in some embodiments, the display ratio range is less than the upper limit of the display ratio, and the display size range is less than the upper limit of the display size. In some embodiments, the display ratio range is greater than the lower limit of the display ratio, and the display object size is greater than the lower limit of the display size.

[0092] In some embodiments, reference Figure 6 , the virtual object 402 includes a bounding box 604 having a plurality of bounding box elements 606. Optionally, the bounding box 604 and the bounding box elements 606 are not rendered together with the virtual object 402. The object size of the virtual object 402 includes a first size S1 of the virtual object 502, a second size S2 of the bounding box 604, and a third size S3 of a subset of the plurality of bounding box elements 606. The electronic device adjusts the object size of the virtual object 402 by adjusting the third size S3 of the subset of the plurality of bounding box elements 606 according to the distance of the virtual object 402 from the camera 510, and making the first size S1 of the virtual object 402 and the second size S2 of the bounding box 604 consistent with the distance of the virtual object 402 from the camera 510.

[0093] In some embodiments, the electronic device obtains a background image 420 of contextual content captured by the camera 260 and runs an augmented reality (AR) application. In an AR environment enabled by the AR application, the virtual object 402 is superimposed on the background image 420.

[0094] In some embodiments, the electronic device creates a background image 420 including contextual content of multiple virtual objects and runs a virtual reality (VR) application. In a VR environment enabled by the VR application, the virtual object 402 is superimposed on the background image 420.

[0095] It should be understood that the Figure 7The specific order of operations in is exemplary only and does not represent the only order in which the operations are to be performed. A person of ordinary skill in the art will recognize the various methods described herein for adaptively adjusting objects in each image. In addition, it should be noted that the above relative Figure 4A-Figure 6 The details of the other processes can also be applied by analogy to the above with respect to Figure 7 The method 700 in FIG. 700 is not described here for the sake of brevity.

[0096] The terms used herein when describing various embodiments are only used to describe specific embodiments and are not restrictive. As used in the description of the various embodiments described and the appended claims, the singular form "one" is also intended to include plural forms unless the context clearly states otherwise. In addition, it should also be understood that the term "and / or" used herein refers to and includes any and all possible combinations of one or more related listed items. It should also be understood that the terms "including", "comprising" and / or "composition" used in this specification refer to the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, it is understood that although the terms "first", "second" and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

[0097] As used herein, the term "if" may be understood as "when" or "in" or "in response to determining" or "in response to detecting" or "according to determining", depending on the context. Similarly, the phrases "if determining" or "if [the condition or event] is detected" may be understood as "in determining" or "in response to determining" or "in detecting [the condition or event]" or "in response to detecting [the condition or event]" or "determined according to detecting [the condition or event]", depending on the context.

[0098] For ease of explanation, the above description is described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the exact forms disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art to grasp.

[0099] Although the various figures illustrate some logical stages in a particular order, stages that are not dependent on the order may be reordered, and other stages may be combined or split. Although some reordering or other groupings are specifically mentioned, other groupings are also obvious to those of ordinary skill in the art, and therefore the ordering and groupings described herein are not an exhaustive list of alternatives. In addition, it should be recognized that these stages can be implemented by hardware, firmware, software, or any combination thereof.

Claims

1. An image rendering method, applied to an electronic device, characterized in that: The method comprises: generating information of a virtual object for display with contextual content from a viewpoint of a camera having a camera position enabling the viewpoint of the camera; Determining the distance between the virtual object and the camera according to the object position and the camera position; Adjusting the object size of the virtual object according to the distance of the virtual object from the camera; and The context content and the virtual object are rendered, wherein the virtual object is displayed at the object position with the adjusted object size in a scene associated with the context content.

2. The method according to claim 1, characterized in that The adjusting the object size of the virtual object according to the distance between the virtual object and the camera includes: Determining an angle size of the virtual object on a display according to the object size of the virtual object and the distance of the virtual object from the camera; and The angle size of the virtual object on the display is controlled within an angle size range.

3. The method according to claim 2, characterized in that The angle size range is between a first angle size and a second angle size that is smaller than the first angle size.

4. The method according to claim 3, characterized in that The step of controlling the angle size of the virtual object on the display within an angle size range includes: According to the determination that the angular size of the virtual object on the display is greater than the first angular size, reducing the object size of the virtual object so that the angular size on the display is equal to the second angular size; and According to the determination that the angular size of the virtual object on the display is smaller than the second angular size, the object size of the virtual object is increased so that the angular size on the display is equal to the second angular size.

5. The method according to claim 2, characterized in that: The angle range is smaller than the upper angle limit.

6. The method according to claim 5, characterized in that The step of controlling the angle size of the virtual object on the display within an angle size range includes: According to the determination that the angle size of the virtual object on the display is greater than the angle upper limit, the object size of the virtual object is reduced so that the angle size is equal to the angle upper limit.

7. The method according to claim 2, characterized in that The angle range is greater than the angle lower limit.

8. The method according to claim 7, characterized in that The step of controlling the angle size of the virtual object on the display within an angle size range includes: According to the determination that the angle size of the virtual object on the display is less than the angle lower limit, the object size of the virtual object is increased so that the angle size is equal to the angle lower limit.

9. The method according to any one of claims 1 to 3, characterized in that: The virtual object is displayed on the display of the electronic device at a display object size, and the object size of the virtual object is adjusted so that the ratio of the display object size to the display size of the display is controlled within a display ratio range.

10. The method according to claim 9, characterized in that The display ratio range is between a first display ratio threshold and a second display ratio threshold that is smaller than the first display ratio threshold, and the display object size is between a first display size and a second display size that is smaller than the first display size.

11. The method according to claim 9, characterized in that The display ratio range is smaller than the display ratio upper limit, and the display object size is smaller than the display size upper limit.

12. The method according to claim 9, characterized in that The display ratio range is larger than the display ratio lower limit, and the display object size is larger than the display size lower limit.

13. The method according to any one of claims 1 to 12, characterized in that: The virtual object includes a bounding box having a plurality of bounding box elements, an object size of the virtual object includes a first size of the virtual object, a second size of the bounding box, and a third size of a subset of the plurality of bounding box elements, and adjusting the object size of the virtual object includes: adjusting a third size of the subset of the plurality of bounding box elements according to a distance of the virtual object from the camera; and The first size of the virtual object and the second size of the bounding box are kept consistent with the distance of the virtual object from the camera.

14. The method according to any one of claims 1 to 13, characterized in that: The method further comprises: Acquire a background image of the context content captured by the camera; and An augmented reality (AR) application is run, wherein the virtual object is superimposed on the background image in an AR environment enabled by the AR application.

15. The method according to any one of claims 1 to 13, characterized in that: The method further comprises: creating a background image of the contextual content including a plurality of virtual objects; and A virtual reality (VR) application is run, wherein the virtual object is superimposed on the background image in a VR environment enabled by the VR application.

16. An electronic device, characterized in that: include: one or more processors; as well as A memory storing instructions, wherein the instructions are executed by the one or more processors to enable the one or more processors to execute the method according to any one of claims 1 to 15.

17. A non-transitory computer-readable medium, characterized in that The non-transitory computer-readable medium stores instructions, and the instructions are executed by one or more processors to enable the one or more processors to perform the method according to any one of claims 1 to 15.

Images