Augmented reality equipment data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an augmented reality equipment data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first image of a real world and a second image of a virtual world of the augmented reality equipment; determining a first depth between a real object in the first image and a user of the augmented reality device, and a second depth between a virtual object in the second image and the user; cutting the first image and the second image based on the first depth, the second depth and a preset depth threshold to obtain a target image, and displaying the target image; wherein the target image is an image comprising a real world and a virtual world. Thus, the image including both the real world and the virtual world can be displayed in the augmented reality device, so that a user can perceive the real world while performing virtual experience, thereby being capable of interacting with the real world while performing virtual experience, and improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technology, and in particular, to a method, apparatus, electronic device, and storage medium for processing data of an extended reality device. Background Art

[0002] Extended Reality (XR) technology is to use computer technology to fuse and reconstruct various information such as vision, hearing, and touch, so as to generate a human-computer interactive virtual scene, and users can obtain an immersive virtual experience through extended reality devices.

[0003] However, when users wear extended reality devices for immersive virtual experiences, they cannot perceive the external environment, which poses a safety hazard. In related technologies, a safety area is preset, and users can perform immersive virtual experiences within the safety area. After users walk out of the safety area, they can see the external environment using a camera.

[0004] However, when performing virtual experiences within the preset safety area, users have no visual perception of the real world, and the auditory perception is not obvious either, which is inconvenient for users to carry out activities in the real world, resulting in poor user experience. Summary of the Invention

[0005] To overcome the problems existing in related technologies, the present disclosure provides a method, apparatus, electronic device, and storage medium for processing data of an extended reality device. To overcome the problems in related technologies that when users perform virtual experiences within the preset safety area, they have no visual perception of the real world, and the auditory perception is not obvious either, which is inconvenient for users to carry out activities in the real world.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a method for processing data of an extended reality device, including:

[0007] Obtaining a first image of the real world and a second image of the virtual world of the extended reality device;

[0008] Determining a first depth between a real object in the first image and a user of the extended reality device, and a second depth between a virtual object in the second image and the user;

[0009] Performing a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and displaying the target image; wherein the target image is an image including the real world and the virtual world.

[0010] In some embodiments, the method of cropping the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image includes:

[0011] Cropping the first image based on the first depth and the preset depth threshold to obtain a third image;

[0012] Cropping the second image based on the second depth and the preset depth threshold to obtain a fourth image;

[0013] Obtaining the target image based on the third image and the fourth image; wherein, the objects to be retained in the third image and the fourth image are different.

[0014] In some embodiments, the method of cropping the first image based on the first depth and the preset depth threshold to obtain a third image includes:

[0015] When the first depth is less than the preset depth threshold, determining the real object to be retained in the first image;

[0016] Cropping the first image according to the real object to be retained to obtain the third image; wherein, the third image includes the real object to be retained.

[0017] In some embodiments, the method of cropping the second image based on the second depth and the preset depth threshold to obtain a fourth image includes:

[0018] When the second depth is greater than or equal to the preset depth threshold, determining the virtual object to be retained in the second image;

[0019] Cropping the second image according to the virtual object to be retained to obtain the fourth image; wherein, the fourth image includes the virtual object to be retained.

[0020] In some embodiments, the method of obtaining the target image based on the third image and the fourth image includes:

[0021] Performing a splicing process on the third image and the fourth image according to pixel points based on the preset depth threshold to obtain the target image.

[0022] In some embodiments, the first image is multiple frames; the method further includes:

[0023] Respectively obtaining the target depth between the real object in each frame of the first image and the user;

[0024] Determine a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth; wherein, the moving speed of the target real object is greater than a preset speed threshold;

[0025] Based on the target depth between the target real object and the user, superimpose the image corresponding to the target real object on the target image.

[0026] In some embodiments, the determining a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth includes:

[0027] Determine the moving speed of each of the real objects according to the target depth and the acquisition time;

[0028] Among each of the real objects in each frame of the first image, determine as the target real object the real object whose moving speed is greater than the preset speed threshold.

[0029] According to a second aspect of the embodiments of the present disclosure, there is provided an extended reality device data processing apparatus, including:

[0030] A first acquisition module configured to acquire a first image of the real world and a second image of the virtual world of the extended reality device;

[0031] A first determination module configured to determine a first depth between a real object in the first image and a user of the extended reality device, and a second depth between a virtual object in the second image and the user;

[0032] A first display module configured to perform cropping processing on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and display the target image; wherein, the target image is an image including the real world and the virtual world.

[0033] In some embodiments, the first display module includes:

[0034] A determination unit configured to perform cropping processing on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image;

[0035] A display unit configured to display the target image;

[0036] Wherein, the determination unit includes:

[0037] A first cropping subunit configured to crop the first image based on the first depth and the preset depth threshold to obtain a third image;

[0038] A second cropping subunit, configured to crop the second image based on the second depth and the preset depth threshold to obtain a fourth image;

[0039] A target image determination subunit, configured to obtain the target image based on the third image and the fourth image; wherein, the objects to be retained in the third image and the fourth image are different.

[0040] In some embodiments, the first cropping subunit is configured to:

[0041] When the first depth is less than the preset depth threshold, determine the real object to be retained in the first image;

[0042] Crop the first image according to the real object to be retained to obtain the third image; wherein, the third image includes the real object to be retained.

[0043] In some embodiments, the second cropping subunit is configured to:

[0044] When the second depth is greater than or equal to the preset depth threshold, determine the virtual object to be retained in the second image;

[0045] Crop the second image according to the virtual object to be retained to obtain the fourth image; wherein, the fourth image includes the virtual object to be retained.

[0046] In some embodiments, the target image determination subunit is configured to:

[0047] Based on the preset depth threshold, splice the third image and the fourth image pixel by pixel to obtain the target image.

[0048] In some embodiments, the first image is multiple frames; the apparatus further includes:

[0049] A second acquisition module, configured to respectively acquire the target depth between the real object in each frame of the first image and the user;

[0050] A second determination module, configured to determine a target real object from each real object according to the acquisition time of each frame of the first image and the corresponding target depth; wherein, the moving speed of the target real object is greater than a preset speed threshold;

[0051] A second display module, configured to superimpose and display the image corresponding to the target real object in the target image based on the target depth between the target real object and the user.

[0052] In some embodiments, the second determination module is configured to:

[0053] Determine the moving speed of each of the real objects according to the target depth and the acquisition time;

[0054] Among the real objects in each frame of the first image, determine, as the target real object, a real object whose moving speed is greater than the preset speed threshold.

[0055] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0056] A processor;

[0057] A memory for storing instructions executable by the processor;

[0058] Wherein, the processor is configured to: when executed, implement the steps in any of the extended reality device data processing methods in the first aspect above.

[0059] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an extended reality device data processing apparatus, enables the apparatus to execute any of the extended reality device data processing methods in the first aspect above.

[0060] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0061] Obtain a first image of the real world and a second image of the virtual world of the extended reality device, determine a first depth between a real object in the first image and a user of the extended reality device, and a second depth between a virtual object in the second image and the user. In this way, the distances between each object included in the first image and the second image and the user are obtained. Crop the first image and the second image according to the first depth, the second depth, and a preset depth threshold to obtain a target image including the real world and the virtual world, and display the target image.

[0062] Compared with the related art, when a user performs a virtual experience within a preset safe area, there is no visual perception of the real world, and the auditory perception is not obvious, which is not convenient for the user to carry out activities in the real world. The extended reality device of this solution can display an image that includes both the real world and the virtual world, enabling the user to have a perception of the real world while performing a virtual experience, so that the user can interact with the real world while performing the virtual experience, improving the user experience.

[0063] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Brief Description of the Drawings

[0064] The drawings herein are incorporated into and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0065] Figure 1 It is a flowchart of a method for processing data of an extended reality device shown according to an exemplary embodiment.

[0066] Figure 2 It is a block diagram of the hardware included in the front of an extended reality device shown according to an exemplary embodiment.

[0067] Figure 3A It is a block diagram of the hardware included in the back of an extended reality device shown according to an exemplary embodiment.

[0068] Figure 3B It is a schematic diagram of a target image shown according to an exemplary embodiment.

[0069] Figure 3C It is a schematic diagram of an image formed by superimposing and displaying an image corresponding to a target real object in a target image shown according to an exemplary embodiment.

[0070] Figure 4 It is a flowchart of a method for processing data of an extended reality device shown according to an exemplary embodiment.

[0071] Figure 5 It is a block diagram of a data processing apparatus of an extended reality device shown according to an exemplary embodiment.

[0072] Figure 6 It is a block diagram of the hardware of an electronic device 800 shown according to an exemplary embodiment. Detailed Description of the Embodiments

[0073] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] Figure 1 It is a flowchart of a method for processing data of an extended reality device shown according to an exemplary embodiment. As Figure 1 shown, this method is used in an extended reality device and mainly includes the following steps:

[0075] In step 101, obtain a first image of the real world and a second image of the virtual world of the extended reality device.

[0076] In step 102, determine a first depth between a real object in the first image and the user of the extended reality device, and a second depth between a virtual object in the second image and the user.

[0077] In step 103, perform a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and display the target image; wherein, the target image is an image including the real world and the virtual world.

[0078] Here, Extended Reality (XR) can be understood as a technology that combines the real and the virtual through a computer to create a virtual environment for human-computer interaction, and can include technologies such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). The extended reality device can include a head-mounted display device, a handheld mobile device, etc., which are not limited herein.

[0079] Here, the real world can be the real world where the user of the extended reality device is located. The first image can include an image captured by a left-eye camera and / or a right-eye camera. The real object in the first image can be an object in the real world included in the first image.

[0080] The first depth between the real object in the first image and the user of the extended reality device can be the distance between the real object included in the first image and the user of the extended reality device, or the distance between the real object included in the first image and the camera of the extended reality device. The first depth can be represented by a first depth map corresponding to the first image, and the first depth map can include the depth of each pixel point in the first image.

[0081] Figure 2 is a hardware structure block diagram included in the front of an extended reality device shown according to an exemplary embodiment. As Figure 2 shown, the front of the extended reality device can include a left-eye camera 201, a right-eye camera 202, and a Time of Flight (TOF) sensor 203. Among them, the left-eye camera 201 and the right-eye camera 202 are used to obtain the first image, and the camera can be a color camera. For example, a red green blue (RGB) camera, which can simultaneously capture the red, green, and blue three main color channels of an image, thereby generating a complete color image. The TOF sensor is used to determine the first depth.

[0082] Here, the TOF sensor can be a sensor capable of implementing TOF ranging, which can calculate the distance between the camera of the extended reality device (such as the left-eye camera 201 and the right-eye camera 202) and the real object in the first image by measuring the "time of flight" of signals such as ultrasonic waves, microwaves, and light. The "time of flight" can be calculated by methods such as the phase difference method and the time difference method.

[0083] In this embodiment, the virtual world can be the world presented by the extended reality device to the user. For example, a completely virtual world, a world combining the real world and a virtual environment, etc. The second image can include a left-eye virtual image and / or a right-eye virtual image. The virtual object in the second image can be an object in the virtual world included in the second image.

[0084] Here, the second depth between the virtual object in the second image and the user can be the distance between the virtual object included in the second image and the user of the extended reality device. The second depth can be represented by a second depth map corresponding to the second image, and the second depth map can include the depth of each pixel point in the second image.

[0085] In this embodiment, the second image and the second depth can be obtained through an application programming interface when the extended reality device performs virtual content rendering. The application can be an application capable of XR experience, such as a game application, a social application, an entertainment application, etc.

[0086] Here, virtual content rendering can be the process of converting the objects and environment in the virtual scene into two-dimensional images that can be displayed on the extended reality device.

[0087] Exemplarily, in a scenario where the user of the extended reality device experiences a game through the extended reality device, the processor of the extended reality device can obtain the second image of the virtual world of the extended reality device and the second depth between the virtual object in the second image and the user of the extended reality device through the game engine.

[0088] Here, the cropping process can be to divide an image into several parts by cutting or trimming. The number of parts of the image obtained after cropping can be determined according to the cutting or trimming method, and this embodiment does not limit this.

[0089] In this embodiment, after obtaining the first image, the second image, the first depth, and the second depth, the processor of the extended reality device crops the first image according to the first depth and a preset depth threshold to obtain an image including some real objects in the first image, crops the second image according to the second depth and the preset depth threshold to obtain an image including some virtual objects in the second image, and processes the cropped images, that is: processes the image including some real objects in the first image and the image including some virtual objects in the second image, and an image that includes both some real objects in the first image and some virtual objects in the second image can be obtained. Since the first image corresponds to the real world and the second image corresponds to the virtual world of the extended reality device, then, based on the above process, a target image that includes both the real world and the virtual world can be obtained. After obtaining the target image, the target image can be displayed through the display of the extended reality device.

[0090] Here, the preset depth threshold can be a value customized by the user of the extended reality device or a value pre-set in the extended reality device, and no specific limitation is made here. The display of the target image can be implemented through Video See Through (VST) technology, and its implementation method can be:

[0091] Collect the first image of the real world through the left-eye camera 201 and the right-eye camera 202 of the extended reality device, determine the first depth between the real object in the first image and the user of the device through the TOF sensor 203 of the extended reality device. When the extended reality device renders virtual content, obtain the second image through the application interface and determine the second depth. Combining computer image technology, crop the first image and the second image based on the first depth, the second depth, and the preset depth threshold to obtain the target image, and present the target image to the user through a display carrier such as a head-mounted helmet or a display.

[0092] Figure 3A It is a hardware structure block diagram included on the reverse side of an extended reality device shown according to an exemplary embodiment. As Figure 3A shown, the reverse side of the extended reality device may include a left display lens 301 and a right display lens 302. Among them, the left display lens 301 and the right display lens 302 are used to display the target image so that the user of the extended reality device can see it.

[0093] It should be noted that in the case where the first image includes the images collected by the left-eye camera and the right-eye camera, and the second image includes the left-eye virtual image and the right-eye virtual image, the target image also includes a left-eye target image and a right-eye target image, that is, the images seen by the left and right eyes of the user of the extended reality device respectively.

[0094] Here, the left display lens 301 and the right display lens 302 can be convex lenses, whose purpose is to refract light rays and image the picture on the display of the extended reality device closer to the retina position, so that the eyes of the user of the extended reality device can clearly see the display screen in front of them.

[0095] In the embodiments of the present disclosure, the processor of the extended reality device acquires the first image of the real world and the second image of the virtual world of the extended reality device, determines the first depth between the real object in the first image and the user of the extended reality device, and the second depth between the virtual object in the second image and the user. In this way, the distances between each object included in the first image and the second image and the user are obtained. The first image and the second image are cropped according to the first depth, the second depth, and a preset depth threshold to obtain a target image including the real world and the virtual world, and the target image is displayed.

[0096] Compared with the related art, when the user performs a virtual experience within a preset safe area, there is no visual perception of the real world, and the auditory perception is not obvious, which is inconvenient for the user to carry out activities in the real world. The extended reality device of the present solution can display an image that includes both the real world and the virtual world, enabling the user to have a perception of the real world while performing a virtual experience, so that the user can interact with the real world while performing a virtual experience, improving the user experience.

[0097] In some embodiments, the cropping the first image and the second image according to the first depth, the second depth, and a preset depth threshold to obtain a target image includes:

[0098] Cropping the first image according to the first depth and the preset depth threshold to obtain a third image;

[0099] Cropping the second image according to the second depth and the preset depth threshold to obtain a fourth image;

[0100] Obtaining the target image based on the third image and the fourth image; wherein, the objects to be retained in the third image and the fourth image are different.

[0101] Here, since the first depth can represent the first distance between each real object in the first image and the user of the extended reality device, the object to be retained in the third image is the real object in the real world included in the first image that the extended reality device wants to display when the distance from the user is less than the preset depth threshold when displaying the target image.

[0102] Then, according to the magnitude relationship between the first depth and the preset depth threshold, real objects in the first image with a first depth less than the preset depth threshold can be determined. These real objects are the real objects to be retained in the first image, and the remaining real objects in the first image are the real objects not to be retained. Based on this, by cropping the first image, a third image including the real objects to be retained in the first image can be obtained, that is: the cropped first image.

[0103] Since the second depth can represent the second distance between each virtual object in the second image and the user of the extended reality device, the objects to be retained in the fourth image are the virtual objects in the virtual world included in the second image that the extended reality device wants to be able to display when showing the target image at a distance greater than or equal to the preset depth threshold from the user.

[0104] Then, according to the magnitude relationship between the second depth and the preset depth threshold, virtual objects in the second image with a second depth greater than or equal to the preset depth threshold can be determined. These virtual objects are the virtual objects to be retained in the second image, and the remaining virtual objects in the second image are the virtual objects not to be retained. Based on this, by cropping the second image, a fourth image including the virtual objects to be retained in the second image can be obtained, that is: the cropped second image.

[0105] After obtaining the third image and the fourth image, through image processing techniques, image processing is performed on the third image including the real objects to be retained in the first image and the fourth image including the virtual objects to be retained in the second image, and an image that includes both the real objects to be retained in the first image and the virtual objects to be retained in the second image can be obtained. Since the first image corresponds to the real world and the second image corresponds to the virtual world of the extended reality device, then, based on the above process, an image including the real world and the virtual world can be obtained, that is: the target image.

[0106] In this embodiment, the target image is obtained through the above method. Since the target image is an image including the real world and the virtual world, therefore, an image that includes both the real world and the virtual world can be displayed on the display of the extended reality device.

[0107] In some embodiments, the cropping of the first image based on the first depth and the preset depth threshold to obtain the third image includes:

[0108] When the first depth is less than the preset depth threshold, determining the real objects to be retained in the first image;

[0109] Crop the first image according to the real object to be retained, to obtain the third image; wherein, the third image includes the real object to be retained.

[0110] In this embodiment, the processor of the extended reality device determines the magnitude relationship between the first depth between each real object in the first image and the user of the extended reality device and a preset depth threshold. When the first depth is less than the preset depth threshold, it is determined that the real object with the first depth less than the preset depth threshold is the real object to be retained in the first image, and the real object with the first depth greater than or equal to the preset depth threshold is the real object not to be retained in the first image.

[0111] Here, the real object to be retained is the real object in the real world included in the first image that the extended reality device wants to be able to display when the distance from the user is less than the preset depth threshold when displaying the target image, so that the user of the extended reality device can interact with these real objects. After determining the real object to be retained, perform a cropping process on the real objects not to be retained in the first image other than the real object to be retained, and the obtained image is the third image.

[0112] In this way, the subsequent obtained target image includes the image of the real object to be retained in the first image (i.e., the third image), and does not include the image of the real object not to be retained in the first image, so that when the extended reality device subsequently displays the target image, it can display the third image when the distance from the user is less than the preset depth threshold, for the user of the extended reality device to be able to interact with the real objects in the real world included in the third image.

[0113] In this embodiment, the real object in the first image with the first depth less than the preset depth threshold is determined as the real object to be retained in the first image, and the first image is cropped based on the real object to be retained to obtain the third image. Since the third image is part of the target image, the above solution can enable the display of the extended reality device to display the corresponding screen of the third image when the distance from the user is less than the preset depth threshold, and this screen is the screen of the real world.

[0114] In some embodiments, the cropping the second image based on the second depth and the preset depth threshold to obtain a fourth image includes:

[0115] When the second depth is greater than or equal to the preset depth threshold, determine the virtual object to be retained in the second image;

[0116] Crop the second image according to the virtual object to be retained, to obtain the fourth image; wherein, the fourth image includes the virtual object to be retained.

[0117] In this embodiment, the processor of the extended reality device determines the magnitude relationship between the second depth of each virtual object in the second image and the user of the extended reality device and a preset depth threshold. When the second depth is greater than or equal to the preset depth threshold, the virtual object with the second depth greater than or equal to the preset depth threshold is determined as the virtual object to be retained in the second image, and the virtual object with the second depth less than the preset depth threshold is determined as the virtual object not to be retained in the second image.

[0118] Here, the virtual object to be retained is a virtual object in the virtual world included in the second image that the extended reality device wants to display when showing the target image when the distance from the user is greater than or equal to the preset depth threshold, so that the user of the extended reality device can have a virtual experience through these virtual objects.

[0119] After determining the virtual object to be retained, crop the virtual objects not to be retained in the second image except for the virtual object to be retained, and the obtained image is the fourth image.

[0120] In this way, the subsequent target image includes the image of the virtual object to be retained in the second image (i.e., the fourth image) and does not include the image of the virtual object not to be retained in the second image, so that when the subsequent extended reality device displays the target image, it can display the fourth image when the distance from the user is greater than or equal to the preset depth threshold, so that the user of the extended reality device can achieve the effect of virtual experience based on the virtual objects in the virtual world included in the fourth image.

[0121] In this embodiment, the virtual object with the second depth greater than or equal to the preset depth threshold in the second image is determined as the virtual object to be retained in the second image, and the second image is cropped based on the virtual object to be retained to obtain the fourth image. Since the fourth image is also part of the target image, the above solution can enable the display of the extended reality device to display the corresponding picture of the fourth image, which is a picture of the virtual world, when the distance from the user is greater than or equal to the preset depth threshold.

[0122] In some embodiments, obtaining the target image based on the third image and the fourth image includes:

[0123] Based on the preset depth threshold, splice the third image and the fourth image according to pixel points to obtain the target image.

[0124] Here, since the third image includes the real objects to be retained in the first image and does not include the real objects that do not need to be retained in the first image, compared with the first image, the third image lacks a part of the content of the first image. And the fourth image includes the virtual objects to be retained in the second image and does not include the virtual objects that do not need to be retained in the second image. Compared with the second image, the fourth image lacks a part of the content of the second image.

[0125] The preset depth threshold is used as the standard for determining the real objects to be retained in the first image and the virtual objects to be retained in the second image. After obtaining the third image and the fourth image based on this standard, according to the preset depth threshold, the pixel points in the third image with depth values less than the preset depth threshold and the pixel points in the fourth image with depth values greater than or equal to the preset threshold are stitched together according to the positions of their corresponding pixel points, and the target image is obtained.

[0126] In this embodiment, the target image is obtained through the above process, which is simple, fast, and easy to implement.

[0127] Exemplarily, Figure 3B is a schematic diagram of a target image shown according to an exemplary embodiment. As Figure 3B shown, the target image includes two parts. The objects included in one part 303 are: the real objects included in the third image, and the objects included in the other part 304 are: the virtual objects included in the fourth image.

[0128] It should be noted that: Figure 3B the real objects included in the third image and the virtual objects included in the fourth image shown in are only for exemplary illustration and are not used for limitation.

[0129] In some embodiments, the first image is multiple frames; the method further includes:

[0130] respectively obtaining the target depth between the real objects in each frame of the first image and the user;

[0131] determining the target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth; wherein, the moving speed of the target real object is greater than a preset speed threshold;

[0132] based on the target depth between the target real object and the user, superimposing and displaying the image corresponding to the target real object in the target image.

[0133] In this embodiment, when the first image is multiple frames, the processor of the extended reality device respectively obtains the target distances, that is, the target depths, between each real object included in each frame of the first image and the user of the extended reality device. Then, based on the acquisition time of each frame of the first image and the target depths between each real object included in each frame of the first image and the user, the processor of the extended reality device determines, from each real object, the real object whose moving speed is greater than a preset speed threshold, that is: the target real object.

[0134] Here, the target real object is a real object in the real world that is not displayed on the display of the extended reality device. However, since the processor of the extended reality device determines that the moving speed of the target real object is greater than the preset speed threshold, it indicates that the target real object may pose a danger to the user of the extended reality device. Therefore, it is necessary to display the target real object on the display of the extended reality device.

[0135] To display the target real object on the display of the extended reality device so that the user can see the target real object for reminding the user, the processor of the extended reality device can determine the corresponding display position of the target real object on the display based on the target depth between the target real object and the user, and superimpose and display the image corresponding to the target real object on the target object based on this display position. In this way, the user of the extended reality device can see the target real object, thereby taking safety protection measures to avoid danger.

[0136] Here, the preset speed threshold can be a value customized by the user or other values, and this embodiment does not limit this.

[0137] In this embodiment, by superimposing and displaying the image corresponding to the target real object on the target image through the above method, it is possible to prompt the user of the extended reality device to indicate that the target real object superimposed and displayed in the target image may cause harm to the user, thereby achieving a warning effect, which can avoid accidents, protect the safety of the user of the extended reality device, and further improve the user experience and satisfaction.

[0138] Figure 3C is a schematic diagram of an image formed by superimposing and displaying the image corresponding to the target real object on the target image according to an exemplary embodiment. Figure 3C is on the basis of Figure 3B after superimposing and displaying the image corresponding to the target real object, and Figure 3C 305 in

[0139] It should be noted that: Figure 3C 305 in

[0140] In some embodiments, determining a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth includes:

[0141] Determining the moving speed of each of the real objects according to the target depth and the acquisition time;

[0142] Among each of the real objects in each frame of the first image, determining as the target real object a real object whose moving speed is greater than the preset speed threshold.

[0143] In this embodiment, the processor of the extended reality device can determine the moving speed of each real object according to the acquisition time of each frame of the first image and the target depth between each real object included in each frame of the first image and the user.

[0144] After obtaining the moving speed of each real object in each frame of the first image, comparing the moving speed of each real object with the preset speed threshold, and based on this size relationship, among each of the real objects in each frame of the first image, determining as the target real object a real object whose moving speed is greater than the preset speed threshold, for subsequent superimposing and displaying the image corresponding to the target real object in the target image. This method of determining the target real object is simple and fast, with simple implementation, and can pre-determine real objects in the real world that may cause harm to the user, for subsequent superimposing and displaying the image corresponding to the target real object in the target image, so that the user can see the target real object, thereby reminding the user.

[0145] Figure 4 is a flowchart of a method for processing data of an extended reality device shown according to an exemplary embodiment. As Figure 4 shown, this method is applied to obtain a target image and mainly includes the following steps:

[0146] In step 401, acquire a first image of the real world and a second image of the virtual world of the extended reality device.

[0147] In step 402, determine a first depth between a real object in the first image and the user of the extended reality device, and a second depth between a virtual object in the second image and the user.

[0148] In step 403, based on the first depth and a preset depth threshold, crop the first image to obtain a third image.

[0149] In step 404, based on the second depth and a preset depth threshold, crop the second image to obtain a fourth image.

[0150] In step 405, a target image is obtained based on the third image and the fourth image, and the target image is displayed; wherein, the objects to be retained in the third image and the fourth image are different.

[0151] Figure 5 is a block diagram of a data processing device for an extended reality device shown according to an exemplary embodiment. As Figure 5 shown, the extended reality device data processing device 500 mainly includes:

[0152] A first acquisition module 501, configured to acquire a first image of the real world and a second image of the virtual world of the extended reality device;

[0153] A first determination module 502, configured to determine a first depth between a real object in the first image and the user of the extended reality device, and a second depth between a virtual object in the second image and the user;

[0154] A first display module 503, configured to perform a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and display the target image; wherein, the target image is an image including the real world and the virtual world.

[0155] In some embodiments, the first display module 503 includes:

[0156] A determination unit, configured to perform a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image;

[0157] A display unit, configured to display the target image;

[0158] Wherein, the determination unit includes:

[0159] A first cropping subunit, configured to crop the first image based on the first depth and the preset depth threshold to obtain a third image;

[0160] A second cropping subunit, configured to crop the second image based on the second depth and the preset depth threshold to obtain a fourth image;

[0161] A target image determination subunit, configured to obtain the target image based on the third image and the fourth image; wherein, the objects to be retained in the third image and the fourth image are different.

[0162] In some embodiments, the first cropping subunit is configured to:

[0163] When the first depth is less than the preset depth threshold, determine the real objects to be retained in the first image;

[0164] Crop the first image according to the real objects to be retained to obtain the third image; wherein, the third image includes the real objects to be retained.

[0165] In some embodiments, the second cropping subunit is configured to:

[0166] When the second depth is greater than or equal to the preset depth threshold, determine the virtual objects to be retained in the second image;

[0167] Crop the second image according to the virtual objects to be retained to obtain the fourth image; wherein, the fourth image includes the virtual objects to be retained.

[0168] In some embodiments, the target image determination subunit is configured to:

[0169] Based on the preset depth threshold, splice the third image and the fourth image pixel by pixel to obtain the target image.

[0170] In some embodiments, the first image is multiple frames; the apparatus 500 further includes:

[0171] A second acquisition module, configured to respectively acquire the target depth between the real objects in each frame of the first image and the user;

[0172] A second determination module, configured to determine a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth; wherein, the moving speed of the target real object is greater than a preset speed threshold;

[0173] A second display module, configured to superimpose and display the image corresponding to the target real object in the target image based on the target depth between the target real object and the user.

[0174] In some embodiments, the second determination module is configured to:

[0175] Determine the moving speed of each of the real objects according to the target depth and the acquisition time;

[0176] Determine, as the target real object, the real object among each of the real objects in each frame of the first image whose moving speed is greater than the preset speed threshold.

[0177] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0178] Figure 6 FIG. 4 is a block diagram of the hardware structure of an electronic device 800 shown according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0179] Referring to Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0180] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0181] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0182] The power component 806 provides power to various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0183] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0184] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0185] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0186] The sensor component 814 includes one or more sensors for providing a status assessment of various aspects of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or a CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0187] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a communication standard-based wireless network, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0188] In an exemplary embodiment, the electronic device 800 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0189] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the above instructions can be executed by a processor 820 of the electronic device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0190] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor, is capable of executing an extended reality device data processing method, the method including:

[0191] Obtaining a first image of the real world and a second image of the virtual world of the extended reality device;

[0192] Determining a first depth between a real object in the first image and the user of the extended reality device, and a second depth between a virtual object in the second image and the user;

[0193] Performing a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and displaying the target image; wherein the target image is an image including the real world and the virtual world.

[0194] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0195] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for processing data of an extended reality device, characterized in that, Including: Obtaining a first image of the real world and a second image of the virtual world of the extended reality device; Determining a first depth between a real object in the first image and a user of the extended reality device, and a second depth between a virtual object in the second image and the user; Performing a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and displaying the target image; wherein, the target image is an image including the real world and the virtual world.

2. The method according to claim 1, wherein The performing a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image includes: Cropping the first image based on the first depth and the preset depth threshold to obtain a third image; Cropping the second image based on the second depth and the preset depth threshold to obtain a fourth image; Obtaining the target image based on the third image and the fourth image; wherein, the objects to be retained in the third image and the fourth image are different.

3. The method according to claim 2, wherein The cropping the first image based on the first depth and the preset depth threshold to obtain a third image includes: Determining a real object to be retained in the first image when the first depth is less than the preset depth threshold; Cropping the first image according to the real object to be retained to obtain the third image; wherein, the third image includes the real object to be retained.

4. The method according to claim 2, wherein The cropping the second image based on the second depth and the preset depth threshold to obtain a fourth image includes: Determining a virtual object to be retained in the second image when the second depth is greater than or equal to the preset depth threshold; Cropping the second image according to the virtual object to be retained to obtain the fourth image; wherein, the fourth image includes the virtual object to be retained.

5. The method according to claim 2, wherein The obtaining the target image based on the third image and the fourth image includes: Performing a stitching process on the third image and the fourth image according to pixel points based on the preset depth threshold to obtain the target image.

6. The method according to any one of claims 1-5, characterized in that The first image is multiple frames; the method further includes: Respectively obtaining a target depth between a real object in each frame of the first image and the user; Determining a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth; wherein, the moving speed of the target real object is greater than a preset speed threshold; Based on the target depth between the target real object and the user, superimposing and displaying the image corresponding to the target real object in the target image.

7. The method according to claim 6, characterized in that, The determining a target real object from each of the real objects according to the acquisition time of each frame of the first image and the corresponding target depth includes: Determining the moving speed of each of the real objects according to the target depth and the acquisition time; Among the real objects in each frame of the first image, a real object with a moving speed greater than the preset speed threshold is determined as the target real object.

8. An extended reality device data processing apparatus, characterized in that, It includes: A first acquisition module configured to acquire a first image of the real world and a second image of the virtual world of the extended reality device; A first determination module configured to determine a first depth between a real object in the first image and a user of the extended reality device, and a second depth between a virtual object in the second image and the user; A first display module configured to perform a cropping process on the first image and the second image based on the first depth, the second depth, and a preset depth threshold to obtain a target image, and display the target image; wherein the target image is an image including the real world and the virtual world.

9. An electronic device, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to: when executed, implement the steps in any one of the extended reality device data processing methods in claims 1 to 7 above.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the extended reality device data processing apparatus, the apparatus can execute any one of the extended reality device data processing methods in claims 1 to 7 above.

Images