Display method of augmented reality equipment and augmented reality equipment

Abstract

The invention provides a display method of augmented reality equipment and the augmented reality equipment, the augmented reality equipment comprises a plurality of cameras, lenses of different cameras face different directions, and the display method of the augmented reality equipment comprises the following steps: collecting environment images by adopting the plurality of cameras; performing image splicing processing on the environment images of the plurality of cameras at the same moment to obtain a spliced image; detecting an object in the spliced image, and generating reminding information according to the detected information of the object; and generating a reminding image according to the reminding information, and displaying the reminding image in a window mode. According to the invention, environment perception dead angles can be reduced, accurate analysis of objects in the environment can be ensured, a user can effectively avoid the objects through the reminding information, and the practicability of the augmented reality equipment and the user safety are improved.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of display, in particular to a display method of an extended reality device and the extended reality device. BACKGROUND

[0002] In the existing XR (Extended Reality) device, the environment perception technology mainly relies on fixed sensors (such as depth cameras, infrared sensors) and a single camera for space scanning and object detection. However, these methods have significant limitations: the field of view of the fixed sensor is limited, and the single camera is difficult to cover the global scene, resulting in the problem of dead angle of environment perception, so as to fail to ensure the accurate analysis of the objects in the environment, and the XR user may have a safety hazard in the real scene due to insufficient perception. SUMMARY

[0003] The embodiment of the application provides a display method of an extended reality device and the extended reality device, and is used for solving the problem that the existing XR device environment perception has a dead angle, so as to fail to ensure the accurate analysis of the objects in the environment, and the XR user may have a safety hazard in the real scene due to insufficient perception.

[0004] In order to solve the above technical problems, the application is implemented as follows:

[0005] In a first aspect, the embodiment of the application provides a display method of an extended reality device, the extended reality device comprising a plurality of cameras, the lenses of different cameras being directed to different directions, and the display method of the extended reality device comprising:

[0006] collecting environment images by using the plurality of cameras;

[0007] performing image stitching processing on the environment images of the same moment of the plurality of cameras to obtain a stitched image;

[0008] detecting objects in the stitched image, and generating reminding information according to the information of the detected objects;

[0009] generating a reminding image according to the reminding information, and displaying the reminding image in a window mode.

[0010] Optionally, the plurality of cameras are arranged in a surrounding manner, and the fields of view of adjacent cameras at least partially overlap.

[0011] Optionally, the stitched image is a top-view perspective image, and the performing image stitching processing on the environment images of the same moment of the plurality of cameras to obtain a stitched image further comprises:

[0012] For each camera, obtain a source image containing a calibration cloth taken by the camera, and obtain pixel coordinates of a plurality of calibration coordinate points in the source image and a target image, wherein a coordinate system in the target image is a coordinate system of a stitched image;

[0013] According to the pixel coordinates of the plurality of calibration coordinate points in the source image containing the calibration cloth and the target image, a projection transformation relationship between the source image containing the calibration cloth and the target image is determined.

[0014] The image stitching processing of the environment images of the plurality of cameras at the same time includes:

[0015] According to the projection transformation relationship of the cameras, the environment images of the cameras are converted into top-view perspective images.

[0016] The image stitching processing of the environment images of the plurality of cameras at the same time includes:

[0017] The image stitching processing of the environment images of the plurality of cameras at the same time includes:

[0018] The image stitching processing includes: for two environment images to be stitched, dividing the two environment images to be stitched into an overlapping region and a non-overlapping region according to an intersection of the two environment images to be stitched; calculating a distance of each pixel in the overlapping region of the two environment images to be stitched from a boundary of the overlapping region and the non-overlapping region, determining a weight of each pixel in the overlapping region according to the distance to obtain a normalized weight image, wherein the closer the pixel in the overlapping region is to the boundary, the greater the weight is; obtaining a fused overlapping region according to the overlapping region of the two environment images to be stitched and the normalized weight image; and stitching the fused overlapping region with the non-overlapping regions of the two environment images to be stitched respectively to obtain a local stitched image.

[0019] If not, the local stitched image is taken as a new environment image to be stitched to continue the image stitching processing, otherwise, the local stitched image is taken as the stitched image.

[0020] Optionally, the generating of the reminder information according to the detected information of the object includes:

[0021] The information of the object includes at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object.

[0022] determine a safe area and / or a dangerous area in the stitching image according to the information of the object, the dangerous area including at least one of: an area including the object and a distance between the object and the user being less than or equal to a preset distance threshold, an area containing the dynamic object; the safe area including at least one of: an area not including an object, an area including an object and a distance between the object and the user being greater than the preset distance threshold;

[0023] take the safe area and / or the dangerous area as the reminder information.

[0024] Optionally, the generating reminder information according to the detected information of the object further includes:

[0025] taking the indication information of whether the object is a static object or a dynamic object as the reminder information.

[0026] Optionally, the displaying the reminder image in a window mode includes:

[0027] displaying the reminder image in a window mode when a condition for triggering window display is met;

[0028] wherein the condition for triggering window display includes at least one of:

[0029] the user wearing the extended reality device is in a moving state;

[0030] the reminder information contains a dynamic object;

[0031] the reminder information contains a dangerous area;

[0032] receiving a display confirmation operation of the user, the display confirmation operation being used to indicate displaying the window.

[0033] Optionally, the method further includes:

[0034] generating a vibration reminder information when the user approaches the dangerous area.

[0035] Optionally, the generating reminder information according to the detected information of the object includes:

[0036] obtaining information of the object, the information of the object including at least one of: position information of the object, distance information between the object and the user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object;

[0037] generating a planned path according to the information of the object;

[0038] taking the planned path as the reminder information.

[0039] Optionally, the generating the planning path according to the information of the object further comprises:

[0040] generating voice reminding information according to the planning path, the voice reminding information comprising at least one of the following: environmental information of a current position of the user, action suggestion for the user.

[0041] Optionally, the generating the reminding image according to the reminding information further comprises:

[0042] labeling the reminding information into the stitched image to obtain the reminding image;

[0043] or,

[0044] determining a position of the object corresponding to the reminding information, and generating a map image containing the reminding information according to the position of the object corresponding to the reminding information and a position of the user, and taking the map image as the reminding image.

[0045] Optionally, the displaying the reminding image in a window mode further comprises:

[0046] in a case where a selection operation of selecting a target object in the reminding image is received from the user, performing a target processing operation, the target processing operation comprising at least one of the following:

[0047] obtaining an enlarged image of an image region in which the target object is located in the reminding image and displaying the enlarged image;

[0048] obtaining an environment image before stitching corresponding to the target object, and obtaining an enlarged image of an image region in which the target object is located in the environment image before stitching and displaying the enlarged image;

[0049] recognizing the target object, and displaying information of the target object in the reminding image or the enlarged image.

[0050] Optionally, the collecting the environment images by using the plurality of cameras further comprises:

[0051] dividing the plurality of cameras into a plurality of camera groups; wherein, every preset number of cameras, one camera is selected to form a camera group;

[0052] opening the plurality of camera groups one by one in time sequence, or opening each camera group according to a preset period, or opening the camera groups according to a control instruction, wherein, only at least one camera group is enabled in each time period, and other camera groups are in a dormant state;

[0053] collecting environment images in real time by using the opened camera groups.

[0054] Optionally, the collecting the environment image by using the plurality of cameras comprises:

[0055] the camera corresponding to the area without change in the environment for more than the preset time length is closed, and the camera corresponding to the area with continuous change in the environment is continuously opened;

[0056] the environment image is collected by using the opened camera.

[0057] Optionally, the method further comprises:

[0058] if the environment does not change in the preset time length, the step of performing the image splicing processing on the environment images of the plurality of cameras at the same time is not performed, or the frame rate of the camera for collecting the environment image is reduced.

[0059] In a second aspect, an embodiment of the present application provides an extended reality device, comprising:

[0060] a plurality of cameras, lenses of different cameras are directed to different directions, and the plurality of cameras are used to collect environment images;

[0061] a data processing module, used to perform image splicing processing on the environment images of the plurality of cameras at the same time, to obtain a spliced image;

[0062] an environment perception module, used to detect objects in the spliced image, and generate reminding information according to information of the detected objects;

[0063] a display module, used to generate a reminding image according to the reminding information, and display the reminding image in a window mode.

[0064] In a third aspect, an embodiment of the present application provides an extended reality device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the program is executed by the processor, the steps of the display method of the extended reality device are implemented.

[0065] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method of the extended reality device are implemented.

[0066] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, and when the computer instructions are executed by a processor, the steps of the display method of the extended reality device are implemented.

[0067] In the embodiment of the present application, a plurality of cameras are arranged in the extended reality device, the lenses of different cameras are directed to different directions, the environment images are collected through the plurality of cameras, and the environment images of the plurality of cameras at the same time are subjected to image stitching processing to obtain a stitched image, the stitched image can cover more fields of view, thereby reducing the dead angle of environment perception, overcoming the limitations of the existing environment perception technology, in addition, the objects in the stitched image are detected, and a reminder image containing reminder information generated based on the objects is generated, so that not only the immersion of the user can be improved, but also the user can accurately perceive the surrounding environment and safely avoid the objects through the clear and easy-to-understand reminder information, higher safety and adaptability are provided, and great technical advantages and application potential are shown. BRIEF DESCRIPTION OF DRAWINGS

[0068] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is merely meant to teach a person of ordinary skill a way to make and use the application. It is not meant to put limitations on the scope of the application. Moreover, the same reference numerals are intended to denote the same components throughout the text. In the drawings:

[0069] Figure 1 A flowchart of a display method of an extended reality device according to an embodiment of the present application;

[0070] Figure 2 A structural diagram of a multi-camera system according to an embodiment of the present application;

[0071] Figure 3 A state diagram of a normal image space and a distorted image space;

[0072] Figure 4 A diagram of an image calibration method according to an embodiment of the present application;

[0073] Figure 5 A diagram of converting an environment image into an overhead view according to an embodiment of the present application;

[0074] Figure 6 A diagram of image fusion according to an embodiment of the present application;

[0075] Figure 7 A diagram of stitching two environment images to be stitched to obtain a local stitched image according to an embodiment of the present application;

[0076] Figure 8 A diagram of a reminder image displayed according to an embodiment of the present application;

[0077] Figure 9 A structural diagram of an extended reality device according to an embodiment of the present application;

[0078] Figure 10Figure 2 is a structural schematic diagram of an extended reality device according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0080] Reference is made to Figure 1 The display method of the extended reality device provided in the embodiments of the present application can be a display method of a VR (Virtual Reality) device, an AR (Augmented Reality) device or a MR (Mixed Reality) device. The extended reality device comprises a plurality of cameras, and the lenses of different cameras are directed to different directions, that is, the field of view ranges of different cameras are different.

[0081] In some embodiments, the plurality of cameras form a multi-camera system, which is a neck-mounted multi-camera system, Figure 2 Figure 2 In the neck-mounted multi-camera system, 21 is a neck-mounted carrier, and 22 is a camera arranged on the neck-mounted carrier. The plurality of cameras are arranged in a surrounding manner, and the field of view of adjacent cameras at least partially overlaps.

[0082] In some embodiments, the plurality of cameras form a multi-camera system, which is a neck-mounted multi-camera system,

[0083] In the embodiments of the present application, the field of view of the plurality of cameras can cover 360 degrees, so that the requirement of obtaining a perception panorama image can be met, and efficient data support is provided for subsequent other processing (such as path planning or dynamic obstacle avoidance).

[0084] In the embodiments of the present application, the plurality of cameras are high-frame-rate cameras, for example, cameras capable of supporting image acquisition at a speed of 1000-10000 frames per second.

[0085] In the embodiments of the present application, the display method of the extended reality device comprises:

[0086] S1: acquiring an environment image by using the plurality of cameras;

[0087] ​Step S2: image stitching processing is performed on the environment images of the plurality of cameras at the same time to obtain a stitched image;

[0088] In the embodiment of the present application, before the image stitching processing is performed on the environment images of the plurality of cameras at the same time, the environment images collected by the plurality of cameras can also be time-synchronized to ensure that the plurality of environment images correspond to the same environment state at the same time point, thereby avoiding the image content misplacement problem caused by time deviation.

[0089] In the embodiment of the present application, the image captured by one camera can be referred to as a single-view image. The image formed by stitching the images of different views captured by the plurality of cameras can be referred to as a multi-view stitched image.

[0090] In the embodiment of the present application, when the view angles of the plurality of cameras can cover 360 degrees and all the cameras are turned on at the same time, the omnidirectional environment image can be collected, and the stitched image obtained from the environment images collected by the plurality of cameras at the same time is a panoramic image without dead angle, so as to eliminate the view angle blind area and comprehensively cover the environment around the user.

[0091] Step S3: detecting the object in the stitched image, and generating the reminding information according to the information of the detected object,

[0092] Step S4: generating the reminding image according to the reminding information, and displaying the reminding image in a window mode.

[0093] In the embodiment of the present application, compared with the single-view image, the stitched image has more information, more uniform target distribution, clearer geometric information and less target occlusion.

[0094] In the embodiment of the present application, the reminding image can be displayed in the form of a small window in the user's field of view, for example, the reminding image is displayed in the form of a window at the upper right corner of the user's field of view for the user to refer to.

[0095] In the embodiment of the present application, a plurality of cameras are arranged in the extended reality device, the lenses of different cameras are directed to different directions, the environment images are collected by the plurality of cameras, the image stitching processing is performed on the environment images of the plurality of cameras at the same time to obtain a stitched image, the stitched image can cover more field of view, thereby reducing the dead angle of environment perception, overcoming the limitations of the existing environment perception technology, in addition, the object in the stitched image is detected, and the reminding image containing the reminding information generated based on the object is generated, so as to not only improve the immersion of the user, but also help the user to accurately perceive the surrounding environment and safely avoid the object through the clear and easy-to-understand reminding information, provide higher safety and adaptability, and show great technical advantages and application potential.

[0096] In some embodiments, the generating the reminding image according to the reminding information comprises:

[0097] annotating the reminding information into the spliced image to obtain the reminding image.

[0098] In this embodiment, the reminding information is annotated into the real environment image to obtain the reminding image.

[0099] In some other embodiments, the generating the reminding image according to the reminding information comprises:

[0100] determining the position of the object corresponding to the reminding information, generating a map image containing the reminding information according to the position of the object corresponding to the reminding information and the position of the user, and taking the map image as the reminding image.

[0101] In this embodiment, the reminding information is not annotated into the real environment, but a map image is generated, in which the real environment is not displayed, and only the position of the object corresponding to the reminding information is annotated, so that it is more concise.

[0102] In some embodiments, the spliced image is a top-view perspective image. In the embodiment of the present application, the normal perspective image needs to be converted into a top-view perspective image, then spliced, and the environment image does not need to be reconstructed (the required calculation power is large, and it is usually executed in the cloud), the calculation power required for converting the normal perspective image into a top-view perspective image is small, and it can be completed on the extended reality device side, in addition, the top-view perspective image can also make the user more easily observe the objects on the ground.

[0103] In some embodiments, the spliced image is a top-view panoramic image. That is, the environment images around the user collected by the plurality of cameras are converted into a top-view panoramic image in real time, and the object information in the environment is intuitively presented.

[0104] In some embodiments, the spliced image is a top-view perspective image, and the image splicing processing of the environment images of the plurality of cameras at the same time comprises:

[0105] Step S21: performing image correction on the plurality of environment images, the image correction comprising at least one of the following: image distortion correction, image brightness normalization, image contrast normalization, and image definition enhancement.

[0106] Image distortion refers to the shape distortion of an image caused by factors such as camera lens, sensor, imaging conditions, etc. during image acquisition. Common types of image distortion include radial distortion and tangential distortion. Radial distortion is caused by uneven radial magnification of the lens, which usually manifests as stretching or squeezing of the image edges; tangential distortion is caused by a slight misalignment between the lens and the imaging plane, which causes the image to tilt with the edges. Image distortion correction is a correction process for the above shape distortion. Please refer to Figure 3 , Figure 3 The normal image space and the spatial state of the distorted image are shown in the figure. Image distortion correction can make the problem image return to the normal state.

[0107] In the embodiments of the present application, the following formula can be used for image distortion correction:

[0108] x distorted =x·(1+k1r 2 +k2r 4 +k3r 6 )+[2p1xy+p2(r 2 +2x 2 )]

[0109] y distorted =y·(1+k1r 2 +k2r 4 +k3r 6 )+[p1(r 2 +2y 2 )+2p2xy]

[0110] Where x, y are the normalized pixel coordinates after distortion correction (relative to the optical center); x distorted , y distorted are the normalized pixel coordinates after distortion. k1, k2, k3 are radial distortion coefficients, and p1, p2 are tangential distortion coefficients.

[0111] Image brightness normalization is a technique for adjusting image pixel values to a uniform range, mainly used to eliminate lighting differences and improve subsequent processing results. Common methods include maximum and minimum value normalization, color space conversion, gamma correction, etc.

[0112] Contrast usually refers to the standard deviation of pixels in an image or image region. Contrast normalization includes global contrast normalization and local contrast normalization to reduce the amount of variation in the data.

[0113] By performing image correction processing on the environment image, the environment image is more suitable for generating and fusing the stitched image.

[0114] In some embodiments, the image stitching of the environment images of the plurality of cameras at the same time is performed to obtain a stitched image, and the method further comprises:

[0115] Step S01: For each camera, a source image containing a calibration cloth photographed by the camera is obtained, and pixel coordinates of a plurality of calibration coordinate points in the source image and a target image are obtained, wherein a coordinate system in the target image is a coordinate system of a stitched image.

[0116] The target image can be a checkerboard image or the like.

[0117] In the embodiments of the present application, each camera only needs to be calibrated once.

[0118] Step S02: A projection transformation relationship between the source image and the target image is determined according to the pixel coordinates of the plurality of calibration coordinate points in the source image and the target image.

[0119] Optionally, the number of calibration coordinate points is greater than or equal to 4.

[0120] Please refer to Figure 4 , Figure 4 The left side of the target image is a target image, and the right side is a calibrated environment image. As can be seen, the target image includes four calibration coordinate points, and the calibrated environment image also includes four corresponding calibration coordinate points.

[0121] Please refer to Figure 4 , the position coordinates of the coordinate points 1, 2, 3 and 4 in the target image can be known through the calibration cloth. By clicking on each corresponding point on the calibration cloth in the source image, Figure 4 the coordinates of the coordinate points 1, 2, 3 and 4 in the source image can be obtained.

[0122] In the embodiments of the present application, the length of each square grid in the calibration cloth is x cm (but not limited to 30 cm), and x cm = y pixel in the target image. Based on the above correspondence, the pixel number of the user to the edge of an object in the environment stitched image can be obtained, and the real distance between the two can be estimated without using complex algorithms such as binocular distance measurement method, thereby improving the efficiency.

[0123] In the embodiments of the present application, the projection transformation relationship can be a homography matrix (or a fitting projection matrix). The homography matrix is a key concept in projective geometry, which is used to describe the perspective transformation relationship between two planes.

[0124] In some embodiments, the formula of the projection transformation relationship can be:

[0125]

[0126] wherein (x, y) is a point in the source image (environment image), (x', y') is the corresponding point on the target image. H is the homography matrix (or called the fitted projection matrix) of each camera.

[0127] In the embodiments of the present application, optionally, to ensure that the relative position remains unchanged and the projection transformation relationship remains unchanged, a gimbal or other physical image stabilization method can be provided for each camera.

[0128] The multi-camera system in the embodiments of the present application can also include an inertial sensor (IMU) for obtaining real-time attitude information of the XR device, such as rotation angle and / or position offset. The real-time attitude information can be used to adjust the homography matrix to correspond to different poses of the multi-camera system.

[0129] wherein the image stitching processing of the environment images of the plurality of cameras at the same time is performed to obtain a stitched image, comprising:

[0130] Step S21': converting the camera environment image into a top-view perspective image according to the camera projection transformation relationship;

[0131] Please refer to Figure 5 , Figure 5 The left side is the calibrated environment image, and the right side is the top-view perspective image.

[0132] Step S22': performing image stitching processing on the top-view perspective images of the plurality of cameras to obtain a stitched image.

[0133] Image stitching technology is a technology of combining multiple images with overlapping parts into a seamless panoramic image or a high-resolution image. The core process includes image registration and image fusion. The former establishes a geometric transformation relationship through feature point extraction and matching, and the latter uses weighted averaging, multi-band fusion and other methods to eliminate stitching marks.

[0134] In the embodiments of the present application, optionally, the image fusion can use at least one of the following: a pixel-level fusion-based algorithm, a feature fusion-based algorithm, and a deep learning-based fusion algorithm. The ultimate purpose of using these algorithms is not limited to effectively integrating complementary information in multiple images to generate a fused image with smooth seams, weak artifacts, and rich details, providing clear and dead-angle-free visual information for environment perception.

[0135] The image stitching method of the embodiments of the present application is described below taking the pixel-level fusion-based algorithm as an example. The pixel-level fusion-based algorithm can effectively improve the fusion speed and reduce the product power consumption.

[0136] In some embodiments, the image stitching processing on the environment images of the same moment of the plurality of cameras, to obtain a stitched image, comprises:

[0137] Step S21'': performing image stitching processing, the image stitching processing comprising: for two environment images to be stitched, dividing the environment images to be stitched into an overlapping region and a non-overlapping region according to the intersection of the two environment images to be stitched; calculating the distance of each pixel in the overlapping region of the environment images to be stitched from the boundary of the overlapping region and the non-overlapping region, determining the weight of each pixel in the overlapping region according to the distance to obtain a normalized weight image, wherein the closer the pixel in the overlapping region is to the boundary, the greater the weight; obtaining a fused overlapping region according to the overlapping region of the two environment images to be stitched and the normalized weight image; stitching the fused overlapping region with the non-overlapping region of the two environment images to be stitched respectively to obtain a local stitched image.

[0138] In the embodiment of the application, the distance can be Euclidean distance, etc.

[0139] In the embodiment of the application, for two environment images to be stitched, dividing the environment images to be stitched into an overlapping region and a non-overlapping region according to the intersection of the two environment images to be stitched can comprise: converting the two environment images to be stitched (usually RGB images) into two grayscale images, and calculating the intersection of the two grayscale images to divide the environment images to be stitched into an overlapping region and a non-overlapping region.

[0140] In the embodiment of the application, the farther the pixel in the overlapping region is from the boundary (the boundary of the overlapping region and the non-overlapping region), the smaller the weight, and the closer the distance, the greater the weight.

[0141] In the embodiment of the application, the fused overlapping region can be obtained according to the overlapping region of the two environment images to be stitched and the normalized weight image, specifically, multiplying the overlapping region by the normalized weight image of the two environment images to be stitched to obtain the fused overlapping region.

[0142] Please refer to Figure 6 , Figure 6 for the schematic diagram of image fusion.

[0143] Please refer to Figure 7 , Figure 7 for the schematic diagram of the local stitched image obtained after stitching the two environment images to be stitched.

[0144] Step S22'': judging whether the splicing is completed, if not, continuing to execute the image splicing processing by taking the local spliced image as a new environment image to be spliced, otherwise, taking the local spliced image as the spliced image. If all the cameras of the multi-camera system are turned on, the final obtained spliced image is a global spliced image.

[0145] In the embodiment of the present application, whether the splicing is completed is judged by judging whether the images to be spliced have been used up. When the images to be spliced have been used up, it is determined that the splicing is completed, otherwise, it is determined that the splicing is not completed.

[0146] In the embodiment of the present application, the following splicing strategies can be used for image splicing.

[0147] Scheme one: two-by-two splicing of two adjacent images.

[0148] For example, assuming that the multi-camera system includes 12 cameras, which are numbered in sequence as: 01, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 14.

[0149] The splicing scheme can be:

[0150] (01, 03) (04, 05) (06, 07) (08, 09) (10, 11) (12, 14)→(0103, 0405) (0607, 0809) (1011, 1214)→(01030405, 06070809) (1011, 1214)→(0103040506070809, 10111214)→(010304050607080910111214).

[0151] Scheme two: splicing in sequence according to the sequence of the cameras.

[0152] The splicing scheme can be:

[0153] (01, 03)→(01, 03, 04)→(01, 03, 04, 05)→(01, 03, 04, 05, 06)→(01, 03, 04, 05, 06, 07)→(01, 03, 04, 05, 06, 07, 08)→(01, 03, 04, 05, 06, 07, 08, 09)→(01, 03, 04, 05, 06, 07, 08, 09, 10)→(01, 03, 04, 05, 06, 07, 08, 09, 10, 11)→(01, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12)→(01, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 14).

[0154] It should be noted that the splicing scheme is not limited to the above two.

[0155] The multi-camera system in the embodiment of the application can further comprise an inertial sensor (IMU) for acquiring real-time attitude information of the XR device, such as a rotation angle and / or a position offset. The real-time attitude information can be used as a registration basis for consistency of local spliced images to global spliced images, to ensure that multiple environment images can be spliced into a large spliced image.

[0156] In the embodiment of the application, optionally, a target detection model (YOLO, Mask R-CNN, etc.) can be used to process objects in the spliced image. By counting the number of pixels occupied by the target and the shortest number of pixels between the target frame edge and the user, the size, distance, and other key information of the object can be obtained.

[0157] In some embodiments, optionally, the generation of the reminder information according to the detected information of the object comprises:

[0158] Step S31: acquiring information of the object, the information of the object comprising at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object;

[0159] In the embodiment of the application, a deep learning model can be used to identify the information of the object. Through depth information and object estimation on the environment spliced image, the time complexity and space complexity of the algorithm can be significantly reduced, and the real-time performance of the algorithm is improved.

[0160] Step S32: determining a safe area and / or a dangerous area in the spliced image according to the information of the object, the dangerous area comprising at least one of the following: an area comprising the object and having a distance between the object and the user less than or equal to a preset distance threshold, and an area comprising the dynamic object; and the safe area comprising at least one of the following: an area not comprising the object, and an area comprising the object and having a distance between the object and the user greater than the preset distance threshold.

[0161] Step S33: taking the safe area and / or the dangerous area as the reminder information.

[0162] For example, the dangerous area can be displayed by highlighting a certain area.

[0163] In the embodiment of the application, the safe area and / or the dangerous area are determined according to the detected information of the object, and the safe area and / or the dangerous area are taken as the reminder information, so as to remind the user to avoid the object.

[0164] In some embodiments, the generating the reminding information according to the information of the object detected, optionally, further includes:

[0165] The step S31' includes: taking the indication information of whether the object is a static object or a dynamic object as the reminding information.

[0166] In some embodiments, the displaying the reminding image in a window mode, optionally, includes:

[0167] The reminding image is displayed in a window mode when the condition of triggering the window display is met.

[0168] The condition of triggering the window display includes at least one of the following:

[0169] The user wearing the extended reality device is in a moving state.

[0170] The reminding information contains a dynamic object.

[0171] The reminding information contains a dangerous area.

[0172] The display confirmation operation of the user is received, and the display confirmation operation is used to indicate to display the window.

[0173] It should be noted that if the user is in a stationary state, the condition is not met, and the window is not displayed.

[0174] That is, in the embodiments of the present application, the reminding image can not be displayed all the time, but the reminding image is displayed when a certain triggering condition is met, thereby saving the power consumption of the device.

[0175] In some embodiments, the generating the reminding information according to the information of the object detected, optionally, includes:

[0176] The step S31'' includes: obtaining the information of the object, and the information of the object includes at least one of the following: the position information of the object, the distance information between the object and the user wearing the extended reality device, the size information of the object, and the indication information of whether the object is a static object or a dynamic object.

[0177] The step S32'' includes: generating a planned path according to the information of the object.

[0178] The planned path is used to guide the user to move forward.

[0179] In the embodiments of the present application, the planned path can be generated by using a path planning algorithm, and the path planning algorithm includes but is not limited to A* algorithm or dynamic window algorithm (DWA).

[0180] Optionally, in the embodiment of the present application, when the path planning is performed, the passable region in the direction in which the user is heading is preferentially selected.

[0181] Step S33'': taking the planned path as the reminding information.

[0182] In the embodiment of the present application, the planned path can be taken as the reminding information in the form of an indication line or the like, so as to facilitate the user to understand and follow.

[0183] In the embodiment of the present application, the method further comprises:

[0184] When a new object is detected in the stitched image, the planned path is updated in real time (i.e., steps S31'' and S32'' are re-executed), so as to avoid collision risks caused by environmental changes and ensure the safety and continuity of the user during the travel.

[0185] In the embodiment of the present application, the planned path can be dynamically adjusted according to the information of the identified object, so as to ensure the safety of the user.

[0186] Please refer to Figure 8 , Figure 8 The schematic diagram of the displayed reminding image in the embodiment of the present application is shown in FIG. 2. Figure 8 As can be seen from FIG. 2, the reminding image includes reminding information that the region is a dangerous region, reminding information that the object is a static object or a dynamic object, and a planned path.

[0187] In some embodiments, the method further comprises:

[0188] When the user approaches the dangerous region, vibration reminding information is generated.

[0189] In the embodiment of the present application, the user approaching the dangerous region may, for example, mean that the distance between the user and the dangerous region is less than or equal to a preset distance threshold.

[0190] In the embodiment of the present application, different vibration frequencies and / or intensities can be used to represent the degree of danger, for example, the more frequent the vibration, the closer the user is to the dangerous region, and the greater the vibration intensity, the closer the user is to the dangerous region.

[0191] In the embodiment of the present application, vibration feedback is used to help the user to safely avoid obstacles and provide the user with an intuitive and efficient interactive experience.

[0192] In some embodiments, the method further comprises:

[0193] Step S34'': generating voice reminding information according to the planned path, the voice reminding information including at least one of the following: environmental information of the current position of the user, and suggestions for the action of the user.

[0194] The voice reminding information may be, for example, "there is an obstacle in front of you, please move to the right" and the like.

[0195] In the embodiment of the present application, voice feedback is used to help avoid obstacles and provide intuitive and efficient interactive experience for users.

[0196] In the embodiment of the present application, when the view angles of the plurality of cameras can cover 360 degrees and all the cameras are turned on at the same time, the spliced image obtained from the environment images collected by the plurality of cameras is an environment panoramic image. However, turning on all the cameras at the same time will cause high power consumption.

[0197] In some embodiments, the reminding image is optionally displayed in a window mode, and the method further includes:

[0198] In a case where a selection operation of selecting a target object in the reminding image is received, a target processing operation is performed, and the target processing operation includes at least one of the following:

[0199] An enlarged image of an image region in which the target object is located in the reminding image is obtained and displayed;

[0200] An environment image before splicing corresponding to the target object is obtained, and an enlarged image of an image region in which the target object is located in the environment image before splicing is obtained and displayed;

[0201] The target object is recognized, and information of the target object is displayed in the reminding image or the enlarged image.

[0202] For example, when the user sees the reminding image with the reminding information (such as a dangerous area, a dynamic object, etc.) marked therein, the user may not be able to clearly see the object in the dangerous area, and thus the target object can be enlarged for viewing by selecting the target object in the reminding image. Especially for the top-down spliced image, the object in the top-down spliced image may be severely deformed, affecting the viewing effect, and thus the enlarged viewing can improve the viewing effect of the user, or the enlarged image of the target object in the environment image before splicing (the environment image without top-down view angle transformation) is viewed, further facilitating the user to identify the target object.

[0203] In some embodiments, in order to save the power consumption of the multi-camera system and prolong the use time of the device, the method of collecting environment images by using the plurality of cameras includes:

[0204] Step S11: dividing the plurality of cameras into a plurality of camera groups; wherein every preset number of cameras, one camera is selected to form a camera group;

[0205] Step S12: sequentially open the plurality of camera groups in time sequence, or open each camera group according to a preset period, or open the camera group according to a control instruction, wherein at least one camera group is enabled in each time period, and other camera groups are in a dormant state.

[0206] Step S13: real-time collection of environment images by using the opened camera group.

[0207] For example, the camera groups are sequentially opened in time periods using a round-robin strategy. Assuming that there are 12 cameras, 4 cameras form a group, and only one group of cameras is enabled each time, and other camera groups are in a dormant state. Every certain time (for example, 500 ms to 1 second), the currently working camera group is switched, so that the multi-camera system always maintains part of the cameras in a working state, thereby reducing power consumption.

[0208] In the embodiment of the application, optionally, every preset number of cameras (for example, every 3 cameras) can be selected to form a camera group, so as to cover more fields of view as much as possible.

[0209] In some embodiments, in order to save the power consumption of the multi-camera system and prolong the use time of the device, optionally, the collection of the environment images by using the plurality of cameras comprises:

[0210] Step S11': the cameras corresponding to the regions without changes in the environment for more than a preset time period are closed, and the cameras corresponding to the regions with continuous changes in the environment are continuously opened;

[0211] Step S12': the opened cameras are used to collect environment images.

[0212] In some embodiments, in order to save the power consumption of the multi-camera system and prolong the use time of the device, optionally, the method further comprises: if the environment does not change within a preset time period, the step of image splicing processing of the environment images at the same time by the plurality of cameras is not performed (for example, only the object detection step can be retained, and further, the above image correction step can not be performed), or the frame rate of the collection of the environment images by the cameras is reduced, for example, the frame rate of the image collection is reduced from 60 frames per second to 10 frames per second.

[0213] In summary, the display method of the extended reality device in the embodiment of the application has the following advantages:

[0214] Global coverage: the multi-camera collects environment information, eliminates the blind area of the field of view, and comprehensively covers the environment around the user;

[0215] High-precision perception: the scale of the bird's-eye panoramic image is aligned with the actual scene (calibration), which ensures accurate analysis of objects and environmental layout;

[0216] Real-time dynamic processing: using deep learning models to identify static and dynamic objects, dynamically adjusting path planning to ensure user safety;

[0217] Multi-modal feedback: providing users with intuitive and efficient interactive experiences through visual, voice, and / or vibration feedback of XR devices.

[0218] Please refer to Figure 9 The embodiment of the application also provides an extended reality device 100, comprising:

[0219] A multi-camera system 101, comprising a plurality of cameras, the lenses of different cameras facing different directions, for collecting environment images;

[0220] A data processing module 102, for performing image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image;

[0221] An environment perception module 103, for detecting objects in the stitched image and generating reminder information according to the information of the detected objects;

[0222] A display module 104, for generating a reminder image according to the reminder information and displaying the reminder image in a window mode.

[0223] In the embodiment of the application, a plurality of cameras are provided in the extended reality device, the lenses of different cameras face different directions, environment images are collected by the plurality of cameras, image stitching processing is performed on the environment images of the plurality of cameras at the same time to obtain a stitched image, the stitched image can cover more fields of view, thereby reducing the dead angle of environment perception and overcoming the limitations of existing environment perception technologies. In addition, objects in the stitched image are detected, and a reminder image containing reminder information generated based on the objects is generated, so that not only the immersion of users can be improved, but also users can accurately perceive the surrounding environment and safely avoid objects through clear and easy-to-understand reminder information, providing higher safety and adaptability, and showing great technical advantages and application potential.

[0224] In some embodiments, the display module 104 can optionally label the reminder information into the stitched image to obtain the reminder image.

[0225] In some embodiments, the display module 104 can optionally determine the position of the object corresponding to the reminder information, generate a map image containing the reminder information according to the position of the object corresponding to the reminder information and the position of the user, and take the map image as the reminder image.

[0226] In some embodiments, the multi-camera system 101 can be a wrap-around neck-mounted multi-camera system, or the multi-camera system can be a wrap-around head-mounted multi-camera system.

[0227] The plurality of cameras of the multi-camera system 101 are arranged in a surround manner, and the fields of view of adjacent cameras at least partially overlap.

[0228] In some embodiments, the stitched image is a top-view image.

[0229] In some embodiments, the extended reality device 100 further comprises:

[0230] The calibration module is configured to, for each camera, acquire a source image containing a calibration cloth taken by the camera, acquire pixel coordinates of a plurality of calibration coordinate points in the source image and in a target image, the coordinate system in the target image being a coordinate system of a stitched image, and determine a projection transformation relationship between the source image containing the calibration cloth and the target image according to the pixel coordinates of the plurality of calibration coordinate points in the source image and in the target image.

[0231] The data processing module 102 further comprises:

[0232] The conversion submodule is configured to convert the environment image of the camera into a top-view image according to the projection transformation relationship of the camera.

[0233] The stitching submodule is configured to perform image stitching processing on the top-view images of the plurality of cameras to obtain a stitched image.

[0234] Optionally, the stitching submodule is configured to perform image stitching processing, and the image stitching processing comprises: for two environment images to be stitched, dividing the two environment images to be stitched into an overlapping region and a non-overlapping region according to an intersection of the two environment images to be stitched; calculating a distance of each pixel in the overlapping region of the two environment images to be stitched to a boundary of the overlapping region and the non-overlapping region, determining a weight of each pixel in the overlapping region according to the distance to obtain a normalized weight image, wherein the closer the pixel in the overlapping region is to the boundary, the greater the weight is; obtaining a fused overlapping region according to the overlapping regions of the two environment images to be stitched and the normalized weight image; stitching the fused overlapping region with the non-overlapping regions of the two environment images to be stitched respectively to obtain a local stitched image; determining whether the stitching is completed, and if not, continuing to perform the image stitching processing by taking the local stitched image as a new environment image to be stitched, otherwise, taking the local stitched image as the stitched image.

[0235] Optionally, the environment perception module 103 is configured to acquire the object information, the object information comprising at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object; determine a safe area and / or a dangerous area in the stitching image according to the object information, the dangerous area comprising at least one of the following: an area including the object and having a distance between the object and the user less than or equal to a preset distance threshold, and an area including the dynamic object; the safe area comprising at least one of the following: an area not including the object, and an area including the object and having a distance between the object and the user greater than the preset distance threshold; and use the safe area and / or the dangerous area as the reminder information.

[0236] Optionally, the environment perception module 103 is configured to use the indication information of whether the object is a static object or a dynamic object as the reminder information.

[0237] Optionally, the display module 104 is configured to display the reminder image in a window mode when a condition for triggering window display is met.

[0238] display the reminder image in a window mode when a condition for triggering window display is met.

[0239] The condition for triggering window display comprises at least one of the following:

[0240] The user wearing the extended reality device is in a moving state.

[0241] The reminder information includes the dynamic object.

[0242] The reminder information includes the dangerous area.

[0243] A display confirmation operation of the user is received, the display confirmation operation being used to indicate display of the window. Optionally, the extended reality device 100 further comprises:

[0244] A first reminder module is configured to generate a vibration reminder information when the user approaches the dangerous area.

[0245] Optionally, the environment perception module 103 is configured to acquire the object information, the object information comprising at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object; generate a planned path according to the object information; and use the planned path as the reminder information.

[0246] Optionally, the extended reality device 100 further comprises:

[0247] A second reminding module is configured to generate voice reminding information according to the planning path, the voice reminding information comprising at least one of the following: environmental information of the current position of the user, and action suggestions for the user.

[0248] In some embodiments, optionally, the extended reality device 100 further comprises:

[0249] A processing module is configured to perform a target processing operation if a selection operation of selecting a target object in the reminding image is received, the target processing operation comprising at least one of the following:

[0250] Obtaining an enlarged image of an image region in the reminding image where the target object is located and displaying the enlarged image;

[0251] Obtaining an environment image before image stitching corresponding to the target object, and obtaining an enlarged image of an image region in the environment image before image stitching where the target object is located and displaying the enlarged image;

[0252] Identifying the target object, and displaying information of the target object in the reminding image or the enlarged image.

[0253] Optionally, the extended reality device 100 further comprises:

[0254] A first control module is configured to divide the plurality of cameras into a plurality of camera groups; wherein every preset number of cameras, one camera is selected to form a camera group; the plurality of camera groups are opened one by one in time sequence, or each camera group is opened according to a preset period, or the camera groups are opened according to a control instruction, wherein only at least one camera group is enabled in each time period, and other camera groups are in a dormant state; the environment image is collected in real time by using the opened camera groups.

[0255] Optionally, the extended reality device 100 further comprises:

[0256] A second control module is configured to close the cameras corresponding to the regions where the environment has not changed for more than a preset time period, and to keep the cameras corresponding to the regions where the environment has changed continuously open; the environment image is collected in real time by using the opened cameras.

[0257] Optionally, the extended reality device 100 further comprises:

[0258] A third control module is configured to, if the environment has not changed for a preset time period, not perform the step of performing image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image, or to reduce the frame rate of the cameras for collecting environment images.

[0259] Please refer to Figure 10 The embodiment of the present application also provides an extended reality device 200, which comprises a processor 201, a memory 202, and a computer program stored in the memory 202 and executable on the processor 201, wherein the computer program is executed by the processor 201 to realize each process of the display method embodiment of the extended reality device and achieve the same technical effects. To avoid repetition, details are not described herein.

[0260] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to realize each process of the display method embodiment of the extended reality device and achieve the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0261] The embodiment of the present application also provides a computer program product, which comprises computer instructions, wherein the computer instructions are executed by a processor to realize each process of the display method embodiment of the extended reality device and achieve the same technical effects. To avoid repetition, details are not described herein. Figure 1

[0262] It should be noted that, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0263] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in each embodiment of the present application.

[0264] ​The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection of the present application.

Claims

1. A display method of an extended reality device, characterized by, The extended reality device includes a plurality of cameras, lenses of different cameras are directed to different directions, and a display method of the extended reality device includes: acquiring environment images by using the plurality of cameras; performing image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image; detecting an object in the stitched image, and generating reminder information according to information of the detected object; generating a reminder image according to the reminder information, and displaying the reminder image in a window mode.

2. The method of claim 1, wherein the plurality of cameras are arranged in a ring shape, and fields of view between adjacent cameras at least partially overlap.

3. The method of claim 1, wherein, The stitched image is a top-view image, and the image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image further includes: for each camera, obtaining a source image containing a calibration cloth captured by the camera, and obtaining pixel coordinates of a plurality of calibration coordinate points in the source image and a target image, the coordinate system in the target image being a coordinate system of the stitched image; determining a projection transformation relationship between the source image containing the calibration cloth and the target image according to the pixel coordinates of the plurality of calibration coordinate points in the source image and the target image; wherein the image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image includes: converting the environment image of the camera into a top-view image according to the projection transformation relationship of the camera; performing image stitching processing on the top-view images of the plurality of cameras at the same time to obtain a stitched image.

4. The method of claim 1, wherein, The image stitching processing on the environment images of the plurality of cameras at the same time to obtain a stitched image includes: performing image stitching processing, which includes: for two environment images to be stitched, dividing the environment images to be stitched into an overlapping region and a non-overlapping region according to an intersection of the two environment images to be stitched; calculating a distance of each pixel in the overlapping region of the environment images to be stitched from a boundary of the overlapping region and the non-overlapping region, determining a weight of each pixel in the overlapping region according to the distance to obtain a normalized weight image, wherein the closer the pixel in the overlapping region is to the boundary, the greater the weight; obtaining a fused overlapping region according to the overlapping regions of the two environment images to be stitched and the normalized weight image; and stitching the fused overlapping region with the non-overlapping regions of the two environment images to be stitched respectively to obtain a local stitched image; determining whether the stitching is completed, and if not, continuing to perform the image stitching processing by taking the local stitched image as a new environment image to be stitched, otherwise, taking the local stitched image as the stitched image.

5. The method of claim 1, wherein, The generating of the reminder information according to the information of the detected object includes: obtaining information of the object, the information of the object comprising at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object; determining a safe area and / or a dangerous area in the stitched image according to the information of the object, the dangerous area comprising at least one of the following: an area including the object and having a distance to the user less than or equal to a preset distance threshold, and an area including the dynamic object; and the safe area comprising at least one of the following: an area not including the object, and an area including the object and having a distance to the user greater than the preset distance threshold; taking the safe area and / or the dangerous area as the reminder information.

6. The method of claim 5, wherein, The method of generating reminder information according to the detected information of the object further comprises: taking the indication information of whether the object is a static object or a dynamic object as the reminder information.

7. The method of claim 6, wherein, The method of displaying the reminder image in a window manner further comprises: displaying the reminder image in a window manner when a condition of triggering window display is met; wherein the condition of triggering window display comprises at least one of the following: the user wearing the extended reality device is in a moving state; the reminder information includes the dynamic object; the reminder information includes the dangerous area; a display confirmation operation of the user is received, the display confirmation operation being used to indicate to display the window.

8. The method of claim 5, wherein, The method further comprises: generating vibration reminder information when the user approaches the dangerous area.

9. The method of claim 1, wherein, The method of generating reminder information according to the detected information of the object further comprises: obtaining information of the object, the information of the object comprising at least one of the following: position information of the object, distance information between the object and a user wearing the extended reality device, size information of the object, and indication information of whether the object is a static object or a dynamic object; generating a planned path according to the information of the object; taking the planned path as the reminder information.

10. The method of claim 9, wherein, The method of generating a planned path according to the information of the object further comprises: generating voice reminder information according to the planned path, the voice reminder information comprising at least one of the following: environmental information of a current position of the user, and an action suggestion for the user.

11. The method according to any one of claims 1 to 10, characterized in that, The method of generating a reminder image according to the reminder information further comprises: labeling the reminder information into the stitched image to obtain the reminder image; or determining a position of an object corresponding to the reminder information, generating a map image containing the reminder information according to the position of the object corresponding to the reminder information and a position of the user, and taking the map image as the reminder image. The method of displaying the reminder image in a window manner further comprises:

12. The method according to any one of claims 1 to 10, characterized in that, in a case where a selection operation of a target object in the reminder image is received from the user, performing a target processing operation, the target processing operation comprising at least one of the following: obtaining and displaying an enlarged image of an image area in which the target object is located in the reminder image; ​ Acquire the pre-stitching environmental image corresponding to the target object, and acquire the magnified image of the image area where the target object is located in the pre-stitching environmental image and display the magnified image; Identify the target object, and display the information of the target object in the prompt image or the magnified image.

13. The method according to any one of claims 1 to 10, characterized in that, The method for acquiring the environmental image by using the plurality of cameras comprises: The plurality of cameras are divided into a plurality of camera groups; wherein, every preset number of cameras, one camera is selected to form a camera group; According to the time sequence, the plurality of camera groups are opened one by one, or each camera group is opened according to a preset period, or the camera groups are opened according to a control instruction, wherein, in each time period, only at least one camera group is enabled, and other camera groups are in a dormant state; The environmental image is acquired in real time by using the opened camera groups.

14. The method according to any one of claims 1 to 10, characterized in that, The method for acquiring the environmental image by using the plurality of cameras comprises: The cameras corresponding to the area where the environment has not changed for more than a preset time are closed, and the cameras corresponding to the area where the environment has changed continuously are opened continuously; The environmental image is acquired by using the opened camera.

15. The method according to any one of claims 1 to 10, characterized in that, Further comprising: If the environment has not changed within a preset time, the step of performing image stitching processing on the environmental images of the plurality of cameras at the same time is not performed, or the frame rate of the cameras for acquiring the environmental images is reduced.

16. An extended reality device, comprising: Comprise: A plurality of cameras, the lenses of different cameras are directed to different directions, for acquiring environmental images; A data processing module, for performing image stitching processing on the environmental images of the plurality of cameras at the same time, to obtain a stitched image; An environmental perception module, for detecting objects in the stitched image, and generating prompt information according to the information of the detected objects; A display module, for generating a prompt image according to the prompt information, and displaying the prompt image in a window mode.

17. An extended reality device, comprising: Comprise: A processor, a memory, and a program stored on the memory and executable on the processor, when the program is executed by the processor, the steps of the display method of the extended reality device according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and when the computer program is executed by the processor, the steps of the display method of the extended reality device according to any one of claims 1 to 15 are implemented.

19. A computer program product, characterised in that, The computer instructions are executed by the processor, and the steps of the display method of the extended reality device according to any one of claims 1 to 15 are implemented.

Images