Display method, device, equipment and system based on head-mounted equipment and medium

Abstract

The invention discloses a display method, device, equipment and system based on head-mounted equipment and a medium, and relates to the technical field of head-mounted, and the method is applied to the head-mounted equipment, and comprises the steps: obtaining a first environment image and a second environment image under the condition that the current sight of a wearer of the head-mounted equipment is shielded by an obstacle, the first environment image is collected by a first external camera arranged on the head-mounted equipment, and the second environment image is collected by a second external camera which is arranged in the environment and has an image collection range covering the environment; according to the first environment image and the second environment image, determining a first fusion environment image of the wearer at the current visual angle; adjusting an obstacle sub-image corresponding to an obstacle in the first fusion environment image into an obstacle sub-image with perspectivity to obtain a second fusion environment image; and displaying the second fused environment image. According to the method, the viewing efficiency of the wearer is improved.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more specifically, to a display method, apparatus, device, system, and medium based on a head-mounted device. Background Technology

[0002] Head-mounted devices are widely used in various fields due to their portability and intelligence.

[0003] Currently, when users use devices such as AR headsets, such as Figure 1 As shown, if a user's current line of sight is blocked by an obstacle, the user will not be able to see the device behind the obstacle. If the user wants to see the device behind the obstacle, they need to go around to the back of the obstacle, which results in low viewing efficiency. Summary of the Invention

[0004] One objective of this application is to provide a new technical solution for a display based on a head-mounted device.

[0005] According to a first aspect of this application, a display method based on a head-mounted device is provided, applied to a head-mounted device, comprising:

[0006] When the wearer's current line of sight is obstructed by an obstacle, a first environmental image and a second environmental image are acquired. The first environmental image is captured by a first external camera installed on the head-mounted device, and the second environmental image is captured by a second external camera installed in the environment and whose image acquisition range covers the environment.

[0007] Based on the first environmental image and the second environmental image, a first fused environmental image is determined from the wearer's current perspective;

[0008] The obstacle sub-images corresponding to the obstacles in the first fused environment image are adjusted to obstacle sub-images with perspective to obtain the second fused environment image;

[0009] Display the second fused environment image.

[0010] Optionally, before acquiring the first environmental image and the second environmental image when the wearer's current line of sight is obstructed by an obstacle, the method further includes:

[0011] Acquire an eye image from the head-mounted device, the eye image being captured by an internal camera mounted on the head-mounted device;

[0012] Based on the eye image and the first environment image, determine the environmental objects in the wearer's current line of sight;

[0013] Output a prompt indicating whether the environmental object is determined to be an obstacle;

[0014] If the wearer determines that the environmental object is an obstacle, it is determined that the wearer's current line of sight is blocked by the obstacle, and the obstacle is the environmental object.

[0015] Optionally, adjusting the obstacle sub-image corresponding to the obstacle in the first fused environment image to a perspective-oriented obstacle sub-image to obtain the second fused environment image includes:

[0016] Extract the edges of the obstacles from the first fused environment image;

[0017] The obstacle sub-image in the first fused environment image is replaced with the edge of the obstacle to obtain the second fused environment image.

[0018] Optionally, the edge is an edge formed by dashed lines.

[0019] Optionally, when the wearer's current line of sight is obstructed by an obstacle, acquiring a first environmental image and a second environmental image includes:

[0020] Obtain the current operating mode of the head-mounted device;

[0021] When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, a first environmental image and a second environmental image are acquired.

[0022] Optionally, when the current working mode is obstacle shielding mode and the wearer's current line of sight is obstructed by an obstacle, acquiring the first environmental image and the second environmental image includes:

[0023] When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, a communication connection is established between the head-mounted device and the first external camera.

[0024] Acquire the first environmental image;

[0025] In addition, a second environmental image is acquired based on the communication connection.

[0026] According to a second aspect of this application, a display device based on a head-mounted device is provided, applied to a head-mounted device, comprising:

[0027] The acquisition module is used to acquire a first environmental image and a second environmental image when the current line of sight of the wearer of the head-mounted device is blocked by an obstacle. The first environmental image is captured by a first external camera installed on the head-mounted device, and the second environmental image is captured by a second external camera installed in the environment and whose image acquisition range covers the environment.

[0028] The determining module is used to determine a first fused environment image from the wearer's current perspective based on the first environment image and the second environment image;

[0029] The adjustment module is used to adjust the obstacle sub-image corresponding to the obstacle in the first fused environment image to an obstacle sub-image with perspective, so as to obtain the second fused environment image;

[0030] The display module is used to display the second fusion environment image.

[0031] According to a third aspect of this application, a head-mounted device is provided, the head-mounted device comprising the means as described in the second aspect and a first external camera, the first external camera being used to capture a first environmental image; or,

[0032] The head-mounted device includes the first external camera, a memory, and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method as described in any one of the first aspects.

[0033] According to a fourth aspect of this application, a display system based on a head-mounted device is provided, comprising: a head-mounted device as described in the third aspect and a second external camera, wherein the second external camera is communicatively connected to the head-mounted device and is used to send a second environmental image to the head-mounted device.

[0034] According to a fifth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0035] This application provides a display method based on a head-mounted device, applied to the head-mounted device, comprising: acquiring a first environmental image and a second environmental image when the wearer's current line of sight is obstructed by an obstacle; the first environmental image being captured by a first external camera mounted on the head-mounted device, and the second environmental image being captured by a second external camera mounted in the environment and whose image acquisition range covers the environment; determining a first fused environmental image from the wearer's current perspective based on the first and second environmental images; adjusting the obstacle sub-images corresponding to the obstacles in the first fused environmental image to obstruction sub-images with perspective to obtain a second fused environmental image; and displaying the second fused environmental image. Through this method, the wearer can not only see the obstacles to know their true location, but also see through the obstacles to see the environment obstructed by them. Thus, while ensuring that the wearer sees the real environment, the wearer can see the image of the environment obstructed by the obstacles without having to walk around them, thereby improving the wearer's viewing efficiency.

[0036] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0038] Figure 1 This is a schematic diagram of a scenario provided in this application;

[0039] Figure 2 This is a framework for the hardware configuration of a head-mounted device that implements a display method based on a head-mounted device, according to an embodiment of this application. Figure 1 ;

[0040] Figure 3 This is a flowchart illustrating a method for implementing a display based on a head-mounted device according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of a structure for implementing a head-mounted display device according to an embodiment of this application;

[0042] Figure 5 This is a framework for the hardware configuration of a head-mounted device that implements a display method based on a head-mounted device, according to an embodiment of this application. Figure 2 ;

[0043] Figure 6 This is a schematic diagram of a display system based on a head-mounted device provided in an embodiment of this application. Detailed Implementation

[0044] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0045] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0046] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0047] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0049] Figure 2 This is a framework for the hardware configuration of a head-mounted device that implements a display method based on a head-mounted device, according to an embodiment of this application. Figure 1 Among them, the head-mounted device 2000 can be an AR device, a MR device, or an XR device, etc.

[0050] The head-mounted device 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, a display device 2500, an input device 2600, a speaker 2700, a microphone 2800, and an image acquisition device 2900, etc. The processor 2100 may be a central processing unit (CPU), a microprocessor (MCU), etc. The memory 2200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 2300 may include, for example, a USB interface, a headphone jack, etc. The communication device 2400 may be capable of wired or wireless communication. The display device 2500 may be, for example, an LCD screen, a touch screen, etc. The input device 2600 may include, for example, a touch screen, a keyboard, etc. Users can input / output voice information through the speaker 2700 and the microphone 2800. The image acquisition device 2900 may include an external camera and an internal camera. The external camera is used to acquire environmental images of the external environment in which the head-mounted device 2000 is located, and the internal camera is used to acquire eye images of the wearer of the head-mounted device 2000.

[0051] Despite Figure 2 The head-mounted device 2000 shows multiple devices, but this application may only involve some of them. For example, the head-mounted device 2000 may only involve the memory 2200, the processor 2100, and the image acquisition device 2900.

[0052] In the embodiments applied in this application, the memory 2200 of the head-mounted device 2000 is used to store instructions for controlling the processor 2100 to execute the display method based on the head-mounted device provided in the embodiments of this application.

[0053] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this application. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0054] The display method based on head-mounted devices provided in this application is applicable to, for example... Figure 2 The head-mounted device shown. This method allows the wearer to see an image of the environment obstructed by an obstacle along the direction of their current line of sight when their current line of sight is blocked. For example... Figure 3 As shown, the display method based on a head-mounted device provided in this application includes the following steps S3100 to S3400.

[0055] Step S3100: When the wearer's current line of sight is obstructed by an obstacle, acquire a first environmental image and a second environmental image.

[0056] The first environmental image is captured by a first external camera mounted on the head-mounted device, and the second environmental image is captured by a second external camera positioned in the environment and whose image capture range covers the environment.

[0057] In this embodiment, the external camera included in the image acquisition device of the head-mounted device is referred to as the first external camera, and the environmental image acquired by the first external camera is referred to as the first environmental image.

[0058] A camera, designated as the second external camera, is installed in the environment to capture environmental images. The environmental images captured by the second external camera are referred to as the second environmental image. The image capture range of the second external camera covers the entire environment. Therefore, the second external environmental image captured by the second external camera includes environmental images obscured by obstacles along the current line of sight.

[0059] It is understandable that, in order to achieve the goal of covering the environment with the image acquisition range of the second external camera, the second external camera may specifically consist of at least one external sub-camera.

[0060] In one example, where the head-mounted device is used in a production environment, the image capture range of the second external camera covers the entire production environment.

[0061] In addition, in one embodiment of this application, the display method based on a head-mounted device provided in this application, prior to step S3100 above, further includes a step of determining whether the wearer's current line of sight is obstructed by an obstacle and locating the obstacle. This step can be implemented by the wearer directly informing the user, or it can be implemented through steps S3110 to S3113 below.

[0062] Step S3110: Obtain an eye image from the head-mounted device.

[0063] The eye images are captured by an internal camera mounted on the head-mounted device.

[0064] Step S3111: Based on the eye image and the first environmental image, determine the environmental objects in the wearer's current line of sight.

[0065] In one embodiment of this application, the environmental object where the wearer's current gaze point is located can be determined based on the eye image, the first environmental image, and the eye-tracking algorithm, and denoted as the environmental object in the current gaze direction in step S3111 above.

[0066] Step S3112: Output a prompt message indicating whether the environmental object is an obstacle.

[0067] After identifying environmental objects in the wearer's current line of sight, the head-mounted device outputs a prompt to indicate whether the wearer has determined that the environmental object is an obstacle. The prompt can be text or voice; this application does not limit the type of prompt.

[0068] Step S3113: If the wearer determines that the environmental object is an obstacle, it is determined that the wearer's current line of sight is blocked by the obstacle and that the obstacle is an environmental object.

[0069] Once the wearer has identified an environmental object in their current line of sight, they inform the head-mounted device that the object is an obstacle. At this point, the head-mounted device determines that the wearer's current line of sight is obstructed by the obstacle, and that the obstacle is an environmental object.

[0070] Step S3200: Determine the first fused environment image from the wearer's current perspective based on the first environment image and the second environment image.

[0071] After obtaining the first environmental image and the second environmental image based on the above step S3200, the first environmental image and the second environmental image are merged, deduplicated and viewpoint transformed to obtain the first fused environmental image.

[0072] Because the second environmental image includes environmental images obscured by obstacles along the current line of sight, while the first environmental image does not, the first fused environmental image obtained from the first and second environmental images includes environmental images obscured by obstacles along the current line of sight, and the corresponding image viewpoint is the wearer's current viewpoint.

[0073] Step S3300: Adjust the obstacle sub-images corresponding to the obstacles in the first fused environment image to obstacle sub-images with perspective lines to obtain the second fused environment image.

[0074] In one embodiment of this application, the above step S3300 can be achieved by adjusting the transparency of the obstacle sub-image corresponding to the obstacle in the first fused environment image to less than 1.

[0075] Step S3400: Display the second fused environment image.

[0076] Because the first fused environmental image still includes obstacle sub-images corresponding to obstacles, and the image perspective is the wearer's current perspective, although the first fused environmental image contains environmental images obscured by obstacles along the current line of sight, the wearer still cannot see these obscured environmental images based on the first fused environmental image. To address this, by adjusting the obstacle sub-images corresponding to obstacles in the first fused environmental image to have perspective, the wearer can see through the obstacle sub-images and view the environmental images obscured by obstacles along the current line of sight. Simultaneously, the wearer can still see the obstacle sub-images. Based on this, when the second fused environmental image is displayed, the wearer can not only see the obstacles to know their true location, but also see through the obstacles to see the environment obscured by them. This ensures that the wearer sees the real environment, and allows the wearer to see the image of the environment obscured by obstacles without having to navigate around them, thus improving the wearer's viewing efficiency.

[0077] Based on the above, for example Figure 1 In the scenario shown, based on the display method of the head-mounted device provided in the embodiments of this application, the wearer can see through obstacles and see the device behind the obstacles.

[0078] Furthermore, the head-mounted display method provided in this application can also be used in scenarios where the driver is making an inward turn. Specifically, the driver wears a head-mounted device and, when making an inward turn by looking towards the passenger side door, cannot see the environment outside the passenger side door. At this time, a second external camera mounted on the vehicle or located elsewhere captures images of the environment outside the passenger side door to obtain a second environmental image. The head-mounted device's first external camera captures the first environmental image. After the head-mounted device executes the display method provided in this application, when the driver looks towards the passenger side door, the images seen visually along the current line of sight are, in sequence: an image inside the passenger side door, a transparent image of the passenger side door, and an image outside the passenger side door. Thus, the wearer can be aware of whether there are obstacles outside the passenger side door that might hinder the inward turn.

[0079] Of course, the display method based on head-mounted devices provided in this application can also be applied to other scenarios, such as reversing scenarios, and this application does not limit it to these scenarios.

[0080] This application provides a display method based on a head-mounted device, applied to the head-mounted device, comprising: acquiring a first environmental image and a second environmental image when the wearer's current line of sight is obstructed by an obstacle; the first environmental image being captured by a first external camera mounted on the head-mounted device, and the second environmental image being captured by a second external camera mounted in the environment and whose image acquisition range covers the environment; determining a first fused environmental image from the wearer's current perspective based on the first and second environmental images; adjusting the obstacle sub-images corresponding to the obstacles in the first fused environmental image to obstruction sub-images with perspective to obtain a second fused environmental image; and displaying the second fused environmental image. Through this method, the wearer can not only see the obstacles to know their true location, but also see through the obstacles to see the environment obstructed by them. Thus, while ensuring that the wearer sees the real environment, the wearer can see the image of the environment obstructed by the obstacles without having to walk around them, thereby improving the wearer's viewing efficiency.

[0081] In one embodiment of this application, step S3300 can be specifically implemented by the following steps S3310 and S3311.

[0082] Step S3310: Extract the edges of obstacles from the first fused environment image.

[0083] In one embodiment of this application, the edges of obstacles can be extracted from a first fused environment image using an edge detection algorithm.

[0084] Step S3311: Replace the obstacle sub-image in the first fused environment image with the edge of the obstacle to obtain the second fused environment image.

[0085] In one embodiment of this application, the edge of an obstacle can be represented by a dashed line, that is, the edge in step S3311 above is an edge composed of dashed lines. This can improve the wearer's visual sensitivity to obstacles.

[0086] In this embodiment, the obstacle sub-image in the first fused environmental image is replaced with the outline of the obstacle. Thus, after the second fused environmental image is displayed based on the above step S3400, the obstacles in the second fused environmental image seen by the wearer are represented by the edges of the obstacles. That is, the portion outside the edges of the obstacle sub-image has zero transparency. At this time, the wearer can not only know the true location of the obstacle, but also clearly see the environment obscured by the obstacle.

[0087] In one embodiment of this application, step S3100 is specifically implemented through the following steps S3120 and S3121.

[0088] Step S3120: Obtain the current working mode of the head-mounted device.

[0089] Step S3121: When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, acquire the first environmental image and the second environmental image.

[0090] In this embodiment, the head-mounted device has the function of shielding obstacles as described in steps S3100 to S3400 above. The operating mode of the head-mounted device when performing steps S3100 to S3400 is referred to as the obstacle shielding mode. When the wearer wants to view an environmental image obscured by obstacles along the current line of sight, the head-mounted device is instructed to activate the obstacle shielding mode. Based on this, the head-mounted device performs step S3100 above.

[0091] Corresponding to step S3121 above, if the current working mode of the head-mounted device is a mode other than the obstacle shielding mode, the head-mounted device will operate according to the other mode.

[0092] In one embodiment of this application, step S3121 is specifically implemented by steps S31210 to S31212.

[0093] Step S31210: When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, establish a communication connection between the head-mounted device and the first external camera.

[0094] Step S31211: Obtain the first environmental image.

[0095] Step S31212: Obtain the second environmental image based on the communication connection.

[0096] In this embodiment, when the current operating mode is obstacle shielding mode and the wearer's current line of sight is obstructed by an obstacle, a communication connection is first established between the head-mounted device and the first external camera. This communication connection can be, for example, a Bluetooth wireless communication connection, but it can also be other types. Further, a second environmental image is acquired from a second external camera based on this communication connection. Simultaneously, a first environmental image is acquired.

[0097] In this embodiment, a communication connection between the head-mounted device and the first external camera is established only when the current operating mode is obstacle shielding mode and the wearer's current line of sight is obstructed by an obstacle. This avoids unnecessary communication connections between the head-mounted device and the first external camera, thereby saving power consumption of the head-mounted device.

[0098] This application also provides a display device 400 based on a head-mounted device, applied to a head-mounted device, such as... Figure 4 As shown, it includes:

[0099] The acquisition module 410 is used to acquire a first environmental image and a second environmental image when the current line of sight of the wearer of the head-mounted device is blocked by an obstacle. The first environmental image is captured by a first external camera installed on the head-mounted device, and the second environmental image is captured by a second external camera installed in the environment and whose image acquisition range covers the environment.

[0100] The determining module 420 is used to determine a first fused environment image from the wearer's current perspective based on the first environment image and the second environment image;

[0101] The adjustment module 430 is used to adjust the obstacle sub-image corresponding to the obstacle in the first fused environment image to an obstacle sub-image with perspective, so as to obtain the second fused environment image;

[0102] Display module 440 is used to display the second fusion environment image.

[0103] In one embodiment of this application, the acquisition module 410 is further configured to: acquire an eye image of the head-mounted device, the eye image being captured by an internal camera mounted on the head-mounted device;

[0104] The determining module 420 is further configured to determine environmental objects in the wearer's current line of sight based on the eye image and the first environmental image;

[0105] In this embodiment, the head-mounted display device 400 provided in this application further includes:

[0106] The output module is used to output a prompt message indicating whether the environmental object is determined to be an obstacle;

[0107] In this embodiment, the determining module 420 is further configured to determine, upon receiving a determination from the wearer that the environmental object is an obstacle, that the wearer's current line of sight is blocked by an obstacle, and that the obstacle is the environmental object.

[0108] In one embodiment of this application, the adjustment module 430 is specifically used for:

[0109] Extract the edges of the obstacles from the first fused environment image;

[0110] The obstacle sub-image in the first fused environment image is replaced with the edge of the obstacle to obtain the second fused environment image.

[0111] In one embodiment of this application, the edge is an edge formed by dashed lines.

[0112] In one embodiment of this application, the acquisition module 410 is specifically used for:

[0113] Obtain the current operating mode of the head-mounted device;

[0114] When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, a first environmental image and a second environmental image are acquired.

[0115] In one embodiment of this application, the acquisition module 410 is specifically used to: establish a communication connection between the head-mounted device and the first external camera when the current working mode is obstacle shielding mode and the current line of sight of the wearer of the head-mounted device is blocked by an obstacle;

[0116] Acquire the first environmental image;

[0117] In addition, a second environmental image is acquired based on the communication connection.

[0118] This application also provides another head-mounted device 500, which includes any of the head-mounted device-based display devices 400 provided in the above-described device embodiments and a first external camera, the first external camera being used to capture a first environmental image.

[0119] Or, such as Figure 5As shown, the head-mounted device 500 includes a first external camera 510, a memory 520, and a processor 530. The memory 520 is used to store computer instructions, and the processor 530 is used to call the computer instructions from the memory 520 to execute any of the head-mounted device-based display methods provided in the above method embodiments.

[0120] This application also provides a display system 600 based on a head-mounted device, such as... Figure 6 As shown, the device includes a head-mounted device 500 as provided in the above-described device embodiment and a second external camera 610, wherein the second external camera 610 is communicatively connected to the head-mounted device 500 and is used to send a second environmental image to the head-mounted device 500.

[0121] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the head-mounted device-based display methods provided in the above-described method embodiments.

[0122] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0123] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0124] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0125] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0126] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0130] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A display method based on a head-mounted device, characterized in that, Used in head-mounted devices, including: When the wearer's current line of sight is obstructed by an obstacle, a first environmental image and a second environmental image are acquired. The first environmental image is captured by a first external camera installed on the head-mounted device, and the second environmental image is captured by a second external camera installed in the environment and whose image acquisition range covers the environment. Based on the first environmental image and the second environmental image, a first fused environmental image is determined from the wearer's current perspective; The obstacle sub-images corresponding to the obstacles in the first fused environment image are adjusted to obstacle sub-images with perspective to obtain the second fused environment image; Display the second fused environment image.

2. The method according to claim 1, characterized in that, Before acquiring the first environmental image and the second environmental image when the wearer's current line of sight is obstructed by an obstacle, the method further includes: Acquire an eye image from the head-mounted device, the eye image being captured by an internal camera mounted on the head-mounted device; Based on the eye image and the first environment image, determine the environmental objects in the wearer's current line of sight; Output a prompt indicating whether the environmental object is determined to be an obstacle; If the wearer determines that the environmental object is an obstacle, it is determined that the wearer's current line of sight is blocked by the obstacle, and the obstacle is the environmental object.

3. The method according to claim 1, characterized in that, The step of adjusting the obstacle sub-image corresponding to the obstacle in the first fused environment image to an obstacle sub-image with perspective to obtain the second fused environment image includes: Extract the edges of the obstacles from the first fused environment image; The obstacle sub-image in the first fused environment image is replaced with the edge of the obstacle to obtain the second fused environment image.

4. The method according to claim 3, characterized in that, The edge is defined by dashed lines.

5. The method according to claim 1, characterized in that, When the wearer's current line of sight is obstructed by an obstacle, acquiring a first environmental image and a second environmental image includes: Obtain the current operating mode of the head-mounted device; When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, a first environmental image and a second environmental image are acquired.

6. The method according to claim 5, characterized in that, The step of acquiring a first environmental image and a second environmental image when the current working mode is obstacle shielding mode and the wearer's current line of sight is obstructed by an obstacle includes: When the current working mode is obstacle shielding mode and the wearer's current line of sight is blocked by an obstacle, a communication connection is established between the head-mounted device and the first external camera. Acquire the first environmental image; In addition, a second environmental image is acquired based on the communication connection.

7. A display device based on a head-mounted device, characterized in that, Used in head-mounted devices, including: The acquisition module is used to acquire a first environmental image and a second environmental image when the current line of sight of the wearer of the head-mounted device is blocked by an obstacle. The first environmental image is captured by a first external camera installed on the head-mounted device, and the second environmental image is captured by a second external camera installed in the environment and whose image acquisition range covers the environment. The determining module is used to determine a first fused environment image from the wearer's current perspective based on the first environment image and the second environment image; The adjustment module is used to adjust the obstacle sub-image corresponding to the obstacle in the first fused environment image to an obstacle sub-image with perspective, so as to obtain the second fused environment image; The display module is used to display the second fusion environment image.

8. A head-mounted device, characterized in that, The head-mounted device includes the apparatus as described in claim 7 and a first external camera, the first external camera being used to capture a first environmental image; or... The head-mounted device includes a first external camera, a memory, and a processor. The memory stores computer instructions, and the processor retrieves the computer instructions from the memory to perform the method as described in any one of claims 1-6.

9. A display system based on a head-mounted device, characterized in that, include: The head-mounted device and the second external camera as described in claim 8, wherein the second external camera is communicatively connected to the head-mounted device and is used to send a second environmental image to the head-mounted device.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images