Safety boundary adjustment method, display device, display system and computer storage medium

Abstract

The present disclosure relates to the technical field of display, and provides a safety boundary adjustment method, a display device, a display system and a computer storage medium, capable of solving the existing problem of large potential safety hazards for users due to impact of external environments being not considered during the defining of safety boundaries. The adjustment method of the present disclosure comprises: obtaining an environmental image of the surroundings of a display device, the environmental image being acquired by an acquisition apparatus; on the basis of the environmental image at the current moment and a target safety boundary of the display device, determining whether an object is present in an area defined by the target safety boundary; and in response to an object is present in the area defined by the target safety boundary, adjusting the target safety boundary on the basis of position information of the object. User safety can be protected to the maximum extent.

Description

Method for adjusting safety boundary, display device, display system and computer storage medium TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of display, and particularly relates to a method for adjusting a safety boundary, a display device, a display system and a computer storage medium. BACKGROUND

[0002] VR (Virtual Reality) is a technology that creates and experiences a virtual world using computer technology. It simulates human visual, auditory, tactile and other sensory experiences, allowing users to immerse themselves in a virtual environment and interact with virtual objects. VR technology usually requires the use of special equipment such as VR headsets, gloves, and handsets to provide an immersive experience. For example, when a user wears a VR headset, a virtual scene is displayed on the screen, and the user interacts with the virtual environment through head tracking technology and handsets. VR technology has a wide range of applications in gaming, education, training, healthcare, entertainment and other fields. It can provide immersive learning and experience, helping people better understand and master knowledge and skills, and also provide a more immersive entertainment experience. With the continuous development of technology, the cost of VR technology is gradually decreasing, and the application scenarios are becoming more and more widespread. At the same time, VR technology is constantly evolving and improving, such as higher resolution, lower latency, and better interaction methods, providing users with a more realistic and comfortable experience.

[0003] When a user wears a VR headset to experience a virtual environment, the user's eyes cannot see the real world around them. In order to avoid the problem of the user colliding with the virtual environment during interaction, a safety boundary needs to be defined to ensure the safety of the user.

[0004] SUMMARY

[0005] The present disclosure aims to at least solve one of the technical problems in the prior art, and provides a method for adjusting a safety boundary, a display device, a display system and a computer storage system.

[0006] In a first aspect, a technical solution for solving the technical problem of the present disclosure is a method for adjusting a safety boundary, wherein the adjustment method is applied to a display device, and the adjustment method comprises:

[0007] determining an initial safety boundary of the display device based on a safety mode type;

[0008] acquiring an environment image of the display device surrounding the display device collected by a collection device, and determining a target safety boundary based on the collected environment image and the initial safety boundary;

[0009] determining whether there is an object in the area defined by the target safety boundary based on the target safety boundary and the environment image at the current time;

[0010] adjusting the target safety boundary based on the position information of the object in response to the existence of the object in the area defined by the target safety boundary.

[0011] In some embodiments, the adjusting method further comprises: receiving a safety mode type selected by a user; when the received safety mode type selected by the user is a stay-in-place mode, the step of determining the initial safety boundary of the display device based on the safety mode type comprises:

[0012] determining the initial position information of the display device, and taking the initial position information as a reference point to obtain boundary information of an area containing a preset range of the reference point, thereby obtaining the initial safety boundary.

[0013] In some embodiments, the adjusting method further comprises: receiving a safety mode type selected by a user; when the received safety mode type selected by the user is a self-defined mode, the step of determining the initial safety boundary of the display device based on the safety mode type comprises:

[0014] receiving safety boundary information set by a user, obtaining image information within the user-set safety boundary information, and determining the initial safety boundary.

[0015] In some embodiments, the step of determining whether there is an object in the area defined by the target safety boundary based on the target safety boundary and the environment image at the current time specifically comprises:

[0016] detecting whether there is an object in the environment image at the current time;

[0017] in response to the existence of the object in the environment image at the current time, obtaining depth information of the object to obtain position information of the object relative to the target safety boundary;

[0018] determining whether there is an object in the area defined by the target safety boundary based on the position information of the object.

[0019] In some embodiments, the environment image comprises a plurality of sub-environment images, and the step of obtaining the environment image around the display device collected by the collection device specifically comprises:

[0020] obtaining a plurality of sub-environment images around the display device collected by the collection device, wherein there is partial overlap of scenes between the plurality of sub-environment images;

[0021] stitching the plurality of sub-environment images to obtain the environment image.

[0022] In some embodiments, the adjusting the target safety boundary based on the position information of the object in response to the existence of the object in the area defined by the target safety boundary specifically comprises:

[0023] In response to the existence of the object in the area defined by the target safety boundary, generating first prompt information to prompt the user;

[0024] In response to the user not clearing the object in the area defined by the target safety boundary, adjusting the target safety boundary based on the position information of the object.

[0025] In some embodiments, the adjusting method further comprises:

[0026] Obtaining a first distance between the display device and the target safety boundary;

[0027] In response to the display device being in the area defined by the target safety boundary and the first distance being less than or equal to a preset distance, generating second prompt information to prompt the user.

[0028] In some embodiments, the first prompt information comprises display information and voice information; and / or,

[0029] The second prompt information comprises display information and voice information.

[0030] In some embodiments, the adjusting method further comprises:

[0031] In response to the display device being outside the area defined by the target safety boundary, switching the virtual environment picture to a perspective picture.

[0032] In a second aspect, the embodiments of the present disclosure further provide a display device, wherein the display device comprises a memory, a processor, and a safety boundary setting program stored in the memory and executable on the processor, and the safety boundary adjusting program is executed by the processor to implement the steps of the safety boundary adjusting method of any one of the above first aspect.

[0033] In a third aspect, the embodiments of the present disclosure further provide a display system, wherein the display system comprises a processor and an acquisition device, and the acquisition device is used to acquire an environment image around the display device;

[0034] The processor is configured to perform the following steps:

[0035] Based on the safety mode type, determining an initial safety boundary of the display device;

[0036] acquire an environment image around the display device collected by the collection device, and determine a target safety boundary based on the acquired environment image and the initial safety boundary;

[0037] determine whether there is an object in the area defined by the target safety boundary based on the target safety boundary and the environment image at the current time;

[0038] in response to the existence of the object in the area defined by the target safety boundary, adjust the target safety boundary based on the position information of the object.

[0039] In some embodiments, the environment image includes a plurality of sub-environment images, and the collection device includes a plurality of collection components arranged around the display device,

[0040] Each of the collection components is configured to collect a respective sub-environment image around the display device, and the plurality of sub-environment images have some overlapping scenes.

[0041] The processor is further configured to stitch the plurality of sub-environment images to obtain the environment image.

[0042] In some embodiments, the collection device includes a projector and a receiver.

[0043] The projector is configured to project a light pattern onto the surface of the object to obtain a projection pattern.

[0044] The receiver is configured to receive the light pattern and the projection pattern, and obtain the depth information of the object based on the change of the projection pattern relative to the light pattern.

[0045] In some embodiments, the collection device includes a projector and a receiver.

[0046] The projector is configured to emit a light signal and cause the light signal to be reflected on the surface of the object.

[0047] The receiver is configured to receive the reflected signal of the light signal reflected on the surface of the object.

[0048] The receiver is further configured to calculate the time of flight of the light signal and calculate the distance between the projector and the object to obtain the depth information of the object.

[0049] In some embodiments, the processor is further configured to receive a safety mode type selected by a user.

[0050] When the received safety mode type selected by the user is the in-place mode, the processor is specifically configured to:

[0051] Determine initial position information of the display device, and take the initial position information as a reference point, acquire boundary information of a region containing a preset range of the reference point, and obtain the initial safety boundary.

[0052] In some embodiments, the processor is further configured to receive a safety mode type selected by a user;

[0053] When the received safety mode type selected by the user is a custom mode, the processor is specifically configured to:

[0054] Receive safety boundary information set by the user, acquire image information in the user-set safety boundary information, and determine an initial safety boundary.

[0055] In some embodiments, the processor is specifically configured to perform the following steps:

[0056] In response to the existence of an object in a region defined by the target safety boundary, generate first prompt information to prompt the user;

[0057] In response to the user not clearing the object in the region defined by the target safety boundary, adjust the target safety boundary based on position information of the object.

[0058] In some embodiments, the processor is further configured to perform the following steps:

[0059] Acquire a first distance between the display device and the target safety boundary;

[0060] In response to the display device being in the region defined by the target safety boundary and the first distance being less than or equal to a preset distance, generate second prompt information to prompt the user.

[0061] In some embodiments, the display system further includes a handle device for setting an initial safety boundary at an initial time.

[0062] In a fourth aspect, the embodiments of the present disclosure further provide a computer storage medium, wherein the computer storage medium stores a safety boundary setting program and a safety boundary adjustment program, and the safety boundary adjustment program is executed by a processor to implement the steps of the safety boundary adjustment method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0063] FIG. 1 is a flowchart of a safety boundary adjustment method according to an embodiment of the present disclosure;

[0064] FIG. 2 is a flowchart of another adjustment method according to an embodiment of the present disclosure;

[0065] FIGS. 3a and 3b are schematic diagrams of initial safety boundaries in two safety modes, respectively.

[0066] FIGS. 4a-4b are schematic diagrams of two scenarios before and after adjusting the current safety boundary, respectively;

[0067] FIGS. 5a-5c are schematic diagrams of three types of prompt information, respectively;

[0068] FIG. 6 is a structural block diagram of a display system 600 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] For those skilled in the art to better understand the technical solutions of the present application, the present application will be described in further detail below in conjunction with the drawings and specific embodiments.

[0070] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning understood by a person skilled in the art to which the present disclosure belongs. The terms "first", "second", and similar terms used in the present disclosure do not represent any order, number, or importance, but are only used to distinguish different components. Similarly, the terms "one", "an", or "the" and similar terms do not represent a quantity limitation, but represent the existence of at least one. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0071] In the related technical solutions, when the safety boundary is drawn, the influence of the objects in the environment is not considered, which poses a great safety risk to the user.

[0072] In the in-place mode, when the display system draws the safety boundary, the user needs to look around 360 degrees to see if there are any objects; if there are objects around, the user needs to ensure that the objects have been cleared to avoid the user's arms, legs, or head from colliding with any objects, which obviously does not meet the user's real needs. In addition, if the user does not completely clear the objects, the display system will ignore the objects when automatically drawing the safety boundary, and still directly draw the safety boundary, which poses a great safety risk to the user himself.

[0073] In the self-defined mode, when the display system draws the safety boundary, if there are objects within the drawn safety boundary, the display system will prompt the user to clear the objects, and if the user does not clear the objects, the display system will ignore the objects and still directly draw the safety boundary, which also poses a great safety risk to the user himself.

[0074] In addition, after the safety boundary is delimited, the display system only determines whether the user himself (the position of the head-mounted display) or the gesture (the position of the hand) defined by the user approaches or crosses the safety boundary during the user experience. If the user crosses the safety boundary, the display system reminds the user. However, the existing display system does not consider that there may be an active object in the environment entering the safety boundary, such as a floor cleaning robot, a cat, a dog pet, and the like entering the safety boundary. This also has a great security risk for the user himself.

[0075] Based on the above problems, the embodiments of the present disclosure provide a safety boundary adjustment method, which can be applied to a display device.

[0076] FIG. 1 is a flowchart of a safety boundary adjustment method provided by the embodiments of the present disclosure. FIGS. 4a-4b are schematic diagrams of the current safety boundary before and after adjustment, respectively. As shown in FIG. 1, the adjustment method includes the following steps:

[0077] S100, determining an initial safety boundary of the display device based on a safety mode type.

[0078] S101, acquiring an environment image around the display device collected by a collection device, and determining a target safety boundary based on the collected environment image and the initial safety boundary.

[0079] S103, judging whether there is an object in the area limited by the target safety boundary based on the target safety boundary and the environment image at the current time. If there is an object in the area limited by the target safety boundary, S104 is executed; if there is no object in the area limited by the target safety boundary, S105 is executed.

[0080] S104, in response to the existence of the object in the area limited by the target safety boundary, adjusting the target safety boundary based on the position information of the object.

[0081] S105, in response to the non-existence of the object in the area limited by the target safety boundary, not processing the target safety boundary.

[0082] Specifically, the display device can be a VR device, for example, the display device can be a VR helmet, a VR glove, etc., and of course the display device can also be other display devices. In the process of wearing the display device to experience the virtual environment, the eyes of the user cannot see the real world around them. In order to avoid the problem that the user may collide in the process of interacting with the virtual environment, a safety solution is needed to ensure the safety of the user. In the adjustment method provided by the present disclosure, at the initial moment after the display device is started and before the experience application, the initial safety boundary at the initial moment is usually set in different ways according to the different types of safety modes selected by the user, in order to ensure the safety of the user. Among them, the safety mode type includes two modes of in-place mode and custom mode. The in-place mode refers to setting the initial safety boundary based on the initial position of the display device at the initial moment; the custom mode refers to setting the initial safety boundary at the initial moment based on the indication of the user, wherein the indication of the user can be an indication issued based on the will of the user himself, or an indication issued by the user based on a certain rule, which is not limited by the present disclosure.

[0083] The environmental image around the display device refers to the image around the display device 360° and within a preset range. There can be objects within the area defined by the initial safety boundary, and there can also be active objects that enter the area defined by the initial safety boundary after the initial safety boundary is set, thereby affecting the safety of the user. The embodiment of the present disclosure obtains the environmental image around the display device collected by the collection device to determine whether there are objects that affect the safety of the user within the area defined by the initial safety boundary. If there are objects that affect the safety of the user within the area defined by the initial safety boundary, the initial safety boundary is adjusted to obtain a target safety boundary, so as to exclude the objects that affect the safety of the user from the target safety boundary; otherwise, if there are no objects that affect the safety of the user within the area defined by the initial safety boundary, the initial safety boundary is taken as the target safety boundary.

[0084] It should be noted that the object mentioned in the embodiment of the present disclosure refers to any obstacle that affects the safety of the user except the user of the display device, wherein the obstacle can include at least one of active objects such as a sweeping robot, a cat, a dog, etc., and at least one of fixed objects such as a table, a chair, etc.

[0085] Optionally, the acquisition device can be a camera or other imaging components. Taking a camera as an example, the camera can be integrated in front of the display device. In this case, the user needs to simply look around in a 360° range so that the acquisition device can capture a complete environmental image of the area around the display device. Alternatively, a ring of cameras can be integrated into the display device to acquire sub-images from various angles around the display device. This way, the user does not need to turn their head to observe, and a 360° environmental image of the area around the display device can still be captured. The acquisition device can also exist independently of the display device.

[0086] The area defined by the target safety boundary of the display device refers to the range of movement that the display device can make while wearing it, i.e., the safety zone of the display device. As shown in Figures 4a and 4b, optionally, the area defined by the target safety boundary can be a regular area or an irregular area, which can be determined according to the position of the object in the specific scene.

[0087] In addition, since there may be moving objects in the external environment, these moving objects may enter the safe zone defined by the target safety boundary at any time. These objects include robot vacuum cleaners, cats, dogs, and even people, which may cause unsafe factors to exist in the originally safe zone and affect the safety of users.

[0088] In the adjustment method provided in this embodiment, the display device acquires a 360° environmental image of its surroundings and detects whether there is an object within the area defined by the target safety boundary based on the environmental image. When the display device detects that there is an object within the area defined by the target safety boundary, it automatically adjusts the target safety boundary according to the specific location information of the object, so as to automatically remove the object from the area defined by the target safety boundary and maximize the safety of the user.

[0089] Figure 2 is a flowchart of another adjustment method provided in an embodiment of this disclosure. Figures 3a and 3b are schematic diagrams of the initial security boundary under two security modes, wherein Figure 3a is a schematic diagram of the initial security boundary in the in-situ mode; and Figure 3b is a schematic diagram of the initial security boundary in the automatic mode.

[0090] As shown in Figures 1 and 2, in some embodiments, the adjustment method includes not only steps S100, S101, S103, S104 and S105, but also: S106, receiving the security mode type selected by the user.

[0091] In some embodiments, when the received user-selected security mode type is the in-place mode, the step S100 of determining the initial security boundary of the display device based on the security mode type specifically comprises: S1061, determining the initial position information of the display device, and taking the initial position information as a reference point, obtaining boundary information of a region containing a preset range of the reference point to obtain the initial security boundary.

[0092] Specifically, the position information of the display device is the position information of the user. When the security mode type is the in-place mode, at the initial moment, the display device takes its initial position as the center and automatically generates the initial security boundary based on the boundary information. Optionally, the region defined by the initial security boundary can be a regular region, for example, the region defined by the initial security boundary is a circular region, and the boundary information can be the radius of the circular region; the region defined by the initial security boundary is a rectangular region, and the boundary information can be the side length information of the rectangular region. Optionally, the region defined by the initial security boundary can also be an irregular region, and the boundary information can be coordinate information with the initial position of the irregular security boundary as the coordinate origin, etc.

[0093] As shown in FIG. 3a, taking the region defined by the initial security boundary as a circular region as an example, in the in-place mode, the specific process of determining the initial security boundary is: taking the current position of the user as the center, determining a first preset value based on the boundary information of the region containing the preset range of the reference point, and automatically determining a circular region with the first preset value as the radius as the security region, and the boundary of the circular region is the initial security boundary. Optionally, the first preset value can be determined according to the specific model of the display device. Taking the display device Pico4 as an example, the security boundary supports three sizes of 1.7m, 2.5m and 3.5m, that is, the first preset value can be set to 1.7m, 2.5m and 3.5m. Optionally, the first preset value can also be determined according to the external environment in which the display device is located. For example, if the current external environment in which the display device is located is relatively open, the first preset value can be set to be larger, and vice versa, if the current external environment in which the display device is located is relatively complex, the first preset value can be set to be smaller to ensure the safety of the user. The determination of the first preset value is not limited in the present disclosure.

[0094] In the embodiments of the present disclosure, the display device acquires an external environment image, and after the initial safety boundary is drawn, the display device automatically determines whether there is an object in the area defined by the initial safety boundary according to the external environment image. If it is detected that there is an object in the area defined by the initial safety boundary, the display device automatically adjusts the initial safety boundary according to the specific position information of the object, and takes the adjusted initial safety boundary as the target safety boundary, so as to automatically exclude the object from the area defined by the target safety boundary. In the prior art, when the display device selects the in-place mode, the user needs to look around 360 degrees to see whether there is an object, and if there is an object, the user needs to remove the object to ensure that the object in the safety area is removed, so as to prevent the arms, legs or head from colliding with any object. The adjustment method in the prior art obviously does not meet the real needs of the user. Compared with the prior art, for the in-place mode, the adjustment method provided in the embodiments of the present disclosure changes from active identification of the object to passive receiving of the reminder from the perspective of the user, which is more in line with the actual use demand.

[0095] In some embodiments, when the received user-selected safety mode type is the custom mode, the step S100 of determining the initial safety boundary of the display device based on the safety mode type specifically includes: S1062, receiving the user-set safety boundary information, acquiring image information in the user-set safety boundary information, and determining the initial safety boundary.

[0096] Specifically, as shown in FIG. 3b, in the custom mode, when the initial safety boundary is drawn, the user needs to personally rotate a circle using the handle, and the display device receives the user-set safety boundary information to draw the initial safety boundary. Preferably, the display device can also acquire image information in the user-set safety boundary information to determine the initial safety boundary. For example, some display devices also support automatic drawing of the custom initial safety boundary. Taking the display device Meta Quest as an example, when the user wearing the device rotates a circle, the device can acquire the depth information of the environment in real time, and the depth information can be used to identify the object in the environment, so as to automatically draw the custom safety boundary according to the user-set safety boundary information and the object information in the environment, and then the user confirms, for example, the user can use the handle to adjust the safety boundary and complete the confirmation, and finally draw the initial safety boundary of the display device at the initial moment.

[0097] In the embodiments of the present disclosure, in the custom mode, the display device acquires an external environment image, and automatically determines whether an object exists in a safety region corresponding to the safety boundary information set by the user according to the external environment image. If the display device detects that the object exists in the current safety region, the display device automatically adjusts the safety boundary corresponding to the safety boundary information set by the user according to the specific position information of the object, so as to automatically exclude the object from the safety region, and takes the adjusted safety boundary as the initial safety boundary. In the prior art, when the display device selects the custom mode, the initial safety boundary is automatically determined after the safety boundary information set by the user is received. If it is detected that the object exists in the initial safety boundary, the user needs to remove the object. If the user does not remove the object, the initial safety boundary is still directly determined. Thus, the object exists in the safety region determined by the initial safety boundary, which brings great security risks to the user. Compared with the prior art, the embodiments of the present disclosure can automatically adjust the safety boundary corresponding to the safety boundary information set by the user when it is detected that the object exists in the safety boundary, so as to obtain the initial safety boundary, and automatically exclude the object from the safety region, thereby maximizing the safety of the user.

[0098] As shown in FIGS. 1 and 2, in some embodiments, step S103 includes the following steps:

[0099] S1031, detecting whether an object exists in the environment image at the current time. In response to the fact that no object exists in the environment image at the current time, step S105 of not processing the target safety boundary is performed. In response to the fact that an object exists in the environment image at the current time, step S1032 of determining whether an object exists in the region limited by the target safety boundary is performed.

[0100] Specifically, the region where the environment image is located is generally greater than or equal to the region limited by the target safety boundary, that is, the region where the environment image is located includes the region limited by the target safety boundary. After the display device acquires the environment image, it is first determined whether an object exists in the environment image. If no object exists in the environment image, it is certain that no object exists in the region limited by the target safety boundary. Conversely, if an object exists in the environment image, it is further determined whether the object is located in the region limited by the target safety boundary according to the specific position of the object.

[0101] In the embodiments of the present disclosure, by judging whether there is an object in the environment image, the display device can detect in real time whether the object in the environment penetrates into the target safety boundary when it is determined that there is a moving object in the environment image, so that the target safety boundary can be adjusted in time when the moving object penetrates into the target safety boundary, thereby maximizing the safety of the user.

[0102] It can be understood that the display device can also directly acquire an external image in which the area defined by the target safety boundary is located, and directly judge whether there is an object in the external image in which the area defined by the target safety boundary is located.

[0103] In some embodiments, the step S1031 of detecting whether there is an object in the environment image at the current time specifically comprises: identifying the environment image at the current time based on a preset image recognition algorithm, and judging whether there is an object in the environment image at the current time according to the identification result.

[0104] Optionally, the preset image recognition algorithm can be any one of a target detection algorithm, a target segmentation algorithm, a semantic segmentation algorithm, and a panoramic segmentation algorithm. Based on the preset image recognition algorithm, it can be identified that the object in the environment image is specifically what kind of object, for example, the object is a moving object such as a cat or a dog, or a fixed object such as a column.

[0105] Among them, target detection refers to identifying and locating targets of interest in an image or video, such as people, vehicles, animals, etc., and target detection usually returns the bounding box and class label of the target; target segmentation refers to segmenting the target from the background in an image or video, i.e. separating the target from the background, and target segmentation usually returns the mask or contour of the target; semantic segmentation refers to assigning each pixel in an image or video to a predefined class, such as road, building, tree, etc., semantic segmentation does not distinguish different target instances, but divides the entire image into different semantic regions; panoramic segmentation is a more advanced form of semantic segmentation, which not only segments the image into different semantic regions, but also attempts to distinguish target instances of the same class, and panoramic segmentation usually returns the mask or contour of each target instance. Among them, target detection is the simplest image recognition algorithm, and semantic segmentation algorithm is a more complex image recognition algorithm, but the recognition result is relatively more accurate.

[0106] When the display device detects the object, the specific position information of the object can be further determined according to the depth information of the object. In some embodiments, the step S1032 specifically can comprise: in response to the existence of the object in the environment image at the current time, acquiring the depth information of the object to obtain the position information of the obstacle region relative to the target safety boundary; and judging whether there is an object in the domain defined by the safety boundary at the previous time according to the position information of the object.

[0107] Optionally, a camera can be integrated on the display device, and the depth information can be obtained based on the camera.

[0108] Specifically, since the specific position of the object needs to be located, when the camera is integrated, the scene depth can be acquired in an active or passive manner. The active manner refers to directly acquiring depth data by using a structured light or a TOF (Time of flight) depth camera. The structured light camera usually consists of a projector and a receiver. The projector projects a light pattern onto the surface of an object, and the receiver captures the deformation of the pattern. By analyzing the deformation, the depth of the object can be obtained. The TOF camera also consists of a projector and a receiver. When the light source emits a light signal, the light signal is reflected on the surface of the object and then received by the receiver. By measuring the flight time of the light signal, the distance between the light source and the object can be calculated. This type of camera provides a complete scene depth map through one-time imaging, has no scanning device, has a fast imaging speed, has a low calculation load, and is widely applied. The passive manner refers to calculating the depth data by using a binocular camera. The principle is to compare the parallax of the same object captured by the left and right cameras, and calculate the distance between the object and the camera according to the parallax.

[0109] For the integration of a ring of cameras, taking a camera with a field of view angle of 90° as an example, if an active camera is used, four cameras need to be integrated; if a passive camera is used, eight cameras need to be integrated.

[0110] In some embodiments, the environment image includes a plurality of sub-environment images, and the environment image around the display device collected by the collection device in step S101 is specifically obtained by: collecting a plurality of sub-environment images collected by the collection device, wherein the plurality of sub-environment images have partial scene overlap; and splicing the plurality of sub-environment images to obtain the environment image.

[0111] Specifically, if the display device integrates a ring of cameras, each camera collects a sub-image of a corresponding region, and the plurality of sub-images need to be spliced. Image splicing refers to combining two or more images with partial scene overlap to generate an image with a wider viewing angle or a 360° panoramic viewing angle, thereby providing more information to support various subsequent processing, and thus has a wide application in the field of machine vision, such as motion detection and tracking, augmented reality, image jitter processing, resolution enhancement, video compression, and the like.

[0112] In some embodiments, the plurality of sub-environment images are spliced to obtain the environment image, specifically by: splicing the plurality of sub-environment images based on a first preset algorithm to obtain the environment image.

[0113] In some embodiments, the plurality of sub-environment images are spliced based on a first preset algorithm to obtain an environment image, specifically comprising: image preprocessing on the plurality of sub-environment images; respectively detecting and matching feature points on the plurality of processed sub-images; solving a homography matrix and performing perspective transformation; image post-processing is performed to splice the plurality of sub-images to obtain a complete environment image.

[0114] Specifically, image preprocessing mainly refers to image enhancement and denoising, which is used to remove noise points in the image and enhance the detail texture features of the image, and improve the accuracy of feature detection; feature point detection and matching refers to extracting key feature points such as corner points, edges, and textures from each image, and determining the corresponding relationship between the images; solving a homography matrix and performing perspective transformation refers to solving the mapping relationship between the to-be-registered image and the reference image through the matched image coordinates of the feature point pairs, the purpose being to convert the perspective of the to-be-registered image into the perspective of the reference image, to ensure correct splicing between the images and to generate a synthesized image with spatial consistency and natural viewing experience; image post-processing refers to using image fusion technology to eliminate possible seams between images after splicing, and to improve the synthesis quality of the images.

[0115] In the embodiments of the present disclosure, a plurality of sub-environment images are respectively collected, and the plurality of sub-environment images are fused based on a first preset algorithm to obtain a complete environment image, without the need for user cooperation in collecting the environment image, which reduces the difficulty of environment image collection, improves the efficiency of safety boundary setting, and maximizes the safety of the user.

[0116] In some embodiments, as shown in FIG. 2, step S104, in response to the existence of an object in the area defined by the target safety boundary, adjusting the target safety boundary based on the position information of the object, specifically comprising: in response to the existence of an object in the area defined by the target safety boundary, generating first prompt information to prompt the user; in response to the user not clearing the object in the area defined by the target safety boundary, adjusting the target safety boundary based on the position information of the object.

[0117] Specifically, as shown in FIG. 2, if the display device determines that there is an object in the area defined by the target safety boundary, it generates first prompt information to prompt the user to clean up the object in step S107. The display device further performs step S108 to determine whether the user has cleared the object in the area defined by the target safety boundary; if the user pays attention to the reminder and clears the object, step S105 is performed; if the user ignores the reminder and insists on not clearing the object in the area defined by the target safety boundary, step S104 is performed: the display device automatically adjusts the target safety boundary to automatically exclude the object from the area defined by the target safety boundary, thereby maximizing the safety of the user.

[0118] It can be understood that the S108 determines whether the user removes the object in the area of the target safety boundary. Specifically, the display device can obtain the environment image of the current time again, and determine whether there is an object in the area limited by the target safety boundary by image recognition on the environment image of the current time, so as to determine whether the user removes the object in the area of the target safety boundary. Of course, the display device can also determine in other ways, for example, if the user removes the object, the user will send an instruction to the display device to inform the display device, and the display device determines based on the received instruction.

[0119] FIGS. 5a-5c are schematic diagrams of three kinds of prompt information.

[0120] In some embodiments, the first prompt information includes display information and voice information. Optionally, the display information includes any one of display line information, display grid information or small window display information. For example, the display device detects in real time whether the object in the environment enters the target safety boundary, and if there is an object entering the safety area, the user will be reminded to pay attention in the form of a small window, as shown in FIG. 5c. Of course, the user can also be reminded by displaying a line or a grid, as shown in FIGS. 5a and 5b, or by issuing an alarm sound, and the present disclosure does not limit this.

[0121] In some embodiments, the adjustment method includes steps S100, S101, S103, S104 and S105, and the adjustment method further includes: obtaining a first distance h1 between the display device and the target safety boundary; and in response to the display device being in the area limited by the target safety boundary and the first distance h1 being less than or equal to a preset distance, generating second prompt information to prompt the user. In some embodiments, the adjustment method includes steps S100, S101, S103, S104 and S105, and the adjustment method further includes: in response to the display device being outside the area limited by the target safety boundary, switching the virtual environment picture to a perspective picture.

[0122] In some embodiments, the second prompt information includes display information and voice information, and the display information includes any one of display line information, display grid information or small window display information.

[0123] Specifically, as shown in FIGS. 4a and 4b, the preset distance can be determined according to a specific experience scenario, where the position of the display device is also the position of the user himself / herself. After the safety boundary is delimited, the display device detects the position of the user himself / herself in real time. If it is detected that the user himself / herself (the position of the head-mounted display) or the gesture (the position of the handle) is close to the boundary and within a certain distance, the second prompt information such as a line or a grid is displayed for prompting. If the user receives the second prompt information and moves away from the safety boundary, the second prompt information disappears. If the user ignores the second prompt information and insists on crossing the target safety boundary, the perspective picture is switched to display the real environment around to ensure the safety of the user. In addition, the display device automatically pauses the application.

[0124] In some embodiments, the first prompt information is different from the second prompt information. In this way, the user can make different responses according to different prompt information to ensure the safety of the user himself / herself.

[0125] In some embodiments, as shown in FIGS. 4a and 4b, the range of the minimum distance h2 between the adjusted target safety boundary and the object is 10 cm to 30 cm. In this way, the safety of the user can be ensured to the maximum extent without excessively narrowing the area defined by the target safety boundary.

[0126] The adjustment method provided by the embodiments of the present disclosure can ensure the safety of the user himself / herself to the maximum extent by acquiring the environmental image of a 360° preset range around the display device, detecting that there is an object in the area defined by the target safety boundary based on the environmental image, and automatically adjusting the target safety boundary according to the specific position information of the object to automatically exclude the object from the area defined by the target safety boundary.

[0127] In a second aspect, based on the same inventive concept, the embodiments of the present disclosure further provide a display device, which comprises a memory, a processor, and a safety boundary adjustment program stored in the memory and executable on the processor. When the safety boundary adjustment program is executed by the processor, the steps of the safety boundary adjustment method in any of the above embodiments are implemented.

[0128] In a third aspect, based on the same inventive concept, the embodiments of the present disclosure further provide a display system. FIG. 6 is a structural block diagram of a display system 600 provided by the embodiments of the present disclosure. As shown in FIG. 6, the display system 600 comprises a processor 601 and an acquisition device 602.

[0129] The processor 601 is configured to perform the following steps: determining an initial security boundary of the display device based on the security mode type; obtaining an environment image of the surroundings of the display device collected by the collection device 602, determining a target security boundary based on the collected environment image and the initial security boundary; judging whether there is an object in the region defined by the target security boundary based on the target security boundary and the environment image at the current time; and adjusting the target security boundary based on the position information of the object in response to the existence of the object in the region defined by the target security boundary. In some embodiments, the processor 601 is specifically configured to generate first prompt information to prompt the user in response to the existence of the object in the region defined by the target security boundary; and adjust the target security boundary based on the position information of the object in response to the user not clearing the object in the region defined by the target security boundary.

[0130] The collection device 602 is used to collect the environment image of the surroundings of the display device and send it to the processor 601. Optionally, the collection device 602 can be integrated on the display device, for example, the collection device 602 can be integrated in front of the display device or around the display device. Optionally, the collection device 602 can also exist separately and then send the collected environment image of the surroundings of the display device to the processor 601 for subsequent processing.

[0131] In some embodiments, the environment image includes multiple sub-environment images, the collection device 602 includes multiple collection components arranged around the display device, each collection component is used to collect a respective sub-environment image of the surroundings of the display device, and there is partial overlap between the multiple sub-environment images; and the processor 601 is further configured to splice the multiple sub-environment images to obtain the environment image.

[0132] Specifically, the collection device 602 can be a camera, and can also be other camera components. Taking the collection device 602 as a camera as an example, the camera can be integrated in front of the display device, at this time, the user needs to cooperate with simple viewing around 360° to enable the collection device 602 to collect a complete environment image. In addition, a ring of cameras can also be integrated on the display device to obtain sub-environment images of the surroundings of the display device at each angle, so that the user does not need to turn his head to observe, and a 360° environment image of the surroundings of the display device can also be collected.

[0133] In some embodiments, the collection device 602 includes a projector and a receiver; the projector is used to project a light pattern onto the surface of an object to obtain a projection pattern; the receiver is used to receive the light pattern and the projection pattern, and obtain the depth information of the object according to the change of the projection pattern relative to the light pattern.

[0134] Specifically, the acquisition device 602 can be a structured light camera, which is usually composed of a projector and a receiver. The projector projects a light pattern onto the surface of an object, and the receiver captures the deformation of the pattern. By analyzing the deformation, the depth information of the object can be obtained.

[0135] In some embodiments, the acquisition device 602 includes a projector and a receiver. The projector is configured to emit a light signal and reflect the light signal on the surface of an object. The receiver is configured to receive a reflected signal of the light signal reflected on the surface of the object. The receiver is further configured to calculate the time of flight of the light signal and calculate the distance between the projector and the object to obtain the depth information of the object.

[0136] Specifically, the acquisition device 602 can be a TOF (Time of flight) depth camera. The TOF camera is also composed of a projector and a receiver. When the light source emits a light signal, the light signal is reflected on the surface of an object and then received by the receiver. By measuring the time of flight of the light signal, the distance between the light source and the object can be calculated. This type of camera provides a complete scene depth map through one-time imaging, has no scanning device, has fast imaging speed, low computing load, and is widely used.

[0137] It can be understood that, in addition to the structured light camera or the TOF camera, the acquisition device 602 can also be other cameras. For example, a binocular camera can also be used to calculate depth data. The principle is to compare the parallax of the same object captured by the left and right cameras, and calculate the distance between the object and the camera according to the parallax.

[0138] In some embodiments, the processor 601 is further configured to receive a security mode type selected by a user. When the received security mode type selected by the user is a stay-in-place mode, the processor 601 is specifically configured to determine initial position information of the display device, take the initial position information as a reference point, obtain boundary information of a region containing a preset range of the reference point, and obtain an initial security boundary.

[0139] In some embodiments, the processor 601 is further configured to receive a security mode type selected by a user. When the received security mode type selected by the user is a self-defined mode, the processor 601 is specifically configured to receive security boundary information set by the user, obtain image information in the security boundary information set by the user, and determine an initial security boundary.

[0140] In some embodiments, the processor 601 is further configured to obtain a first distance between the display device and a target security boundary, and in response to the display device being in a region defined by the target security boundary and the first distance being less than or equal to a preset distance, generate second prompt information to prompt the user.

[0141] In some embodiments, the display system 600 comprises not only the processor 601 and the sampling device 602, but also the handle device 603 for setting the initial safety boundary.

[0142] Specifically, in the in-place mode, the processor 601 determines its initial position information, and after obtaining the boundary information of the area containing the preset range of the reference point based on the initial position information, the handle device 603 sets the initial safety boundary based on the initial position information and the boundary information. The initial position information of the display device is the position information of the user. For example, the handle device 603 takes the current position of the user as the center, and according to the boundary information, a circular area is automatically delineated as the initial safety boundary.

[0143] In the custom mode, the handle device 603 can rotate according to the user's indication to delineate the initial safety boundary. Alternatively, in the custom mode, the processor 601 receives the user-set safety boundary information and obtains the image information within the user-set safety boundary information, and the handle device 603 sets the initial safety boundary according to the image information within the user-set safety boundary information. For example, when the display device is a VR headset, the sampling device 602 is integrated on the display device, and the user wears the display device to rotate a circle, and the sampling device 602 collects the depth information of the environment around the display device. The processor 601 can identify the objects in the environment by using the depth information, and automatically generate the custom initial safety boundary information according to the objects in the environment, and then the handle device 603 sets the initial safety boundary according to the initial safety boundary information. The details of the display system 600 provided in the embodiments of the present disclosure are similar to those of the above-mentioned adjustment method of the safety boundary, and are not repeated here.

[0144] The display system 600 provided in the embodiments of the present disclosure integrates the sampling device 602 on the display device, which can collect the environmental image around the display device. The processor 601 detects whether there is an object within the area defined by the target safety boundary based on the environmental image, and adjusts the target safety boundary in response to the existence of the object within the area defined by the target safety boundary to exclude the object from the safety area, thereby ensuring the safety of the user.

[0145] Based on the same inventive concept, the present disclosure also provides a computer storage medium, wherein the computer storage medium stores a safety boundary adjustment program, and the safety boundary setting program is executed by the processor to implement the steps of the safety boundary adjustment method of any one of the above-mentioned embodiments.

[0146] It is understood that the above embodiments are only exemplary for illustrating the principles of the present application, and the present application is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and scope of the present application, and these modifications and improvements are also considered as the protection scope of the present application.

Claims

1. A method of adjusting a security boundary, wherein, The adjustment method is applied to a display device, and the adjustment method comprises: determining an initial safety boundary of the display device based on a safety mode type; acquiring an environment image of the display device surrounding collected by a collection device, and determining a target safety boundary based on the collected environment image and the initial safety boundary; judging whether an object exists in a region defined by the target safety boundary based on the target safety boundary and the environment image at a current time; in response to the object existing in the region defined by the target safety boundary, adjusting the target safety boundary based on position information of the object.

2. The adjustment method of claim 1, wherein, The adjustment method further comprises: receiving a safety mode type selected by a user; and when the received safety mode type selected by the user is a stay-in-place mode, the step of determining the initial safety boundary of the display device based on the safety mode type comprises: determining initial position information of the display device, and acquiring boundary information of a region within a preset range of the initial position information as a reference point to obtain the initial safety boundary.

3. The adjustment method of claim 1, wherein, The adjustment method further comprises: receiving a safety mode type selected by a user; and when the received safety mode type selected by the user is a self-defined mode, the step of determining the initial safety boundary of the display device based on the safety mode type comprises: receiving safety boundary information set by a user, acquiring image information within the user-set safety boundary information, and determining the initial safety boundary.

4. The adjustment method of claim 1, wherein, The step of judging whether the object exists in the region defined by the target safety boundary based on the target safety boundary and the environment image at the current time specifically comprises: detecting whether the object exists in the environment image at the current time; in response to the object existing in the environment image at the current time, acquiring depth information of the object to obtain position information of the object relative to the target safety boundary; judging whether the object exists in the region defined by the target safety boundary according to the position information of the object.

5. The adjustment method of claim 1, wherein, The environment image comprises a plurality of sub-environment images, and the step of acquiring the environment image of the display device surrounding collected by the collection device specifically comprises: acquiring a plurality of sub-environment images of the display device surrounding collected by the collection device, wherein there is partial overlap of scenes between the plurality of sub-environment images; stitching the plurality of sub-environment images to obtain the environment image.

6. The adjustment method of claim 1, wherein, The step of adjusting the target safety boundary based on the position information of the object in response to the object existing in the region defined by the target safety boundary specifically comprises: in response to the object existing in the region defined by the target safety boundary, generating first prompt information to prompt the user; in response to the user not clearing the object in the region defined by the target safety boundary, adjusting the target safety boundary based on the position information of the object.

7. The adjustment method of claim 6, wherein, The adjustment method further comprises: acquiring a first distance between the display device and the target safety boundary; In response to the display device being within the area defined by the target safety boundary and the first distance being less than or equal to the preset distance, a second prompt information is generated to prompt the user.

8. The adjustment method of claim 7, wherein, The first prompt information includes display information and voice information; and / or, The second prompt information includes display information and voice information.

9. The adjustment method according to any one of claims 1 to 8, wherein, The adjustment method further includes: In response to the display device being outside the area defined by the target safety boundary, switching a virtual environment picture to a perspective picture.

10. A display device, wherein, The display device includes a memory, a processor, and a safety boundary adjustment program stored on the memory and executable on the processor, and the safety boundary adjustment program, when executed by the processor, implements the steps of the safety boundary adjustment method according to any one of claims 1 to 9.

11. A display system, wherein, The display device includes a processor and a collection device for collecting an environment image around the display device; The processor is configured to perform the following steps: Based on the safety mode type, determine the initial safety boundary of the display device; Obtain the environment image around the display device collected by the collection device, and based on the collected environment image and the initial safety boundary, determine the target safety boundary; Based on the target safety boundary and the environment image at the current time, determine whether there is an object within the area defined by the target safety boundary; In response to the object being within the area defined by the target safety boundary, adjust the target safety boundary based on the position information of the object.

12. The display system of claim 11, wherein, The environment image includes a plurality of sub-environment images, and the collection device includes a plurality of collection components arranged around the display device, Each collection component is configured to collect a corresponding sub-environment image around the display device, and the plurality of sub-environment images have overlapping scenes; The processor is further configured to stitch the plurality of sub-environment images to obtain the environment image.

13. The display system of claim 11, wherein, The collection device includes a projector and a receiver; The projector is configured to project a light pattern onto the object surface to obtain a projection pattern; The receiver is configured to receive the light pattern and the projection pattern, and based on the change of the projection pattern relative to the light pattern, obtain the depth information of the object. The collection device includes a projector and a receiver; 14. The display system of claim 11, wherein, The projector is configured to emit a light signal and reflect the light signal on the object surface; The receiver is configured to receive the reflected signal of the light signal on the object surface; The receiver is further configured to calculate the time of flight of the light signal and calculate the distance between the projector and the object to obtain the depth information of the object. The processor is further configured to receive a safety mode type selected by a user; 15. The display system of claim 11, wherein, When the received safety mode type selected by the user is the in-place mode, the processor is specifically configured to: Determine the initial position information of the display device, and take the initial position information as a reference point to obtain boundary information of an area within a preset range of the reference point to obtain the initial safety boundary. The processor is further configured to receive a safety mode type selected by a user; 16. The display system of claim 11, wherein, ​ When the received user-selected security mode type is a custom mode, the processor is specifically configured to: receive user-set security boundary information, acquire image information within the user-set security boundary information, and determine the initial security boundary.

17. The display system of claim 11, wherein, The processor is specifically configured to perform the following steps: in response to the presence of the object within the area defined by the target security boundary, generate first prompt information to prompt the user; in response to the user not clearing the object within the area defined by the target security boundary, adjust the target security boundary based on the position information of the object.

18. The display system of claim 11, wherein, The processor is further configured to perform the following steps: acquire a first distance between the display device and the target security boundary; in response to the display device being within the area defined by the target security boundary and the first distance being less than or equal to a preset distance, generate second prompt information to prompt the user.

19. The display system of claim 11, wherein, The display system further includes a handle device for setting an initial security boundary at an initial time.

20. A computer storage medium, wherein, The computer storage medium stores a security boundary adjustment program, and the security boundary adjustment program, when executed by the processor, implements the steps of the security boundary adjustment method according to any one of claims 1 to 9.

Images