A safety reminder method, device and head-mounted integrated device based on virtual reality
By collecting environmental images and posture information in real time in VR head-mounted all-in-one machine, detecting and monitoring motion and static obstacles, and generating reminder information, the safety hazards of the existing technology that cannot remind motion obstacles are solved, and the various Real-time safety reminders for obstacles improve safety when wearing.
Patent Information
- Application Number
- CN202010848679.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-19
AI Technical Summary
When worn by existing VR headsets, they cannot effectively remind users of safety hazards of collisions with movement obstacles (such as pets or children). The existing technology mainly targets stationary obstacles, and cannot monitor and remind movement obstacles in real time.
The head-mounted all-in-one machine collects environmental images and posture information in real time, performs feature point detection and spatial position calculation, monitors motion and static obstacles in the environment, generates reminder information, and uses the See-Through function to remind users in real time.
Real-time monitoring and reminding of motion and static obstacles is realized, improving the safety of users when wearing VR headsets and avoiding the risk of collision with various obstacles.
Smart Images

Figure CN112114664B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a virtual reality-based safety reminder method, device and head-mounted all-in-one device. Background Art
[0002] Virtual Reality (VR) technology uses computers to fuse and reconstruct multiple types of information to generate a three-dimensional interactive virtual environment to provide users with an immersive experience. When a user wears a head-mounted all-in-one device and walks in a space with obstacles such as tables and chairs, because they cannot see the surrounding environment, they are likely to collide with obstacles in the environment, affecting the user's safety.
[0003] At present, various VR headset manufacturers usually set a fixed safety area virtual fence in the virtual scene. When the user moves freely in the virtual scene, the safety area fence will be displayed at the boundary of the safety area virtual fence to remind the user that there may be unsafe risks ahead and ask the user not to walk out of the safety area fence. Alternatively, through the See-Through function of the headset, the user can see the external real space environment based on the See-Through function, and customize the safety area fence through user-prompted operation interaction.
[0004] The methods in the prior art can, to a certain extent, avoid the physical safety hazards that users experience when using head-mounted all-in-one devices. However, since the existing solutions all use pre-established safety area fences, they can only avoid collisions with stationary obstacles. If a user is wearing the head-mounted all-in-one device and suddenly a pet such as a cat or dog passes by or plays with them, or a child passes by and plays with them, there is a high possibility of physical collision with these moving obstacles, which will pose certain safety hazards to both parties. Summary of the invention
[0005] The present application provides a virtual reality-based safety reminder method, device and head-mounted all-in-one device, which are used to solve or partially solve the above-mentioned problems.
[0006] On the one hand, the present application provides a virtual reality-based safety reminder method, comprising:
[0007] Using the head-mounted integrated device to collect an environmental image of the space environment in real time, and obtaining position information of the head-mounted integrated device corresponding to the environmental image;
[0008] Performing feature point detection on the environment image to obtain a feature point set;
[0009] Determine spatial position information of each feature point in the feature point set relative to the spatial environment according to the posture information;
[0010] Obstacles in the spatial environment are monitored according to the spatial position information of the feature points, and reminder information is generated based on the monitored obstacles, and the reminder information is used to remind the user.
[0011] On the other hand, the present application provides a virtual reality-based safety reminder device, comprising:
[0012] An information acquisition unit, used to use the head-mounted integrated device to collect an environmental image of the space environment in real time, and obtain position information of the head-mounted integrated device corresponding to the environmental image;
[0013] A feature point detection unit, used to perform feature point detection on the environment image to obtain a feature point set;
[0014] A spatial position calculation unit, used to determine the spatial position information of each feature point in the feature point set relative to the spatial environment according to the posture information;
[0015] The safety reminder unit monitors obstacles in the spatial environment according to the spatial position information of the feature point, generates reminder information based on the monitored obstacles, and uses the reminder information to remind the user.
[0016] On the other hand, the present application provides a head-mounted all-in-one device, including a binocular camera, a memory, and a processor;
[0017] Binocular camera, collecting environmental images;
[0018] a memory storing computer executable instructions;
[0019] The processor, when the computer executable instructions are executed, causes the processor to execute the above-mentioned virtual reality-based safety reminder method.
[0020] On the other hand, the present application provides a computer-readable storage medium having one or more computer programs stored thereon, which implement the above-mentioned virtual reality-based safety reminder method when executed.
[0021] The beneficial effects of the present application are as follows: the present application detects feature points of the environmental image by real-time acquisition of the environmental image and the posture information corresponding to the environmental image, and uses the position and posture information of the head-mounted integrated device relative to the spatial environment carried by the posture information to perform spatial transformation on the detected feature point set, uses the spatial position information of the feature points to monitor the moving obstacles and stationary obstacles in the spatial environment, and generates corresponding reminder information to remind the user. That is, the present application monitors the moving obstacles and stationary obstacles in the spatial environment based on the See-Through function of the head-mounted integrated device and the posture information of the head-mounted integrated device, and provides safety reminders for various obstacles in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 is a flow chart of a virtual reality-based safety reminder method shown in one embodiment of the present application;
[0023] Figure 2 is a structural block diagram of a virtual reality-based safety reminder device shown in one embodiment of the present application;
[0024] Figure 3 is a structural block diagram of a head-mounted integrated device shown in one embodiment of the present application;
[0025] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium shown in one embodiment of the present application. DETAILED DESCRIPTION
[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0027] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0028] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0029] The embodiments of the present application are used to protect and guide users when the head-mounted integrated device is used in actual scenarios, such as guiding users in common home scenarios such as living rooms and bedrooms. Due to the advancement and development of VR technology, head-mounted integrated devices can currently support 6DoF (degree of freedom) scenarios, that is, users wearing the head-mounted integrated device can walk around freely and experience various experience contents in the head-mounted integrated device.
[0030] Since the user is wearing a head-mounted all-in-one device, when the user is playing at home, the user is unaware of the real environment. In this case, when walking back and forth in the real environment, it is very likely to collide with obstacles, such as walls, tables, stools and other stationary obstacles, or moving obstacles, such as passing pet cats and dogs, or passing children, which will bring some dangerous hidden dangers to the user's physical safety.
[0031] Based on the above description, the embodiment of the present application provides a safety reminder mechanism for users of the head-mounted integrated device. Before the user is about to collide with an object in the real environment, some friendly safety hazard reminders are displayed in the head-mounted integrated device to simultaneously ensure the user's safety.
[0032] Figure 1 FIG. 1 is a flow chart of a virtual reality-based safety reminder method according to an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application includes the following steps:
[0033] Step S110, using the head-mounted integrated device to collect environmental images of the spatial environment in real time, and obtaining the position and posture information of the head-mounted integrated device corresponding to the environmental image.
[0034] The head-mounted integrated device in this embodiment is equipped with two external binocular cameras, which can be two visible light cameras, for simulating human eyes and capturing environmental images of the space environment in real time, wherein the space environment is the space environment in the actual use scenario of the head-mounted integrated device. The head-mounted integrated device in this embodiment is also provided with a sensor module, such as a built-in IMU 9-axis inertial navigation sensor, which uses the built-in sensor module to collect the posture information of the head-mounted integrated device in real time.
[0035] Among them, the posture information of the head-mounted integrated device corresponding to the environmental image can be understood as: posture information with the same or similar timestamp information as the environmental image. For example, if the environmental image is acquired at time A1, then the posture information corresponding to the environmental image is the posture information of the head-mounted integrated device acquired at time A1±10ms. ±10ms is an example, and in the application process, it can be specifically set according to the accuracy requirements, and this embodiment does not impose specific restrictions on this.
[0036] Step S120: perform feature point detection on the environment image to obtain a feature point set.
[0037] In this step, a preset detection algorithm can be used to perform feature point detection, and feature point detection can be performed using feature point detection algorithms such as FAST (Features From Accelerated Segment Test), SIFT (Scale-invariant feature transform), and SURF (Speeded Up Robust Features). The position information represented by the obtained feature point set is the spatial position information in the camera coordinate system.
[0038] Step S130, determining the spatial position information of each feature point in the feature point set relative to the spatial environment according to the posture information.
[0039] Since the posture information of the head-mounted integrated device carries the position and posture information of the head-mounted integrated device relative to the spatial environment, this embodiment uses the posture information of the head-mounted integrated device to convert the coordinate point set in the camera coordinate system (i.e., the head-mounted integrated device) into the coordinate point set in the world coordinate system (i.e., the spatial environment).
[0040] Step S140: monitor obstacles in the spatial environment according to the spatial position information of the feature point, generate reminder information based on the monitored obstacles, and use the reminder information to remind the user.
[0041] Since this step calculates the obstacle information at each image acquisition moment based on the coordinate point set in the world coordinate system, real-time detection of moving obstacles and stationary obstacles can be achieved.
[0042] Depend on Figure 1 As shown, this embodiment detects feature points of the environmental image by collecting the environmental image and the posture information corresponding to the environmental image in real time, and uses the position and posture information of the head-mounted integrated device relative to the spatial environment carried by the posture information to perform spatial transformation on the detected feature point set. The spatial position information of the feature points can be used to monitor moving obstacles and stationary obstacles in the spatial environment, and generate corresponding reminder information to remind the user.
[0043] This embodiment monitors moving obstacles and stationary obstacles in the spatial environment based on the See-Through function of the head-mounted integrated device and the posture information of the head-mounted integrated device, and provides safety reminders for various obstacles in real time.
[0044] The following takes the obstacle monitoring of a VR head-mounted all-in-one device as an example. Figure 1 The implementation steps of the security reminder method in are specifically explained.
[0045] The head-mounted all-in-one device in this embodiment has an external binocular camera and a built-in sensor module. The binocular camera is used to collect environmental images of the spatial environment, and each frame of the environmental image includes a left-eye image and a right-eye image. The built-in sensor module is used to collect the posture information of the head-mounted all-in-one device, and the posture information includes the position and posture information of the head-mounted all-in-one device.
[0046] The environment image and the position information have the same or similar timestamp information, so that the environment image and the position information can be used to monitor various obstacles in the space environment in real time.
[0047] Before using the head-mounted integrated device to collect environmental images, the binocular cameras are calibrated. This embodiment adopts the traditional Zhang Zhengyou calibration method to obtain the internal parameters and distortion parameters of the left and right cameras and the rotation and translation matrix parameters of the left camera relative to the right camera.
[0048] In this way, the environmental image can be subjected to anti-distortion processing according to the calibration parameters and distortion parameters of the camera used by the head-mounted integrated device to collect the environmental image, so as to obtain the environmental image after anti-distortion processing; feature point detection is performed on any one of the left-eye image and the right-eye image included in the environmental image after anti-distortion processing, so as to obtain a group of feature points; the position of the group of feature points on the other image is calculated according to the principle of binocular stereo imaging, so as to obtain another group of feature points; the three-dimensional coordinates of the two groups of feature points on the left-eye image and the right-eye image in the camera coordinate system are calculated according to the principle of stereo triangulation positioning, so as to obtain the feature point set.
[0049] In one example, after using a binocular camera to collect an environmental image of a spatial environment, the environmental image is dedistorted using internal parameters and distortion parameters to eliminate the distortion effect of the image and reduce image errors. The feature points on the left image are detected using the FAST algorithm. Based on the principle of binocular stereo imaging, the position of each feature point on the left image corresponding to the right image is obtained through the Normalized Cross Correlation (NCC) algorithm and the rotation and translation matrix parameters of the left camera relative to the right camera. At the same time, the three-dimensional coordinates of the feature points on the left and right images in the camera coordinate system are calculated through the principle of stereo triangulation.
[0050] After the feature point set is detected, a rotation matrix and a translation vector are calculated according to the position information, and the feature points in the feature point set are transformed into a coordinate system using the rotation matrix and the translation vector to obtain the three-dimensional coordinates of each feature point in the world coordinate system.
[0051] For example, the three-dimensional coordinates of each feature point in the world coordinate system are calculated according to the following formula (1):
[0052] Object w =R HMD *Object+T HMD (1)
[0053] In the above formula (1), Objecpoint w Represents the feature point set in the world coordinate system, Objecpoint represents the feature point set in the camera coordinate system, R HMD represents the rotation matrix, T HMD Represents the translation vector.
[0054] After calculating the spatial position information of each feature point relative to the spatial environment, the feature points in the feature point set are clustered according to the spatial position information of the feature points, that is, the three-dimensional coordinates of the feature points, to obtain one or more clustering results, each clustering result corresponding to an obstacle; the size of each obstacle is calculated according to the spatial position information of the feature points contained in each clustering result; reminder information is generated according to the number of obstacles monitored and the size of each obstacle, and the reminder information is used to remind the user.
[0055] Since there are a large number of feature points detected based on the feature point detection algorithm, directly monitoring obstacles will occupy a large amount of computing resources, and the detected feature points include various feature points at close range and long range in the environmental image, among which only the feature points at close range can detect obstacle information that poses a safety hazard to the user. Therefore, before clustering the feature points in the feature point set, this embodiment selects feature points whose distance from the head-mounted integrated device is less than a preset distance from the feature point set according to the spatial position information of the feature points, and obtains the filtered feature point set; the obstacles in the environment are monitored according to the spatial position information of each feature point in the filtered feature point set. Under the premise of ensuring the reliability of obstacle monitoring, computing resources are saved, the obstacle detection speed is improved, and the delay is reduced.
[0056] In one example, the feature point set Object can be sorted based on a preset distance. wThe feature points in the image are filtered, for example, the feature points within 50cm from the head-mounted integrated device are filtered. It should be noted that the preset distance can be set according to the complexity of the actual environment. The safety alarm range shown in this example is 50CM. If there is an obstacle within 50CM of the user, the user needs to be reminded in real time. Obviously, the safety alarm range can also be set to other values, and this application does not make specific restrictions on this.
[0057] In the process of selecting feature points, the feature point set Objecpoint can be selected based on the following formula (2): w Filter the feature points in:
[0058]
[0059] In the above formula (2), Dist represents the distance from the head-mounted integrated device.
[0060] The feature point set Object w The feature points whose Dist is less than the preset distance are screened out to obtain the feature point set ObstaclePoint. The feature point set ObstaclePoint is clustered by the KNN clustering algorithm to obtain the N obstacle information ObsPoint in the feature point set ObstaclePoint, ObsPoint = {Obstacle1, Obstacle2, Obstacle3, .... Obstacle n}, N is a natural number greater than or equal to 1, Obstacle n is the nth obstacle.
[0061] The size of each obstacle can be calculated according to formula (3):
[0062]
[0063] In the above formula (3), Obstacle i Indicates the clustering result corresponding to the i-th obstacle, Width i and Height i Represent the width and height of the i-th obstacle respectively.
[0064] In some embodiments, after obtaining the clustering results, the center position of each clustering result is also calculated, and the distance information of the obstacle from the head-mounted integrated device is obtained based on the center position, that is, the distance information of the obstacle from the head-mounted integrated device can be calculated according to the above formula (2). Then, the reminder information is generated according to the number of obstacles monitored, the size of each obstacle, and the distance information of each obstacle from the head-mounted integrated device, that is, the generated reminder information carries the number of obstacles, the size of each obstacle, and the distance of each obstacle. Therefore, based on various interactive modes of the head-mounted integrated device, the generated reminder information about obstacles can be displayed to the user in real time, so as to achieve the purpose of reminding the user to pay attention to safety in real time.
[0065] Based on the above description, this embodiment can implement safety reminders in a virtual reality environment.
[0066] Corresponding to the above method, the present application also provides a safety reminder device based on virtual reality, Figure 2 FIG. 1 is a block diagram of a virtual reality-based safety reminder device according to an embodiment of the present application. Figure 2 As shown, the safety reminder device 200 of this embodiment includes:
[0067] The information acquisition unit 210 is used to use the head-mounted integrated device to collect an environmental image of the space environment in real time, and obtain the position information of the head-mounted integrated device corresponding to the environmental image;
[0068] A feature point detection unit 220 is used to perform feature point detection on the environment image to obtain a feature point set;
[0069] A spatial position calculation unit 230, configured to determine spatial position information of each feature point in the feature point set relative to the spatial environment according to the posture information;
[0070] The safety reminder unit 240 monitors obstacles in the spatial environment according to the spatial position information of the feature points, generates reminder information based on the monitored obstacles, and uses the reminder information to remind the user.
[0071] In some embodiments, the safety reminder unit 240 includes an obstacle monitoring module and a reminder module;
[0072] The obstacle monitoring module is used to cluster the feature points in the feature point set according to the spatial position information of the feature points to obtain one or more clustering results, each clustering result corresponds to an obstacle; and the size of each obstacle is calculated according to the spatial position information of the feature points contained in each clustering result;
[0073] The reminder module is used to generate reminder information according to the number of obstacles monitored and the size of each obstacle, and use the reminder information to remind the user.
[0074] In some embodiments, the obstacle monitoring module is further used to calculate the center position of each clustering result after obtaining the clustering results, and obtain the distance information of the obstacle from the head-mounted integrated device based on the center position.
[0075] Correspondingly, the reminder module is also used to generate reminder information based on the number of obstacles monitored, the size of each obstacle and the distance information of each obstacle from the head-mounted integrated device.
[0076] In some embodiments, the safety reminder unit 240 also includes a screening module, which is used to screen out feature points whose distance from the head-mounted integrated device is less than a preset distance from the feature point set according to the spatial position information of the feature points, so as to obtain a screened feature point set; correspondingly, the obstacle monitoring module is specifically used to monitor obstacles in the environment according to the spatial position information of each feature point in the screened feature point set.
[0077] In some embodiments, the feature point detection unit 220 is further used to perform anti-distortion processing on the environmental image according to the calibration parameters and distortion parameters of the camera used by the head-mounted integrated device to collect the environmental image, so as to obtain the environmental image after anti-distortion processing; perform feature point detection on any one of the left-eye image and the right-eye image included in the environmental image after anti-distortion processing to obtain a group of feature points; calculate the position of the group of feature points on the other image according to the principle of binocular stereo imaging to obtain another group of feature points; calculate the three-dimensional coordinates of the two groups of feature points on the left-eye image and the right-eye image in the camera coordinate system through the principle of stereo triangulation positioning to obtain the feature point set.
[0078] In some embodiments, the spatial position calculation unit 230 is also used to calculate a rotation matrix and a translation vector based on the posture information; and use the rotation matrix and the translation vector to perform coordinate system transformation on the feature points in the feature point set to obtain the three-dimensional coordinates of each feature point in the world coordinate system.
[0079] In summary, this embodiment detects feature points of the environmental image by real-time acquisition of the environmental image and the posture information corresponding to the environmental image, and uses the position and posture information of the head-mounted integrated device relative to the spatial environment carried by the posture information to perform spatial transformation on the detected feature point set. The spatial position information of the feature points can be used to monitor moving obstacles and stationary obstacles in the spatial environment, and generate corresponding reminder information to remind the user. That is, the embodiment of the present application monitors moving obstacles and stationary obstacles in the spatial environment based on the See-Through function of the head-mounted integrated device and the posture information of the head-mounted integrated device, and provides safety reminders for various obstacles in real time.
[0080] It should be noted that:
[0081] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages can be utilized to realize the content of the application described herein, and the description of the specific language above is for the purpose of disclosing the best mode of the application.
[0082] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0083] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.
[0084] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0085] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0086] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components in the virtual reality-based safety reminder device according to the embodiment of the present application. The present application can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0087] For example, Figure 3 : is a structural block diagram of a head-mounted integrated device shown in an embodiment of the present application, and the head-mounted integrated device 300 includes a binocular camera 340, a processor 310, and a memory 320 arranged to store computer executable instructions (computer readable program code). The binocular camera 340 collects environmental images, and the memory 320 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. The memory 320 has a storage space 330 for storing a computer-readable program code 331 for executing any method step in the above method. For example, the storage space 330 for storing computer-readable program code may include various computer-readable program codes 331 respectively used to implement various steps in the above method. The computer-readable program code 331 can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards, or floppy disks. Such computer program products are typically, for example Figure 4 The computer-readable storage medium shown.
[0088] Figure 4A schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable storage medium 400 stores a computer-readable program code 331 for executing the method steps according to the present application, which can be read by the processor 310 of the head-mounted integrated device 300. When the computer-readable program code 331 is executed by the head-mounted integrated device 300, the head-mounted integrated device 300 executes each step of the method described above. Specifically, the computer-readable program code 331 stored in the computer-readable storage medium can execute the method shown in any of the above embodiments. The computer-readable program code 331 can be compressed in an appropriate form.
[0089] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.
Claims
1. A safety reminder method based on virtual reality, characterized in that: include: Using the head-mounted integrated device to collect an environmental image of the space environment in real time, and obtaining the position and posture information of the head-mounted integrated device corresponding to the environmental image; Performing feature point detection on the environment image to obtain a feature point set; Determine, according to the posture information, the spatial position information of each feature point in the feature point set relative to the spatial environment; The spatial position information of the feature point relative to the spatial environment is the three-dimensional coordinates of the feature point in the world coordinate system; Monitoring obstacles in the spatial environment according to the spatial position information of the feature points, generating reminder information based on the monitored obstacles, and using the reminder information to remind the user; Perform feature point detection on the environment image to obtain a feature point set, including: Performing anti-distortion processing on the environmental image according to calibration parameters and distortion parameters of a camera used by the head-mounted all-in-one device to collect the environmental image, to obtain an environmental image after anti-distortion processing; Performing feature point detection on any one of the left-eye image and the right-eye image included in the anti-distortion processed environment image to obtain a set of feature points; The positions of the set of feature points on another image are calculated based on the binocular stereo imaging principle to obtain another set of feature points; The three-dimensional coordinates of two groups of feature points on the left eye image and the right eye image in the camera coordinate system are calculated by the principle of stereo triangulation to obtain the feature point set.
2. The method according to claim 1, characterized in that Monitoring obstacles in the spatial environment according to the spatial position information of the feature point, generating reminder information based on the monitored obstacles, and using the reminder information to remind the user, including: Clustering the feature points in the feature point set according to the spatial position information of the feature points to obtain one or more clustering results, each clustering result corresponding to an obstacle; The size of each obstacle is calculated based on the spatial position information of the feature points contained in each clustering result; Reminder information is generated according to the number of obstacles monitored and the size of each obstacle, and the reminder information is used to remind the user.
3. The method according to claim 2, characterized in that After getting the clustering results, it also includes: The center position of each clustering result is calculated, and the distance information of the obstacle from the head-mounted integrated device is obtained based on the center position.
4. The method according to claim 3, characterized in that According to the number of obstacles detected and the Size generates reminder information, including: Reminder information is generated based on the number of obstacles monitored, the size of each obstacle, and the distance between each obstacle and the head-mounted integrated device.
5. The method according to claim 2, characterized in that Before clustering the feature points in the feature point set, the following steps are also included: According to the spatial position information of the feature points, feature points whose distances from the head-mounted integrated device are less than a preset distance are selected from the feature point set to obtain a selected feature point set; The obstacles in the monitoring environment according to the spatial position information of the feature points are specifically: Obstacles in the environment are monitored based on the spatial position information of each feature point in the filtered feature point set.
6. The method according to claim 1, characterized in that Determining spatial position information of each feature point in the feature point set relative to the spatial environment according to the posture information includes: Calculate a rotation matrix and a translation vector according to the position and posture information; The rotation matrix and the translation vector are used to perform coordinate system transformation on the feature points in the feature point set to obtain the three-dimensional coordinates of each feature point in the world coordinate system.
7. A safety reminder device based on virtual reality, characterized in that: include: An information acquisition unit, used to use the head-mounted integrated device to collect an environmental image of the space environment in real time, and obtain position information of the head-mounted integrated device corresponding to the environmental image; A feature point detection unit, used to perform feature point detection on the environment image to obtain a feature point set; A spatial position calculation unit, used to determine the spatial position information of each feature point in the feature point set relative to the spatial environment according to the position information; the spatial position information of the feature point relative to the spatial environment is the three-dimensional coordinates of the feature point in the world coordinate system; A safety reminder unit, which monitors obstacles in the spatial environment according to the spatial position information of the feature point, generates reminder information based on the monitored obstacles, and uses the reminder information to remind the user; The feature point detection unit is also used to perform anti-distortion processing on the environmental image according to the calibration parameters and distortion parameters of the camera used by the head-mounted integrated device to collect the environmental image, so as to obtain the environmental image after anti-distortion processing; perform feature point detection on any image of the left eye image and the right eye image included in the environmental image after anti-distortion processing, so as to obtain a group of feature points; calculate the position of the group of feature points on the other image according to the principle of binocular stereo imaging, so as to obtain another group of feature points; calculate the three-dimensional coordinates of the two groups of feature points on the left eye image and the right eye image in the camera coordinate system according to the principle of stereo triangulation positioning, so as to obtain the said feature point set.
8. A head-mounted integrated device, characterized in that: include: Stereo camera, memory and processor; Binocular camera, collecting environmental images; a memory storing computer executable instructions; The processor, when the computer executable instructions are executed, causes the processor to execute the virtual reality-based safety reminder method as described in any one of claims 1 to 6.
