Method for automatically generating geofences, real-time detection method and device

Abstract

A method for automatically generating a geofence, a real-time detection method and an apparatus are provided, the method including the steps of: acquiring an environmental image (101), the environmental image being acquired by photographing a scene in which a user is located; generating spatial structure information based on the environmental image (102), the spatial structure information including a 3D point cloud and information about at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; and generating the geofence based on the spatial structure information (103).

Description

[Technical Field]

[0001] TECHNICAL FIELD Embodiments of the present application relate to the field of communications, and in particular to an automatic generation method and real-time detection method and device for geofences. [Background technology]

[0002] In recent years, virtual reality (VR) technology has gained popularity in the industrial and consumer sectors. even more It is attracting attention and is a VR device. against market Essential request has steadily increased While using VR devices, the virtual world is completely isolated from the real world, and users cannot sense their surroundings while using them. Therefore, to protect users' safety, geofences are used as the boundaries of safety areas. When users cross the geofence, they are warned of the risk.

[0003] However, geofences are applied scene The requirements are high, It is not easy for users to use For general use inconvenience is. Summary of the Invention

[0004] The present application provides a method for automatically generating a geofence, a real-time detection method, and an apparatus for automatically generating a geofence. This makes it easier to use. The automatically generated geofence has high accuracy and high security, which can greatly improve the user experience. The real-time detection method and device can update the geofence in a timely manner, thereby further improving the accuracy and security of the geofence.

[0005] According to a first aspect, an embodiment of the present application provides a method for automatically generating a geofence, the method comprising: acquiring an environmental image, the environmental image being a scene and generating spatial structure information based on the environmental image, the spatial structure information including information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; generating the geofence based on the spatial structure information; The present invention provides a method comprising:

[0006] In an embodiment of the present application, when a user needs to use an electronic device, the camera or camera lens of the electronic device is located at the position where the user is. scene to acquire an environmental image, and scene Spatial structure information is generated that can represent different object forms such as 3D objects and the ground in the map, and based on the spatial structure information, a boundary within which the user can safely move is determined, and a geofence can be obtained after connecting the boundary.

[0007] Alternatively, the 3D point cloud in the spatial structure information may be output by a SLAM system and used to describe the shape of the 3D object. The information about the plane may be information about a plane fitted based on an actual 3D point cloud obtained by simulating the object distribution of the 3D point cloud in the spatial structure information, and may represent an actual plane of the object. The information about the plane may be information that can describe the plane, such as the length and width of the plane.

[0008] The electronic device may generate the geofence in different ways based on different spatial structure information. For example, first, a boundaryless reference plane corresponding to the ground is formed, and then, based on the spatial structure information, processes such as detection and identification are performed on objects in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the obstacle area of ​​the object in the space are determined. On the boundaryless reference plane, the boundary of the reference plane and the obstacle area can be connected to obtain a safe area where the user can move safely, and the boundary of the safe area is the geofence.

[0009] In this embodiment of the present application, a geofence can be automatically generated based on an environmental image. Alternatively, the environmental image can be obtained by starting a photography mode or an image capture mode after power-on, without requiring user operation. The geofence generated in this embodiment of the present application has high accuracy and security, and can easily identify the location of the user. scene It is highly adaptable to various geofences and is not limited by the shape of the geofence. Geofencing made easy , the user experience is greatly improved.

[0010] In this application, a plane is a horizontal plane, a vertical plane, or an inclined plane, and an inclined plane is a plane that is not parallel to a horizontal or vertical plane.

[0011] In a possible implementation, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0012] In a possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0013] In a possible implementation, before generating spatial structure information based on the environmental image, the method further includes a step of acquiring inertial measurement unit (IMU) data, the IMU data indicating where the user is located. scene The spatial structure information is obtained by performing an IMU solution for the object at the environmental image. The step of generating spatial structure information based on the environmental image includes generating the spatial structure information based on the environmental image and the IMU data.

[0014] The IMU data indicates the user's location. sceneBased on the measured acceleration signal of the object in the navigation coordinate system, the angular velocity signal of the carrier relative to the navigation coordinate system and the measured angular velocity and acceleration of the object in three-dimensional space are decomposed onto three independent axes of the carrier coordinate system.

[0015] The electronic device may generate the geofence in different ways based on different spatial structure information. For example, first, a boundaryless reference plane corresponding to the ground is formed, and then, based on the spatial structure information, processes such as detection and identification are performed on objects in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the projection of the object in the space, i.e., the obstacle area, are determined. On the boundaryless reference plane, the boundary of the reference plane and the obstacle area can be connected to obtain a safe area where the user can move safely, and the boundary of the safe area is the geofence.

[0016] In a possible implementation, the environmental image is VR Using glasses or an intelligent device with a camera, the user is located scene The user needs a geofence. scene When the power is turned on, the user turns the power on. The camera or camera lens The user is located scene Automatically captures and captures images but Processing or detection will be done. For example, environment detection, terrain detection, environmental data extraction, image descriptor acquisition but line Broken An environmental image including the above various information may be acquired.

[0017] In a possible implementation, the spatial structure information may be pose information, a 3D point cloud, a plane Information about , depth data, mesh identification data, and 3D object identification information, the step of generating the spatial structure information based on the environmental image and the IMU data, and the step of generating the geofence based on the spatial structure information include: acquiring the pose information and the 3D point cloud based on the environmental image and the IMU data; performing plane detection on the pose information and the 3D point cloud to obtain the plane; performing depth detection based on the environment image to obtain depth data; The aforementioned depth processing the mesh identification data and the 3D object identification information based on the data; generating the geofence based on mesh identification data and 3D object identification information processed based on the 3D point cloud, the plane, and the depth data; Includes.

[0018] The spatial structure information may be the orientation information, the 3D point cloud, information about the plane, the depth data, the mesh identification data, and the 3D object identification information. simultaneous Location and mapping ( Simultaneous The depth information may be obtained by performing real-time pose estimation using a SLAM (Simultaneous Location and Mapping) system, and is mainly used for coordinate alignment so that the generated geofence is more consistent with the detected 3D real scene. The depth information may be obtained by depth estimation. After the depth data is input into the electronic device, a depth map is obtained, and the depth map is aligned with a planar image of the environmental image to obtain depth values ​​corresponding to each pixel on the planar image. The environmental image and the IMU data When the depth value is added, the accuracy of the output 3D point cloud is effectively improved and the density of the output 3D point cloud is increased. The mesh identification information is a triangular surface generated after mesh identification, and the mesh identification information is a triangular surface where the user is located. scene The 3D object identification information is used to describe the exterior of the 3D object after the 3D object is identified. dataThe generation of spatial structure information based on the above may be as follows: data The system inputs the environmental image, and performs feature extraction and matching in real time based on the input environmental image to obtain a matching relationship between the planar features of the image. Next, the pose of the camera is estimated based on the IMU data and corresponding positional relationship parameters between the image capture device, e.g., a camera, and the IMU to obtain the original pose, i.e., the pose information of the camera. Next, a 3D point cloud is generated using an algorithm, e.g., a triangulation algorithm, based on the pose information and the matching relationship between the planar features, resulting in the 3D point cloud output by the SLAM system. Next, planes in space are obtained by plane detection. Next, depth estimation is performed using the environmental image as input to obtain depth data. The depth data may be used as input for other data detection. For example, mesh identification or 3D object identification is performed using the depth data as input to obtain mesh identification data and 3D object identification information separately. According to the above method, a reference plane and its boundary are obtained based on the 3D point cloud. This 3D point cloud may be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the method, an obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on the information about the plane, the mesh identification data processed based on the depth data, and the 3D object identification information, thereby optimizing the obstacle area to determine the obstacle area where the user is located. scene The geofence can be generated by connecting the surrounding obstacle areas on the reference plane to obtain a more accurate safety activity area, and the boundary of that area becomes the generated geofence.

[0019] In a possible implementation, the step of generating the geofence comprises: scene The coordinate system of the environmental image is set to the plane point of the user position at the center. By aligning , generating the geofence.

[0020] The electronic device connects the surrounding obstacle areas on a reference plane around the user currently using the device to obtain a safe activity area, and the boundary of the obstacle area is aligned with the coordinate system of the boundary of the corresponding obstacle 3D object in the environmental image, and the generated geofence is connected to the location of the user. scene The geofence essentially corresponds to the location of each 3D object in the scene is more suitable for

[0021] In a possible implementation, after generating the geofence, the method further comprises: Store The method further includes the step of:

[0022] After generating the geofence, the electronic device stores the geofence as a historical geofence, and upon next use, the user can search among the stored historical geofences and select the geofence for the next use. scene Therefore, the geofence can be set to the same value every time. scene There is no need to generate it repeatedly for

[0023] In a possible implementation, after the step of acquiring the environmental image, the method further includes the steps of: searching for the environmental image from the stored historical geofences; and if historical geofences with relevant similarity are acquired, calculating weighted scores for the historical geofences and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence; and if the resolution is successful, setting the target geofence based on the difference between the pose of the target geofence and the pose of the environmental image, and aligning the coordinate system of the target geofence with the coordinate system of the environmental image.

[0024] In this embodiment of the present application, weighted clustering is performed on corresponding candidate frames in different historical geofences based on descriptor distance sorting, and a target geofence with the highest weighted score is obtained. A posture is resolved for the target geofence. If the resolution is successful, a coordinate transformation relationship is calculated based on the difference between the successfully resolved posture and the posture of the target geofence and the posture of the environment image, and the target geofence is loaded based on the coordinate transformation relationship. The coordinate system of the target geofence is determined based on the geofence corresponding to the environment image, i.e., the geofence the user needs to use. Geofence The coordinate system is aligned with that of

[0025] In a possible embodiment, the depth data is time of flight TOF data.

[0026] The depth data output by the TOF sensor is preferably TOF data obtained by the TOF sensor, since the accuracy of the depth data is much higher than that of the depth estimation result. For example, the depth map output by the TOF sensor is used as the input for mesh identification and 3D object detection, and more accurate mesh identification data and 3D object identification information are separately obtained. According to the above method, the reference plane and the boundary of the reference plane are obtained based on a 3D point cloud. This 3D point cloud can be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the above method, an obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on information about the plane, the mesh identification data processed based on the depth data, and the 3D object identification information, thereby optimizing the obstacle area to locate the user. scene The geofence can be generated by connecting the surrounding obstacle areas on the reference plane to obtain a more accurate safety activity area, and the boundary of that area becomes the generated geofence.

[0027] In a possible implementation, the step of generating the geofence based on the spatial structure information includes: constructing an optimization formula based on the 3D point cloud and information about the plane to obtain an optimization plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground plane; projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region on the reference horizontal plane; and generating the geofence based on the reference horizontal plane and the obstacle region.

[0028] An optimization formula is constructed based on the 3D point cloud and information about the plane. The plane is optimized based on the optimization formula to obtain an optimized plane. The obtained optimized plane can not only truly represent various objects in the environment where the user is located, but also contribute to some improvements in generating a geofence based on information about the plane. scene Therefore, it is possible to optimize the collision prevention of small planar objects that do not affect the user's use, so that the optimized plane is more accurate and the generated geofence better meets the usage requirements. The step of generating the geofence based on the reference horizontal plane and the obstacle area may be as follows: the currently used device is taken as the center, and the surrounding obstacle area on the reference horizontal plane is connected to obtain a safe activity area, and the boundary of this area can be the generated geofence.

[0029] According to a second aspect, an embodiment of the present application provides a method for automatically generating an offense, the method comprising: acquiring an environmental image, the environmental image being a scene and generating spatial structure information based on the environmental image, the spatial structure information including a 3D point cloud and 3D object identification information; identifying a reference plane corresponding to the ground based on the 3D point cloud; determining a projection of a 3D object on the reference plane whose distance to the ground exceeds a first threshold based on the 3D object identification information, and determining a boundary and an obstacle area of ​​the reference plane; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; The present invention provides a method for identifying a reference plane corresponding to a ground plane based on the 3D point cloud. For details of the method, see the previous example. Next, a 3D object whose distance to the ground exceeds a first threshold is projected onto the reference plane to determine an obstacle region. After separating the obstacle region from the reference plane, a boundary of the reference plane is connected to obtain a safe region, and the boundary of the safe region is used as the geofence.

[0030] According to a third aspect, an embodiment of the present application provides a method for automatically generating an offense, the method comprising: acquiring an environmental image, the environmental image being a scene and generating spatial structure information based on the environmental image, the spatial structure information including mesh identification data and 3D object identification information; The mesh Identification Data identifying a reference plane corresponding to the ground based on determining a projection of a 3D object on the reference plane whose distance to the ground exceeds a first threshold based on the 3D object identification information, and determining a boundary and an obstacle area of ​​the reference plane; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; The present invention provides a method comprising:

[0031] First, the mesh Identification Data The reference plane corresponding to the ground may be identified based on the mesh identification data. For example, the mesh identification data obtained by mesh identification is a triangular surface corresponding to the ground. After connecting the triangular surfaces, an initial ground area may be obtained and determined as the reference plane. Next, the mesh triangular surface and a 3D object whose distance to the ground exceeds a first threshold are projected onto the reference plane to determine an obstacle area. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. For example, after separating the obstacle area from the reference plane, the boundary of the reference plane is connected to obtain a safe area, and the boundary of the safe area is determined as the geofence.

[0032] In this way, the method of determining the geofence by combining the contents of two types of spatial structure information can minimize the solution's requirements for computing power and obtain an accurate safety activity area, on the premise of ensuring the main function.

[0033] According to a fourth aspect, an embodiment of the present application provides a method for detecting a geofence in real time, comprising: The user is located scene Periodically captures real-time environmental images Capture and When it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, updating the geofence based on the spatial structure information generated based on the real-time environmental image; The present invention provides a method comprising:

[0034] The user is located using the camera scene Periodically captures real-time environmental images Capture Or you can use other imaging equipment to record in real time. scene in real time and periodically acquires real-time environmental images. CaptureThe real-time spatial structure information may be extracted from the image. The real-time spatial structure information is acquired based on the real-time environmental image, existing spatial structure information is acquired based on the environmental image, and a local comparison between the real-time spatial structure information and the existing spatial structure information is performed in real time. The type of change in the spatial structure information may be fusion of the real-time spatial structure information and the existing spatial structure information based on different fusion weight values. For example, adding a spatial structure, changing the size or orientation of an existing spatial structure, or modifying the original spatial structure may be performed. non-existence These all belong to the category of changes in spatial structure information.

[0035] In this embodiment of the present application, if new spatial structure information is added to the real-time environmental image, the geofence should be updated with the difference in time to ensure the safety of the usage process. If any spatial structure information changes in the real-time environmental image, different weight values ​​should be set based on the position, size, and absolute value of the change in the structure to prevent detection errors from affecting the update of the geofence. A larger difference in change indicates a larger weight value. If some spatial structure information in the real-time environmental image is deleted, data checks for multiple frames are required. After confirming the loss of spatial structure information multiple times and completing fusion, global optimization is performed based on the weight values ​​corresponding to each spatial structure information, and then the geofence is updated based on the spatial structure information used by the device.

[0036] In this way, there is no need to update the geofence in real time and generate a new geofence, but instead, the fusion is completed on the existing basis and the added or subtracted part is reflected in the geofence in real time, thereby efficiently improving the security of the geofence.

[0037] According to a fifth aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to acquire an image of an environment, the image of the environment being an image of a location where a user is located; scenean input module, a processing module configured to generate spatial structure information based on the environment image, the spatial structure information including information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; Including, The apparatus provides an apparatus in which the processing module is further configured to generate the geofence based on the spatial structure information.

[0038] In a possible implementation, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0039] In a possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0040] In a possible implementation, the input module is further configured to acquire Infrared Measurement Unit (IMU) data, the IMU data being used to determine where the user is located. scene is obtained by performing an IMU solution on the object at The processing module is further configured to generate the spatial structure information based on the environment image and the IMU data.

[0041] In a possible implementation, the input module may include a scene The device is further configured to acquire the environmental image by capturing an image of the environment.

[0042] In a possible implementation, the processing module comprises: constructing an optimization formula based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground plane; Projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region on the reference horizontal plane; generating the geofence based on the horizontal reference plane and the obstacle region; It is further configured as follows.

[0043] In a possible implementation, the processing module includes a pose estimation unit, a 3D point cloud processing unit, a pose estimation unit, a depth estimation unit, a mesh identification unit, and a 3D object identification unit; The pose estimation unit, based on the environment image and the IMU data, figure configured to acquire trend information, The 3D point cloud processing unit processes the 3D point cloud based on the environmental image and the IMU data. 3 further configured to acquire a D point cloud; the plane detection unit is further configured to perform plane detection on the pose information and the 3D point cloud to obtain the plane; The depth estimation unit performs depth detection based on the environment image, Oku configured to retrieve outgoing data, The mesh identification unit is configured to: M configured to process the flash identification data; The 3D object identification unit is configured to: 3 D. configured to process object identity information; J The offense generation unit generates a processed image based on the 3D point cloud, the plane, and the depth data. M Bush identification data and 3 DBased on object identification information J The device is further configured to generate an offense.

[0044] In a possible embodiment, the depth data is time of flight TOF data.

[0045] In a possible implementation, the geofence generation unit is configured to generate a geofence based on the location of the user. scene and generating the geofence using an alignment with the coordinate system of the environmental image, centered on a plane point of the user's position at

[0046] In a possible embodiment, the device further comprises a storage module configured to store the geofence as a historical geofence.

[0047] In a possible implementation, the processing module further includes a historical geofence search unit; Historical Geofence search The unit extracts the environmental image from the historical geofence stored in the storage module. search and when historical geofences having relevant similarities are obtained, calculating weighted scores of the historical geofences and determining the historical geofence with the highest weighted score as the target geofence; If the pose estimation unit successfully resolves the pose of the target geofence, the geofence generation unit: The posture of the target geofence and the environmental image posture Between The target geofence is set based on the difference, and the coordinate system of the target geofence is aligned with the coordinate system of the environmental image.

[0048] According to a sixth aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to acquire the environmental image, the environmental image being a representation of a location where a user is located; scene an input module, 3 D. Identify a reference plane corresponding to the ground based on the point cloud, 3a processing module configured to determine, based on the 3D object identification information, a projection of the 3D object on the reference plane whose distance to the ground exceeds a first threshold, and to determine a boundary of the reference plane and the obstacle region; Including, The apparatus provides that the processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

[0049] According to a seventh aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to acquire the environmental image, the environmental image being a representation of a location where a user is located; scene an input module, 3 Based on the 3D object identification information, the 3D object whose distance to the ground exceeds a first threshold is detected. three a processing module configured to determine a projection onto a reference plane and to determine a boundary of said reference plane and said obstacle region; Including, The apparatus provides that the processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

[0050] According to an eighth aspect, an embodiment of the present application provides a device for detecting a geofence in real time, the device comprising: The user is located scene Periodically captures real-time environmental images Capture an input module further configured to: a geofence update module configured to update the geofence based on spatial structure information generated based on the real-time environmental image when it is detected that a difference between spatial structure information corresponding to the real-time environmental image and existing spatial structure information exceeds a second threshold; The present invention provides equipment including:

[0051] The fifth, sixth, seventh, and eighth aspects and any of their implementations correspond to the first, second, third, and fourth aspects and any of their implementations, respectively. For technical effects corresponding to the fifth, sixth, seventh, and eighth aspects and any of their implementations, please refer to the technical effects corresponding to the first, second, third, and fourth aspects and any of their implementations. Details will not be described again here.

[0052] According to a ninth aspect, an embodiment of the present application is an electronic device, comprising: one or more processors; a memory configured to store one or more programs; Including, The present invention provides an electronic device, wherein when one or more programs including the one or more processors are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of the first to fourth aspects.

[0053] According to a tenth aspect, an embodiment of the present application provides a computer-readable storage medium comprising a computer program, the computer program being configured to, when executed on a computer, enable the computer to perform a method according to any one of the implementations of the first to fourth aspects.

[0054] According to an eleventh aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computer program code, the computer program code being used to perform a method according to any of the implementations of the first to fourth aspects when the computer program code is run on a computer. [Brief explanation of the drawings]

[0055] In order to more clearly describe the technical solutions in the embodiments of the present application, the following briefly describes the accompanying drawings used in describing the embodiments of the present application. Obviously, the accompanying drawings in the following description only illustrate some embodiments of the present application, and those skilled in the art can still derive other drawings from these accompanying drawings without creative efforts.

[0056] [Figure 1] 1 is a diagram of the structure of an electronic device capable of automatically generating a geofence according to an embodiment of the present application. FIG.

[0057] [Figure 2] 1 is a flowchart of a method for automatically generating a geofence according to an embodiment of the present application.

[0058] [Figure 3] 1 is a flowchart of another method for automatically generating a geofence according to an embodiment of the present application.

[0059] [Figure 4] 1 is a flowchart of a method for detecting a geofence in real time according to an embodiment of the present application;

[0060] [Figure 5A] FIG. 1 is a diagram illustrating a procedure for automatically generating a geofence. [Figure 5B] FIG. 1 is a diagram illustrating a procedure for automatically generating a geofence. [Figure 5C] FIG. 1 is a diagram illustrating a procedure for automatically generating a geofence.

[0061] [Figure 6] 10 is a diagram illustrating the effect of 3D object discrimination in the same scene.

[0062] [Figure 7] 1 is a diagram of spatial information fusion in automatic generation of geofences.

[0063] [Figure 8]FIG. 10 is a scene detection effect diagram.

[0064] [Figure 9] This is an original diagram of the geofence effect.

[0065] [Figure 10] FIG. 10 is a diagram illustrating the detection effect after an object is added to the scene.

[0066] [Figure 11] 10 is an updated geofence effect diagram.

[0067] [Figure 12] FIG. 10 is a diagram of another procedure for automatic generation of a geofence.

[0068] [Figure 13] FIG. 10 illustrates yet another procedure for automatic generation of a geofence.

[0069] [Figure 14] FIG. 10 is a diagram of another procedure for automatic generation of a geofence.

[0070] [Figure 15] FIG. 1 is a block diagram of the structure of a device for automatically generating a geofence according to an embodiment of the present application.

[0071] [Figure 16] FIG. 10 is a block diagram of the structure of another device for automatically generating a geofence according to an embodiment of the present application.

[0072] [Figure 17] FIG. 10 is a block diagram of yet another device architecture for automatically generating a geofence according to an embodiment of the present application.

[0073] [Figure 18] FIG. 10 is a block diagram of another device architecture for automatically generating a geofence according to an embodiment of the present application.

[0074] [Figure 19] FIG. 1 is a block diagram of the structure of a device for detecting a geofence in real time according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0075] The following will clarify the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. theory Obviously, the described embodiments are only a part, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts should fall within the protection scope of the present invention.

[0076] The term "and / or" in this specification refers only to the association relationship that describes the associated objects, and indicates that three relationships may exist. For example, A and / or B may represent three cases: only A exists, both A and B exist, and only B exists.

[0077] In the present application, in the specification and claims, the terms "first," "second," etc. are intended to distinguish between different objects and do not refer to a particular order of the objects. For example, a first target object, a second target object, etc. are intended to distinguish between different target objects and do not refer to a particular order of the target objects.

[0078] In the embodiments of the present application, the words "example" or "for example" are used to indicate an example, illustration, or explanation. Any embodiment or design solution described in the embodiments of the present application as an "example" or "for example" should not be described as being preferred or having more advantages than another embodiment or design solution. Rather, the use of the words "example," "for example," etc. is intended to present the relevant concept in a particular way.

[0079] In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two. For example, a plurality of processing units means two or more processing units, and a plurality of systems means two or more systems.

[0080] In the conventional technology, due to the widespread use of VR devices, some VR devices have geofences with specific shapes pre-set. There are. For example, a geofence can be defined as a circular area with a diameter of 4m, or as a square area with a diagonal length of 4m. When using a geofence with a pre-designed shape, users must first search for a safe site that meets the shape requirements and then power on the device in the center of the site. in teeth, The area within the geofence is a safe area by default, The user region You can move freely 。 When it detects that the user's body has crossed or is about to cross the geofence, it will display prompt information on the display interface and / or sound an alarm. directly The system sends a notification to the user to inform them that a danger may occur and that they should return to the safe area within the geofence. Presetting a geofence with a specific shape to protect the user's safety has high requirements for site selection, such as home life and work. scene So, we have a set shape like this scene It is not easy to find such a geofence. scene However, there is a problem that the geofence is limited and it is inconvenient for users to use it. Some VR devices obtain the geofence by guiding the user to perform an operation. For example, after the VR device activates the VR eye, the VR device allows the user to adjust the height of the virtual plane displayed by the VR device so that it is as close to the ground as possible in the egocentric activity scene. Then, the VR device controllerThe system uses a virtual ray to draw a line in a flat, obstacle-free area around the VR device. After connecting both ends with a line, the VR eye generates a geofence in the corresponding area. This requires user participation, which incurs a learning cost. In the process of generating a geofence, the height of the plane measured by the user and the line drawn by the user directly affect the accuracy of the geofence. This results in poor usability and a poor user experience.

[0081] Based on the above problem, an embodiment of the present application provides an electronic device that can automatically generate a geofence. The electronic device may be a device such as VR glasses, a motion-sensing game device configured with VR, or a wheelchair configured with a geofence. Since the geofence can be automatically generated, the geofence can be easily generated. is easily used It has high accuracy and security, and can greatly improve the user experience.

[0082] 1 is a diagram of an electronic device 100 according to an embodiment of the present application. As shown in FIG. 1, the electronic device 100 includes a processor 110, a display 120, a sensor 130, a memory 140, a handle 150, a speaker 160, a microphone 170, and a controller 180, indicator 1 Including 90.

[0083] The processor 110 may include one or more central processing units, one central processing unit and one graphics processing unit, or an application processor and a coprocessor (e.g., a microcontroller unit or a neural network processor), and may further include buffers and registers. When the processor 110 includes multiple processors, the multiple processors may be integrated on the same chip, or each of the multiple processors may be an independent chip. A processor may include one or more physical cores, where a physical core is the smallest processing module. The processor 110 may be configured as a simultaneous localization and mapping (SLAM) system. The SLAM system may perform location and posture configuration to configure an object shape. In the following embodiments, the processor 110 may be configured to identify spatial structure information, perform fitting optimization, and perform weight-based fusion.

[0084] The display 120 is configured to display images, videos, etc. The display 120 includes a display panel, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or an active matrix organic light-emitting diode (OLED). ( Active-matrix organic light emitting diode (AMOLED), flexible light emitting diode (flex ible light-emitting diode (FLED), mini LED, micro LED, micro OLED, quantum dot light emitting diode , QLED).

[0085] The sensors 130 may sense the current state of the system, such as open / closed state, position, whether the system is in contact with a user, direction, and acceleration / deceleration. Furthermore, the sensors 130 may generate sensed signals used to control the operation of the system. The sensors 130 may include a visual sensor 131 and an inertial sensor (IMU, Inertial Measurement Unit) 132, or may include a depth sensor 133 or a laser sensor 134. The visual sensor 131, such as a camera, camera lens, depth camera, lidar, or millimeter wave radar, typically: scene The camera 132 may be configured to acquire planar image information of the object. The camera may be configured to capture still images or video. An optical image of the object is generated through a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then sent to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. The IMU sensor 132 may include three single-axis accelerometers and three single-axis gyroscopes, and is generally configured to obtain motion information of the device, such as linear acceleration and angular velocity. The depth sensor 133 or laser sensor 134 generally scene The depth sensor 133 may be a Time Of Flight (TOF) sensor and is typically configured to obtain a more accurate depth map.

[0086] The memory 140 can be configured to store computer-executable program code. The executable program code includes instructions. The internal memory 321 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function and image playback function), etc. The data storage area can store data (e.g., descriptors corresponding to historical geofences or images) created in the process of using the electronic device, e.g., the VR device. Furthermore, the memory 140 may include high-speed random access memory, or may further include non-volatile memory, e.g., at least one disk storage device, flash memory, or universal flash storage (UFS). Processor 110 instructions stored in memory 140 and / or the processor 110 Memory in 140 and executes instructions stored in the electronic device 100 to perform various functional applications and data processing functions.

[0087] The electronic device 100 can perform audio functions such as voice control, music playback, and recording using the speaker 160, microphone 170, etc. The speaker 160 is also called a "loudspeaker" and is configured to convert an audio electrical signal into a sound signal. The electronic device 100 can use the speaker 160 to hear sounds. For example, VR glasses can use the speaker to The user An audible warning can be sent to the user indicating that the geofence has been crossed.

[0088] The microphone 170 is also called a "microphone" or a "mic" and is configured to convert an audio signal into an electrical signal. A user can input an audio signal into the microphone 170 by speaking near the microphone 170 through their mouth. CaptureIn addition to providing a high-quality audio signal, the microphone 170 may also perform noise reduction functions. In some embodiments, multiple microphones 170 are arranged in the electronic device to capture audio signals. Capture It can perform functions such as reducing noise, identifying sound sources, and performing directional recording.

[0089] controller 180 can be configured to determine the motion posture of the electronic device 100 or input a user request signal.

[0090] indicator 1 90 is the indicator light and may be configured to indicate charging status and power source changes; or Lights up to indicate when a user crosses the geofence Configured to show message death That's fine.

[0091] The interface is an interface that complies with a standard specification, and may specifically be a mini USB interface, a micro USB interface, a USB Type-C interface, etc. The interface may be configured to connect to a charger to charge the electronic device 100, to transmit data between the electronic device and another device, or to connect to a headset to play audio through the headset. Alternatively, the interface may be configured to connect to another mobile phone, such as an AR device.

[0092] It can be understood that the structures shown in the embodiments of the present invention do not constitute specific limitations on the control device. In some other embodiments of the present application, the control device may include more or fewer components than those shown in the figures, combine some components, separate some components, or have a different component arrangement. The components in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0093] The electronic device 100 determines the user's location using the following method: scene The geofence may be automatically generated based on the above-mentioned method, and a highly secure geofence may be automatically provided to the user without any user operation. centre 2, the method includes the following steps 101, 102, and 103.

[0094] Step 101: Acquire an environment image.

[0095] The environmental image is scene For example, the environmental image is acquired by capturing the image of the user's location using virtual reality (VR) glasses or an intelligent device with a camera. scene For example, a visual sensor such as a camera or a camera lens may be used to capture an image of the environment in which the user is located. The image may be captured automatically upon the user's power-on operation. The image may be captured by taking multiple photos in different directions, or by capturing a short video and extracting frames.

[0096] In some examples, the electronic device may also acquire IMU data. scene It is obtained by performing IMU decomposition on the object at

[0097] The IMU data may represent the object's pose, where the user is located on three independent axes of the carrier's coordinate system measured by the accelerometers. scene The IMU is obtained by performing IMU decomposition based on the acceleration signal of the object in the navigation coordinate system, the angular velocity signal of the carrier relative to the navigation coordinate system measured by the gyroscope, and the measured angular velocity and acceleration of the object in three-dimensional space.

[0098] In some examples, if a user has previously used a device provided herein, geofences generated when the user previously used the device may be stored in historical geofences. Each time the user powers on the device for use, environmental images may be compared to the historical geofences to determine whether adaptive historical geofences are available. For example, a VR device configured with the device may retrieve historical geofences stored in the VR device for comparison, or another intelligent device that does not store historical geofences may retrieve historical geofences from another device that stores historical geofences for comparison via Bluetooth, cloud sharing, or another transmission method. Alternatively, a location may be further determined based on the user's location and a positioning function such as GPS or BeiDou, and the stored historical geofences may be retrieved for comparison as the user moves within that location.

[0099] FIG. 3 illustrates another method for automatically generating a geofence according to an embodiment of the present application. centre 3, the method includes steps 101, 102, 103, 104, 105, and 106.

[0100] Step 104: Retrieve the environmental image from the stored historical geofence. search 4. Determine whether a historical geofence with the relevant similarity has been obtained. If so, perform step 105; if not, perform step 102.

[0101] In some examples, the environmental image is read from the stored historical geofence. Global feature extraction is first performed based on the input environmental image to obtain the descriptor and the current frame, and the global descriptor is obtained based on the stored historical geofence. A shorter descriptor distance indicates more similar images. Therefore, the first N historical geofence images with the minimum descriptor distance are compared and selected as candidate frames. Here, 0 < N < 1000, for example, N is 100. When N candidate frames are obtained, it is determined that the historical geofences with relevant similarities have been obtained.

[0102] Step 105: Calculate the weighted score of the historical geofence and determine the historical geofence with the highest weighted score as the target geofence.

[0103] Perform weighted clustering on the corresponding candidate frames among different historical geofences based on the descriptor distance sorting to obtain the target geofence with the highest weighted score.

[0104] Step 106: Solve the pose of the target geofence. If the solution is successful, align the coordinate system of the target geofence with the coordinate system of the environmental image. The posture of the target geofence and the environmental image Pose Between Set the target geofence based on the difference.

[0105] Solve the pose for the target geofence. If the solution is successful, calculate the coordinate transformation relationship based on the difference between the pose of the target geofence and the pose of the environmental image, and load the target geofence based on the coordinate transformation relationship. The coordinate system of the target geofence is aligned with the coordinate system of the environmental image, that is, the coordinate system of the geofence that the user needs to use.

[0106] In some examples, after obtaining the target geodefense corresponding to the frame with the highest weighted score, consecutive candidate frames can be further calculated to obtain a more accurate coordinate transformation relationship. After the pose solution is successful, if the candidate frame that succeeded in the solution and the frame with the highest weighted score correspond to the same target geodefense, the coordinate transformation relationship can be obtained based on the average value of multiple frames. For example, an environmental image can be retrieved from the stored historical geodefenses, and the first 600 historical geodefense images with the smallest descriptor distance can be obtained as candidate frames. 100 candidate frames correspond to historical geodefense A, 200 candidate frames correspond to historical geodefense B, and 300 candidate frames correspond to historical geodefense C. For different 600 candidate frames, weighted clustering is performed based on the descriptor distance sorting. If the frame with the highest weighted score corresponds to target geodefense A, the pose of target geodefense A is solved. If the solution is successful, the poses of consecutive images can be further solved among the 100 candidate frames. If M consecutive frames are successfully solved, where 1 < M < 20. For example, M is 3, and it can be determined whether all 3 candidate frames correspond to historical geodefense A. If all 3 candidate frames correspond to historical geodefense A, each candidate frame and target geodefense A Between calculates the coordinate transformation relationship, and then, in order to obtain a more accurate coordinate correspondence, a matrix average value can be obtained using 3 consecutive frames. And based on its coordinates correspondence by loading and using target geodefense A for the user, the read historical geodefense A has a high degree of consistency with the environment where the user is located and has high twist restorability.

[0107] The descriptors obtained by extracting global and local features may be obtained using a pre-trained artificial intelligence (AI) model, or may be obtained using conventional feature extraction methods, such as the Oriented FAST and Rotated BRIEF (Oriented FAST and Rotated BRIEF) algorithm or the Scale-Invariant Feature Transform (SIFT) algorithm.

[0108] Step 102: Generate spatial structure information based on the environment image, where the spatial structure information includes information about the 3D point cloud and at least one plane, where the plane is determined by the distribution of points of the 3D point cloud on the plane.

[0109] If the electronic device in the above embodiment further acquires IMU data, in step 102, spatial structure information may be generated based on the environmental image and the IMU data.

[0110] The 3D point cloud may be output by a SLAM system, scene It may be used to describe the shape of a 3D object in a given space. The plane is not obtained by projecting the 3D point cloud onto a location, but by fitting the surface of a contiguous dense point cloud cluster obtained by segmentation clustering based on the actual distribution of the 3D point cloud.

[0111] In some examples, the at least one plane includes at least one horizontal plane and another plane, and the other plane includes a horizontal plane or a vertical plane. That is, the planes obtained by 3D point cloud clustering must form at least one infinite plane corresponding to the ground. Otherwise, to obtain the planes, the environment image is reacquired and the 3D point cloud is reacquired.

[0112] The user is located scene If there is a large outside object like a wall FurthermoreIf present, the plane obtained based on the distribution of 3D point clouds must include at least one horizontal plane and at least one vertical plane representing the ground and walls. Certainly, in practical applications, a large number of planes obtained based on 3D point clouds are required, and both horizontal and vertical planes of objects in space can be obtained based on the 3D point clouds, allowing for more accurate geofence generation.

[0113] In addition, to make the obtained geofence more accurate, the spatial structure information can further include other data such as pose information, depth data, mesh identification data, and 3D object identification information. The pose information is obtained by performing real-time pose estimation using the configured SLAM system and is mainly used for coordinate alignment, so that the generated geofence has a higher degree of consistency with the detected 3D real scene. The depth information can also be obtained by depth estimation. After the depth data is input into the electronic device, a depth map is obtained, and the depth map is aligned with a planar image of the environment image to obtain depth values ​​corresponding to each pixel on the planar image. Environment Image and IMU data When the depth value is added, the accuracy of the output 3D point cloud is effectively improved, and the density of the output 3D point cloud is increased. The mesh identification information is the triangular surface generated after mesh identification, and it indicates where the user is located. scene 3D object identification information is information used to describe the exterior of a 3D object after it has been identified.

[0114] In some examples, the generation of spatial structure information based on an environmental image may be as follows: an environmental image is input to the SLAM system, and feature extraction and matching are performed in real time based on the input image to obtain a matching relationship between planar features of the image. Then, other detection or processing methods such as environment detection, terrain detection, and environmental data are used to obtain a posture difference, and the spatial structure information is adjusted based on the posture difference to obtain the spatial structure information as a location where the user is located. scene It is relatively realistic scenecan be restored.

[0115] In some examples, the generation of spatial structure information based on the environmental image and IMU data may be as follows: the environmental image and IMU data are input into the SLAM system, and feature extraction and matching are performed in real time based on the input image to obtain a matching relationship between the planar features of the image. Next, the camera pose is estimated based on the IMU data and the corresponding positional relationship parameters between the imaging device, for example, the camera and the IMU, to obtain the original pose, i.e., the camera pose information. Next, based on the pose information and the matching relationship between the planar features, an algorithm, for example, a triangulation algorithm, is used to generate a 3D point cloud to obtain the 3D point cloud. Then, based on the 3D point cloud output by the SLAM system, plane detection is performed to obtain information about horizontal and vertical planes in the space. For example, scene Detect that the horizontal plane of the table in is 60cm x 50cm.

[0116] Furthermore, the image is used as input to perform depth estimation based on a pre-trained AI model to obtain depth data. The depth data can also be used as input for other data detection. For example, mesh identification or 3D object identification can be performed using the depth data as input to obtain mesh identification data and 3D object identification information, respectively. Alternatively, mesh identification data and 3D object identification information can be obtained by detection based only on the environmental image and IMU data, but this will not produce as accurate detection results as adding depth data.

[0117] Step 103: Generate a geofence based on the spatial structure information.

[0118] For example, the electronic device connects the surrounding obstacle areas on a reference plane with the user currently using the device as the center to obtain a safe activity area. The boundary of the obstacle area is aligned with the coordinate system of the boundary of the corresponding obstacle 3D object in the environmental image, and the generated geofence is connected to the location where the user is located. sceneThe geofence essentially matches the location of each 3D object in the map, so the geofence can determine where the user is located. scene is more suitable for

[0119] The geofence may be generated in different ways based on different spatial structure information. For example, first, a boundaryless reference plane corresponding to the ground is formed, and then, based on the spatial structure information, processing such as detection and identification is performed on objects in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the projection of the object in the space, i.e., the obstacle area, are obtained. On the boundaryless reference plane, the boundary of the reference plane and the obstacle area are connected to obtain a safe area where the user can move safely. Then, the boundary of the reference plane and the obstacle area where the user is located are obtained. scene The coordinate system of the boundary is aligned with the coordinate system of the corresponding object in the environmental image, with the plane point on the reference plane of the user position at the center, and the boundary of the safe area becomes the geofence.

[0120] For example, if the spatial structure information includes 3D point cloud and plane information, an optimization formula is constructed based on the 3D point cloud and plane information. The plane is optimized based on the optimization formula to obtain an optimized plane. A reference plane is determined based on the optimized plane corresponding to the ground. In the gravity direction, the optimized plane representing the boundary is projected onto the reference plane to determine the boundary of the reference plane, and a reference horizontal plane with a boundary corresponding to the ground is obtained. In the gravity direction, the optimized plane representing an obstacle is projected onto the reference horizontal plane to determine the obstacle area of ​​the reference horizontal plane. A geofence is generated based on the reference horizontal plane and the obstacle area.

[0121] For example, VR glasses can obtain 3D point cloud output through a SLAM system. The information about the planes is that if there is a rectangle smaller than 20cm x 30cm in the space, that is, a rectangle with a short side less than 20cm and a long side less than 30cm, the rectangle is a plane that needs to be ignored. An optimization formula is constructed based on the 3D point cloud and the information about the planes to obtain an optimized plane. Here, the optimized plane does not include the planes of objects smaller than a 20cm x 30cm rectangle. The optimization formula can not only optimize small planes, but also further reduce the error value of the plane, so the optimized plane is a plane with a shape that is smaller than the 20cm x 30cm rectangle. scene Approach the object within.

[0122] In the optimized plane, ground The plane of infinity corresponding to the optimized plane is determined as the reference plane, and the plane of the object representing the edge of the space in the optimized plane is projected onto the gravity-direction reference plane. By connecting the planes, a reference plane with a boundary, i.e., a reference horizontal plane, is obtained. scene Other objects in the map are also optimized to fit into the optimized plane, which can represent the object. In the direction of gravity, the optimized plane is projected onto the horizontal reference plane to obtain the obstacle area. The currently used device is placed at the center, and the surrounding obstacle areas on the horizontal reference plane are connected to obtain the safety activity area, and the boundary of that area can be used as the generated geofence.

[0123] Assuming that key functions are ensured, a more accurate safety action area can be obtained while minimizing the computational power requirements of the solution. In some instances, the spatial structure information may be of two types.

[0124] For example, if the spatial structure information is mesh identification data and 3D object identification information, a reference plane corresponding to the ground may be first identified based on the mesh identification data. For example, the mesh identification data obtained by mesh identification is a triangular surface corresponding to the ground. After connecting the triangular surfaces, an initial ground area may be obtained and determined as the reference plane. Next, based on the 3D object identification information, a projection onto the reference plane of 3D objects whose distance to the ground exceeds a first threshold is determined, and a boundary between the reference plane and an obstacle area is determined. For example, if the first threshold is 10 cm, the mesh triangular surface and 3D objects whose distance to the ground exceeds 10 cm are projected onto the reference plane to determine the obstacle area. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. For example, after separating the obstacle area from the reference plane, the boundary of the reference plane is connected to obtain a safe area, and the boundary of the safe area is used as the geofence.

[0125] For example, if the spatial structure information is a 3D point cloud and 3D object identification information, a reference plane corresponding to the ground may be identified based on the 3D point cloud. Based on the 3D object identification information, a projection of a 3D object whose distance to the ground exceeds a first threshold onto the reference plane is determined, and a boundary between the reference plane and an obstacle area is determined. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. The reference plane corresponding to the ground is identified based on the 3D point cloud. For a method, see the previous example. Next, a 3D object whose distance to the ground in space exceeds a first threshold is projected onto the reference plane to determine an obstacle area. After separating the obstacle area from the reference plane, the boundary of the reference plane is connected to obtain a safe area, and the boundary of the safe area is used as the geofence.

[0126] For example, spatial structure information can be pose information, 3D point clouds, planes, Information about When the data includes depth data, mesh identification data, and 3D object identification information, the most accurate geofence can be obtained by integrating multiple types of information. For example, environmental images and IMU dataPose information and a 3D point cloud are obtained based on the pose information and the 3D point cloud. Planes are obtained by plane detection based on the pose information and the 3D point cloud. Depth data is obtained by depth detection based on the environmental image. Mesh identification data and 3D object identification information are processed based on the depth data. A geofence is generated based on the mesh identification data and 3D object identification information processed based on the 3D point cloud, plane, and depth data.

[0127] The generation of spatial structure information based on environmental images and IMU data may be as follows: the environmental images and IMU data are input into the SLAM system, and feature extraction and matching are performed in real time based on the input environmental images to obtain a matching relationship between the planar features of the images. Next, the camera pose is estimated based on the IMU data and the corresponding positional relationship parameters between the imaging device, e.g., the camera and the IMU, to obtain the original pose, i.e., the camera pose information. Next, a 3D point cloud is generated using an algorithm, e.g., a triangulation algorithm, based on the matching relationship between the pose information and the planar features, thereby obtaining the 3D point cloud.

[0128] Then, based on the 3D point cloud output by the SLAM system, information about planes in space is obtained by plane detection. For example, scene If it is detected that the horizontal plane of the table in the image is 60cm x 50cm and is larger than the minimum discriminant plane of 20cm x 30cm, the horizontal plane of the table is secured and the horizontal plane is projected onto the reference plane to obtain the obstacle area.

[0129] Next, depth estimation is performed using the environmental image as input to obtain depth data. The depth data may also be used as input for other data detection. For example, mesh identification or 3D object identification is performed using the depth data as input, and mesh identification data and 3D object identification information are obtained separately. Since the depth data output by a TOF sensor is much more accurate than the depth estimation results, if the VR device is configured with a TOF sensor, the depth data is preferably TOF data obtained by the TOF sensor. For example, the depth map output from the TOF sensor is used as input for mesh identification and 3D object detection to obtain more accurate mesh identification data and 3D object identification information, respectively.

[0130] According to the above method, the boundary between the reference planes is obtained based on a 3D point cloud. This 3D point cloud can be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the above method, an obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on information about the plane, mesh identification data processed based on the depth data, and 3D object identification information, thereby optimizing the obstacle area to fit the area where the user is located. scene The geofence can be generated by connecting the surrounding obstacle areas on the reference plane to obtain a more accurate safety activity area, and the boundary of that area becomes the generated geofence.

[0131] In some embodiments, the depth data is TOF data. Mesh identification and plane detection may be limited in the size of the planes that are fitted or detected, so for example, some small planes described in the example above may be ignored, and some other small area objects may be ignored by mesh identification because their size is below a specified threshold. However, in real applications, protruding objects such as handrails and bars may be included. scene These objects may appear in the image due to their small size or plane. sceneHowever, these objects are not captured as 3D objects within the geofence and cannot be used to form an obstacle area. When a geofence is generated, these objects are ignored, potentially posing a safety risk to the user. To avoid this, depth data captured by a TOF sensor can be used to identify these types of objects and form an obstacle area, effectively solving the above problem and further improving the safety of the detected geofence.

[0132] For example, in a SLAM system that generates a 3D point cloud, after inputting TOF, a depth map can be aligned with a 2D image to obtain the depth value corresponding to each pixel in the 2D image, which can then be used as the initial depth value for the triangulation module. Furthermore, by constraining the optimization module, the accuracy of the 3D point cloud can be effectively improved, resulting in a higher density in the output 3D point cloud and higher accuracy in the position, size, and contour of the object represented by the 3D point cloud.

[0133] When the depth data is TOF data, the accuracy of the acquired depth data is improved, making it possible to identify objects with small areas but large depths. By interpolating such objects into the exterior surface information of the 3D object, more accurate exterior surface information of the 3D object in the space where the user is located can be obtained. In some examples, the accuracy of the 3D point cloud generated by TOF data processing and the results of mesh fitting and 3D object detection performed after inputting the TOF data is significantly improved.

[0134] Optionally, in this embodiment of the present application, a trained AI model may be used to generate spatial structure information based on image information. For example, depth estimation may be performed using image information as input, and mesh identification may be performed based on the estimated depth point cloud data. Similarly, a pre-trained AI model may be used to perform semantic recognition of 3D objects based on images. Then, global spatial structure fusion may be performed on the output plane information, mesh information, etc., for global optimization, thereby obtaining more accurate exterior surface information of 3D objects in the space where the user is located.

[0135] In some examples, after the geofence is generated, the VR device may determine whether the geofence's extent meets the user's current usage requirements. If the VR device determines that the geofence's extent does not meet the requirements, the VR device may prompt the user and guide the user to move to another area to regenerate the geofence by scanning.

[0136] In some examples, the generated geofence may be stored as a historical geofence. If the generated geofence can meet the current usage requirements and there is no longer any need to guide the user to move to another area, the generated geofence is stored as a historical geofence. Therefore, the next time the user enables the device, the electronic device selects a geofence that satisfies the current usage requirements from the plurality of historical geofences stored in the electronic device. scene You can retrieve historical geofences suitable for and recall historical geofences.

[0137] In some everyday scenarios, when a user using VR glasses is playing a game, the user is completely isolated from the real world and cannot feel the change of the surrounding environment during use. When the game starts, the user will sceneEven if the environment changes and potentially harmful objects temporarily appear, the user cannot sense the change. For example, when a visually impaired person uses a wheelchair, the user cannot sense changes in the surrounding environment. In order to help the user determine in real time whether the area in which the user is located is safe, in order to protect the user's safety during the use process, scene The geofence needs to be updated in real time whenever the distance changes. Figure 4 shows a method for detecting a geofence in real time according to an embodiment of the present application. centre 4, the method includes the following steps 107 and 108.

[0138] Step 107: User Location scene Periodically captures real-time environmental images Capture do.

[0139] For example, a camera can be used to locate the user. scene Periodically captures real-time environmental images Capture Or you can use other imaging equipment to record in real time. scene in real time and periodically acquires real-time environmental images. Capture and extract real-time spatial structure information from the images.

[0140] Step 108: If it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, update the geofence based on the spatial structure information generated based on the real-time environmental image.

[0141] In some examples, real-time spatial structure information may be obtained based on a real-time environmental image, or existing spatial structure information may be obtained based on the environmental image, and a local comparison between the real-time spatial structure information and the existing spatial structure information may be performed in real time. The type of change in the spatial structure information may be fusion of the real-time spatial structure information and the existing spatial structure information based on different fusion weight values. For example, adding a spatial structure, changing the size or orientation of the existing spatial structure, or modifying the original spatial structure. non-existence These all belong to the category of changes in spatial structure information.

[0142] For example, when new spatial structure information is added to the real-time environmental image, the difference may be updated in the geofence as needed to ensure safety during use. The weight of the plane, point cloud, or mesh data corresponding to the newly added spatial structure information may be set to 100%, and fusion may be performed based on that data.

[0143] For example, if the spatial structure information in the real-time environmental image changes, different weight values ​​should be set based on the position, size, and absolute value of the change in the structure to prevent detection errors from affecting the geofence update. The larger the change difference, the larger the weight value. The weight value range can be controlled between 30% and 70%. Two to three frames of data are used for smooth fusion to complete the geofence update.

[0144] For example, if spatial structure information in a real-time environmental image is deleted, multiple frames of data must be checked to confirm that the spatial structure information has indeed been deleted. For example, after confirming multiple times that the spatial structure information for three consecutive frames has been deleted, the weight value is set to 33%, and the spatial structure information is deleted using three frames of data, completing the fusion.

[0145] After the fusion is completed, global optimization is performed based on the weight values ​​corresponding to each spatial structure information, and the geofence is updated based on the spatial structure information used by the device, such as posture information, 3D point cloud, plane information, depth information, mesh identification information, and 3D object identification information.

[0146] In this way, there is no need to update the geofence in real time and generate a new geofence, but instead, the fusion is completed on the existing basis and the added or subtracted part is reflected in the geofence in real time, thereby efficiently improving the security of the geofence.

[0147] The following describes how to use the device provided in the above embodiment using an example. For example, the device may be a VR device, equipped with a camera, and use a SLAM system.

[0148] Step 1: First, Geofence search Execute the following to determine whether there is a historical geofence with a relevant similarity. Extract global features from the captured image and perform global feature extraction based on the global features. search Run the global search If successful, project the local structure information, load the corresponding geofence based on the successfully resolved pose, and align the coordinate system of the captured image to the coordinate system of the loaded geofence.

[0149] In some examples, FIGS. 5A-5C illustrate a procedure for automatically generating a geofence. As shown in S1 of FIG. 5B, the VR device may first extract a global descriptor based on a current image and compare the descriptor of the current image with the descriptors corresponding to all images included in the historical geofences. Here, the global descriptors corresponding to the stored historical geofences are extracted and stored. Since a smaller descriptor distance indicates a more similar image, 100 frames of images with the smallest descriptor distance may be selected as candidate frames. Based on the descriptor distance sorting, weighted clustering is performed on the corresponding candidate frames in different historical geofences, and the historical geofence with the highest weighted score is obtained as the target geofence.

[0150] The weighted score may be calculated as shown in Equation 1-1. The weighted score Score corresponding to historical geofence i is i When calculating the weight a j represents whether the j-rank candidate frame belongs to historical geofence i. If the j-rank candidate frame belongs to historical geofence i, then a j The value of is 1.0. If the j-rank candidate frame does not belong to historical geofence i, then a j The value of is 0.0. Weight b j represents the weighting factor corresponding to the similarity ranking of the candidate frame, where the weighting factor for candidate frames ranked 1 to 20 is 2.0, the weighting factor for candidate frames ranked 20 to 50 is 1.5, and the weighting factor for candidate frames ranked 50 to 100 is 1.0. Finally, the weighted score for each historical geofence i is calculated.

number

[0151] After obtaining the candidate frame corresponding to the candidate historical geofence, the 2D features and 3D point clouds corresponding to the candidate frame are aggregated, and the 2D features corresponding to the current input image are extracted as the current frame. The 3D and 2D matching relationship between the candidate frame and the current frame is determined using one or more methods such as feature matching, reprojection, and outlier removal, and the pose T of the current frame in the target geofence is calculated using an algorithm such as PNP. wh If the pose is successfully resolved, the first search As shown in Equation 1-2, the local coordinates T ws and Target Geofence Between The coordinate transformation relationship T hs may be calculated.

[0152] Using the above method, posture resolution is performed for three consecutive candidate frames based on the candidate frame. The posture resolution for all three frames is successful, and the calculated coordinate transformation relationship and the coordinate transformation relationship Ths If the difference between the current and historical geofences is less than 5 degrees and 5 cm, and all three frames and the frame with the highest weighted score correspond to the same target geofence, search The average value of the transformation relationship matrices of the three frames is solved as a transformation matrix, and coordinate transformation is performed on the local coordinate system where the current frame is located, and the local coordinate system is unified with the coordinate system where the geofence currently required by the user is located. In this way, the coordinates of the candidate frame are aligned with the coordinates of the current frame, and the historical geofence corresponding to the candidate frame is aligned with the coordinates of the current user. scene Load it as the geofence you need.

number

[0153] In some examples, both the global descriptors and local features may be extracted using a pre-trained AI model or directly using traditional feature extraction methods (such as ORB or SIFT), although this is not required here.

[0154] Historical Geofencing search If this fails, we need to automatically generate a geofence, as shown in S2 of Figure 5A. The steps are as follows:

[0155] Step 2: scene The VR device performs the reconstruction by combining the acquired environment image with the IMU. data Based on this, pose estimation and 3D point cloud generation are performed, plane identification is performed, and mesh Identification Data and 3D objects Identification information The spatial structure information such as the above can be detected, and fusion and global optimization can be performed based on this information to obtain global spatial structure information.

[0156] In this embodiment, the environmental image and the IMU data SLAM is an example of generating spatial structure information based on the input environmental image. RThe system performs real-time feature extraction and matching to obtain matching relationships between planar features in the image. Based on the original IMU data and external parameters between the camera and IMU, the camera pose is estimated to obtain the original camera pose. Next, based on the pose information and the matching relationships between the planar features, an algorithm, such as a triangulation algorithm, is used to generate a 3D point cloud. Finally, the camera pose and the 3D point cloud are jointly optimized. Based on the plane information and the output 3D point cloud data, horizontal and vertical planes are fitted in space to limit the size of the detected planes. For example, the smallest identifiable plane is a 20cm x 30cm rectangle. This prevents small planes from affecting the output results.

[0157] The VR device performs depth estimation based on a pre-trained AI model and uses the environmental image as input, and inputs the depth estimation result, i.e., depth data, into a pre-trained mesh identification and 3D object detection model to perform mesh and 3D object detection, thereby obtaining mesh identification data and 3D object identification information, respectively.

[0158] Figure 6 shows the effect of 3D object identification in the same scenario. scene Figure 7 shows the results of 3D point clouds, planes, meshes, and 3D object identification in Figure 7. The information fusion module uses the above information as input to perform global fusion optimization and adjust the spatial orientation of the identified planes. Figure 7 is a diagram of spatial information fusion in automatic geofence generation. As shown in Figure 7, black dots represent 3D points, 1 represents planes, 2 represents meshes, and 3 represents 3D object identification results.

[0159] As shown in Figure 6, let us take the plane corresponding to the top surface of the carton as an example. The equation of the plane is shown in Equation 1-3. 3D point p i and the distance dp between the plane i is shown in Equation 1-4. The corner points m of each triangular face j Distance d from the plane of the mesh result mjThe upper surface t of the rectangular parallelepiped obtained by 3D object identification is also calculated using Equation 1-4. k The distance between and the plane is d tk Therefore, there is an error equation between the plane and the neighboring i 3D points, j triangular faces, and k 3D objects. The overall error e can be minimized by optimizing the plane equations 1-5, resulting in a more precisely optimized plane.

number

[0160] Equations 1-3 are the plane equations, and A, B, C, and D are the plane equation coefficients. The coordinates (x, y, z) of any 3D point on the plane satisfy the plane equations. i The coordinates of (x i ,y i ,z i ) and use Equation 1-4 to find the spatial distance from the 3D point to the plane corresponding to Equation 1-3. In Equation 1-5, α, β, and γ represent error weighting factors for the 3D point, mesh triangular surface, and 3D object plane, respectively. In some embodiments, the values ​​of α, β, and γ may be 0.3, 0.3, and 0.4, respectively. Alternatively, the weights may be adjusted based on the reliability of the 3D point, mesh triangular surface, and 3D object plane. If the sum of the error weighting factors is 1, the weights for more reliable surfaces may be increased accordingly.

[0161] By obtaining optimized spatial structure information using the above formula and generating a geofence based on the optimized spatial structure information, a geofence with higher accuracy can be obtained.

[0162] For example, in order to reduce the requirement for computational power, the above formula can be simplified to obtain optimized information. For example, if the spatial structure information is information about a 3D point cloud and a plane, first formula Use 1-3 to find the plane equation, formulaThe error between the plane and the 3D point cloud is calculated using 1-4, and finally the plane formula 1-6 is optimized to reduce the error e, thereby obtaining a more accurate optimized plane.

number

[0163] Step 3: Generate a geofence. As shown in S3 of Figure 5B, ground detection is performed based on the plane output in Step 2, and obstacles on the ground are detected by referencing other spatial structure information such as the mesh identification data output in Step 2. The outer facades of the obstacles are connected to generate a geofence boundary.

[0164] Figure 8 shows scene As shown in Figure 8, six planes are obtained based on the 3D point cloud detection in space. For example, as shown in Figure 8, scene In this example, six planes are obtained based on 3D point cloud detection in space, labeled "1," "2," "3," "4," "5," and "6." The planes labeled "1," "3," and "6" are horizontal planes, while the planes labeled "2," "4," and "5" are vertical planes. Based on spatial height, horizontal plane "1" can be determined to be the plane corresponding to the ground, i.e., the reference plane. After connecting the vertical planes "2," "4," and "5" based on plane "1," the boundary of the connected area, i.e., the reference plane, shown in the left diagram of Figure 9, can be determined. Based on this, the horizontal planes corresponding to "3" and "6" in space are projected vertically downward to determine the obstacle area. By removing the intersection area between the boundary of the reference plane and the obstacle area, the original geofence result, shown in the right diagram of Figure 9, is obtained. Figure 9 shows the original geofence effect.

[0165] Alternatively, the numbers "3" and "6" in space may be identified and projected based on the mesh identification data and the 3D object identification information. In this method, fusion optimization is performed on the numbers "3" and "6" in step 2. Therefore, this step is an example of identifying a geofence primarily based on a plane.

[0166] It should be noted that after the geofence generated using this method is provided to the VR device, the device must determine whether the range and size of the geofence meets the user's usage requirements. For example, whether the geofence is suitable for activities such as games or user movements. scene If it is determined that the requirements cannot be met, a prompt should be provided on the program interface and the user should be directed to another area to re-create the geofence.

[0167] Step 4: Perform difference detection. As shown in S4 of Figure 5B, the real-time spatial structure can be locally compared with the existing spatial structure. The real-time spatial structure is represented by the current frame, and the existing spatial structure is represented by the historical frame, and the existing spatial structure is from the current geofence. If the detected difference exceeds a specified threshold, a geofence update is triggered. Major detected differences include, but are not limited to, the following examples:

[0168] 1. 3D point cloud difference. The ICP (Iterative Closest Point) algorithm is a commonly used point cloud matching algorithm. This method is used to calculate the overlap rate between the 3D point cloud in the field of view of the current frame and the 3D point cloud in the same area in the historical frame. The overlap rate is below a specified threshold. If the specified threshold is typically 80%, it may be determined that the environment has changed and a geofence update should be triggered.

[0169] 2. Planar difference. If the area of ​​a plane in the field of view of the current frame exceeds a specified threshold, e.g., zoom rate If the difference in attitude exceeds a specified threshold, e.g., 10 cm at 3 degrees, it may be determined that the environment has changed and a geofence update should be triggered. Alternatively, if the difference in attitude exceeds a specified threshold, e.g., 10 cm at 3 degrees, it may be determined that the environment has changed and a geofence update should be triggered.

[0170] 3. Mesh Difference. For the mesh detected in the field of view of the current frame, the specified threshold is as follows: the distance between some triangular faces and the original triangular faces is greater than 10 cm, and the area obtained after connecting the triangular faces is greater than 20 cm x 30 cm. That is, an object appears whose short side is greater than 20 cm and whose long side is greater than 30 cm. If the specified threshold is exceeded, it may be determined that the environment has changed, and a geofence update must be triggered.

[0171] 4. 3D Object Difference. If the change in size or pose of a 3D object detected in the field of view of the current frame exceeds a specified threshold, e.g., if the change in size is greater than 10 cm, the zoom rate If the change in orientation is greater than 10%, or greater than 10cm at 3 degrees, then there is considered to be a change in the map and the geofence update function should be triggered.

[0172] In particular, from a safety perspective, the decision threshold for removing the original spatial structure should be stricter than the threshold for adding or updating the spatial structure. To avoid the impact of false positives, a geofence update should be triggered after the same difference between at least two consecutive frames is detected.

[0173] Figure 10 shows scene 10 shows the detection effect after an object is added to the VR device. After the original geofence is established, the user can use the VR device in the safe area within the geofence, as shown in Figure 10. During the usage process, the user obtains the latest spatial structure information in real time, compares it with the spatial structure information of the safe area corresponding to the existing geofence, and triggers an update of the geofence if the difference exceeds a threshold.

[0174] As shown in Figure 10 sceneis used as an example. After the initial geofence is established, the real-time spatial structure obtained by real-time detection is different from the spatial structure corresponding to the existing geofence. That is, a vertical plane "7" is added, and the vertical plane is located in the safety area corresponding to the existing geofence. In this case, a geofence update needs to be triggered.

[0175] Step 5: Update the geofence. As shown in S5 of Figure 5C, the real-time spatial structure is used as input and fused with the difference domain of the historical spatial structure. After global optimization, the geofence is regenerated. For example, it is detected in step 4 that the spatial structure corresponding to an existing geofence changes. In this case, the geofence update function is triggered.

[0176] As shown in Figure 10 scene is used as an example. After the initial geofence is established, the real-time spatial structure is different from the spatial structure corresponding to the existing geofence, that is, a vertical plane "7" is added, and the vertical plane is located in the safety area corresponding to the existing geofence. In this case, a geofence update needs to be triggered.

[0177] After the spatial structure information fusion is completed, global optimization is performed based on the weight values ​​corresponding to each spatial structure information. For the optimization method, please refer to the previous example. Step 3 is performed again for the changed object, and the geofence update is completed. The updated geofence is shown in Figure 11. Figure 11 is a diagram of the geofence update effect.

[0178] Step 6: Store the geofence. As shown in S6 of FIG. 5C, before a geofence is initially generated, before a geofence is updated, or before the program terminates, the geofence and its corresponding information are stored for subsequent use. For example, when a geofence is read in step 1, the geofence may be used as a historical geofence.

[0179] During use of this method, the geofence and its corresponding information are stored periodically. The information is also stored before the program terminates. While the VR device is up and running, the geofence and its corresponding information can be called up as needed. The information is stored as the device stored it in step 1. search The information is called when the device is run for difference detection in step 4.

[0180] Typically, data is compressed before the spatial structures are saved, limiting the amount of geofences and spatial structures stored. For example, to reduce memory usage, we limit the number of historical geofences stored to a maximum of 10. Store can.

[0181] In some instances, a TOF sensor can be configured for a VR device. Figure 12 shows another step in the automatic generation of a geofence. figure As shown in Figure 12, when generating 3D point clouds, mesh identification data, and 3D object identification information using a VR device, adding TOF data acquired by a TOF sensor significantly improves the accuracy of the results obtained.

[0182] After inputting depth data into a SLAM system that generates a 3D point cloud, the depth map can be aligned with a planar environment image to obtain a depth value corresponding to each pixel in the planar environment image. Using these depth values ​​as the initial depth values ​​for the triangulation algorithm effectively improves the accuracy of the 3D point cloud, increasing the density of the 3D point cloud output by the SLAM system and improving the accuracy of the position, size, and contour of the spatial objects described. Because the depth data output by a TOF sensor is significantly more accurate than typical depth estimation results, the depth map output by the TOF sensor can be used as depth data. Adding depth data to plane detection, mesh identification, and 3D object identification effectively avoids misidentification, further improving the security of the resulting geofence.

[0183] In some examples, the above-described method of calculating a geofence using multiple pieces of spatial structure information places high demands on the algorithm, resulting in a relatively complex algorithm. Since the current VR device has limited computing power, simplification based on the above-described example can minimize the device's computing power requirements while maintaining key functions. Figure 13 is a diagram of yet another procedure for automatically generating a geofence. As shown in Figure 13, in this embodiment, scene Except for the differences in step 2 of reconstruction and step 4 of difference detection from embodiment 1, this embodiment is basically the same as embodiment 1. The differences are as follows.

[0184] Step 2: scene Reconstruction: In step 3, only the 3D point cloud results on which plane detection has been performed are used as the basis for generating the geofence.

[0185] Step 4: Difference detection. The real-time 3D point cloud is used as input and registered to the historical 3D point cloud. The real-time 3D point cloud is obtained based on environmental images acquired in real time, and the historical 3D point cloud is the 3D point cloud used to generate the geofence. The difference between the 3D point cloud corresponding to the current frame and the 3D point cloud in the same area in the historical 3D point cloud is calculated. If the difference exceeds a specified threshold, a geofence update is triggered. The ICP algorithm is used to calculate the overlap rate between the 3D point cloud of the current frame and the historical 3D point cloud. If the overlap rate is less than a specified value, typically 80%, the geofence needs to be updated.

[0186] In some examples, different VR devices have different requirements for geofence detection solutions, so different types of spatial structure information combinations can be used to detect and generate geofences. Figure 14 is a diagram of another procedure for automatically generating a geofence. As shown in Figure 14, how spatial structure information is combined is mainly as follows:

[0187] The spatial structure information includes a 3D point cloud and 3D object identification information. A reference plane corresponding to the ground is identified based on the 3D point cloud, and 3D objects that exist in space and are more than 10 cm away from the ground are projected onto the plane based on the 3D object identification information to determine an obstacle area. After the obstacle area is cut out, a safety area is obtained. The boundary of the safety area is the geofence. In the difference detection, only the 3D point cloud and 3D object identification information acquired in real time in the current frame are compared with the corresponding spatial structure information when the geofence is generated, and whether an update is required is determined based on the difference value. The comparison method has been described in the previous embodiment.

[0188] For example, if the spatial structure information is mesh identification data and 3D object identification information, a reference plane corresponding to the ground may be first identified based on the mesh identification data. After connecting the triangular surfaces, an initial ground area, which is the reference plane, is obtained. Then, based on the 3D object identification information, 3D objects whose distance from the triangular surfaces to the ground is more than 10 cm are projected onto the ground to determine obstacle areas. After the obstacle areas are removed, a safe area is obtained. In other words, a geofence can be generated. Alternatively, in the difference detection, the mesh identification data and 3D object identification information in the current frame may be compared with the corresponding spatial structure information when the geofence is generated, and whether an update is required may be determined based on the difference value. The comparison method has been described in the previous embodiment.

[0189] In the geofence generation and update method, different spatial structure information can be combined and then used to meet the customization requirements of different hardware and different device vendors for the geofence detection method, and further enhance the versatility of the method for devices with different capabilities and different configurations.

[0190] 15 is a block diagram of an apparatus for automatically generating a geofence according to an embodiment of the present application. The apparatus for automatically generating a geofence 200 is shown in FIG. 15 in the above example, and includes an input module 201 and a processing module 202.

[0191] The input module 201 is configured to acquire an environment image, the environment image being a location where a user is located. scene It is obtained by taking a picture of

[0192] For example, the input module 201 may use virtual reality glasses or a camera to detect where the user is located. scene may be photographed to obtain an environmental image.

[0193] The processing module 202 is configured to generate spatial structure information based on the environmental image, and generate a geofence based on the spatial structure information.

[0194] The spatial structure information includes information about the 3D point cloud and at least one plane, where the plane is determined by the distribution of points of the 3D point cloud on the plane, and the processing module 202 processes the 3D point cloud and the plane in the above example. Information about Generate a geofence according to the method for generating a geofence based on a

[0195] In some examples, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0196] Furthermore, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0197] In some examples, the input module 201 is further configured to acquire inertial measurement unit (IMU) data, the IMU data being used to determine where the user is located. scene It is obtained by performing IMU decomposition on the object in

[0198] The processing module 202 is further configured to generate spatial structure information based on the environment image and the IMU data.

[0199] In some examples, the spatial structure information includes only information about the 3D point cloud and the plane.

[0200] The processing module 202 is specifically configured to: construct an optimization equation based on the 3D point cloud and information about the plane; optimize the plane based on the optimization equation to obtain an optimized plane; determine a reference plane based on the optimized plane corresponding to the ground; project the optimized plane representing a boundary onto the reference plane in the gravity direction to determine a boundary of the reference plane; obtain a reference horizontal plane having a boundary corresponding to the ground; project the optimized plane representing an obstacle onto the reference horizontal plane in the gravity direction to determine an obstacle area on the reference horizontal plane; and generate a geofence based on the reference horizontal plane and the obstacle area.

[0201] 16 is a block diagram of the structure of another device for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 16, the processing module 202 further includes a 3D point cloud processing unit 2021, a geofence generation unit 2022, a plane detection unit 2023, a pose estimation unit 2024, a depth estimation unit 2025, a mesh identification unit 2026, and a 3D object identification unit 2027.

[0202] The posture estimation unit 2024 uses the environment image and the IMU data The posture information is acquired based on the

[0203] The 3D Point Cloud Processing Unit 2021 processes environmental images and IMU data and further configured to obtain a 3D point cloud based on

[0204] The plane detection unit 2023 is further configured to obtain a plane by plane detection based on the pose information and the 3D point cloud.

[0205] Depth Estimation Unit 202 5 is configured to perform depth detection based on the environment image to obtain depth data.

[0206] Mesh identification unit 2026 is configured to process the mesh identification data based on the depth data.

[0207] 3D Object Identification Unit 202 7 is configured to process 3D object identification information based on the depth data.

[0208] The geofence generation unit 2022 is further configured to generate a geofence based on the mesh identification data and the 3D object identification information processed based on the 3D point cloud, plane, and depth data.

[0209] In some examples, the device 200 for automatically generating a geofence is configured to identify a reference plane corresponding to the ground using the 3D point cloud processing unit 2021 or the mesh identification unit 2025, then determine, based on the 3D object identification information, projections of 3D objects whose distances to the ground exceed a first threshold on the reference plane as boundaries of the reference plane and obstacle areas using the 3D object identification unit 2026, and finally generate a geofence based on the reference plane, boundaries of the reference plane, and obstacle areas using the geofence generation unit.

[0210] In one example, the depth estimation module is a TOF sensor and the depth data is time-of-flight TOF data.

[0211] In some examples, the geofence generation unit 2022 may generate a geofence based on the location of the user. scene The system is further configured to generate a geofence by aligning the coordinate system of the environmental image with a plane point of the user's position at the center.

[0212] 17 is a block diagram of the structure of yet another device for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 17, the device 200 further includes a storage module 203.

[0213] The storage module 203 is configured to store the geofence as a historical geofence.

[0214] 18 is a block diagram of another device structure for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 18, the processing module 202 further includes a history geofence search unit 2028, which searches for a history geofence. search The unit 2028 is configured to read the environment images from the historical geofences stored in the storage module 203, and if historical geofences with relevant similarities are obtained, calculate weighted scores of the historical geofences, and determine the historical geofence with the highest weighted score as the target geofence.

[0215] If the pose estimation unit 2024 successfully resolves the pose of the target geofence, the geofence generation unit 2022 sets the target geofence based on the difference between the pose of the target geofence and the pose of the environment image such that the coordinate system of the target geofence is aligned with the coordinate system of the environment image.

[0216] 19 is a block diagram of the structure of a device for detecting a geofence in real time according to an embodiment of the present application. As shown in FIG. 19, the device 30 includes an input module 301 and a geofence update module 302. The device 300 may be integrated with any of the devices 200 described above, but the modules included in this example embodiment are not limited thereto.

[0217] The input module 301 further includes a scene Periodically captures real-time environmental images Capture It is configured to:

[0218] The geofence update module 302 is configured to update the geofence based on the spatial structure information generated based on the real-time environmental image when it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold.

[0219] The device provided in the embodiment of the present invention realizes automatic detection and generation of geofences, solves the geofence limitations of VR devices in the field, reduces the learning cost for users to manually plan geofences, and greatly improves the user experience of VR devices.

[0220] In this method, after the geofence is generated, the differential changes of the objects in the space are further compared in real time, and the geofence is automatically updated using methods such as differential detection and fusion optimization. The geofence that is updated in real time has high timeliness and can greatly improve the safety during the usage process.

[0221] The virtual device provided in the embodiments of the present invention may be integrated into the electronic device of the aforementioned embodiments, and the geofence is generated using the method of the aforementioned embodiments.

[0222] In combination with the examples described in the embodiments disclosed herein, those skilled in the art may recognize that the units and algorithms can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of the present application.

[0223] For the purpose of convenience and concise description, the detailed operating processes of the aforementioned systems, devices and units may be clearly understood by those skilled in the art by referring to the corresponding processes in the aforementioned method embodiments, and the details will not be described again here.

[0224] In some embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the described device embodiments are merely examples. For example, the division into units is merely a logical functional division, and other divisions may occur during actual implementation. For example, multiple units or components may be combined or integrated into another system, or some functions may be omitted or not performed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

Claims

1. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scene in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; generating the geofence based on the spatial structure information; storing the geofence as a historical geofence; Retrieving the environmental image from the stored historical geofences, and if historical geofences with relevant similarity are obtained, calculating weighted scores of the historical geofences, and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence, and if the resolution is successful, setting the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image, and aligning a coordinate system of the target geofence with a coordinate system of the environmental image; A method comprising:

2. The method of claim 1 , wherein the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

3. Before the step of generating spatial structure information based on the environmental image, the method further comprises: acquiring inertial measurement unit (IMU) data, the IMU data being acquired by performing an IMU solution on an object in a scene in which the user is located; The step of generating spatial structure information based on the environmental image includes: The method of claim 1 or 2, further comprising generating the spatial structure information based on the environmental image and the IMU data.

4. The method according to any one of claims 1 to 2, wherein the environmental image is acquired by photographing the scene in which the user is located using virtual reality (VR) glasses or an intelligent device with a camera.

5. The step of generating the geofence based on the spatial structure information includes: constructing an optimization formula based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground plane; projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region on the reference horizontal plane; generating the geofence based on the horizontal reference plane and the obstacle region; The method according to any one of claims 1 to 2, comprising:

6. When the spatial structure information further includes posture information, depth data, mesh identification data, and 3D object identification information, the step of generating the spatial structure information based on the environmental image and the IMU data and the step of generating the geofence based on the spatial structure information include: acquiring the pose information and the 3D point cloud based on the environmental image and the IMU data; performing plane detection on the pose information and the 3D point cloud to obtain the plane; performing depth detection based on the environment image to obtain the depth data; processing the mesh identification data and the 3D object identification information based on the depth data; generating the geofence based on mesh identification data and 3D object identification information processed based on the 3D point cloud, the plane, and the depth data; The method of claim 3, comprising:

7. The method of claim 6 , wherein the depth data is time-of-flight (TOF) data.

8. The method according to any one of claims 1 to 2, wherein the step of generating the geofence includes a step of generating the geofence centered on a plane point of a user position in a scene in which the user is located, and aligned with the coordinate system of the environmental image.

9. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scene in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including a 3D point cloud and 3D object identification information; identifying a reference plane corresponding to the ground based on the 3D point cloud; determining a projection of a 3D object whose distance to the ground exceeds a first threshold on the reference plane based on the 3D object identification information, and determining a boundary and an obstacle area of ​​the reference plane; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; storing the geofence as a historical geofence; Retrieving the environmental image from the stored historical geofences, and if historical geofences with relevant similarity are obtained, calculating weighted scores of the historical geofences, and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence, and if the resolution is successful, setting the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image, and aligning a coordinate system of the target geofence with a coordinate system of the environmental image; A method comprising:

10. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scene in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including mesh identification data and 3D object identification information; identifying a reference plane corresponding to the ground plane based on the mesh identification data; determining a projection of a 3D object whose distance to the ground exceeds a first threshold on the reference plane based on the 3D object identification information, and determining a boundary and an obstacle area of ​​the reference plane; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; storing the geofence as a historical geofence; Retrieving the environmental image from the stored historical geofences, and if historical geofences with relevant similarity are obtained, calculating weighted scores of the historical geofences, and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence, and if the resolution is successful, setting the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image, and aligning a coordinate system of the target geofence with a coordinate system of the environmental image; A method comprising:

11. 1. A method for detecting a geofence in real time, comprising: Retrieving environmental images acquired by photographing a scene where a user is located from the stored historical geofences, and if historical geofences with relevant similarity are acquired, calculating weighted scores of the historical geofences, and determining the historical geofence with the highest weighted score as the geofence; resolving the pose of the geofence, and if the resolution is successful, setting the geofence based on a difference between the pose of the geofence and the pose of the environmental image, and aligning a coordinate system of the geofence with a coordinate system of the environmental image; periodically capturing real-time environmental images of the scene in which the user is located; When it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, updating the geofence based on the spatial structure information generated based on the real-time environmental image; A method comprising:

12. A device that automatically generates a geofence, an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scene in which a user is located; a processing module configured to generate spatial structure information based on the environmental image and generate the geofence based on the spatial structure information, the spatial structure information including information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; a storage module configured to store the geofence as a historical geofence; Including, The processing module includes: Read the environmental image from the historical geofences stored in the storage module, and if historical geofences with relevant similarity are obtained, calculate weighted scores of the historical geofences, and determine the historical geofence with the highest weighted score as the target geofence; If the pose of the target geofence is successfully resolved, set the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image, and align a coordinate system of the target geofence with a coordinate system of the environmental image. The device further comprises:

13. The apparatus of claim 12 , wherein the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

14. the input module is further configured to acquire inertial measurement unit (IMU) data, the IMU data being acquired by performing IMU solving on an object in a scene in which the user is located; The apparatus of claim 12 or 13, wherein the processing module is further configured to generate the spatial structure information based on the environmental image and the IMU data.

15. 14. The device of claim 12 or 13, wherein the input module is further configured to acquire the environment image by photographing a scene in which the user is located using virtual reality (VR) glasses or an intelligent device with a camera.

16. The processing module includes: constructing an optimization formula based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground plane; Projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region on the reference horizontal plane; generating the geofence based on the horizontal reference plane and the obstacle region; 14. The device according to claim 12 or 13, configured to:

17. the processing module includes a pose estimation unit, a 3D point cloud processing unit, a plane detection unit, a depth estimation unit, a mesh identification unit, and a 3D object identification unit; the pose estimation unit is configured to obtain pose information based on the environment image and the IMU data; the 3D point cloud processing unit is further configured to acquire the 3D point cloud based on the environment image and the IMU data; the plane detection unit is further configured to perform plane detection on the pose information and the 3D point cloud to obtain the plane; the depth estimation unit is configured to perform depth detection based on the environment image to obtain depth data; the mesh identification unit is configured to process mesh identification data based on the depth data; the 3D object identification unit is configured to process 3D object identification information based on the depth data; the geofence generation unit is further configured to generate the geofence based on mesh identification data and 3D object identification information processed based on the 3D point cloud, the plane, and the depth data.

15. The device of claim 14.

18. The device of claim 17 , wherein the depth data is time-of-flight (TOF) data.

19. The device of claim 17 , wherein the geofence generation unit is further configured to generate the geofence centered on a plane point of a user position in a scene in which the user is located and aligned with a coordinate system of the environmental image.

20. A device that automatically generates a geofence, an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scene in which a user is located; a processing module configured to: identify a reference plane corresponding to the ground based on the 3D point cloud; determine projections of 3D objects whose distances to the ground exceed a first threshold on the reference plane based on 3D object identification information; determine a boundary of the reference plane and an obstacle region; and generate the geofence based on the reference plane, the boundary of the reference plane, and the obstacle region; a storage module configured to store the geofence as a historical geofence; Including, The processing module is further configured to: read the environment image from the historical geofences stored in the storage module; if historical geofences with relevant similarity are obtained, calculate weighted scores for the historical geofences; determine the historical geofence with the highest weighted score as a target geofence; if the pose of the target geofence is successfully resolved, set the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image; and align a coordinate system of the target geofence with a coordinate system of the environmental image.

21. A device that automatically generates a geofence, an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scene in which a user is located; a processing module configured to: determine, based on the 3D object identification information, a projection onto a reference plane of a 3D object whose distance to the ground exceeds a first threshold; determine a boundary of the reference plane and an obstacle region; and generate the geofence based on the reference plane, the boundary of the reference plane, and the obstacle region; a storage module configured to store the geofence as a historical geofence; Including, The processing module is further configured to: read the environment image from the historical geofences stored in the storage module; if historical geofences with relevant similarity are obtained, calculate weighted scores for the historical geofences; determine the historical geofence with the highest weighted score as a target geofence; if the pose of the target geofence is successfully resolved, set the target geofence based on a difference between the pose of the target geofence and the pose of the environmental image; and align a coordinate system of the target geofence with a coordinate system of the environmental image.

22. A device for detecting a geofence in real time, a processing module configured to retrieve environmental images obtained by photographing a scene where a user is located from stored historical geofences; if historical geofences with relevant similarity are obtained, calculate weighted scores for the historical geofences, and determine the historical geofence with the highest weighted score as the geofence; if the pose of the geofence is successfully resolved, set the geofence based on a difference between the pose of the geofence and the pose of the environmental image, and align a coordinate system of the geofence with a coordinate system of the environmental image; an input module configured to periodically capture real-time environmental images of the scene in which the user is located; a geofence update module configured to update the geofence based on spatial structure information generated based on the real-time environmental image when it is detected that a difference between spatial structure information corresponding to the real-time environmental image and existing spatial structure information exceeds a second threshold; Equipment including.

23. 1. An electronic device comprising: one or more processors; a memory configured to store one or more programs; Including, An electronic device, wherein the one or more programs, when executed by the one or more processors, enable the one or more processors to perform the method of any one of claims 1-2.

24. A computer-readable storage medium comprising a computer program, which, when run on a computer, enables the computer to carry out the method of any one of claims 1 to 2.

25. A computer program, which when run on a computer, enables the computer to carry out the method according to any one of claims 1 to 2.

Images