Method for automatically generating geofences, method and device for real-time detection

Abstract

A method for automatically generating a geofence, a real-time detection method and an apparatus are provided, the method including the steps of: acquiring an environmental image (101), the environmental image being acquired by photographing a scene in which a user is located; generating spatial structure information based on the environmental image (102), the spatial structure information including a 3D point cloud and information about at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; and generating the geofence based on the spatial structure information (103).

Description

[Technical field]

[0001] The embodiments of the present application relate to the field of communications, and in particular to an automatic generation method and real-time detection method and device for geofences. [Background technology]

[0002] In recent years, Virtual Reality (VR) technology has been gaining popularity in the industrial and consumer sectors. even more It is attracting attention, and VR equipment for market Essential request has steadily increased During the use of the VR device, the virtual world is completely isolated from the real world, and the user cannot feel the surrounding environment during use. Therefore, to protect the user's safety, a geofence is used as the boundary of the safety area. When the user crosses the geofence, the user is prompted of the risk.

[0003] However, geofences are applied scene The requirements are high, It is not easy for users to use For general use inconvenience It is. Summary of the Invention

[0004] The present application provides a method for automatically generating a geofence, a real-time detection method, and an apparatus for automatically generating a geofence. This makes it easier to use. The automatically generated geofence has high accuracy and high security, and can greatly improve the user experience. The real-time detection method and device can timely update the geofence, thereby further improving the accuracy and security of the geofence.

[0005] According to a first aspect, an embodiment of the present application provides a method for automatically generating a geofence, comprising: acquiring an image of an environment, the image of the environment being a location of a user; scene and generating spatial structure information based on the environment image, the spatial structure information including a 3D point cloud and information about at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; generating the geofence based on the spatial structure information; The present invention provides a method comprising:

[0006] In an embodiment of the present application, when a user needs to use an electronic device, the camera or camera lens of the electronic device is located at the position where the user is. scene to obtain an environmental image, and scene Based on the spatial structure information, a boundary within which the user can safely move is determined, and a geofence can be obtained after connecting the boundary.

[0007] The 3D point cloud in the spatial structure information may alternatively be output by a SLAM system and may be used to describe the shape of the 3D object. The information about the plane may be information about a plane fitted based on an actual 3D point cloud obtained by simulating the object distribution of the 3D point cloud in the spatial structure information, and may represent an actual plane of the object. The information about the plane may be information that can describe the plane, for example the length and width of the plane.

[0008] The electronic device may generate the geofence in different ways based on different spatial structure information. For example, first, a boundary-free reference plane corresponding to the ground is formed, and then, based on the spatial structure information, a process such as detection and identification is performed on the object in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the obstacle area of ​​the object in the space are obtained. On the boundary-free reference plane, the boundary of the reference plane and the obstacle area can be connected to obtain a safe area where the user can move safely, and the boundary of the safe area is the geofence.

[0009] In this embodiment of the present application, a geofence can be automatically generated based on an environmental image. Alternatively, the environmental image can be obtained by starting a shooting mode or an image capture mode after power-on, without requiring user operation. The geofence generated in this embodiment of the present application has high accuracy, high safety, and can be used to locate the user's location. scene It is highly adaptable to the geofence and is not limited by its shape. Geofencing made easy , the user experience is greatly improved.

[0010] A plane in this application may be a horizontal plane, a vertical plane, or an inclined plane, where an inclined plane is a plane that is not parallel to a horizontal or vertical plane.

[0011] In a possible implementation, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0012] In a possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0013] In a possible implementation, before generating spatial structure information based on the environmental image, the method further includes a step of acquiring inertial measurement unit (IMU) data, the IMU data being an indication of where the user is located. scene The spatial structure information is obtained by performing an IMU solution for the object at the environment. Generating spatial structure information based on the environment image includes generating the spatial structure information based on the environment image and the IMU data.

[0014] The IMU data indicates where the user is located. sceneBased on the measured acceleration signal of the object in the navigation coordinate system, the angular velocity signal of the carrier relative to the navigation coordinate system and the measured angular velocity and measured acceleration of the object in three-dimensional space are decomposed onto three independent axes of the carrier coordinate system.

[0015] The electronic device may generate the geofence in different ways based on different spatial structure information. For example, first, a boundary-free reference plane corresponding to the ground is formed, and then, based on the spatial structure information, a process such as detection and identification is performed on the object in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the projection of the object in the space, that is, the obstacle area, are obtained. On the boundary-free reference plane, the boundary of the reference plane and the obstacle area can be connected to obtain a safe area where the user can move safely, and the boundary of the safe area is the geofence.

[0016] In a possible implementation, the environment image is VR Using glasses or an intelligent device with a camera, the user is located scene The user needs a geofence. scene When the user turns on the power, A camera or a camera lens The user is located scene Automatically captures and captures images but Processing or Detection will be done. For example, environment detection, terrain detection, environmental data extraction, image descriptor acquisition but line Broken An environmental image including the above various information may be acquired.

[0017] In a possible implementation, the spatial structure information may be pose information, a 3D point cloud, a plane Information about , depth data, mesh identification data, and 3D object identification information, the step of generating the spatial structure information based on the environmental image and the IMU data, and the step of generating the geofence based on the spatial structure information include: acquiring the pose information and the 3D point cloud based on the environmental image and the IMU data; performing plane detection on the pose information and the 3D point cloud to obtain the plane; performing depth sensing based on the environment image to obtain depth data; The above depth processing the mesh identification data and the 3D object identification information based on the data; generating the geofence based on mesh identification data and 3D object identification information processed based on the 3D point cloud, the plane, and the depth data; Includes.

[0018] The spatial structure information may be the pose information, the 3D point cloud, information about the plane, the depth data, the mesh identification data, and the 3D object identification information. simultaneous Location and Mapping ( Simultaneous The depth information may be obtained by performing real-time pose estimation by a Surface Location and Mapping (SLAM) system, which is mainly used for coordinate alignment so that the generated geofence is more consistent with the detected 3D real scene. The depth information may be obtained by depth estimation. After the depth data is input into the electronic device, a depth map is obtained, and the depth map is aligned to a planar image of the environment image to obtain a depth value corresponding to each pixel on the planar image. The environment image and the IMU data When the depth value is added, the accuracy of the output 3D point cloud is effectively improved and the density of the output 3D point cloud is increased. The mesh identification information is a triangular surface generated after mesh identification, and the mesh identification information is a triangular surface on which the user is located. scene The 3D object identification information is information used to describe the exterior of a 3D object after the 3D object is identified. dataThe generation of spatial structure information based on the above may be as follows: data Then, feature extraction and matching are performed in real time based on the input environmental image to obtain a matching relationship of planar features of the image. Next, the pose of the camera is estimated based on the IMU data and the corresponding positional relationship parameters between an imaging device, for example, a camera and the IMU, to obtain an original pose, i.e., pose information of the camera. Next, a 3D point cloud is generated using an algorithm, for example, a triangulation algorithm, based on the pose information and the matching relationship of the planar features, to obtain a 3D point cloud output by the SLAM system. Next, a plane in space is obtained by plane detection. Next, depth estimation is performed using the environmental image as an input to obtain depth data. The depth data may be used as an input for other data detection. For example, mesh identification or 3D object identification is performed using the depth data as an input to obtain mesh identification data and 3D object identification information separately. According to the above method, the boundary between the reference plane and the reference plane is obtained based on the 3D point cloud. This 3D point cloud may be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the method, an obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on the information about the plane, the mesh identification data processed based on the depth data, and the 3D object identification information, thereby optimizing the obstacle area to fit the obstacle area where the user is located. scene The geofence can be generated by connecting the surrounding obstacle regions on the reference plane to obtain a more accurate safety activity region, whose boundary is the generated geofence.

[0019] In a possible implementation, the step of generating a geofence comprises: scene The coordinate system of the environmental image is set to the plane point of the user position at the center. By aligning , generating the geofence.

[0020] The electronic device connects the surrounding obstacle areas on a reference plane centered on the user currently using the device to obtain a safety activity area. The boundaries of the obstacle areas are aligned to the coordinate system of the boundaries of the corresponding obstacle 3D objects in the environmental image, and the generated geofence is connected to the coordinate system of the boundary of the user located in the environment image. scene Since the geofence essentially corresponds to the location of each 3D object in the scene is more suitable for

[0021] In a possible implementation, after generating the geofence, the method further comprises: Store The method further includes the step of:

[0022] After generating the geofence, the electronic device stores the geofence as a historical geofence, and when used next time, the user can search through the stored historical geofences and select the geofence for the next use. scene Therefore, the geofence can be set to the same historical geofence every time. scene There is no need to generate iteratively for

[0023] In a possible implementation, after the step of acquiring the environmental image, the method further includes the steps of: searching for the environmental image from the stored historical geofences; if historical geofences with relevant similarity are acquired, calculating weighted scores of the historical geofences and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence, and if the resolution is successful, setting the target geofence based on the difference between the pose of the target geofence and the pose of the environmental image, and aligning the coordinate system of the target geofence to the coordinate system of the environmental image.

[0024] In this embodiment of the present application, a weighted clustering is performed on the corresponding candidate frames in different historical geofences based on the descriptor distance sorting, and a target geofence with the highest weighted score is obtained. A pose is resolved for the target geofence, and if the resolution is successful, a coordinate transformation relationship is calculated based on the successfully resolved pose and the difference between the pose of the target geofence and the pose of the environment image, and the target geofence is loaded based on the coordinate transformation relationship. The coordinate system of the target geofence is the coordinate system of the geofence corresponding to the environment image, i.e., the one the user needs to use. Geofence The coordinate system is aligned with that of

[0025] In a possible embodiment, the depth data is time of flight TOF data.

[0026] Since the depth data output by the TOF sensor has much better accuracy than the depth estimation result, the depth data is preferably TOF data obtained by the TOF sensor. For example, the depth map output by the TOF sensor is used as the input for mesh identification and 3D object detection, and more accurate mesh identification data and 3D object identification information are obtained separately. According to the above method, the reference plane and the boundary of the reference plane are obtained based on the 3D point cloud. This 3D point cloud can be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the above method, an obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on information about the plane, the mesh identification data processed based on the depth data, and the 3D object identification information, thereby optimizing the obstacle area to a location where the user is located. scene The geofence can be generated by connecting the surrounding obstacle regions on the reference plane to obtain a more accurate safety activity region, whose boundary is the generated geofence.

[0027] In a possible implementation, the step of generating the geofence based on the spatial structure information includes: constructing an optimization equation based on the 3D point cloud and information about the plane to obtain an optimization plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground surface; determining a boundary of the reference plane by projecting an optimization plane representing the boundary onto the reference plane in the direction of gravity to obtain a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in a gravity direction to determine an obstacle region of the reference horizontal plane; and generating the geofence based on the reference horizontal plane and the obstacle region.

[0028] An optimization formula is constructed based on the 3D point cloud and information about the plane. The plane is optimized based on the optimization formula to obtain an optimized plane. The optimized plane obtained can not only truly represent various objects in the environment where the user is located, but also can improve some aspects of the geofence generation based on the information about the plane. scene Therefore, the collision of small planar objects that do not affect the user's use can also be optimized, so that the optimized plane is more accurate and the generation of the geofence better meets the usage requirements. The step of generating the geofence based on the reference horizontal plane and the obstacle area may be as follows: the currently used device is taken as the center, and the surrounding obstacle area on the reference horizontal plane is connected to obtain a safe activity area, and the boundary of the area is the generated geofence.

[0029] According to a second aspect, an embodiment of the present application is a method for automatically generating an offense, comprising: The step of acquiring an image of the environment, the image of the environment being scene and generating spatial structure information based on the environmental image, the spatial structure information including a 3D point cloud and 3D object identification information; determining a reference plane corresponding to a ground surface based on the 3D point cloud; determining a projection of a 3D object on the reference plane, the projection being greater than a first threshold distance from the ground based on the 3D object identification information, and determining a boundary of the reference plane and an obstacle region; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; A method is provided, comprising: identifying a reference plane corresponding to a ground based on the 3D point cloud; see the above example for a method; and then projecting a 3D object whose distance to the ground exceeds a first threshold onto the reference plane to determine an obstacle region. After cutting off the obstacle region from the reference plane, connecting a boundary of the reference plane to obtain a safe region, and the boundary of the safe region is the geofence.

[0030] According to a third aspect, an embodiment of the present application is a method for automatically generating an offense, comprising: The step of acquiring an image of the environment, the image of the environment being scene and generating spatial structure information based on the environmental image, the spatial structure information including mesh identification data and 3D object identification information; The mesh Identification Data determining a reference plane corresponding to the ground based on determining a projection of a 3D object on the reference plane, the projection being greater than a first threshold distance from the ground based on the 3D object identification information, and determining a boundary of the reference plane and an obstacle region; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; The present invention provides a method comprising:

[0031] First, the mesh Identification Data The reference plane corresponding to the ground may be identified based on the mesh identification. For example, the mesh identification data obtained by mesh identification is a triangular surface corresponding to the ground. After connecting the triangular surfaces, an initial ground area may be obtained and the area may be determined as the reference plane. Next, the mesh triangular surface and a 3D object whose distance to the ground exceeds a first threshold are projected onto the reference plane to determine an obstacle area. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. For example, after separating the obstacle area from the reference plane, the boundary of the reference plane is connected to obtain a safety area, and the boundary of the safety area is set as the geofence.

[0032] In this way, the method of combining the contents of two kinds of spatial structure information to obtain a geofence can minimize the solution's requirements for computing power and obtain an accurate safety activity area, on the premise of ensuring the main function.

[0033] According to a fourth aspect, an embodiment of the present application provides a method for detecting a geofence in real time, comprising: The user is located scene Periodically captures real-time environmental images Capture and updating the geofence based on the spatial structure information generated based on the real-time environmental image when it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold; The present invention provides a method comprising:

[0034] The user is located using the camera scene Periodically captures real-time environmental images Capture Or you can use other imaging equipment to record in real time. scene in real time and periodically acquires real-time environmental images Captureand extracting real-time spatial structure information from the image. The real-time spatial structure information is obtained based on the real-time environmental image, and existing spatial structure information is obtained based on the environmental image, and a local comparison between the real-time spatial structure information and the existing spatial structure information is performed in real time. As a type of change in the spatial structure information, the real-time spatial structure information and the existing spatial structure information may be fused based on different fusion weight values. For example, a spatial structure may be added, the size or orientation of the existing spatial structure may be changed, or the original spatial structure may be fused. non-existence All of these belong to changes in spatial structure information.

[0035] In this embodiment of the present application, when there is newly added spatial structure information in the real-time environmental image, the geofence should be updated with a difference in time to ensure the safety of the use process. If any spatial structure information in the real-time environmental image is changed, different weight values ​​should be set according to the position, size and absolute value of the change in the changed structure, so that the detection error will not affect the update of the geofence. The larger the change difference, the larger the weight value. If some spatial structure information in the real-time environmental image is deleted, data checks for multiple frames need to be performed. After the fusion is completed by confirming the disappearance of spatial structure information multiple times, global optimization is performed according to the weight values ​​corresponding to each spatial structure information, and then the geofence is updated according to the spatial structure information used by the device.

[0036] In this way, there is no need to update the geofence in real time and generate a new geofence, instead, the fusion is completed on an existing basis and the added or subtracted part is reflected in the geofence in real time, so that the security of the geofence can be efficiently improved.

[0037] According to a fifth aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to acquire an image of an environment, the image of the environment being an image of a location where a user is located; scenean input module, a processing module configured to generate spatial structure information based on the environment image, the spatial structure information comprising information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; Including, The apparatus provides an apparatus, wherein the processing module is further configured to generate the geofence based on the spatial structure information.

[0038] In a possible implementation, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0039] In a possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0040] In a possible implementation, the input module is further configured to obtain Infrared Measurement Unit (IMU) data, the IMU data being indicative of a location where the user is located. scene is obtained by performing an IMU solution on the object in The processing module is further configured to generate the spatial structure information based on the environment image and the IMU data.

[0041] In a possible implementation, the input module is scene The device is further configured to acquire the environment image by capturing an image of the environment.

[0042] In a possible implementation, the processing module comprises: constructing an optimization formula based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground surface; projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region of the reference horizontal plane; generating the geofence based on the reference horizontal plane and the obstacle region; The method is further configured as follows.

[0043] In a possible implementation, the processing module includes a pose estimation unit, a 3D point cloud processing unit, a pose estimation unit, a depth estimation unit, a mesh identification unit, and a 3D object identification unit; The pose estimation unit, based on the environment image and the IMU data, figure configured to obtain trend information, The 3D point cloud processing unit processes the 3D point cloud based on the environmental image and the IMU data. 3 further configured to obtain a D point cloud; the plane detection unit is further configured to perform plane detection on the pose information and the 3D point cloud to obtain the plane; The depth estimation unit performs depth detection based on the environment image, Oku configured to obtain outgoing data, The mesh identification unit is configured to: M configured to process the flash identification data; The 3D object identification unit is configured to: 3 D. configured to process object identity information; J The offense generation unit processes the 3D point cloud, the plane, and the depth data. M Item identification data and 3 DBased on object identification information, J The device is further configured to generate an offense.

[0044] In a possible embodiment, the depth data is time of flight TOF data.

[0045] In a possible implementation, the geofence generation unit is configured to generate a geofence based on the location of the user. scene The method is further configured to generate the geofence using an alignment with the coordinate system of the environment image, centered on a plane point of the user's position at

[0046] In a possible embodiment, the device further includes a storage module configured to store the geofence as a historical geofence.

[0047] In a possible implementation, the processing module further includes a historical geofence search unit; Historical Geofence search The unit extracts the environmental image from the historical geofences stored in the storage module. search and when historical geofences having relevant similarity are obtained, calculating weighted scores of the historical geofences and determining the historical geofence having the highest weighted score as the target geofence; If the pose estimation unit is successful in resolving the pose of the target geofence, the geofence generation unit: The posture of the target geofence and the environmental image posture Between The target geofence is set based on the difference, and a coordinate system of the target geofence is aligned with a coordinate system of the environmental image.

[0048] According to a sixth aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to obtain an image of the environment, the image of the environment being indicative of a position where a user is located; scene an input module, 3 D Based on the point cloud, a reference plane corresponding to the ground is identified, 3a processing module configured to determine a projection of a 3D object on the reference plane, the projection being greater than a first threshold distance to the ground, based on the 3D object identification information, and to determine a boundary of the reference plane and the obstacle region; Including, The apparatus provides, in which the processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

[0049] According to a seventh aspect, an embodiment of the present application provides an apparatus for automatically generating a geofence, the apparatus comprising: an input module configured to obtain an image of the environment, the image of the environment being indicative of a position where a user is located; scene an input module, 3 Based on the object identification information, a 3D object having a distance from the ground exceeding a first threshold is detected. three a processing module configured to determine a projection onto a reference plane and to determine a boundary of the reference plane and the obstacle region; Including, The apparatus provides, in which the processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

[0050] According to an eighth aspect, an embodiment of the present application provides an apparatus for detecting a geofence in real time, comprising: The user is located scene Periodically captures real-time environmental images Capture an input module further configured to: a geofence update module configured to update the geofence based on spatial structure information generated based on the real-time environmental image when a difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold; The present invention provides an apparatus including:

[0051] The fifth, sixth, seventh, and eighth aspects and any of their implementations correspond to the first, second, third, and fourth aspects and any of their implementations, respectively. For technical effects corresponding to the fifth, sixth, seventh, and eighth aspects and any of their implementations, please refer to the technical effects corresponding to the first, second, third, and fourth aspects and any of their implementations. Details will not be described again here.

[0052] According to a ninth aspect, an embodiment of the present application is an electronic device, comprising: one or more processors; a memory configured to store one or more programs; Including, The present invention provides an electronic device, comprising: one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement a method according to any one of the implementations of the first to fourth aspects.

[0053] According to a tenth aspect, an embodiment of the present application provides a computer readable storage medium comprising a computer program, the computer program being capable of executing a method according to any one of the implementations of the first to fourth aspects when executed on a computer.

[0054] According to an eleventh aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computer program code, the computer program code being used to perform a method according to any of the implementations of the first to fourth aspects, when the computer program code is executed on a computer. [Brief description of the drawings]

[0055] In order to more clearly describe the technical solutions in the embodiments of the present application, the following briefly describes the accompanying drawings used in describing the embodiments of the present application. Obviously, the accompanying drawings in the following description only show some embodiments of the present application, and those skilled in the art may still derive other drawings from these accompanying drawings without creative efforts.

[0056] [Figure 1] FIG. 1 is a diagram of the structure of an electronic device capable of automatically generating a geofence according to an embodiment of the present application.

[0057] [Diagram 2] 1 is a flowchart of a method for automatically generating a geofence according to an embodiment of the present application.

[0058] [Diagram 3] 1 is a flowchart of another method for automatically generating a geofence according to an embodiment of the present application.

[0059] [Figure 4] 1 is a flowchart of a method for detecting a geofence in real time according to an embodiment of the present application.

[0060] [Figure 5A] FIG. 1 is a diagram of a procedure for automatically generating a geofence. [Figure 5B] FIG. 1 is a diagram of a procedure for automatically generating a geofence. [Figure 5C] FIG. 1 is a diagram of a procedure for automatically generating a geofence.

[0061] [Figure 6] FIG. 11 shows the effect of 3D object discrimination in the same scene.

[0062] [Figure 7] FIG. 1 is a diagram of spatial information fusion in automatic generation of geofences.

[0063] [Figure 8]FIG. 13 is a scene detection effect diagram.

[0064] [Figure 9] This is an original effect diagram of a geofence.

[0065] [Figure 10] FIG. 13 is a diagram showing the detection effect after an object is added to the scene.

[0066] [Figure 11] 13 is an updated effect diagram of a geofence.

[0067] [Figure 12] FIG. 1 illustrates another procedure for automatic generation of a geofence.

[0068] [Figure 13] FIG. 13 illustrates yet another procedure for automatic generation of a geofence.

[0069] [Figure 14] FIG. 13 illustrates another procedure for automatic generation of a geofence.

[0070] [Figure 15] FIG. 2 is a block diagram of the structure of a device for automatically generating a geofence according to an embodiment of the present application.

[0071] [Figure 16] FIG. 2 is a block diagram of the structure of another device for automatically generating a geofence according to an embodiment of the present application.

[0072] [Figure 17] FIG. 13 is a block diagram of another device structure for automatically generating a geofence according to an embodiment of the present application.

[0073] [Figure 18] FIG. 2 is a block diagram of another device structure for automatically generating a geofence according to an embodiment of the present application.

[0074] [Figure 19] FIG. 2 is a block diagram of the structure of a device for detecting a geofence in real time according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0075] The following is to clarify the technical solutions in the embodiments of the present application with reference to the accompanying drawings. Theory Obviously, the described embodiments are only a part, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts should fall within the protection scope of the present invention.

[0076] The term "and / or" in this specification refers to only the association relationship that describes the associated objects, and indicates that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, and only B exists.

[0077] In the present application, in the specification and claims, the terms "first", "second", etc. are intended to distinguish between different objects and do not indicate a particular order of the objects. For example, a first target object, a second target object, etc. are intended to distinguish between different target objects and do not describe a particular order of the target objects.

[0078] In the embodiments of the present application, the words "example" or "for example" are used to indicate an example, illustration, or explanation. Any embodiment or design solution described in the embodiments of the present application as an "example" or "for example" should not be described as being preferred or having more advantages over another embodiment or design solution. Rather, use of the words "example," "for example," etc. is intended to present the relevant concept in a particular way.

[0079] In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two. For example, a plurality of processing units is two or more processing units, and a plurality of systems is two or more systems.

[0080] In the conventional technology, due to the widespread use of VR devices, some VR devices are pre-configured with a geofence of a specific shape. There are. For example, a circular area with a diameter of 4m can be defined as a geofence, or a square area with a diagonal length of 4m can be defined as a geofence. When using a geofence with a pre-designed shape, the user needs to search for a safe site that meets the shape requirements in advance and power on the device in the center of the site in order to use the geofence successfully. VR device in teeth, The area within the geofence is the safe area by default. The user region You can move freely 。 When it detects that the user's body has crossed or is about to cross the geofence, it will display prompt information on the display interface or sound an alarm. directly The system sends a notification to inform the user that danger may occur and the user should return to the safe area within the geofence. Presetting a geofence with a specific shape to protect the user's safety has high requirements for site selection. scene So, let's create a shape like this that satisfies the set shape. scene It is not easy to find such a geofence. scene However, there is a problem that the geofence is limited, and it is inconvenient for users to use the geofence. Some VR devices obtain the geofence by guiding the user to perform an operation. For example, after the VR device enables the VR eye, the VR device allows the user to adjust the height of the virtual plane displayed by the VR device so that it is as close to the ground as possible in the egocentric activity scene. Then, the VR device controllerA virtual ray is passed through the VR eye to draw a line in a flat, obstacle-free area around the VR device. After connecting both ends with a line segment, the VR eye generates a geofence in the corresponding area. In this case, the user needs to participate, which incurs a learning cost. In the process of generating a geofence, the height of the plane measured by the user and the line drawn by the user directly affect the accuracy of the geofence. This makes it difficult to use and reduces the user experience.

[0081] Based on the above problem, the embodiment of the present application provides an electronic device that can automatically generate a geofence. The electronic device may be a device such as VR glasses, a motion-sensing game device configured with VR, or a wheelchair configured with a geofence. Since the geofence can be automatically generated, the geofence can be easily generated. is easily used It has high accuracy and security, and can greatly improve the user experience.

[0082] 1 is a diagram of an electronic device 100 according to an embodiment of the present application. As shown in FIG. 1, the electronic device 100 includes a processor 110, a display 120, a sensor 130, a memory 140, a handle 150, a speaker 160, a microphone 170, and a memory 140. controller 180, indicator 1 Includes 90.

[0083] The processor 110 may include one or more central processing units, or may include one central processing unit and one graphics processing unit, or may include an application processor and a coprocessor (e.g., a microcontroller unit or a neural network processor), and may further include buffers and registers. When the processor 110 includes multiple processors, the multiple processors may be integrated into the same chip, or each of the multiple processors may be an independent chip. A processor may include one or more physical cores, and a physical core is a minimum processing module. The processor 110 may be configured with a simultaneous localization and mapping (SLAM) system. The SLAM system may perform location and posture configuration to configure an object shape. The processor 110 may be configured to identify spatial structure information, perform fitting optimization, and perform fusion based on weights in the following embodiments.

[0084] The display 120 is configured to display images, videos, etc. The display 120 includes a display panel, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (OLED), or a liquid crystal display (LCD). ( active-matrix organic light emitting diode, AMOLED, flexible light emitting diode (flex ible light-emitting diode, FLED, mini LED, micro LED, micro OLED, quantum dot light emitting diode , QLED, etc.

[0085] The sensors 130 may sense the current state of the system, such as open / closed state, position, whether the system is in contact with a user, direction, and acceleration / deceleration. Furthermore, the sensors 130 may generate sensing signals that are used to control the operation of the system. The sensors 130 may include a visual sensor 131 and an inertial sensor (IMU, Inertial Measurement Unit) 132, and may also include a depth sensor 133 or a laser sensor 134. The visual sensor 131, such as a camera, a camera lens, a depth camera, a lidar, or a millimeter wave radar, typically: scene The camera may be configured to capture still images or video. An optical image of the object is generated through a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then sent to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. The IMU sensor 132 may include three single-axis accelerometers and three single-axis gyroscopes, and is generally configured to obtain motion information of the device, such as linear acceleration and angular velocity. The depth sensor 133 or laser sensor 134 generally includes a depth sensor 134 for detecting the motion of the device, a depth sensor 135 for detecting the motion of the device, and a depth sensor 136 for detecting the motion of the device. scene The depth sensor 133 may be a Time Of Flight (TOF) sensor and is typically configured to obtain a more accurate depth map.

[0086] The memory 140 may be configured to store computer executable program code. The executable program code includes instructions. The internal memory 321 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function and image playback function), etc. The data storage area may store data (e.g., descriptors corresponding to historical geofences or images) created in the process of using the electronic device, e.g., the VR device. Furthermore, the memory 140 may include a high-speed random access memory, or may further include a non-volatile memory, e.g., at least one disk storage device, flash memory, or Universal Flash Storage (UFS). Processor 110 may be implemented using instructions stored in memory 140 and / or a processor. 110 Memory in 140 and executes instructions stored in the memory to perform various functions of the electronic device 100, applications, and data processing.

[0087] The electronic device 100 can perform audio functions such as voice control, music playback, and recording using the speaker 160, microphone 170, etc. The speaker 160 is also called a "loudspeaker" and is configured to convert an audio electrical signal into a sound signal. The electronic device 100 can use the speaker 160 to hear sounds. For example, VR glasses can use the speaker to The user An audible warning can be sent to the user indicating that the geofence has been crossed.

[0088] The microphone 170 is also called a "microphone" or "mic" and is configured to convert an audio signal into an electrical signal. A user can speak near the microphone 170 through his / her mouth and input the audio signal to the microphone 170. CaptureIn addition to providing a high-quality audio signal, the microphone 170 may also perform a noise reduction function. In some embodiments, multiple microphones 170 may be arranged in the electronic device to capture audio signals. Capture The present invention can perform functions such as detecting the presence of a microphone, reducing noise, identifying sound sources, and performing directional recording.

[0089] controller 180 may be configured to determine the motion posture of the electronic device 100 or to input a user request signal.

[0090] indicator 1 90 is the indicator light and may be configured to indicate charging status and power source changes; or Lights up to indicate that the user has crossed the geofence Configure to show message death This is also fine.

[0091] The interface is an interface that complies with a standard specification, and may specifically be a mini USB interface, a micro USB interface, a USB Type-C interface, etc. The interface may be configured to connect to a charger to charge the electronic device 100, to transmit data between the electronic device and another device, or to connect to a headset to play audio through the headset. The interface may alternatively be configured to connect to another mobile phone, such as an AR device.

[0092] It may be understood that the structures shown in the embodiments of the present invention do not constitute specific limitations on the control device. In some other embodiments of the present application, the control device may include more or fewer components than those shown in the figures, combine some components, separate some components, or have a different component arrangement. The components in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0093] The electronic device 100 determines the user's location using the following method: scene A geofence may be automatically generated based on the above-mentioned, and a highly secure geofence may be automatically provided to a user without user operation. centre 2, the method includes the following steps 101, 102, and 103.

[0094] Step 101: Acquire an environment image.

[0095] The environmental image is scene For example, the environment image is captured by capturing the image of the user's location using virtual reality (VR) glasses or an intelligent device with a camera. scene The environmental image may be acquired by taking a picture of the user's location. For example, a visual sensor such as a camera or a camera lens may be used to take a picture of the scenario in which the user is located to acquire the environmental image. The picture may be taken automatically based on the user's power-on operation. The picture may be taken by taking multiple pictures in different directions, or by using a method such as taking a short video and extracting frames.

[0096] In some examples, the electronic device may further acquire IMU data. The IMU data may include information about the location of the user. scene It is obtained by performing IMU decomposition on the object at

[0097] The IMU data may represent the pose of the object, where the user is located on three independent axes of the carrier's coordinate system measured by the accelerometers. scene The IMU decomposition is obtained based on the acceleration signal of the object at the navigation coordinate system, the angular velocity signal of the carrier relative to the navigation coordinate system measured by the gyroscope, and the measured angular velocity and measured acceleration of the object in three-dimensional space.

[0098] In some examples, if a user has previously used the device provided herein, the geofences generated when the user previously used the device may be stored in the historical geofences. Each time the user turns on the device for use, the environmental image may be compared to the historical geofences to determine whether an adaptive historical geofence is available. For example, a VR device configured with the device may call up the historical geofences stored in the VR device for comparison, and another intelligent device that does not store historical geofences may retrieve the historical geofences from another device that stores historical geofences for comparison via Bluetooth, cloud sharing, or another transmission method. Alternatively, the location may be further determined based on the user's location and a positioning function such as GPS or BeiDou, and the stored historical geofences may be called up for comparison as the user moves around the location.

[0099] FIG. 3 illustrates another method for automatically generating a geofence according to an embodiment of the present application. centre 3, the method includes step 101, step 102, step 103, step 104, step 105 and step 106.

[0100] Step 104: Retrieve the environment image from the stored historical geofence. search 4. Determine whether a historical geofence with an associated similarity has been obtained. If so, perform step 105; if not, perform step 102.

[0101] In some examples, the environmental image is read from the stored historical geofence. Global feature extraction is first performed based on the input environmental image to obtain the descriptor and the current frame, and the global descriptor is obtained based on the stored historical geofence. A shorter descriptor distance indicates more similar images. Therefore, the first N historical geofence images with the minimum descriptor distance are compared and selected as candidate frames. Here, 0 < N < 1000, for example, N is 100. When N candidate frames are obtained, it is determined that the historical geofence with the relevant similarity has been obtained.

[0102] Step 105: Calculate the weighted score of the historical geofence and determine the historical geofence with the highest weighted score as the target geofence.

[0103] Perform weighted clustering on the corresponding candidate frames among different historical geofences based on the descriptor distance sorting to obtain the target geofence with the highest weighted score.

[0104] Step 106: Solve the pose of the target geofence. If the solution is successful, align the coordinate system of the target geofence with the coordinate system of the environmental image. The posture of the target geofence and the environment image Pose Between Set the target geofence based on the difference.

[0105] Solve the pose for the target geofence. If the solution is successful, calculate the coordinate transformation relationship based on the difference between the pose of the target geofence and the pose of the environmental image, and load the target geofence based on the coordinate transformation relationship. The coordinate system of the target geofence is aligned with the coordinate system of the environmental image, that is, the coordinate system of the geofence that the user needs to use.

[0106] In some examples, after obtaining the target geodefense corresponding to the frame with the highest weighted score, subsequent candidate frames can be further calculated to obtain a more accurate coordinate transformation relationship. After successful pose resolution, if the candidate frame that succeeded in the resolution and the frame with the highest weighted score correspond to the same target geodefense, the coordinate transformation relationship can be obtained based on the average value of multiple frames. For example, an environmental image can be retrieved from the stored historical geodefenses to obtain the first 600 historical geodefense images with the minimum descriptor distance as candidate frames, where 100 candidate frames correspond to historical geodefense A, 200 candidate frames correspond to historical geodefense B, and 300 candidate frames correspond to historical geodefense C. For different 600 candidate frames, weighted clustering is performed based on the descriptor distance sorting. If the frame with the highest weighted score corresponds to target geodefense A, the pose of target geodefense A is resolved. If the resolution is successful, the poses of consecutive images can be further resolved among the 100 candidate frames. If M consecutive frames succeed in the resolution, where 1 < M < 20. For example, M is 3, and it can be determined whether all three candidate frames correspond to historical geodefense A. If all three candidate frames correspond to historical geodefense A, each candidate frame and target geodefense A Between calculates the coordinate transformation relationship, and then, in order to obtain a more accurate coordinate correspondence, the matrix average value can be obtained using three consecutive frames. And based on its coordinates correspondence the target geodefense A is loaded and used by the user. As a result, the retrieved historical geodefense A has a high degree of consistency with the environment where the user is located and has high twist restorability.

[0107] The descriptors obtained by extracting global and local features may be obtained using a pre-trained Artificial Intelligence (AI) model or using traditional feature extraction methods, such as the Oriented FAST and Rotated BRIEF (Oriented FAST and Rotated BRIEF) or the Scale-Invariant Feature Transform (SIFT) algorithm.

[0108] Step 102: Generate spatial structure information based on the environment image, where the spatial structure information includes information about the 3D point cloud and at least one plane, where the plane is determined by a distribution of points of the 3D point cloud on the plane.

[0109] If the electronic device in the above embodiment further acquires IMU data, in step 102, the electronic device may generate spatial structure information based on the environment image and the IMU data.

[0110] The 3D point cloud may be output by a SLAM system, scene It may be used to describe the shape of a 3D object in a space. The plane is not obtained by projecting the 3D point cloud onto a location, but by fitting the surface of a contiguous dense point cloud cluster obtained by segmentation clustering based on the actual distribution of the 3D point cloud.

[0111] In some examples, the at least one plane includes at least one horizontal plane and another plane, and the other plane includes a horizontal plane or a vertical plane. That is, the plane obtained by the 3D point cloud clustering must form at least one infinite plane corresponding to the ground. Otherwise, the environment image is reacquired and the 3D point cloud is reacquired to obtain the plane.

[0112] The user is located scene If there is a large external object like a wall FurthermoreIf present, the plane obtained based on the distribution of 3D point clouds should include at least one horizontal plane and at least one vertical plane representing the ground and walls. Indeed, in real applications, a large number of planes obtained based on 3D point clouds are required, and both horizontal and vertical planes of objects in space can be obtained based on the 3D point clouds, so that more accurate geofences can be generated.

[0113] In addition, in order to make the obtained geofence more accurate, the spatial structure information can further include other data such as pose information, depth data, mesh identification data, 3D object identification information, etc. The pose information is obtained by performing real-time pose estimation by the configured SLAM system and is mainly used for coordinate alignment, so that the generated geofence has a higher degree of consistency with the detected 3D real scene. The depth information may be obtained by depth estimation. After the depth data is input into the electronic device, a depth map is obtained, and the depth map is aligned to the planar image of the environment image to obtain a depth value corresponding to each pixel on the planar image. Environment Image and IMU data When the input is input, adding the depth value effectively improves the accuracy of the output 3D point cloud, and the density of the output 3D point cloud is increased. The mesh identification information is the triangular surface generated after mesh identification, and the user is located scene 3D object identification information is information that is used to describe the exterior of a 3D object after the 3D object has been identified.

[0114] In some examples, the generation of spatial structure information based on an environmental image may be as follows: input an environmental image into the SLAM system, perform feature extraction and matching in real time based on the input image, and obtain a matching relationship of the planar features of the image. Then, obtain the posture difference using other detection or processing methods such as environment detection, terrain detection, and environmental data, and adjust the spatial structure information based on the posture difference to obtain the spatial structure information according to the position where the user is located. scene It is relatively realistic. scenecan be restored.

[0115] In some examples, the generation of spatial structure information based on the environmental image and IMU data may be as follows: input the environmental image and IMU data into the SLAM system, perform feature extraction and matching in real time based on the input image, and obtain the matching relationship of the planar features of the image. Then, estimate the camera pose based on the IMU data and the corresponding positional relationship parameters between the shooting device, for example, the camera and the IMU, and obtain the original pose, i.e., the camera pose information. Then, generate a 3D point cloud using an algorithm, for example, a triangulation algorithm, based on the pose information and the matching relationship of the planar features, and obtain a 3D point cloud. Then, obtain information about the horizontal and vertical planes in the space by plane detection based on the 3D point cloud output by the SLAM system. For example, scene Detect that the horizontal plane of the table in is 60cm x 50cm.

[0116] Furthermore, the image is used as an input to perform depth estimation based on a pre-trained AI model to obtain depth data. The depth data may be used as an input for other data detection. For example, mesh identification or 3D object identification is performed using the depth data as an input to obtain mesh identification data and 3D object identification information, respectively. Alternatively, mesh identification data and 3D object identification information may be obtained only by detection based on the environmental image and IMU data, but the detection results are not as accurate as when depth data is added.

[0117] Step 103: Generate a geofence based on the spatial structure information.

[0118] For example, the electronic device may connect the surrounding obstacle areas on a reference plane centered on the user currently using the device to obtain a safe activity area. The boundaries of the obstacle areas are aligned to the coordinate system of the boundaries of the corresponding obstacle 3D objects in the environmental image, and the generated geofence is generated based on the location of the user. sceneSince the geofence essentially matches the location of each 3D object in the map, the geofence can determine where the user is located. scene is more suitable for

[0119] The geofence may be generated in different ways based on different spatial structure information. For example, first, a boundary-free reference plane corresponding to the ground is formed, and then, based on the spatial structure information, processing such as detection and identification is performed on the objects in the space where the user is located. Based on the processed spatial structure information, the boundary of the reference plane and the projection of the objects in the space, that is, the obstacle area, are obtained. On the boundary-free reference plane, the boundary of the reference plane and the obstacle area can be connected to obtain a safe area where the user can move safely. Then, the boundary of the reference plane and the obstacle area where the user is located can be obtained. scene The boundary coordinate system is aligned with the coordinate system of the corresponding object in the environmental image, centered on the plane point on the reference plane of the user's position, and the boundary of the safe area becomes the geofence.

[0120] For example, when the spatial structure information includes 3D point cloud and plane information, an optimization formula is constructed based on the 3D point cloud and plane information. The plane is optimized based on the optimization formula to obtain an optimized plane. A reference plane is determined based on the optimized plane corresponding to the ground. In the gravity direction, the optimized plane representing the boundary is projected onto the reference plane to determine the boundary of the reference plane, and a reference horizontal plane with a boundary corresponding to the ground is obtained. In the gravity direction, the optimized plane representing the obstacle is projected onto the reference horizontal plane to determine the obstacle area of ​​the reference horizontal plane. A geofence is generated based on the reference horizontal plane and the obstacle area.

[0121] For example, the VR glasses can obtain the 3D point cloud output by the SLAM system. The information about the plane is that if there is a rectangle smaller than 20cm x 30cm in the space, that is, a rectangle with a short side less than 20cm and a long side less than 30cm, the rectangle is a plane that needs to be ignored. An optimization formula is constructed based on the 3D point cloud and the information about the plane to obtain an optimized plane. Here, the optimized plane will not include the plane of the object smaller than a 20cm x 30cm rectangle. The optimization formula can not only optimize the small plane, but also further reduce the error value of the plane, so that the optimized plane has a shape that is smaller than the 20cm x 30cm rectangle. scene Approach the object within.

[0122] In the optimized plane, ground The plane at infinity corresponding to is determined as the reference plane, and the plane of the object that represents the edge of the space in the optimized plane is projected onto the gravity-directed reference plane. By connecting the planes, a reference plane with a boundary, i.e., the reference horizontal plane, is obtained. scene Other objects in the are also optimized to the optimized plane that can represent the objects. In the gravity direction, the optimized plane is projected onto the reference horizontal plane to obtain the obstacle area. The currently used device is taken as the center and the surrounding obstacle areas on the reference horizontal plane are connected to obtain the safety activity area, and the boundary of the area can be the generated geofence.

[0123] On the premise of ensuring the main functions, a more accurate safety action area can be obtained while minimizing the computational power requirements of the solution. In some instances, the spatial structure information may be of the above two types.

[0124] For example, when the spatial structure information is mesh identification data and 3D object identification information, a reference plane corresponding to the ground may be identified based on the mesh identification data. For example, the mesh identification data obtained by mesh identification is a triangular surface corresponding to the ground. After connecting the triangular surfaces, an initial ground area may be obtained and the area may be determined as the reference plane. Next, based on the 3D object identification information, a projection onto the reference plane of a 3D object whose distance to the ground exceeds a first threshold is determined, and a boundary between the reference plane and an obstacle area is determined. For example, when the first threshold is 10 cm, a mesh triangular surface and a 3D object whose distance to the ground exceeds 10 cm are projected onto the reference plane, and an obstacle area is determined. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. For example, after separating the obstacle area from the reference plane, the boundary of the reference plane is connected to obtain a safe area, and the boundary of the safe area is set as a geofence.

[0125] For example, when the spatial structure information is a 3D point cloud and 3D object identification information, a reference plane corresponding to the ground may be identified based on the 3D point cloud. Based on the 3D object identification information, a projection of a 3D object whose distance to the ground exceeds a first threshold onto the reference plane is determined, and a boundary between the reference plane and an obstacle area is determined. A geofence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area. A reference plane corresponding to the ground is identified based on the 3D point cloud. For a method, see the above example. Then, a 3D object whose distance to the ground in space exceeds a first threshold is projected onto the reference plane to determine an obstacle area. After the obstacle area is separated from the reference plane, the boundary of the reference plane is connected to obtain a safe area, and the boundary of the safe area is the geofence.

[0126] For example, spatial structure information can be pose information, 3D point clouds, planes, Information about When the data includes depth data, mesh identification data, and 3D object identification information, the most accurate geofence can be obtained by fusing multiple types of information. For example, the environmental image and IMU datapose information and a 3D point cloud are obtained based on the pose information and the 3D point cloud. A plane is obtained by plane detection based on the pose information and the 3D point cloud. Depth data is obtained by depth detection based on the environment image. Mesh identification data and 3D object identification information are processed based on the depth data. A geofence is generated based on the mesh identification data and 3D object identification information processed based on the 3D point cloud, the plane, and the depth data.

[0127] The generation of spatial structure information based on the environmental image and IMU data may be as follows: input the environmental image and IMU data into the SLAM system, perform feature extraction and matching in real time based on the input environmental image, and obtain the matching relationship of the planar features of the image. Then, estimate the camera pose based on the IMU data and the corresponding positional relationship parameters between the shooting device, for example, the camera and the IMU, to obtain the original pose, that is, the camera pose information. Then, generate a 3D point cloud using an algorithm, for example, a triangulation algorithm, based on the matching relationship between the pose information and the planar features, and obtain a 3D point cloud.

[0128] Then, based on the 3D point cloud output by the SLAM system, information about planes in the space is obtained by plane detection. For example, scene If it is detected that the horizontal plane of the table in is 60cm x 50cm and is larger than the minimum discriminant plane of 20cm x 30cm, the horizontal plane of the table is obtained and the horizontal plane is projected onto the reference plane to obtain the obstacle area.

[0129] Next, depth estimation is performed using the environmental image as an input to obtain depth data. The depth data may be used as an input for other data detection. For example, mesh identification or 3D object identification is performed using the depth data as an input to obtain mesh identification data and 3D object identification information separately. Since the depth data output by the TOF sensor is much more accurate than the depth estimation result, when the VR device is configured with a TOF sensor, the depth data is preferably TOF data obtained by the TOF sensor. For example, the depth map output from the TOF sensor is used as an input for mesh identification and 3D object detection to obtain more accurate mesh identification data and 3D object identification information, respectively.

[0130] According to the method, the boundary between the reference planes is obtained based on the 3D point cloud, which may be denser and more accurate than the 3D point cloud obtained after depth estimation. Next, according to the method, the obstacle area is obtained based on the 3D point cloud representing the obstacle, and the obstacle area is optimized based on the information about the plane, the mesh identification data processed based on the depth data, and the 3D object identification information, thereby optimizing the obstacle area to fit the obstacle area where the user is located. scene The geofence can be generated by connecting the surrounding obstacle regions on the reference plane to obtain a more accurate safety activity region, whose boundary is the generated geofence.

[0131] In some embodiments, the depth data is TOF data. Mesh identification and plane detection may be limited in the size of the planes that are fitted or detected, so for example, some small planes described in the above example may be ignored, and some other small area objects may be ignored by mesh identification because their size is below a specified threshold. However, in real applications, protruding objects such as handrails and bars may be included in the mesh. scene These objects may appear in the image due to their small size and plane. sceneIn the geofence, these objects are not captured as 3D objects and obstacle regions cannot be formed. When a geofence is generated, these objects will be ignored, which may cause safety risks to users. In order to avoid such cases, the depth data captured by the TOF sensor can be used to identify this kind of object and form an obstacle region, which can effectively solve the above problem and further improve the safety of the detected geofence.

[0132] For example, in a SLAM system that generates a 3D point cloud, after inputting the TOF, the depth map can be aligned with the 2D image to obtain the depth value corresponding to each pixel on the 2D image, and this depth value can be used as the initial depth value of the triangulation module. In addition, by constraining the optimization module, the accuracy of the 3D point cloud can be effectively improved, so that the density of the output 3D point cloud is higher, and the accuracy of the position, size, and contour of the object represented by the 3D point cloud is higher.

[0133] When the depth data is TOF data, the accuracy of the acquired depth data is improved, and objects with small areas but large depths can be identified, and such objects can be supplemented with the exterior information of the 3D objects to obtain more accurate exterior information of the 3D objects in the space where the user is located. In some examples, the accuracy of the 3D point cloud generated by TOF data processing and the results of mesh fitting and 3D object detection after inputting the TOF data is significantly improved.

[0134] Optionally, in this embodiment of the present application, a trained AI model may be used to generate spatial structure information based on image information. For example, depth estimation is performed using image information as input, and then mesh identification is performed based on the estimated depth point cloud data. Similarly, a pre-trained AI model may be used to perform semantic recognition of 3D objects based on images. Then, global spatial structure fusion is performed on the output plane information, mesh information, etc., and global optimization is performed to obtain more accurate exterior surface information of 3D objects in the space where the user is located.

[0135] In some examples, after the geofence is generated, the VR device may determine whether the extent of the geofence meets the user's current usage requirements. If the VR device determines that the extent of the geofence does not meet the requirements, the VR device may prompt the user and guide the user to move to another area to regenerate the geofence by scanning.

[0136] In some examples, the generated geofence may be stored as a history geofence. If the generated geofence can meet the current usage requirements and there is no need to guide the user to move to another area, the generated geofence is stored as a history geofence. Thus, when the user enables the device next time, the electronic device selects a history geofence from the multiple history geofences stored in the electronic device to determine the history geofence that is the most suitable for the usage of the electronic device. scene It can read out historical geofences suitable for and recall historical geofences.

[0137] In some everyday scenarios, when a user using VR glasses is playing a game, the user is completely isolated from the real world and the user cannot feel the change of the surrounding environment during the game. When the game starts, the user can see the surrounding environment where the user is located. sceneEven if the environment changes and potentially harmful objects temporarily appear, the user cannot sense the change. For example, in the process of using a wheelchair by a visually impaired person, the user cannot sense the changes in the surrounding environment. In order to help the user determine in real time whether the area in which the user is located is safe, in order to protect the user's safety during the process of use, scene The geofence needs to be updated in real time whenever the distance changes. FIG. 4 illustrates a method for detecting a geofence in real time according to an embodiment of the present application. centre 4, the method includes the following steps: step 107 and step 108.

[0138] Step 107: User is located scene Periodically captures real-time environmental images Capture do.

[0139] For example, a camera may be used to determine the user's location. scene Periodically captures real-time environmental images Capture Or you can use other imaging equipment to record in real time. scene in real time and periodically acquires real-time environmental images Capture and extracting real-time spatial structure information from the images.

[0140] Step 108: If it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, update the geofence based on the spatial structure information generated based on the real-time environmental image.

[0141] In some examples, real-time spatial structure information may be obtained based on a real-time environmental image, existing spatial structure information may be obtained based on the environmental image, and a local comparison between the real-time spatial structure information and the existing spatial structure information may be performed in real time. The type of change in the spatial structure information may be a fusion between the real-time spatial structure information and the existing spatial structure information based on different fusion weight values. For example, a spatial structure may be added, the size or orientation of the existing spatial structure may be changed, or the original spatial structure may be changed. non-existence All of these belong to changes in spatial structure information.

[0142] For example, when new spatial structure information is added to the real-time environmental image, the difference may be updated to the geofence in a timely manner to ensure safety during use. The weight of the plane, point cloud, or mesh data corresponding to the newly added spatial structure information may be set to 100%, and fusion may be performed based on that data.

[0143] For example, when the spatial structure information in the real-time environmental image changes, different weight values ​​should be set according to the position, size and absolute value of the change in the changed structure, so that the detection error does not affect the geofence update. The larger the change difference, the larger the weight value. The weight value range may be controlled to 30%-70%. Two to three frames of data are used to smoothly fuse and complete the geofence update.

[0144] For example, if the spatial structure information in the real-time environmental image is deleted, it is necessary to check the data for multiple frames and confirm multiple times whether the spatial structure information has really disappeared. For example, after confirming multiple times that the spatial structure information for three consecutive frames has disappeared, the weight value is set to 33%, and the spatial structure information is deleted using the data for three frames, completing the fusion.

[0145] After the fusion is completed, global optimization is performed based on the weight values ​​corresponding to each spatial structure information, and the geofence is updated based on the spatial structure information used by the device, such as posture information, 3D point cloud, plane information, depth information, mesh identification information, and 3D object identification information.

[0146] In this way, there is no need to update the geofence in real time and generate a new geofence, instead, the fusion is completed on an existing basis and the added or subtracted part is reflected in the geofence in real time, so that the security of the geofence can be efficiently improved.

[0147] The following describes the use of the device provided in the above embodiment with an example. For example, the device may be a VR device, equipped with a camera, and use a SLAM system.

[0148] Step 1: First, Geofence search Execute a step to determine whether there is a historical geofence with a relevant similarity. Extract global features from the captured image and perform a global feature extraction based on the global features. search Run the global search If successful, project the local structure information, load the corresponding geofence based on the successfully resolved pose, and align the coordinate system of the captured image to the coordinate system of the loaded geofence.

[0149] In some examples, FIG. 5A-FIG. 5C are diagrams of a procedure for automatically generating a geofence. As shown in FIG. 5B S1, the VR device may first extract a global descriptor based on a current image, and compare the descriptor of the current image with the descriptors corresponding to all images included in the historical geofence. Here, the global descriptors corresponding to the stored historical geofences are extracted and stored. Since a smaller descriptor distance indicates a more similar image, 100 frames of images with the smallest descriptor distance may be selected as candidate frames. A weighted clustering is performed on the corresponding candidate frames in different historical geofences based on the descriptor distance sorting, and the historical geofence with the highest weighted score is obtained as the target geofence.

[0150] The weighted score may be calculated as shown in Equation 1-1. The weighted score Score corresponding to the historical geofence i may be calculated as follows: i When calculating the weight a j represents whether the j-rank candidate frame belongs to the historical geofence i. If the j-rank candidate frame belongs to the historical geofence i, then a j The value of is 1.0. If the jth rank candidate frame does not belong to the historical geofence i, then a j The value of is 0.0. The weight b j represents the weighting factor corresponding to the similarity ranking of the candidate frames, where the weighting factor for the candidate frames with ranks 1 to 20 is 2.0, the weighting factor for the candidate frames with ranks 20 to 50 is 1.5, and the weighting factor for the candidate frames with ranks 50 to 100 is 1.0. Finally, the weighted score for each historical geofence i is calculated.

number

[0151] After obtaining the candidate historical geofence and the corresponding candidate frame, the 2D features and 3D point clouds corresponding to the candidate frame are aggregated, and the 2D features corresponding to the current input image are extracted as the current frame. The 3D and 2D matching relationship between the candidate frame and the current frame is obtained by using one or more means such as feature matching, reprojection, and outer point removal, and the pose T of the current frame in the target geofence is calculated by using an algorithm such as PNP. wh If the pose is successfully resolved, the first search As shown in Equation 1-2, the local coordinates T ws and Target Geofence Between Coordinate transformation relationship T hs may be calculated.

[0152] Using the above method, pose resolution is performed for three consecutive candidate frames based on the candidate frame, and the poses of all three frames are successfully resolved. The calculated coordinate transformation relationship and the coordinate transformation relationship to the target geofence Ths If the difference between the current and historical geofences is less than 5 degrees and 5 cm, and all three frames and the frame with the highest weighted score correspond to the same target geofence, search It is determined that the conversion is successful. The average value of the conversion relationship matrices of the three frames is solved as the conversion matrix, and coordinate transformation is performed on the local coordinate system in which the current frame is located, and the local coordinate system is unified with the coordinate system in which the geofence currently required by the user is located. In this way, the coordinates of the candidate frame are aligned with the coordinates of the current frame, and the historical geofence corresponding to the candidate frame is aligned with the coordinates of the current user. scene Load it as the geofence you need.

number

[0153] In some examples, both the global descriptors and the local features may be extracted using a pre-trained AI model or directly using traditional feature extraction methods (such as ORB or SIFT), although this is not required here.

[0154] Historical Geofencing search If this fails, we need to automatically generate a geofence, as shown in S2 of Figure 5A. The steps are as follows:

[0155] Step 2: scene The VR device performs the reconstruction by comparing the acquired environment image with the IMU. data Based on this, pose estimation and 3D point cloud generation are performed, plane identification is performed, and mesh Identification Data or 3D objects Identification information The spatial structure information, such as the spatial structure information, can be detected, and fusion and global optimization can be performed based on this information to obtain global spatial structure information.

[0156] In this embodiment, the environmental image and the IMU data We will explain this using an example of generating spatial structure information based on the following. SLAM generates spatial structure information based on the input environmental image. RThe camera performs feature extraction and matching in real time to obtain the matching relationship between the planar features of the image. Based on the original IMU data and the external parameters between the camera and IMU, the camera pose is estimated and the original pose of the camera is obtained. Then, based on the pose information and the matching relationship between the planar features, an algorithm, for example a triangulation algorithm, is used to generate a 3D point cloud, and the joint application obtains the 3D point cloud. Finally, the camera pose and the 3D point cloud are jointly optimized. Based on the information about the plane and the output 3D point cloud data, the horizontal and vertical planes in the space are fitted to limit the size of the detected plane. For example, the smallest identifiable plane is a rectangle of 20 cm x 30 cm. In this case, the impact of small planes on the output result can be avoided.

[0157] The VR device performs depth estimation based on a pre-trained AI model and using an environmental image as input, and inputs the depth estimation result, i.e., depth data, into a pre-trained mesh identification and 3D object detection model to perform mesh and 3D object detection, and obtain mesh identification data and 3D object identification information, respectively.

[0158] FIG. 6 shows the effect of 3D object identification in the same scenario. scene FIG. 7 shows the results of 3D point cloud, plane, mesh, and 3D object identification in Fig. 7. The information fusion module uses the above information as input to perform global fusion optimization and adjust the spatial pose of the identified plane. FIG. 7 is a diagram of spatial information fusion in automatic generation of geofence. As shown in FIG. 7, black dots represent 3D points, 1 represents plane, 2 represents mesh, and 3 represents 3D object identification result.

[0159] As shown in Figure 6, take the plane corresponding to the top surface of the carton as an example. The equation of the plane is shown in Equation 1-3. 3D point p i and the distance dp between the plane i is shown in Equation 1-4. Each triangular face corner point m j Distance d from the mesh result plane mjThe upper surface t of the rectangular parallelepiped obtained by 3D object identification is also calculated using Equation 1-4. k The distance between and the plane is d tk Therefore, there are error equations between the plane and the i nearby 3D points, j nearby triangular faces, and k nearby 3D objects. The overall error e is minimized by optimizing the plane equations 1-5, resulting in a more precisely optimized plane.

number

[0160] Equation 1-3 is the plane equation, A, B, C, D are the equation coefficients of the plane, and the coordinates (x, y, z) of any 3D point on the plane satisfy the plane equation. i The coordinates of (x i ,y i ,z i ) and use Equation 1-4 to find the spatial distance from the 3D point to the plane corresponding to Equation 1-3. In Equation 1-5, α, β, and γ represent error weighting coefficients for the 3D point, mesh triangular surface, and 3D object plane, respectively. In some embodiments, the values ​​of α, β, and γ may be 0.3, 0.3, and 0.4, respectively. Alternatively, the weights may be adjusted based on the reliability of the 3D point, mesh triangular surface, and 3D object plane. If the sum of the error weighting coefficients is 1, the weights with higher reliability may be increased accordingly.

[0161] By obtaining optimized spatial structure information using the above formula and generating a geofence based on the optimized spatial structure information, a geofence with higher accuracy can be obtained.

[0162] For example, in order to reduce the requirements for computing power, the above formula can be simplified to obtain optimized information. For example, if the spatial structure information is information about a 3D point cloud and a plane, first formula Use 1-3 to find the plane equation, formulaThe error between the plane and the 3D point cloud is calculated using 1-4, and finally the plane equation 1-6 is optimized to reduce the error e, thereby obtaining an optimized plane with higher accuracy.

number

[0163] Step 3: Generate a geofence. As shown in S3 of Figure 5B, perform ground detection based on the plane output in step 2, and refer to other spatial structure information such as mesh identification data output in step 2 to detect obstacles on the ground, and connect the outer facades of the obstacles to generate a geofence boundary.

[0164] Figure 8 shows scene As shown in Figure 8, six planes are obtained based on the 3D point cloud detection in space. For example, as shown in Figure 8, scene In the example, six planes are obtained based on 3D point cloud detection in space, which are labeled as "1", "2", "3", "4", "5", and "6". The planes labeled as "1", "3", and "6" are horizontal planes, and the planes labeled as "2", "4", and "5" are vertical planes. The horizontal plane "1" can be determined to be the plane corresponding to the ground, i.e., the reference plane, based on the spatial height. After connecting the vertical planes "2", "4", and "5" based on the plane "1", the boundary of the connected area, i.e., the reference plane, shown in the left diagram of Figure 9, can be obtained. Based on this, the horizontal planes corresponding to "3" and "6" in space are projected vertically downward to obtain the obstacle area. By removing the intersection area between the boundary of the reference plane and the obstacle area, the original result of the geofence, shown in the right diagram of Figure 9, is obtained. Figure 9 is an original effect diagram of the geofence.

[0165] Also, the numbers "3" and "6" in the space may be identified and projected based on the mesh identification data and the 3D object identification information. In this method, fusion optimization is performed on the numbers "3" and "6" in step 2. Therefore, this step is an example of identifying a geofence mainly based on a plane.

[0166] It should be noted that after the geofence generated using this method is provided to the VR device, the device must determine whether the range and size of the geofence meets the user's usage requirements. For example, whether the geofence is suitable for activities such as games or user movements. scene If it is determined that the requirements cannot be met, a prompt should be provided on the program interface and the user should be directed to another area to regenerate the geofence.

[0167] Step 4: Perform difference detection. As shown in S4 of FIG. 5B, the real-time spatial structure can be locally compared with the existing spatial structure. The real-time spatial structure is represented by the current frame, the existing spatial structure is represented by the historical frame, and the existing spatial structure is from the current geofence. If the detected difference exceeds a specified threshold, an update of the geofence is triggered. The main detected differences include, but are not limited to, the following examples:

[0168] 1. 3D point cloud difference. The ICP (Iterative Closest Point) algorithm is a commonly used point cloud matching algorithm. This method is used to calculate the overlap ratio between the 3D point cloud in the field of view of the current frame and the 3D point cloud in the same area in the history frames. The overlap ratio is below a specified threshold. If the specified threshold is typically 80%, it may be determined that the environment has changed and a geofence update should be triggered.

[0169] 2. Plane Difference. If the area of ​​a plane in the field of view of the current frame exceeds a specified threshold, e.g. zoom rate If the difference in pose exceeds 10%, it may be determined that the environment has changed and a geofence update should be triggered. Alternatively, if the difference in pose exceeds a specified threshold, e.g., 10 cm at 3 degrees, it may be determined that the environment has changed and a geofence update should be triggered.

[0170] 3. Mesh Difference. For the meshes detected in the field of view of the current frame, the specified threshold is: the distance between some triangular faces and the original triangular faces is more than 10 cm, and the area obtained after connecting the triangular faces is more than 20 cm x 30 cm. That is, an object appears whose short side is more than 20 cm and whose long side is more than 30 cm. If the specified threshold is exceeded, it may be determined that the environment has changed and a geofence update needs to be triggered.

[0171] 4. 3D Object Difference. If the change in size or orientation of a 3D object detected in the field of view of the current frame exceeds a specified threshold, e.g., if the change in size is greater than 10 cm, the zoom rate If the change in orientation is greater than 10%, or the change in orientation is greater than 3 degrees or 10 cm, then the map is considered to have changed and the geofence update function should be triggered.

[0172] In particular, from a safety perspective, the decision threshold for removing the original spatial structure should be stricter than the threshold for adding or updating the spatial structure. To avoid the impact of false positives, a geofence update should be triggered after the same difference of at least two frames is detected consecutively.

[0173] Figure 10 shows scene 10 is a diagram showing the detection effect after an object is added to the VR device. As shown in FIG. 10, after the original geofence is established, the user can use the VR device in the safe area within the geofence. In the use process, the user obtains the current latest spatial structure information in real time, compares the information with the spatial structure information of the safe area corresponding to the existing geofence, and triggers the update of the geofence when the difference exceeds a threshold.

[0174] As shown in Figure 10 sceneis used as an example. After the initial geofence is established, the real-time spatial structure obtained by real-time detection is different from the spatial structure corresponding to the existing geofence, that is, a vertical plane "7" is added, and the vertical plane is located in the safe area corresponding to the existing geofence. In this case, it is necessary to trigger an update of the geofence.

[0175] Step 5: Update the geofence. As shown in S5 of FIG. 5C, the real-time spatial structure is used as input and fused with the difference domain of the historical spatial structure. After global optimization, the geofence is regenerated. For example, it is detected in step 4 that the spatial structure corresponding to an existing geofence changes. In this case, the geofence update function is triggered.

[0176] As shown in Figure 10 scene is used as an example. After the initial geofence is established, the real-time spatial structure is different from the spatial structure corresponding to the existing geofence, that is, a vertical plane "7" is added, and the vertical plane is located in the safety area corresponding to the existing geofence. In this case, it is necessary to trigger an update of the geofence.

[0177] After the spatial structure information fusion is completed, global optimization is performed based on the weight value corresponding to each spatial structure information. For the optimization method, please refer to the above example. The processing of step 3 is performed again on the changed object, and the geofence update is completed. The updated geofence is shown in Figure 11. Figure 11 is a geofence update effect diagram.

[0178] Step 6: Store the geofence. As shown in S6 of FIG. 5C, before a geofence is initially generated, before a geofence is updated, or before the program ends, the geofence and its corresponding information are stored for subsequent use. For example, when a geofence is read in step 1, the geofence may be used as a historical geofence.

[0179] During use of this method, the geofence and its corresponding information are stored periodically. The information is also stored before the program is terminated. While the VR device is up and running, the geofence and its corresponding information can be recalled as needed. The information is stored when the device detects the geofence in step 1. search The information is called when the device is run for difference detection in step 4.

[0180] Typically, the data is compressed before the spatial structures are saved, and the amount of geofences and spatial structures stored is limited. For example, to reduce memory usage, we limit the number of historical geofences stored. Store can.

[0181] In some examples, a TOF sensor can be configured for a VR device. Figure 12 shows another step in the automatic generation of a geofence. figure As shown in Fig. 12, when generating 3D point clouds, mesh identification data, and 3D object identification information generated by a VR device, adding TOF data acquired by a TOF sensor significantly improves the accuracy of the obtained results.

[0182] After inputting the depth data into the SLAM system that generates the 3D point cloud, the depth map can be aligned to the planar environment image to obtain the depth value corresponding to each pixel of the planar environment image. By using this depth value as the initial depth value of the triangulation algorithm, the accuracy of the 3D point cloud can be effectively improved, and the density of the 3D point cloud output by the SLAM system will be higher, and the accuracy of the position, size, and contour of the described spatial object will be higher. Since the depth data output by the TOF sensor is much more accurate than the general depth estimation result, the depth map output by the TOF sensor can be used as the depth data. By adding the depth data during plane detection, mesh identification, and 3D object identification, misidentification can be effectively avoided, and the security of the obtained geofence can be further improved.

[0183] In some examples, the above-mentioned method of calculating a geofence using multiple spatial structure information has high requirements for the algorithm, and the algorithm is relatively complicated. Since the computational capabilities of current VR devices are limited, the simplification based on the above-mentioned examples can minimize the computational capabilities of the device while ensuring the main functions. FIG. 13 is a diagram of yet another procedure for automatically generating a geofence. As shown in FIG. 13, in this embodiment, scene Except for the differences between the first embodiment and step 2 of reconstruction and step 4 of difference detection, the second embodiment is basically the same as the first embodiment. The differences are as follows.

[0184] Step 2: scene Reconstruction. In step 3, only the 3D point cloud result on which plane detection has been performed is used as the basis for generating the geofence.

[0185] Step 4: Difference detection. The real-time 3D point cloud is used as input and registered to the historical 3D point cloud. The real-time 3D point cloud is obtained based on the environment images acquired in real time, and the historical 3D point cloud is the 3D point cloud for generating the geofence. The difference between the 3D point cloud corresponding to the current frame and the 3D point cloud in the same area in the historical 3D point cloud is calculated. If the difference exceeds a specified threshold, the update of the geofence is triggered. The overlap ratio between the 3D point cloud of the current frame and the historical 3D point cloud is calculated using the ICP algorithm, and if the overlap ratio is less than a specified value, typically 80%, the geofence needs to be updated.

[0186] In some examples, in practice, because different VR devices have different requirements for geofence detection solutions, a combination of different types of spatial structure information can be used to detect and generate a geofence. Figure 14 is a diagram of another procedure for automatic generation of a geofence. As shown in Figure 14, how the spatial structure information is combined is mainly as follows:

[0187] The spatial structure information is a 3D point cloud and 3D object identification information. In addition, a reference plane corresponding to the ground is identified based on the 3D point cloud, and 3D objects that exist in the space and are more than 10 cm away from the ground are projected onto the plane based on the 3D object identification information to determine an obstacle area. After the obstacle area is cut out, a safety area is obtained. The boundary of the safety area is a geofence. In the difference detection, only the 3D point cloud and 3D object identification information acquired in real time in the current frame are compared with the corresponding spatial structure information when the geofence is generated, and whether an update is required is determined based on the difference value. The comparison method has been described in the above embodiment.

[0188] For example, when the spatial structure information is mesh identification data and 3D object identification information, a reference plane corresponding to the ground may be identified based on the mesh identification data first. After connecting the triangular surfaces, an initial ground area, which is a reference plane, is obtained. Then, based on the 3D object identification information, the triangular surfaces and the 3D object whose distance to the ground is more than 10 cm are projected onto the ground to determine the obstacle area. After cutting out the obstacle area, a safety area is obtained. That is, a geofence can be generated. In the difference detection, alternatively, the mesh identification data and the 3D object identification information in the current frame may be compared with the corresponding spatial structure information when the geofence is generated, and whether an update is required may be determined based on the difference value. The comparison method has been described in the above embodiment.

[0189] In the geofence generation and update method, different spatial structure information can be combined and then used to meet the customization requirements of different hardware and different device vendors for the geofence detection method, and further enhance the versatility of the method in devices with different capabilities and different configurations.

[0190] 15 is a block diagram of an apparatus for automatically generating a geofence according to an embodiment of the present application. The apparatus 200 for automatically generating a geofence is shown in FIG. 15 in the above example, and includes an input module 201 and a processing module 202.

[0191] The input module 201 is configured to acquire an image of an environment, the image of the environment being a location of a user. scene It is obtained by taking a picture of the

[0192] For example, the input module 201 may use virtual reality VR glasses or a camera to detect where the user is located. scene may be captured to obtain an environmental image.

[0193] The processing module 202 is configured to generate spatial structure information based on the environmental image, and generate a geofence based on the spatial structure information.

[0194] The spatial structure information includes information about the 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane, and the processing module 202 may Information about Generate a geofence according to the method for generating a geofence based on a

[0195] In some examples, the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

[0196] Additionally, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0197] In some examples, the input module 201 is further configured to acquire inertial measurement unit (IMU) data, the IMU data being used to indicate where the user is located. scene It is obtained by performing IMU decomposition on the object in

[0198] The processing module 202 is further configured to generate spatial structure information based on the environment image and the IMU data.

[0199] In some examples, the spatial structure information includes only information about the 3D point cloud and the plane.

[0200] The processing module 202 is specifically configured to: construct an optimization equation based on the 3D point cloud and information about the plane, optimize the plane based on the optimization equation to obtain an optimized plane, determine a reference plane based on the optimized plane corresponding to the ground, project the optimized plane representing a boundary onto the reference plane in the gravity direction to determine a boundary of the reference plane, obtain a reference horizontal plane having a boundary corresponding to the ground, project the optimized plane representing an obstacle onto the reference horizontal plane in the gravity direction to determine an obstacle area of ​​the reference horizontal plane, and generate a geofence based on the reference horizontal plane and the obstacle area.

[0201] 16 is a block diagram of the structure of another device for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 16, the processing module 202 further includes a 3D point cloud processing unit 2021, a geofence generation unit 2022, a plane detection unit 2023, a pose estimation unit 2024, a depth estimation unit 2025, a mesh identification unit 2026, and a 3D object identification unit 2027.

[0202] The posture estimation unit 2024 estimates the posture of the environment and the IMU. data The system is configured to acquire posture information based on the

[0203] The 3D Point Cloud Processing Unit 2021 processes environmental images and IMU data The method is further configured to obtain a 3D point cloud based on

[0204] The plane detection unit 2023 is further configured to obtain a plane by plane detection based on the pose information and the 3D point cloud.

[0205] Depth Estimation Unit 202 5 is configured to perform depth detection based on the environment image to obtain depth data.

[0206] Mesh Identification Unit 2026 is configured to process the mesh identification data based on the depth data.

[0207] 3D Object Identification Unit 202 7 is configured to process 3D object identification information based on the depth data.

[0208] The geofence generation unit 2022 is further configured to generate a geofence based on the processed mesh identification data and 3D object identification information based on the 3D point cloud, plane, and depth data.

[0209] In some examples, the device 200 for automatically generating a geofence is configured to identify a reference plane corresponding to the ground using the 3D point cloud processing unit 2021 or the mesh identification unit 2025, then determine, based on the 3D object identification information, a projection on the reference plane of a 3D object whose distance to the ground exceeds a first threshold as a boundary of the reference plane and an obstacle area using the 3D object identification unit 2026, and finally generate a geofence based on the reference plane, the boundary of the reference plane, and the obstacle area using the geofence generation unit.

[0210] In one example, the depth estimation module is a TOF sensor and the depth data is time-of-flight TOF data.

[0211] In some examples, the geofence generation unit 2022 may be configured to scene The system is further configured to generate a geofence by aligning the coordinate system of the environment image with a plane point of the user position at the center.

[0212] 17 is a block diagram of the structure of yet another device for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 17, the device 200 further includes a storage module 203.

[0213] The storage module 203 is configured to store the geofence as a historical geofence.

[0214] 18 is a block diagram of another device structure for automatically generating a geofence according to an embodiment of the present application. As shown in FIG. 18, the processing module 202 further includes a history geofence search unit 2028, which searches for a history geofence. search The unit 2028 is configured to read the environment image from the historical geofences stored in the memory module 203, and if historical geofences with associated similarity are obtained, calculate weighted scores of the historical geofences, and determine the historical geofence with the highest weighted score as the target geofence.

[0215] If the pose estimation unit 2024 successfully resolves the pose of the target geofence, the geofence generation unit 2022 sets the target geofence based on the difference between the pose of the target geofence and the pose of the environment image such that the coordinate system of the target geofence is aligned with the coordinate system of the environment image.

[0216] 19 is a block diagram of the structure of a device for detecting a geofence in real time according to an embodiment of the present application. As shown in FIG. 19, the device 30 includes an input module 301 and a geofence update module 302. The device 300 may be integrated with any of the devices 200 described above, but the modules included in the example of this embodiment are not limited.

[0217] The input module 301 further includes a scene Periodically captures real-time environmental images Capture It is configured to:

[0218] The geofence update module 302 is configured to update the geofence based on the spatial structure information generated based on the real-time environmental image when it is detected that a difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold.

[0219] The device provided in the embodiment of the present invention realizes automatic detection and generation of geofences, solves the geofence limitations of VR devices in the field, reduces the learning cost for users to manually plan geofences, and greatly improves the user experience of the VR device.

[0220] In this method, after the geofence is generated, the differential changes of the objects in the space are further compared in real time, and the automatic update of the geofence is completed by using methods such as differential detection method and fusion optimization method. The geofence updated in real time has high timeliness and can greatly improve the safety in the use process.

[0221] The virtual device provided in the embodiment of the present invention may be integrated into the electronic device of the aforementioned embodiment, and the geofence is generated using the method of the aforementioned embodiment.

[0222] Those skilled in the art may recognize that the units and algorithms can be implemented by electronic hardware or a combination of computer software and electronic hardware in combination with the examples described in the embodiments disclosed herein. Whether a function is implemented by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of this application.

[0223] For the purpose of convenience and concise description, the detailed operation processes of the above-mentioned systems, devices and units may be clearly understood by those skilled in the art by referring to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again here.

[0224] In some embodiments provided in the present application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the described device embodiment is merely an example. For example, the division into units is merely a logical functional division, and there may be other divisions during actual implementation. For example, multiple units or components may be combined or integrated into another system, or some functions may be ignored or not performed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

Claims

1. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; generating spatial structure information based on the environment image, the spatial structure information including information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; generating the geofence based on the spatial structure information; The method includes:

2. The method of claim 1 , wherein the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

3. Prior to the step of generating spatial structure information based on the environmental image, the method further comprises: acquiring inertial measurement unit (IMU) data, the IMU data being acquired by performing IMU resolution on objects in a scenario in which the user is located; The step of generating spatial structure information based on the environmental image includes: The method of claim 1 or 2, comprising generating the spatial structure information based on the environmental image and the IMU data.

4. The method according to any one of claims 1 to 3, wherein the environment image is acquired by photographing the scenario in which the user is located using Virtual Reality (VR) glasses or an intelligent device with a camera.

5. The step of generating the geofence based on the spatial structure information includes: constructing an optimization equation based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground surface; determining a boundary of the reference plane by projecting an optimization plane representing the boundary onto the reference plane in the direction of gravity to obtain a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in a gravity direction to determine an obstacle region of the reference horizontal plane; generating the geofence based on the horizontal reference plane and the obstacle region; The method according to any one of claims 1 to 4, comprising:

6. When the spatial structure information further includes posture information, depth data, mesh identification data, and 3D object identification information, the step of generating the spatial structure information based on the environmental image and the IMU data and the step of generating the geofence based on the spatial structure information include: acquiring the pose information and the 3D point cloud based on the environmental images and the IMU data; performing plane detection on the pose information and the 3D point cloud to obtain the plane; performing depth sensing based on the environment image to obtain the depth data; processing the mesh identification data and the 3D object identification information based on the depth data; generating the geofence based on the mesh identification data and the 3D object identification information processed based on the 3D point cloud, the plane, and the depth data; The method of claim 3 , comprising:

7. The method of claim 6 , wherein the depth data is time-of-flight (TOF) data.

8. The method according to any one of claims 1 to 7, wherein the step of generating the geofence includes a step of generating the geofence centered on a plane point of a user position in a scenario in which the user is located, and aligned with a coordinate system of the environmental image.

9. After the step of generating a geofence, the method further comprises: storing the geofence as a historical geofence; The method of any one of claims 1 to 8, further comprising:

10. After the step of acquiring an image of the environment, the method further comprises: Retrieving the environmental image from the stored historical geofences, and when historical geofences with relevant similarity are obtained, calculating weighted scores of the historical geofences, and determining the historical geofence with the highest weighted score as a target geofence; resolving the pose of the target geofence, and if the resolution is successful, setting the target geofence based on an pose difference and aligning a coordinate system of the target geofence with a coordinate system of the environmental image; 10. The method of claim 9 further comprising:

11. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including a 3D point cloud and 3D object identification information; determining a reference plane corresponding to a ground surface based on the 3D point cloud; determining a projection of a 3D object on the reference plane, the projection being greater than a first threshold distance from the ground based on the 3D object identification information, and determining a boundary and an obstacle region of the reference plane; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; The method includes:

12. 1. A method for automatically generating a geofence, comprising: acquiring an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including mesh identification data and 3D object identification information; identifying a reference plane corresponding to a ground surface based on the mesh identification data; determining a projection of a 3D object on the reference plane, the projection being greater than a first threshold distance from the ground based on the 3D object identification information, and determining a boundary of the reference plane and an obstacle region; generating the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region; The method includes:

13. 1. A method for detecting a geofence in real time, comprising: periodically collecting real-time environmental images of a scenario in which a user is located; When it is detected that the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, updating the geofence based on the spatial structure information generated based on the real-time environmental image; The method includes:

14. A device for automatically generating a geofence, comprising: - an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; a processing module configured to generate spatial structure information based on the image of the environment, the spatial structure information comprising information about a 3D point cloud and at least one plane, the plane being determined by a distribution of points of the 3D point cloud on the plane; Including, The processing module is further configured to generate the geofence based on the spatial structure information.

15. 15. The apparatus of claim 14, wherein the at least one plane includes at least one horizontal plane and another plane, the other plane including a horizontal plane or a vertical plane.

16. The input module is further configured to acquire Inertial Measurement Unit (IMU) data, the IMU data being acquired by performing IMU solving on an object in a scenario in which the user is located; 16. The apparatus of claim 14 or 15, wherein the processing module is further configured to generate the spatial structure information based on the environment image and the IMU data.

17. The device according to any one of claims 14 to 16, wherein the input module is further configured to obtain the environment image by photographing a scenario in which the user is located.

18. The processing module includes: constructing an optimization formula based on the 3D point cloud and information about the plane; optimizing the plane based on the optimization formula to obtain an optimized plane; determining a reference plane based on an optimization plane corresponding to the ground surface; projecting an optimization plane representing a boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, and obtaining a reference horizontal plane having a boundary corresponding to the ground; projecting an optimization plane representing an obstacle onto the reference horizontal plane in the direction of gravity to determine an obstacle region of the reference horizontal plane; generating the geofence based on the reference horizontal plane and the obstacle region; The device according to any one of claims 14 to 17, configured so as to

19. The processing module includes a pose estimation unit, a 3D point cloud processing unit, a plane detection unit, a depth estimation unit, a mesh identification unit, and a 3D object identification unit; the pose estimation unit is configured to obtain the pose information based on the environment image and the IMU data; the 3D point cloud processing unit is further configured to acquire the 3D point cloud based on the environment image and the IMU data; the plane detection unit is further configured to perform plane detection on the pose information and the 3D point cloud to obtain the plane; the depth estimation unit is configured to perform depth detection based on the environment image to obtain the depth data; the mesh identification unit is configured to process the mesh identification data based on the depth data; the 3D object identification unit is configured to process the 3D object identification information based on the depth data; the geofence generation unit is further configured to generate the geofence based on the mesh identification data and the 3D object identification information processed based on the 3D point cloud, the plane, and the depth data.

17. The device of claim 16.

20. The apparatus of claim 19 , wherein the depth data is time-of-flight (TOF) data.

21. The device according to any one of claims 14 to 20, wherein the geofence generation unit is further configured to generate the geofence centered on a plane point of a user position in a scenario in which the user is located, aligned with a coordinate system of the environment image.

22. The device comprises: a storage module further configured to store the geofence as a historical geofence; 22. The apparatus of any one of claims 14 to 21, further comprising:

23. The processing module further includes a historical geofence reading unit; The historical geofence reading unit is configured to read the environmental image from the historical geofence stored in the storage module, and when a historical geofence with relevant similarity is obtained, calculate a weighted score of the historical geofence, and determine the historical geofence with the highest weighted score as a target geofence; If the pose estimation unit successfully resolves the pose of the target geofence, the geofence generation unit sets the target geofence based on a pose difference, and aligns a coordinate system of the target geofence with a coordinate system of the environment image.

23. The device of claim 22.

24. A device for automatically generating a geofence, comprising: - an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; a processing module configured to identify a reference plane corresponding to a ground surface based on the 3D point cloud, determine projections of 3D objects whose distance to the ground surface exceeds a first threshold on the reference plane based on the 3D object identification information, and determine a boundary of the reference plane and an obstacle region; Including, The processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

25. A device for automatically generating a geofence, comprising: - an input module configured to acquire an environmental image, the environmental image being acquired by photographing a scenario in which a user is located; a processing module configured to determine, based on the 3D object identification information, a projection onto the reference plane of 3D objects whose distance to the ground exceeds a first threshold, and to determine a boundary of the reference plane and an obstacle region; Including, The processing module is further configured to generate the geofence based on the reference plane, a boundary of the reference plane, and the obstacle region.

26. A device for detecting a geofence in real time, comprising: an input module further configured to periodically collect real-time environmental images of a scenario in which the user is located; a geofence update module configured to update the geofence based on spatial structure information generated based on the real-time environmental image when a difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold; Including equipment.

27. 1. An electronic device comprising: one or more processors; a memory configured to store one or more programs; Including, An electronic device, wherein said one or more programs, when executed by said one or more processors, enable said one or more processors to perform the method according to any one of claims 1 to 13.

28. A computer readable storage medium comprising a computer program, the computer program being enabled, when executed on a computer, to carry out the method according to any one of claims 1 to 13.

29. A computer program product comprising computer program code, the computer program code being enabled, when executed on a computer, to perform the method according to any one of claims 1 to 13.

Images