Electronic fence automatic generation method, real-time detection method and device

Abstract

Embodiments of the present application provide an electronic fence automatic generation method, a real-time detection method and device, comprising: acquiring an environment image, the environment image being obtained by shooting a scene where a user is located; generating space structure information according to the environment image, the space structure information comprising 3D point cloud and information of at least one plane, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane; and generating an electronic fence according to the space structure information. The present application can automatically generate an electronic fence, which can reduce the use threshold of the electronic fence, and the automatically generated electronic fence has high accuracy and strong security, and can greatly improve user experience.

Description

Technical Field

[0001] This application relates to the field of communications, and in particular to an automatic electronic fence generation method, a real-time detection method, and an apparatus. Background Technology

[0002] In recent years, Virtual Reality (VR) technology has received increasing attention from both industry and consumers, with market demand for VR devices steadily growing. VR devices completely isolate the virtual world from the real world during use, making it impossible for users to perceive their surroundings. Therefore, to protect user safety, electronic fences are set as safety zone boundaries. When a user exceeds the electronic fence's range, a risk warning is displayed.

[0003] However, electronic fences have high requirements for the usage scenarios and a high barrier to entry for users, making them inconvenient for daily use. Summary of the Invention

[0004] This application provides an automatic electronic fence generation method, a real-time detection method, and an apparatus. The automatic generation method and apparatus can automatically generate electronic fences, which lowers the barrier to entry for using electronic fences. Furthermore, the automatically generated electronic fences are highly accurate and secure, greatly improving the user experience. The real-time detection method and apparatus can update the electronic fences in a timely manner, further enhancing their accuracy and security.

[0005] In a first aspect, embodiments of this application provide an automatic electronic fence generation method, comprising: acquiring an environmental image, the environmental image being captured by a photograph of a scene in which a user is located; generating spatial structure information based on the environmental image, the spatial structure information including 3D point cloud and information of at least one plane, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane; and generating an electronic fence based on the spatial structure information.

[0006] In this embodiment of the application, when a user needs to use an electronic device, the camera or webcam of the electronic device can capture environmental images of the scene in which the user is located; based on the environmental images, spatial structure information that can characterize the shapes of different objects such as 3D objects and the ground in the scene is generated; then, based on such spatial structure information, the boundary for the user to move safely is determined, and the electronic fence is obtained after connecting these boundaries.

[0007] The 3D point cloud in the spatial structure information can also be output by the SLAM system and can be used to describe the shape of 3D objects; the plane information is a plane fitted by the actual 3D point cloud that simulates the distribution of objects in the space, which can represent the real plane of the object. The plane information refers to the information that can describe the plane, such as the length and width of the plane.

[0008] Electronic devices can generate electronic fences in different ways based on different spatial structure information. For example, a boundless reference plane corresponding to the ground can be generated first. Then, the objects in the user's space can be detected and identified using the aforementioned spatial structure information. Based on the processed spatial structure information, the boundary of the reference plane and the obstacle area of ​​the objects in the space can be obtained. Connecting the boundary of the reference plane and the obstacle area on the boundless reference plane yields a safe area where the user can live safely. The boundary of this safe area is the electronic fence.

[0009] This application embodiment can automatically generate electronic fences based on environmental images. These images can be obtained by starting the camera or video recording mode upon power-on, without requiring user intervention. The electronic fences generated by this application embodiment are highly accurate, secure, and highly adaptable to the user's environment. They are not limited by the shape of the electronic fence, lowering the barrier to entry for users and greatly improving the user experience.

[0010] The plane in this application is a horizontal plane, a vertical plane, or an oblique plane, wherein the oblique plane is a plane that is not parallel to the horizontal plane or the vertical plane.

[0011] In one possible implementation, the at least one plane includes at least one horizontal plane and other planes, the other planes including either a horizontal plane or a vertical plane.

[0012] In one possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0013] In one possible implementation, before generating spatial structure information based on the environmental image, the method further includes: acquiring Inertial Measurement Unit (IMU) data, wherein the IMU data is obtained by IMU calculation of objects in the user's scene; generating spatial structure information based on the environmental image includes: generating the spatial structure information based on the environmental image and the IMU data.

[0014] IMU data is calculated based on the acceleration signals of objects in the user's scene along the independent three axes of the carrier coordinate system, the angular velocity signal of the carrier relative to the navigation coordinate system, and the measured angular velocity and acceleration in three-dimensional space.

[0015] Electronic devices can generate electronic fences in different ways based on different spatial structure information. For example, a boundless reference plane corresponding to the ground can be generated first. Then, the objects in the user's space can be detected and identified using the aforementioned spatial structure information. Based on the processed spatial structure information, the boundary of the reference plane and the projection of the objects in the space are obtained, i.e., the obstacle area. Connecting the boundary of the reference plane and the obstacle area on the boundless reference plane yields a safe area where the user can live safely. The boundary of this safe area is the electronic fence.

[0016] In one possible implementation, the environmental image is obtained by capturing the user's scene through VR glasses or a smart device with a camera. When the user enters a scene requiring an electronic fence, the camera or webcam can automatically capture the user's scene based on the user's power-on operation. The acquired image is then processed or detected, such as performing environmental detection, terrain detection, environmental data extraction, and image descriptor acquisition, to obtain an environmental image containing the aforementioned information.

[0017] In one possible implementation, when the spatial structure information comprises the pose information, the 3D point cloud, the plane, the depth data, the mesh recognition data, and the 3D object recognition information, the step of generating the spatial structure information based on the environmental image and the IMU, and generating the electronic fence based on the spatial structure information, includes: obtaining the pose information and the 3D point cloud based on the environmental image and the IMU; performing plane detection on the pose information and the 3D point cloud to obtain the plane; performing depth detection on the environmental image to obtain the depth data; processing the mesh recognition data and the 3D object recognition information based on the depth data; and generating the electronic fence based on the 3D point cloud, the plane, and the mesh recognition data and 3D object recognition information processed by the depth data.

[0018] Spatial structure information can include pose information, 3D point clouds, planar information, depth data, mesh recognition data, and 3D object recognition information. Pose information can be obtained in real-time by a Simultaneous Localization and Mapping (SLAM) system configured in the electronic device, primarily used for coordinate alignment to improve the consistency between the generated electronic fence and the detected 3D reality. Depth data can be obtained through depth estimation. After inputting depth data into the electronic device, a depth map is generated. By aligning this depth map with the planar image in the environment, the depth value corresponding to each pixel in the planar image is obtained. Including this depth value when inputting the environment image and IMU can effectively improve the accuracy of the output 3D point cloud and make the output 3D point cloud denser. Mesh recognition data consists of triangular patches generated after mesh recognition to describe objects in the user's scene. 3D object recognition information is information about the outer surface of the 3D object after it has been recognized. Generating spatial structure information based on environmental images and IMU (Integrated Device Unit) can be achieved by inputting environmental images and IMU data into the SLAM system. Real-time feature extraction and matching are performed on the input environmental images to obtain the matching relationships of planar features between images. Then, based on IMU data and the corresponding positional relationship parameters between the imaging device (e.g., camera and IMU), the camera's pose is estimated, yielding the camera's original pose information. Next, based on this pose information and the planar feature matching relationships, 3D point clouds are generated using algorithms such as triangulation. This results in the 3D point cloud output by the SLAM system, which is then used for plane detection to obtain the planes in space. Using the environmental images as input, depth estimation is performed to obtain depth data. This depth data can be used as input for other data detection, such as mesh recognition and 3D object detection, yielding mesh recognition data and 3D object recognition information, respectively. The reference plane and its boundary are obtained from the 3D point cloud using the method described above. This 3D point cloud can be a denser and more accurate 3D point cloud obtained after depth estimation. Then, the obstacle region is obtained from the 3D point cloud representing the obstacle using the method described above. The obstacle region is optimized based on the information of the plane and the mesh recognition data and 3D object recognition information processed by the depth data, so that the obstacle region is closer to the real 3D spatial object position, plane and boundary in the user's scene. Then, the surrounding obstacle regions are connected on the reference plane to obtain a more accurate area that can be used for safe activities. The boundary of this area is the generated electronic fence.

[0019] In one possible implementation, generating the electronic fence includes: generating the electronic fence with the plane point of the user's location in the scene as the center and aligned with the coordinate system of the environment image.

[0020] The electronic device centers on the user currently using the device and connects the surrounding obstacle area on the reference plane to obtain a safe area for movement. The boundary of this obstacle area is aligned with the coordinate system of the corresponding 3D obstacle object boundary in the environmental image. The generated electronic fence is basically consistent with the position of each 3D object in the user's scene, making the electronic fence more consistent with the user's scene.

[0021] In one possible implementation, after generating the electronic fence, the method further includes saving the electronic fence as a historical electronic fence.

[0022] After an electronic device generates an electronic fence, it saves it as a historical electronic fence so that when the user uses it next time, they can retrieve and call up a historical electronic fence that is suitable for the next use scenario from multiple saved historical electronic fences, instead of having to repeatedly generate an electronic fence for the same scenario every time.

[0023] In one possible implementation, after acquiring the environmental image, the method further includes: retrieving the environmental image from the saved historical electronic fences; if a historical electronic fence with similarity is obtained, calculating the weighted score of the historical electronic fence and determining the historical electronic fence with the highest weighted score as the target electronic fence; solving the pose of the target electronic fence; if the solution is successful, setting the target electronic fence according to the difference in pose so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

[0024] This application embodiment sorts candidate frames from different historical electronic fences by descriptor distance and performs weighted clustering to obtain the target electronic fence with the highest weighted score. The pose of this target electronic fence is then calculated. If the calculation is successful, a coordinate transformation relationship is calculated based on the pose difference between the target electronic fence and the environmental image. The target electronic fence is then loaded according to this coordinate transformation relationship, aligning its coordinate system with the coordinate system corresponding to the environmental image, i.e., the electronic fence the user needs to use.

[0025] In one possible implementation, the depth data is Time-of-Flight (TOF) data.

[0026] Since the depth data output by a Time-of-Flight (TOF) sensor is far more accurate than depth estimation results, it is best to use TOF data obtained from a TOF sensor. For example, using the depth map output by a TOF sensor as input for mesh recognition and 3D object detection will yield more accurate mesh recognition data and 3D object recognition information, respectively. A reference plane and its boundary are obtained from a 3D point cloud using the method described above. This 3D point cloud can be a denser and more accurate 3D point cloud obtained after depth estimation. Then, obstacle regions are obtained from the 3D point cloud representing obstacles using the same method. The obstacle regions are optimized based on the plane information and the mesh recognition and 3D object recognition information processed from the depth data, making the obstacle regions closer to the actual 3D spatial object positions, planes, and boundaries in the user's scene. Finally, connecting the surrounding obstacle regions on the reference plane yields a more accurate area suitable for safe activity; the boundary of this area is the generated electronic fence.

[0027] In one possible implementation, generating the electronic fence based on the spatial structure information includes: constructing an optimization equation based on the 3D point cloud and the plane information to obtain an optimized plane; optimizing the plane based on the optimization equation to obtain an optimized plane; determining a reference plane based on the optimized plane corresponding to the ground; projecting the optimized plane representing the boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, obtaining a reference horizontal plane with a boundary corresponding to the ground; projecting the optimized plane representing obstacles onto the reference horizontal plane in the direction of gravity to determine the obstacle area of ​​the reference horizontal plane; and generating the electronic fence based on the reference horizontal plane and the obstacle area.

[0028] An optimization equation is constructed based on the information from the 3D point cloud and the plane. The plane is then optimized using this equation to obtain an optimized plane. This optimized plane accurately represents various objects in the user's environment and, through the plane's information, eliminates the erroneous effects of small, non-user-friendly objects on the electronic fence generation, making the generated plane more accurate. Therefore, the generated electronic fence better meets user needs. Alternatively, the electronic fence can be generated based on the reference horizontal plane and the obstacle area. This can be done by centering on the currently used device and connecting it to surrounding obstacle areas on the reference horizontal plane, thus obtaining a safe area for movement. The boundary of this area is the generated electronic fence.

[0029] Secondly, embodiments of this application provide an automatic electronic fence generation method, comprising: acquiring an environmental image, wherein the environmental image is obtained by taking a picture of the scene where the user is located; generating spatial structure information based on the environmental image, wherein the spatial structure information includes, when the spatial structure information is 3D point cloud and 3D object recognition information, a reference plane corresponding to the ground can be identified based on the 3D point cloud; determining the reference plane boundary and obstacle area based on the projection of 3D objects at a distance exceeding a first threshold from the ground based on the 3D object recognition information; and generating an electronic fence based on the reference plane, the reference plane boundary, and the obstacle area. The method of identifying the reference plane corresponding to the ground through 3D point cloud can refer to the above example, further projecting 3D objects at a distance exceeding the first threshold from the ground onto the reference plane to determine the obstacle area, and after cutting off the obstacle area on the reference plane, connecting the reference plane boundary to obtain a safe area, the boundary of the safe area being the electronic fence.

[0030] Thirdly, embodiments of this application provide an automatic electronic fence generation method, comprising: acquiring an environmental image, wherein the environmental image is obtained by taking a picture of the scene where the user is located; generating spatial structure information based on the environmental image, wherein the spatial structure information includes mesh recognition data and 3D object recognition information, and identifying a reference plane corresponding to the ground based on the mesh recognition; determining the boundary of the reference plane and an obstacle area based on the projection of a 3D object at a distance exceeding a first threshold from the ground based on the 3D object recognition information; and generating an electronic fence based on the reference plane, the boundary of the reference plane, and the obstacle area.

[0031] First, a reference plane corresponding to the ground can be identified based on the mesh. For example, the mesh identification data obtained through mesh identification is a triangular facet corresponding to the ground. After connecting the ground triangular facets, an initial ground facet region is obtained, which can be determined as the reference plane. Then, the mesh triangular facets and 3D objects that are more than a first threshold away from the ground are projected onto the reference plane to determine the obstacle region. An electronic fence is generated based on the reference plane, the reference plane boundary, and the obstacle region. For example, after cutting off the obstacle region on the reference plane, connecting the reference plane boundary can obtain a safe area, and the boundary of the safe area is the electronic fence.

[0032] This method of combining two spatial structure information contents to obtain an electronic fence can minimize the computational requirements of the scheme while ensuring the main functions, and at the same time obtain a relatively accurate safe activity area.

[0033] Fourthly, embodiments of this application provide a real-time detection method for electronic fences, comprising: periodically acquiring real-time environmental images of the scene where the user is located; if the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold, then updating the electronic fence based on the spatial structure information generated from the real-time environmental image.

[0034] A camera can be used to periodically capture real-time environmental images of the user's scene. Alternatively, other imaging devices can be used to acquire real-time scene data and periodically capture these environmental images, from which real-time spatial structure information can be extracted. Real-time spatial structure information is obtained from the environmental images, and existing spatial structure information is derived from the environmental images. A local comparison is then made between the real-time and existing spatial structure information. Depending on the type of change in spatial structure information, different fusion weight values ​​can be used to fuse the real-time and existing spatial structure information: changes include the addition of new spatial structures, changes in the size or pose of existing spatial structures, and the disappearance of existing spatial structures.

[0035] In this embodiment of the application, when new spatial structure information is added to the real-time environmental image, the differences should be updated to the electronic fence in a timely manner to ensure safety during use. If the spatial structure information in the real-time environmental image changes, to avoid the impact of detection errors on updating the electronic fence, different weight values ​​should be set according to the position, size, and absolute value of the change of the structure. The greater the difference in change, the higher the weight value. If the spatial structure information in the real-time environmental image is deleted, multi-frame data verification should be performed to confirm that the spatial structure information has indeed disappeared. After fusion, global optimization should be performed based on the weight values ​​corresponding to each spatial structure information. Then, the electronic fence should be updated according to the spatial structure information used by the device.

[0036] This real-time updating of the electronic fence eliminates the need to regenerate it. Instead, it integrates the existing fence with new or removed elements, effectively improving its security.

[0037] Fifthly, embodiments of this application provide an automatic electronic fence generation device, comprising:

[0038] The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located.

[0039] A processing module is configured to generate spatial structure information based on the environmental image, the spatial structure information including information on at least one plane of a 3D point cloud, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane;

[0040] The processing module is also used to generate an electronic fence based on the spatial structure information.

[0041] In one possible implementation, the at least one plane includes at least one horizontal plane and other planes, the other planes including either a horizontal plane or a vertical plane.

[0042] In one possible implementation, the at least one plane includes at least one horizontal plane and at least one vertical plane.

[0043] In one possible implementation, the input module is further configured to acquire measurement unit (IMU) data, which is obtained by IMU calculation of objects in the user's scene; the processing module is further configured to generate the spatial structure information based on the environmental image and the IMU data.

[0044] In one possible implementation, the input module is also used to obtain an environmental image by capturing a scene in which the user is located.

[0045] In one possible implementation, the processing module is specifically configured to: construct an optimization equation based on the information of the 3D point cloud and the plane; optimize the plane according to the optimization equation to obtain an optimized plane; determine a reference plane based on the optimized plane corresponding to the ground; project the optimized plane representing the boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thereby obtaining a reference horizontal plane with a boundary corresponding to the ground; project the optimized plane representing obstacles onto the reference horizontal plane in the direction of gravity to determine the obstacle region of the reference horizontal plane; and generate the electronic fence based on the reference horizontal plane and the obstacle region.

[0046] In one possible implementation, the processing module further includes a pose estimation unit, a depth estimation unit, a mesh recognition unit, and a 3D object recognition unit: the pose estimation unit is used to obtain the pose information based on the environmental image and the IMU data; the 3D point cloud processing unit is further used to obtain the 3D point cloud based on the environmental image and the IMU data; the plane detection unit is further used to perform plane detection on the pose information and the 3D point cloud to obtain the plane; the depth estimation unit is used to perform depth detection based on the environmental image to obtain the depth data; the mesh recognition unit is used to process the mesh recognition data based on the depth data; the 3D object recognition unit is used to process the 3D object recognition information based on the depth data; and the electronic fence generation unit is further used to generate the electronic fence based on the 3D point cloud, the plane, and the mesh recognition data and 3D object recognition information processed by the depth data.

[0047] In one possible implementation, the depth data is Time-of-Flight (TOF) data.

[0048] In one possible implementation, the electronic fence generation unit is further configured to generate the electronic fence with the plane point of the user's location in the scene as the center and aligned with the coordinate system of the environment image.

[0049] In one possible implementation, the device further includes:

[0050] A storage module is used to save the electronic fence as a historical electronic fence.

[0051] In one possible implementation, the processing module further includes a historical electronic fence retrieval unit: the historical electronic fence retrieval unit is used to retrieve the environmental image from the historical electronic fences already saved in the storage module; if a historical electronic fence with similarity association is obtained, the weighted score of the historical electronic fence is calculated, and the historical electronic fence with the highest weighted score is determined as the target electronic fence; if the pose estimation unit successfully calculates the pose of the target electronic fence, the electronic fence generation unit sets the target electronic fence according to the difference in pose, so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

[0052] Sixthly, embodiments of this application provide an automatic electronic fence generation device, comprising:

[0053] The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located.

[0054] The processing module is used to identify a reference plane corresponding to the ground based on the 3D point cloud; and to determine the boundary of the reference plane and the obstacle area based on the projection of 3D objects that are more than a first threshold away from the ground on the reference plane according to the 3D object recognition information.

[0055] The processing module is also used to generate an electronic fence based on the reference plane, the boundary of the reference plane, and the obstacle area.

[0056] Seventhly, embodiments of this application provide an automatic electronic fence generation device, comprising:

[0057] The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located.

[0058] The processing module is used to determine the projection of 3D objects that are more than a first threshold above the ground onto a reference plane based on the 3D object recognition information, and to determine the boundary of the reference plane and the obstacle area.

[0059] The processing module is also used to generate an electronic fence based on the reference plane, the boundary of the reference plane, and the obstacle area.

[0060] Eighthly, embodiments of this application provide a real-time detection device for electronic fences, comprising:

[0061] The input module is also used to periodically collect real-time environmental images of the user's scene;

[0062] The electronic fence update module is used to update the electronic fence based on the spatial structure information generated from the real-time environmental image if the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold.

[0063] The fifth, sixth, seventh, and eighth aspects, and any one of their implementations, correspond to the first, second, third, and fourth aspects, and any one of their implementations, respectively. The technical effects corresponding to the fifth, sixth, seventh, and eighth aspects, and any one of their implementations, can be found in the technical effects corresponding to the first, second, third, and fourth aspects, and any one of their implementations, as described above; they will not be repeated here.

[0064] Ninthly, embodiments of this application provide an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the first to fourth aspects above.

[0065] In a tenth aspect, embodiments of this application provide a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to fourth aspects.

[0066] Eleventhly, embodiments of this application provide a computer program product, the computer program product including computer program code, which, when run on a computer, is used to perform the method described in any one of the first to fourth aspects above. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a structural diagram of an electronic device capable of automatically generating electronic fences according to an embodiment of this application;

[0069] Figure 2 This is a flowchart of a method for automatically generating an electronic fence according to an embodiment of this application;

[0070] Figure 3 This is a flowchart of another method for automatically generating electronic fences according to an embodiment of this application;

[0071] Figure 4 This is a flowchart of a method for real-time detection of an electronic fence according to an embodiment of this application;

[0072] Figure 5 This is a schematic diagram of an automatic electronic fence generation process;

[0073] Figure 6 These are 3D object recognition results in the same scene;

[0074] Figure 7 This is a schematic diagram of spatial information fusion during the automatic generation of electronic fences;

[0075] Figure 8 It is a detection effect diagram of a scene;

[0076] Figure 9 This is the original rendering of the electronic fence;

[0077] Figure 10 This is a screenshot showing the detection results after adding a new object to the scene.

[0078] Figure 11 This is a rendering of the updated electronic fence.

[0079] Figure 12 This is a schematic diagram of another automatic electronic fence generation process;

[0080] Figure 13 This is another schematic diagram of the automatic generation process for electronic fences;

[0081] Figure 14 This is another schematic diagram of the automatic generation process for electronic fences;

[0082] Figure 15 This is a structural block diagram of the automatic electronic fence generation device according to an embodiment of this application;

[0083] Figure 16 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application;

[0084] Figure 17 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application;

[0085] Figure 18 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application;

[0086] Figure 19 This is a structural block diagram of a real-time detection electronic fence device according to an embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0089] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0090] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0091] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0092] Due to the widespread use of VR devices, some VR devices utilize pre-defined electronic fences of a specific shape, such as a circular area with a diameter of 4 meters or a square area with a diagonal length of 4 meters. For these pre-designed electronic fences, users need to find a safe location that meets the shape requirements beforehand and then activate the device at the center of that location. The VR device defaults to the area inside the electronic fence as a safe zone, where users can move freely. If the device detects that a user's limb has exceeded or is about to exceed the electronic fence, it will display a message on the screen or issue a warning sound to alert the user to potential danger and urge them to return to the safe zone within the electronic fence. This type of electronic fence, which protects user safety by pre-setting a specific shape, has high requirements for the choice of usage location. Finding a suitable location in home or work environments is not easy, thus limiting its application and making it inconvenient for users. Other VR devices guide users to generate electronic fences. For example, after activating the VR headset, the user is required to adjust the height of a virtual plane displayed on the VR device, centered on themselves, to make it as close to the ground as possible. Then, a virtual ray is emitted from the controller to draw lines in a flat, unobstructed area. After connecting the lines, the VR headset generates an electronic fence in the corresponding area. In this case, user participation is required, creating a learning curve. The accuracy of the electronic fence is directly affected by the user's measurements of the plane height and the lines drawn, making it inconvenient and resulting in a poor user experience.

[0093] To address the above issues, this application provides an electronic device capable of automatically generating electronic fences. This device can be VR glasses, motion-sensing gaming equipment equipped with VR, or a wheelchair equipped with an electronic fence, among other devices. It can automatically generate electronic fences, lowering the barrier to entry for using electronic fences, offering high accuracy and strong security, and significantly improving the user experience.

[0094] Figure 1 A schematic diagram of an electronic device 100 according to an embodiment of this application is shown, as follows: Figure 1 As shown, the electronic device 100 includes a processor 110, a display 120, a sensor 130, a memory 140, and includes a handle 150, a speaker 160, a microphone 170, a handle 180, and an indicator 190.

[0095] Processor 110 may include one or more central processing units (CPUs), or one CPU and one graphics processing unit (GPU), or an application processor and a coprocessor (e.g., a microcontroller unit or neural network processor), and may also include caches and registers. When processor 110 includes multiple processors, these processors may be integrated on the same chip or be independent chips. A processor may include one or more physical cores, where a physical core is the smallest processing module. Processor 110 may be configured with a simultaneous localization and mapping (SLAM) system, which is capable of localization, pose (position and orientation) construction of object shapes. Processor 110 can be used for spatial structure information recognition, fitting optimization, and weighted fusion in the embodiments described below.

[0096] Display 120 is used to display images, videos, etc. Display 120 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc.

[0097] Sensor 130 can sense the current state of the system, such as open / closed state, position, contact with the user, direction, and acceleration / deceleration, and can generate sensing signals for controlling the operation of the system. Sensor 130 may include a vision sensor 131, an inertial measurement unit (IMU) 132, and may also include a depth sensor 133 or a laser sensor 134. Vision sensor 131, such as a camera, depth camera, lidar, millimeter-wave radar, etc., is typically used to acquire planar image information of a scene. A camera can be used to capture still images or video. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. The IMU sensor 132 may include three single-axis accelerometers and three single-axis gyroscopes, typically used to acquire motion information of the device, such as linear acceleration and angular velocity; the depth sensor 133 or laser sensor 134 is typically used to acquire depth information of the scene. The depth sensor 133 may be a time-of-flight (TOF) sensor, typically used to acquire more accurate depth maps.

[0098] Memory 140 can be used to store computer executable program code, which includes instructions. Internal memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.). The data storage area may store data created by electronic devices, such as VR devices, during use (such as historical electronic fences, image descriptors, etc.). In addition, memory 140 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 120 executes various functional applications and data processing of electronic device 100 by running instructions stored in memory 140 and / or instructions stored in memory 120 located in processor 130.

[0099] Electronic device 100 can perform audio functions through speakers 160 and microphones 170, such as voice control, music playback, and recording. Speaker 160, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 100 can hear sound through speaker 160; for example, VR glasses can use a speaker to emit a warning sound to the user indicating that they have exceeded an electronic fence.

[0100] Microphone 170, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Users can speak by bringing their mouth close to microphone 170, inputting sound signals into it. In addition to collecting sound signals, microphone 170 can also perform noise reduction. In some embodiments, electronic devices can be equipped with multiple microphones 170 to achieve functions such as sound signal collection, noise reduction, sound source identification, and directional recording.

[0101] The handle 180 can be used to determine the motion posture of the electronic device 100 or to input user demand signals.

[0102] Indicator 190 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, flashing to warn users of exceeding the electronic fence range, etc.

[0103] The interface conforms to standard specifications and can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. This interface can be used to connect a charger to charge electronic devices (100% charging), or to transfer data between electronic devices and other devices. It can also be used to connect headphones for audio playback. Furthermore, this interface can be used to connect other mobile phones, such as AR devices.

[0104] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the main control device. In other embodiments of this application, the main control device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0105] The aforementioned electronic device 100 can automatically generate an electronic fence based on the user's location using the following method, providing the user with a highly secure electronic fence without requiring user intervention. Figure 2 This is a flowchart of a method for automatically generating an electronic fence according to an embodiment of this application, such as... Figure 2 As shown, the method includes steps 101, 102, and 103, as follows:

[0106] Step 101: Obtain environmental images.

[0107] Environmental images are captured from the user's surroundings. For example, environmental images can be captured using virtual reality (VR) glasses or smart devices with cameras. Alternatively, visual sensors such as cameras or webcams can be used to capture environmental images of the user's surroundings. This capture can be automatic based on the user's power-on operation, and can involve taking multiple photos from different angles or capturing a frame from a short video.

[0108] In some examples, electronic devices can also acquire IMU data, which is obtained by IMU calculation of objects in the user's scene.

[0109] IMU data can be used to characterize the attitude of an object. It is calculated by the IMU, which can be based on the acceleration signals of the object in the user's scene on the independent three axes of the carrier coordinate system measured by the accelerometer, and the angular velocity and acceleration of the carrier in three-dimensional space measured by the gyroscope based on the angular velocity signal of the carrier relative to the navigation coordinate system.

[0110] In some examples, if a user has previously used the device provided in this application, the geofence generated during that previous use can be saved in the historical geofence database. Each time the device is powered on, the environmental image can be compared with the historical geofence to determine if a suitable historical geofence is available. For example, a VR device equipped with this device can retrieve its own stored historical geofences for comparison. Other smart devices that have not saved historical geofences can also obtain and compare them with historical geofences stored by other devices that have stored them via Bluetooth, cloud sharing, or other transmission methods. Alternatively, the user's location can be determined using GPS or BeiDou positioning functions, and historical geofences stored from when the user was active at that location can be retrieved for comparison.

[0111] Figure 3 This is a flowchart of another method for automatically generating electronic fences according to an embodiment of this application, such as... Figure 3 This includes steps 101, 102, 103, 104, 105, and 106.

[0112] Step 104: Search for environmental images in the saved historical electronic fences. If a similar historical electronic fence is found, proceed to step 105; otherwise, proceed to step 102.

[0113] In some examples, retrieving environmental images from saved historical geofences can begin by performing global feature extraction on the input environmental image to obtain descriptors and the current frame. Global descriptors are then obtained based on the saved historical geofences. Since smaller descriptor distances indicate greater similarity between images, the N historical geofence images with the smallest descriptor distances are selected as candidate frames, where 0 < N < 1000 (e.g., N = 100). If N candidate frames are obtained, the historical geofences with similarity associations are determined.

[0114] Step 105: Calculate the weighted score of the historical electronic fences and determine the historical electronic fence with the highest weighted score as the target electronic fence.

[0115] Based on descriptor distance ranking, weighted clustering is performed on the corresponding candidate frames in different historical electronic fences to obtain the target electronic fence with the highest weighted score.

[0116] Step 106: Solve the pose of the target electronic fence. If the solution is successful, set the target electronic fence according to the pose difference so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

[0117] The pose of the target electronic fence is calculated. If the calculation is successful, the coordinate transformation relationship can be calculated based on the pose difference between the target electronic fence and the environmental image. The target electronic fence is then loaded according to this coordinate transformation relationship and aligned with the coordinate system of the electronic fence corresponding to the environmental image, i.e., the electronic fence that the user needs to use.

[0118] In some examples, to obtain a more accurate coordinate transformation relationship, after the target electronic fence corresponding to the frame with the highest weighted score is obtained, continuous candidate frames may be further calculated. After the pose calculation is successful, if the successfully calculated candidate frames and the frame with the highest weighted score correspond to the same target electronic fence, the coordinate transformation relationship can be obtained through the mean value of these several frames. For example, environmental images may be retrieved from the saved historical electronic fences, and the first 600 historical electronic fence images with the smallest descriptor distance are obtained as candidate frames, where 100 candidate frames correspond to historical electronic fence A, 200 candidate frames correspond to historical electronic fence B, and 300 candidate frames correspond to historical electronic fence C. Weighted clustering is performed on the 600 different candidate frames based on the descriptor distance ranking. If the frame with the highest weighted score corresponds to the target electronic fence A, then the pose of the target electronic fence A is solved. If the solution is successful, the pose of continuous images in the 100 candidate frames may be further solved. If there are M consecutive frames that are successfully solved, where 1<M<20, for example, M is 3, it may be determined whether all the 3 candidate frames correspond to the historical electronic fence A. If yes, the coordinate transformation relationship from each candidate frame to the target electronic fence A can be calculated, and then the matrix average value is obtained through the 3 consecutive frames to obtain a more accurate coordinate correspondence. Then the target electronic fence A is loaded to the user according to the coordinates, so that the retrieved historical electronic fence A has a higher matching degree with the environment where the user is located and stronger restoration performance.

[0119] The descriptors obtained by extracting the global features and the local features may be obtained through a pre-trained Artificial Intelligence (AI) model, and a conventional feature extraction method may also be used, such as the Oriented FAST and Rotated BRIEF algorithm, the Scale-Invariant Feature Transform (SIFT) algorithm, and the like.

[0120] Step 102: Generate spatial structure information according to the environmental image, where the spatial structure information includes a 3D point cloud and information of at least one plane, and the plane is determined by the distribution of points of the 3D point cloud on the plane.

[0121] If the electronic device in the above example further acquires IMU data, step 102 may generate the spatial structure information according to the environmental image and the IMU data.

[0122] The 3D point cloud may be output by a SLAM system, and may be used to describe the shape of a 3D object in a scene. The plane is obtained by performing segmentation and clustering according to the actual distribution of the 3D point cloud to obtain a surface of a relatively continuous and dense point cloud cluster, instead of projecting the 3D point cloud to a certain position.

[0123] In some examples, at least one plane includes at least one horizontal plane and other planes, which may be horizontal or vertical. That is, the planes obtained by clustering 3D point clouds should at least form a wireless plane corresponding to the ground; otherwise, environmental images will be reacquired to obtain 3D point clouds and then planes.

[0124] If the user's scene contains large external objects such as walls, the plane obtained from the distribution of 3D point clouds should include at least one horizontal plane and one vertical plane to represent the ground and walls. In practical applications, however, more planes should be obtained from the 3D point clouds, allowing for the generation of more accurate electronic fences by deriving both the horizontal and vertical planes of objects in the space.

[0125] Furthermore, to make the obtained electronic fence more accurate, the spatial structure information can also include other data, such as pose information, depth data, mesh recognition data, and 3D object recognition information. Pose information can be obtained through real-time pose estimation by the configured SLAM system, mainly used for coordinate alignment to ensure a higher degree of consistency between the generated electronic fence and the detected 3D reality. Depth data can be obtained through depth estimation. After the electronic device inputs depth data, a depth map is obtained. By aligning this depth map with the planar image in the environment image, the depth value corresponding to each pixel in the planar image is obtained. Adding this depth value when inputting the environment image and IMU can effectively improve the accuracy of the output 3D point cloud and make the output 3D point cloud denser. Mesh recognition data consists of triangular patches generated after mesh recognition to describe objects in the user's scene. 3D object recognition information is information about the outer surface of the 3D object after it has been identified.

[0126] In some examples, generating spatial structure information from environmental images can involve inputting environmental images into a SLAM system, performing real-time feature extraction and matching based on the input images to obtain the matching relationship of planar features between images, and then obtaining pose differences based on other detection or processing methods, such as environmental detection, terrain detection, and environmental data. The spatial structure information is then adjusted based on these pose differences to make the spatial structure information closer to the user's scene and to relatively realistically recreate the scene.

[0127] In some examples, generating spatial structure information based on environmental images and IMUs can involve inputting environmental images and IMUs into the SLAM system, performing real-time feature extraction and matching based on the input images to obtain the matching relationships of planar features between images, and then estimating the camera's pose based on IMU data and the corresponding positional relationship parameters between the imaging device, such as the camera and the IMU, to obtain the camera's original pose information. Then, based on this pose information and the planar feature matching relationships, an algorithm such as triangulation is used to generate a 3D point cloud. Finally, based on the 3D point cloud output by the SLAM system, planar detection is used to obtain information about horizontal and vertical planes in space, such as detecting that the horizontal plane of a table in the scene is 60cm*50cm.

[0128] Alternatively, depth estimation can be performed based on a pre-trained AI model, using images as input, to obtain depth data. This depth data can then be used as input for other data detection, such as mesh recognition and 3D object detection, yielding mesh recognition data and 3D object recognition information respectively. Mesh recognition data and 3D object recognition information can also be obtained solely based on the environmental image and IMU, but the detection results are less accurate than those incorporating depth data.

[0129] Step 103: Generate an electronic fence based on the spatial structure information.

[0130] For example, an electronic device can connect the surrounding obstacle area on a reference plane with the user currently using the device as the center, thus obtaining an area where safe activities can be carried out. The boundary of this obstacle area is aligned with the coordinate system of the corresponding 3D obstacle object boundary in the environmental image. The generated electronic fence is basically consistent with the position of each 3D object in the user's scene, making the electronic fence more in line with the user's scene.

[0131] Electronic fences can be generated in different ways based on different spatial structure information. For example, a boundless reference plane corresponding to the ground can be generated first. Then, objects in the user's space can be detected and identified using the aforementioned spatial structure information. Based on the processed spatial structure information, the boundary of the reference plane and the projection of objects in the space are obtained, i.e., the obstacle area. Connecting the boundary of the reference plane and the obstacle area on the boundless reference plane yields a safe area where the user can live safely. Then, taking the plane point of the user's position in the scene on the reference plane as the center, the coordinate system of this boundary is aligned with the coordinate system of the corresponding object in the environmental image. The boundary of this safe area is the electronic fence.

[0132] For example, when the spatial structure information includes 3D point cloud and planar information, an optimization equation is constructed based on the 3D point cloud and planar information; the planar is optimized based on the optimization equation to obtain an optimized planar; a reference plane is determined based on the optimized planar corresponding to the ground; the optimized planar representing the boundary is projected onto the reference plane in the direction of gravity to determine the boundary of the reference plane, resulting in a reference horizontal plane with a boundary corresponding to the ground; the optimized planar representing obstacles is projected onto the reference horizontal plane in the direction of gravity to determine the obstacle area of ​​the reference horizontal plane; and an electronic fence is generated based on the reference horizontal plane and the obstacle area.

[0133] For example, VR glasses can obtain 3D point clouds output by a SLAM system. Plane information can be categorized as follows: if there are rectangles smaller than 20cm x 30cm in space (i.e., rectangles with a shorter side less than 20cm and a longer side less than 30cm), these are considered planes that need to be ignored. An optimization equation is constructed based on the 3D point cloud and plane information to obtain an optimized plane. This optimized plane no longer contains objects with rectangles smaller than 20cm x 30cm. Besides eliminating smaller planes, the optimization equation also reduces the error value of the plane, making the optimized plane closer to the shapes of objects in the scene.

[0134] In the optimized plane, the infinitely large plane corresponding to the opposite side should be determined as the reference plane. Then, in the direction of gravity, the planes representing the edges of the space in the optimized plane are projected onto this reference plane. Connecting these planes yields a bounded reference plane, i.e., the reference horizontal plane. Other objects in the scene have also been optimized into optimized planes that can represent them. In the direction of gravity, these optimized planes are also projected onto the reference horizontal plane to obtain the obstacle area. Then, with the currently used device as the center, connecting the surrounding obstacle areas on the reference horizontal plane yields the area where safe movement is possible. The boundary of this area is the generated electronic fence.

[0135] To minimize the computational demands of the solution while ensuring core functionality, and to obtain a relatively accurate safe activity area, the spatial structure information in some examples can be either of the two types mentioned above.

[0136] For example, when the spatial structure information consists of mesh recognition data and 3D object recognition information, a reference plane corresponding to the ground can be identified first based on the mesh recognition. For instance, the mesh recognition data obtained through mesh recognition is triangular facets corresponding to the ground. After connecting the ground triangular facets, an initial ground facet region is obtained, which can be determined as the reference plane. Then, based on the 3D object recognition information, the projection of 3D objects that are more than 10cm away from the ground onto the reference plane is determined to define the reference plane boundary and obstacle region. For instance, if the first threshold is 10cm, the mesh triangular facets and 3D objects that are more than 10cm away from the ground are projected onto the reference plane to determine the obstacle region. An electronic fence is generated based on the reference plane, the reference plane boundary, and the obstacle region. For instance, after cutting off the obstacle region on the reference plane, connecting the reference plane boundary yields a safe area, and the boundary of the safe area is the electronic fence.

[0137] For example, when the spatial structure information is either 3D point cloud or 3D object recognition information, a reference plane corresponding to the ground can be identified based on the 3D point cloud; the projection of 3D objects exceeding a first threshold from the ground onto the reference plane can be determined based on the 3D object recognition information to define the reference plane boundary and obstacle area; and an electronic fence can be generated based on the reference plane, its boundary, and the obstacle area. The method for identifying the reference plane corresponding to the ground using 3D point cloud can refer to the above example. Then, 3D objects exceeding 10cm from the ground can be projected onto this reference plane to determine the obstacle area. After cutting off the obstacle area on the reference plane, connecting the reference plane boundary yields the safe area, and the boundary of the safe area is the electronic fence.

[0138] In some examples, when spatial structure information includes pose information, 3D point clouds, planes, depth data, mesh recognition data, and 3D object recognition information, the most accurate electronic fence can be obtained through the fusion processing of multiple information sources. For instance, pose information and 3D point clouds are obtained from environmental images and IMU data; planes are obtained from the pose information and 3D point clouds through plane detection; depth data is obtained from the environmental images through depth detection; mesh recognition data and 3D object recognition information are processed from the depth data; and an electronic fence is generated based on the 3D point clouds, planes, and the depth-processed mesh recognition data and 3D object recognition information.

[0139] Generating spatial structure information based on environmental images and IMUs can be achieved by inputting environmental images and IMUs into the SLAM system, performing real-time feature extraction and matching based on the input environmental images to obtain the matching relationship of planar features between images, estimating the camera's pose based on IMU data and the corresponding positional relationship parameters between the imaging device, such as the camera and the IMU, to obtain the camera's original pose, which is the pose information. Then, based on this pose information and the planar feature matching relationship, 3D point clouds are generated according to algorithms such as triangulation algorithms to obtain the 3D point cloud.

[0140] Based on the 3D point cloud output by the SLAM system, information about planes in space is obtained through plane detection. For example, if the horizontal plane of a table in the scene is detected to be 60cm*50cm, which is larger than the smallest recognition plane of 20cm*30cm, then the horizontal plane of the table is retained. The horizontal plane is projected onto the reference plane to obtain an obstacle area.

[0141] Using the environmental image as input, depth estimation is performed to obtain depth data. This depth data can be used as input for other data detection, such as mesh recognition and 3D object detection, yielding mesh recognition data and 3D object recognition information respectively. Since the depth data output by the Time-of-Flight (TOF) sensor is far more accurate than the depth estimation result, if the VR device is equipped with a TOF sensor, the depth data should ideally be TOF data obtained through the TOF sensor. For example, using the depth map output by the TOF sensor as input for mesh recognition and 3D object detection will yield more accurate mesh recognition data and 3D object recognition information.

[0142] The reference plane and its boundary are obtained from the 3D point cloud using the method described above. This 3D point cloud can be a denser and more accurate 3D point cloud obtained after depth estimation. Then, the obstacle region is obtained from the 3D point cloud representing the obstacle using the method described above. The obstacle region is optimized based on the information of the plane and the mesh recognition data and 3D object recognition information processed by the depth data, so that the obstacle region is closer to the real 3D spatial object position, plane and boundary in the user's scene. Then, the surrounding obstacle regions are connected on the reference plane to obtain a more accurate area that can be used for safe activities. The boundary of this area is the generated electronic fence.

[0143] In some embodiments, the depth data is Time-of-Flight (TOF) data. However, when performing mesh recognition or plane detection, there may be limitations on the size of the fitted or detected planes. For example, some small planes, as illustrated above, may be ignored. Other small objects may also be missed by mesh recognition because they are smaller than the built-in threshold. In practical applications, there may be protruding objects such as handrails or sticks in the scene. These objects, due to their small size or planar shape, are not captured as 3D objects in the scene and fail to form obstacle areas. When generating an electronic fence, they will be ignored, potentially posing a safety hazard to users. To avoid this, depth data acquired by a TOF sensor can be used to identify these objects and form obstacle areas, effectively compensating for the above problems and further improving the security of the detected electronic fence.

[0144] For example, in a SLAM system that generates 3D point clouds, after inputting a Time-of-Flight (TOF) image, the depth value corresponding to each pixel in the 2D image can be obtained by aligning the depth map with the 2D image. This depth value is used as the initial depth value for the triangulation module. At the same time, the constraint optimization module can effectively improve the accuracy of the 3D point cloud, resulting in a denser output 3D point cloud with higher accuracy in the position, size, and outline of the objects shown in the 3D point cloud.

[0145] When the depth data is TOF data, the obtained depth data is more accurate, and it can identify objects that are small in area but large in depth. These objects are then added to the outer surface information of 3D objects, resulting in more precise information about the outer surface of 3D objects in the user's space. In some cases, the accuracy of 3D point clouds generated after processing TOF data, or mesh fitting and 3D object detection results obtained after inputting TOF data, are significantly improved.

[0146] Optionally, the spatial structure information generated from image information in this embodiment can be based on a pre-trained AI model. For example, after taking image information as input and performing depth estimation, mesh recognition can be performed based on the estimated depth point cloud data. Similarly, semantic recognition of 3D objects can be performed based on images using a pre-trained AI model. Then, the output plane and mesh information can be fused globally to complete the overall optimization and obtain more accurate information about the outer surface of 3D objects in the user's space.

[0147] In some examples, after the electronic fence is generated, the VR device can determine whether the size of the electronic fence meets the user's needs for this use. If it is determined that the needs are not met, the user can be prompted and guided to move to other areas to scan and generate the electronic fence again.

[0148] In some examples, the generated geofence can be saved as a historical geofence. If the generated geofence meets the needs of the current use case and there is no longer a need to guide the user to move to other areas, it is saved as a historical geofence. This allows the electronic device to retrieve and recall a suitable historical geofence from its stored pool of geofences the next time the user activates the device.

[0149] In some real-life scenarios, such as when users are playing games using VR glasses, they are completely isolated from the real world. They cannot perceive changes in their surroundings. Once the game begins, if the scene changes and potentially harmful objects appear, the user will be unaware. Similarly, for visually impaired individuals using wheelchairs, the user also cannot perceive changes in their environment. To help users assess the safety of their surroundings in real time, the electronic fence needs to be updated promptly whenever the scene changes, thus protecting the user's safety during use. Figure 4 This is a flowchart of a method for real-time detection of an electronic fence according to an embodiment of this application, as follows: Figure 4 As shown, steps 107 and 108 are included:

[0150] Step 107: Periodically collect real-time environmental images of the user's location.

[0151] For example, a camera can be used to periodically capture real-time environmental images of the user's location, or other camera devices can be used to capture real-time scenes and periodically capture these real-time environmental images, from which real-time spatial structure information can be extracted.

[0152] Step 108: If the difference between the spatial structure information corresponding to the real-time environment image and the existing spatial structure information exceeds the second threshold, then update the electronic fence based on the spatial structure information generated from the real-time environment image.

[0153] In some examples, real-time spatial structure information can be obtained from real-time environmental images, and existing spatial structure information can be obtained from the environmental images. A local comparison can then be made between the real-time and existing spatial structure information. Depending on the type of change in spatial structure information, different fusion weight values ​​can be used to fuse the real-time and existing spatial structure information: such as the addition of new spatial structures, changes in the size or pose of existing spatial structures, and the disappearance of existing spatial structures all fall under the category of changes in spatial structure information.

[0154] For example, when new spatial structure information is added to the real-time environmental image, the differences should be promptly updated in the geofence to ensure safety during use. The weights of the plane, point cloud, or mesh data corresponding to the newly added spatial structure information can be set to 100%, and the data fusion can be completed accordingly.

[0155] For example, if the spatial structure information in the real-time environmental image changes, in order to avoid the impact of detection errors on the updating of the electronic fence, different weight values ​​should be set according to the location, size and absolute value of the change. The greater the difference in change, the higher the weight value. The weight value can be controlled within the range of 30%-70%. Through 2-3 frames of data, the fusion is completed smoothly, and the electronic fence is updated.

[0156] For example, if spatial structure information in a real-time environmental image is deleted, multiple frames of data verification should be performed to confirm that the spatial structure information has indeed disappeared. If the spatial structure is confirmed to have disappeared after three consecutive frames of data, the weight value is set to 33%. The deletion of spatial structure information is completed through three frames of data, and the fusion is completed.

[0157] After fusion is completed, global optimization is performed based on the weight values ​​corresponding to each spatial structure information. Then, the electronic fence is updated based on the spatial structure information used by the device, such as pose information, 3D point cloud, planar information, depth data, mesh recognition data, and 3D object recognition information.

[0158] This real-time updating of the electronic fence eliminates the need to regenerate it. Instead, it integrates the existing fence with new or removed elements, effectively improving its security.

[0159] The following examples illustrate how to use the device provided in the above embodiments, such as when the device is configured in a VR device and is equipped with a camera and can use a SLAM system.

[0160] Step 1: First, perform an electronic fence search to determine if there are any historical electronic fences with similarity: Extract global features from the captured images, perform a global search based on the global features, and if the global search is successful, project the local structural information. Based on the successfully calculated pose, load the corresponding electronic fence and align the coordinate systems of the two.

[0161] In some examples, Figure 5 This is a schematic diagram of an automatic electronic fence generation process, such as... Figure 5 As shown in S1, the VR device can first extract global descriptors based on the current image, comparing the descriptors of the current image with the descriptors corresponding to all images included in the historical electronic fence. The global descriptors corresponding to the saved historical electronic fences have already been extracted and saved. Since the smaller the descriptor distance, the more similar the represented images, the 100 images with the smallest descriptor distance can be selected as candidate frames. Based on the descriptor distance ranking, weighted clustering is performed on the candidate frames corresponding to different historical electronic fences, and the historical electronic fence with the highest weighted score is the target electronic fence.

[0162] The weighted score can be calculated as shown in Formula 1-1, which calculates the weighted score corresponding to the historical electronic fence i. i At that time, the weight a j The value of b represents whether a candidate frame of sorted j belongs to the historical electronic fence i; if it does, the value is 1.0, and if it does not, the value is 0.0. j The weighted coefficients representing the similarity ranking of candidate frames are: 2.0 for candidate frames ranked 1-20, 1.5 for candidate frames ranked 20-50, and 1.0 for candidate frames ranked 50-100. Finally, the weighted score of each historical electronic fence i is calculated.

[0163]

[0164] After acquiring candidate historical electronic fences and their corresponding candidate frames, the 2D features and 3D point clouds corresponding to the candidate frames are aggregated. The 2D features corresponding to the current input image are extracted as the current frame. The 3D and 2D matching relationship between the candidate frames and the current frame can be obtained through one or more methods such as feature matching, reprojection, and outlier removal. Then, the pose T of the current frame in the target electronic fence is calculated using algorithms such as PNP. wh If the pose calculation is successful, the first retrieval in the historical electronic fence is confirmed as successful. The local coordinates T of the current frame can then be further calculated as shown in Formula 1-2. ws Coordinate transformation relationship T between the target electronic fence and the target electronic fence hs .

[0165] Based on this candidate frame, the above method is used to solve the pose of three consecutive candidate frames. If all three frames can be solved successfully, and the calculated coordinate transformation relationship T with the target electronic fence is obtained... hs If the difference between the three frames is less than 5cm@5 degrees, and all three frames correspond to the same target electronic fence as the frame with the highest weighted score, then the historical electronic fence retrieval is considered successful. The average value of the transformation relationship matrix of these three frames is taken as the transformation matrix. The local coordinate system of the current frame is transformed and unified to the coordinate system of the electronic fence required by the current user, so that the candidate frame is aligned with the coordinate of the current frame. The historical electronic fence corresponding to the candidate frame is then loaded as the electronic fence required by the current user scenario.

[0166] T hs =T wh -1 *T ws -1 (1-2)

[0167] In some examples, both global descriptors and local feature extraction can be obtained through pre-trained AI models, or traditional feature extraction methods (such as ORB, SIFT, etc.) can be used directly, which is not required here.

[0168] If the historical geofence retrieval fails, a geofence needs to be automatically generated, such as... Figure 5 As shown in S2, the steps are as follows:

[0169] Step 2: Scene Reconstruction: The VR device performs pose estimation and 3D point cloud generation based on the acquired environmental images and IMU, and then performs planar recognition. At the same time, it can also detect spatial structure information such as mesh and 3D objects. After fusing and optimizing the above information, global spatial structure information is obtained.

[0170] This embodiment uses the generation of spatial structure information based on environmental images and IMU (Integrated Device Unit) as an example. SLAM (Simultaneous Localization and Mapping) performs real-time feature extraction and matching based on the input environmental images to obtain the matching relationship of planar features between images. Based on the original IMU data and the external parameters between the camera and the IMU, the camera's pose is estimated to obtain the original camera pose. Then, based on the camera pose and the planar feature matching relationship, a triangulation algorithm is used to generate 3D point clouds. Finally, the camera pose and 3D point clouds are jointly optimized. Based on the planar information, the horizontal and vertical planes in space are fitted according to the output 3D point cloud data, and the size of the detected planes is limited, such as the smallest recognizable plane being a 20cm*30cm rectangle, which avoids the influence of small planes on the output results.

[0171] VR can perform depth estimation based on pre-trained AI models, using environmental images as input. The depth estimation results, i.e., depth data, are then input into pre-trained mesh recognition and 3D object detection models to detect meshes and 3D objects, obtaining mesh recognition data and 3D object recognition information respectively.

[0172] Figure 6 These are 3D object recognition results in the same scene, such as... Figure 6 The images show the results of 3D point cloud, plane, mesh, and 3D object recognition in the same scene. The information fusion module takes the above information as input, performs global fusion optimization, and adjusts the spatial pose of the recognition plane. Figure 7 This is a schematic diagram of spatial information fusion during the automatic generation of electronic fences, such as... Figure 7 As shown in the figure: black dots represent 3D points, 1 represents a plane, 2 represents a mesh, and 3 represents the 3D object recognition result.

[0173] like Figure 6As shown, taking the plane corresponding to the upper surface of the cardboard box as an example, the plane equation is shown in Formula 1-3, and the 3D point p i Distance dp from the plane i As shown in Formula 1-4, the corner point m of each triangle in the Mesh result j Distance to the plane dm j Similarly, derived from formulas 1-4, the plane t above the cuboid for 3D object recognition is... k Distance dt k Thus, the plane and the nearby i 3D points, j triangular facets, and k 3D objects form a joint error equation. By optimizing the plane expression 1-5, the overall error e is minimized, thereby obtaining an optimized plane with higher accuracy.

[0174] Ax + By + Cz + D = 0 (A 2 +B 2 +C 2 =1) (1-3)

[0175] dp i =Ax i +By i +Cz i +D (1-4)

[0176]

[0177] Equations 1-3 show the equations for a plane, where A, B, C, and D are the coefficients of the equation representing the plane. The coordinates (x, y, z) of any 3D point on the plane conform to these equations. (3D point p) i The coordinates in three-dimensional space are (x) i ,y i ,z i If the distance from the 3D point to the plane corresponding to Formula 1-3 is calculated using Formula 1-4, then α, β, and γ in Formula 1-5 represent the error weight coefficients for the 3D point, Mesh triangle, and 3D object plane, respectively. In some embodiments, these coefficients can be 0.3, 0.3, and 0.4, respectively. The weights can also be adjusted based on the reliability of the 3D point, Mesh triangle, and 3D object plane. When the total error weight coefficients are 1, the weights for higher reliability can be increased accordingly.

[0178] The optimized spatial structure information is obtained through the above formula, and the electronic fence is generated based on this information, resulting in a more accurate electronic fence.

[0179] In some examples, to reduce the computational requirements, the above equations can be simplified to obtain optimized information. For example, when the spatial structure information consists of 3D point cloud and plane information, the plane equation can be obtained first through 1-3, then the error between the plane and the 3D point cloud can be calculated through 1-4, and finally the plane expression 1-6 can be optimized to obtain a smaller error e, thereby obtaining an optimized plane with higher accuracy.

[0180]

[0181] Step 3, Electronic Fence Generation: (e.g., ...) Figure 5 As shown in S3, the ground can be detected based on the plane output in step 2. At the same time, obstacles above the ground can be detected by combining the Mesh recognition data and other spatial structure information output in step 2. The outer facades of the obstacles are connected to generate the electronic fence boundary.

[0182] Figure 8 It is a detection effect image of a scene, such as Figure 8 As shown, six planes are obtained based on 3D point cloud detection in space, such as... Figure 8 In the scene shown, six planes can be detected based on 3D point clouds in space, labeled "1", "2", "3", "4", "5", and "6". "1", "3", and "6" are horizontal planes, while "2", "4", and "5" are vertical planes. Based on spatial height, horizontal plane "1" can be identified as the plane corresponding to the ground, i.e., the reference plane. Vertical planes "2", "4", and "5" can be obtained by connecting them with plane "1". Figure 9 The connected region shown in the left-middle image represents the boundary of the reference plane. Based on this, projecting the horizontal planes corresponding to "3" and "6" vertically downwards in space yields the obstacle region. Removing the intersection of the reference plane boundary and the obstacle region results in... Figure 9 The original result of the electronic fence shown in the right figure is as follows. Figure 9 This is the original rendering of the electronic fence.

[0183] The "3" and "6" in space can also be identified and projected based on Mesh recognition data and 3D object recognition information. This method has already fused and optimized them in step 2. Therefore, this step mainly uses the plane to illustrate the recognition of electronic fences.

[0184] It should be noted that after the electronic fence generated by this method is provided to the VR device, the device needs to determine whether the range and size of the electronic fence meet the user's needs. If it is determined that the electronic fence cannot meet the user's needs for the scene in activities such as games or sports, a prompt should be displayed in the program interface, and the user should be guided to another area to regenerate the electronic fence.

[0185] Step 4, Difference Detection: such as Figure 5 S4 can perform local comparisons between real-time spatial structures and existing spatial structures. The real-time spatial structure can be represented by the current frame, and the existing spatial structure can be represented by historical frames, with the existing spatial structure originating from the current geofence. When the detected difference exceeds a set threshold, the geofence is triggered to update. The main differences detected include, but are not limited to, the following examples:

[0186] 1. 3D point cloud differences: The ICP (Iterative Closest Point) algorithm is a commonly used point cloud matching algorithm. This method is used to calculate the overlap rate between the 3D point cloud in the current frame and the 3D point cloud in the same area in the historical frame. If the overlap rate is lower than a specified threshold, which is usually 80%, it can be determined that the environment has changed and the electronic fence needs to be updated.

[0187] 2. Planar Difference: If the area of ​​a plane in the current frame's field of view exceeds a set threshold, such as a scaling of more than 10%, it indicates a change in the environment, requiring an update to the electronic fence. Similarly, if the pose difference exceeds a predetermined threshold, such as 10cm@3 degrees, it indicates a change in the environment, requiring an update to the electronic fence.

[0188] 3. Mesh Difference: In the mesh detected in the current frame's field of view, the threshold is set as follows: if some triangular facets are more than 10cm away from the original triangular facets, and the area of ​​the connected triangular facets exceeds 20cm*30cm, that is, if there are objects with a short side exceeding 20cm and a long side exceeding 30cm, it can be determined that the environment has changed and the electronic fence needs to be updated.

[0189] 4. 3D Object Differences: When the size or pose of a 3D object detected in the current frame exceeds the set threshold, such as a size change greater than 10cm or a scaling greater than 10%, or a pose change greater than 10cm@3 degrees, it is considered that the map has changed and the electronic fence update function needs to be triggered.

[0190] In particular, for security reasons, the threshold for removing existing spatial structures should be more stringent than that for adding or updating spatial structures. To avoid the impact of false detections, the electronic fence should only be updated after at least two consecutive frames of the same difference have been detected.

[0191] Figure 10 This is a screenshot showing the detection results after adding a new object to the scene, such as... Figure 10As shown, once the original electronic fence has been established, users can use VR devices within the safe area of ​​the electronic fence. During the user's use, the latest spatial structure information can be obtained in real time and compared with the spatial structure information of the corresponding safe area of ​​the existing electronic fence. When the difference exceeds the threshold, the electronic fence is updated.

[0192] by Figure 10 Taking the scenario shown as an example, after the initial electronic fence is established, there is a difference between the real-time spatial structure obtained by real-time detection and the spatial structure corresponding to the existing electronic fence, that is, a new vertical plane "7" is added. This vertical plane is located in the security area corresponding to the existing electronic fence. At this time, the address fence update should be triggered.

[0193] Step 5, Electronic Fence Update: (e.g., ...) Figure 5 As shown in S5, the real-time spatial structure is used as input, and the difference areas in the historical spatial structure are merged. After global optimization, the electronic fence is regenerated. If the spatial structure corresponding to the existing electronic fence has changed as detected in step 4 above, the electronic fence update function will be triggered.

[0194] by Figure 10 Taking the scenario shown as an example, after the initial electronic fence is established, there is a difference between the real-time spatial structure and the spatial result corresponding to the existing electronic fence: a new vertical plane "7" has been added. This vertical plane is located within the security area corresponding to the existing electronic fence, and at this time, the address fence update should be triggered.

[0195] After the spatial structure information is fused, global optimization is performed based on the weight values ​​corresponding to each spatial structure information. The optimization method can refer to the example above. Step 3 is then reprocessed on the changed objects to complete the update of the electronic fence. The updated electronic fence is as follows: Figure 11 As shown, Figure 11 This is a rendering of the updated electronic fence.

[0196] Step 6: Save the electronic fence: (e.g.) Figure 5 As shown in S6, before the electronic fence is generated for the first time, before the electronic fence is updated, or before exiting the program, the electronic fence and its corresponding information are saved for future use, such as when performing the electronic fence retrieval in step 1, and can be used as historical electronic fences.

[0197] During the use of this method, the electronic fence and its corresponding information will be saved periodically, and will also be saved before the program exits. The VR device can retrieve the electronic fence and its corresponding information as needed during startup and operation: when the device starts up, the information is retrieved for use in step 1; during device operation, the information is retrieved for use in step 4 to detect differences.

[0198] Generally, data compression is performed before saving spatial structures, and the number of stored geofences and spatial structures is limited, such as setting a maximum of 10 historical geofences to reduce memory usage.

[0199] In some examples, VR devices can be configured with TOF sensors. Figure 12 This is another schematic diagram of the automatic generation process for electronic fences, such as... Figure 12 When VR devices generate 3D point clouds, mesh recognition data, and 3D object recognition information, incorporating TOF data acquired by a TOF sensor can significantly improve the accuracy of the results.

[0200] A SLAM system that generates 3D point clouds, after inputting depth data, obtains the depth value corresponding to each pixel in the planar environment image by aligning the depth map with the planar environment image. Using this depth value as the initial depth value for the triangulation algorithm effectively improves the accuracy of the 3D point cloud. It also makes the 3D point cloud output by the SLAM system denser, describing the position, size, and contour of spatial objects with higher accuracy. Since the depth data output by a Time-of-Flight (TOF) sensor is far more accurate than general depth estimation results, the depth map output by the TOF sensor can be used as depth data. Adding depth data during planar detection, mesh recognition, and 3D object recognition can effectively avoid misidentification, further improving the security of the resulting electronic fence.

[0201] In some examples, since the aforementioned method of obtaining electronic fences using multiple spatial structure information has high requirements for algorithms and is relatively complex, considering the limited computing power of current VR devices, it is possible to simplify the aforementioned examples to minimize the device's computing power requirements while ensuring the main functions. Figure 13 This is another schematic diagram of the automatic generation process for electronic fences, such as... Figure 13 As shown, except for step 2 (scene reconstruction) and step 4 (difference detection), the implementation methods of this embodiment are basically the same as those in Embodiment 1. The specific differences are as follows:

[0202] Step 2, Scene Reconstruction: Only the results of the 3D point cloud detected on the plane are used as the basis for generating the electronic fence in Step 3.

[0203] Step 4, Difference Detection: Using real-time 3D point clouds as input, register them with historical 3D point clouds. The real-time 3D point clouds are obtained from real-time acquired environmental images, while the historical 3D point clouds are those generated when the electronic fence was created. Calculate the difference between the current frame's 3D point cloud and the historical 3D point clouds in the same region. When the difference exceeds a set value, trigger an electronic fence update. Alternatively, use the ICP algorithm to calculate the overlap rate between the current frame's 3D point cloud and the historical 3D point cloud. If the overlap rate is lower than a set value, typically 80%, the electronic fence needs to be updated.

[0204] In some examples, taking into account the different needs of different VR devices for electronic fence detection solutions, different types of spatial structure information combinations can be used for electronic fence detection and generation. Figure 14 This is another schematic diagram of the automatic generation process for electronic fences, such as... Figure 14 As shown, the main ways to combine spatial structure information are:

[0205] The spatial structure information consists of 3D point clouds and 3D object recognition information. The reference plane corresponding to the ground can be identified using the 3D point cloud. Then, based on the 3D object recognition information, 3D objects in space that are more than 10cm away from the ground are projected onto the plane to determine the obstacle area. After shearing the obstacle area, the safe area is obtained, and the boundary of the safe area is the electronic fence. During difference detection, it is only necessary to compare the real-time acquired 3D point cloud and 3D object recognition information in the current frame with the corresponding spatial structure information when the electronic fence was generated. The difference value is used to determine whether an update is needed. The comparison method is as described in the above embodiments.

[0206] The spatial structure information consists of mesh recognition data and 3D object recognition information. Mesh recognition data is used to first identify the corresponding triangular facets on the ground. Connecting these triangular facets yields the initial ground region, i.e., the reference plane. Based on this, triangular facets and 3D objects exceeding 10cm from the ground are projected onto the ground facets using 3D object recognition information to determine obstacle areas. After shearing the obstacle areas, the safe area is obtained, thus generating the electronic fence. Difference detection can also be performed by comparing the mesh recognition data and 3D object recognition information in the current frame with the corresponding spatial structure information used when the electronic fence was generated. The difference value determines whether an update is needed, and the comparison method is as described in the above embodiments.

[0207] The method of generating and updating electronic fences by combining different spatial structure information can meet the customized needs of different hardware and equipment manufacturers for electronic fence detection methods, and can further improve the versatility of this method on devices with different performance and configurations.

[0208] Figure 15This is a structural block diagram of an automatic electronic fence generation device according to an embodiment of this application. The automatic electronic fence generation device 200 is as described in the above example. Figure 15 As shown, it includes: input module 201 and processing module 202.

[0209] The input module 201 is used to acquire environmental images, which are obtained by taking pictures of the scene where the user is located.

[0210] For example, input module 201 can use virtual reality technology (VR glasses) or a camera to capture environmental images of the user's surroundings.

[0211] The processing module 202 is used to generate spatial structure information based on the environmental image and to generate an electronic fence based on the spatial structure information.

[0212] The spatial structure information includes 3D point cloud and information about at least one plane, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane, and the processing module 202 generates the electronic fence according to the method of generating electronic fence by 3D point cloud and plane in the example above.

[0213] In some examples, at least one plane includes at least one horizontal plane and other planes, which may be horizontal or vertical planes.

[0214] Furthermore, at least one plane includes at least one horizontal plane and at least one vertical plane.

[0215] In some examples, the input module 201 is also used to acquire measurement unit (IMU) data, which is obtained by IMU calculation of objects in the user's scene.

[0216] The processing module 202 is also used to generate spatial structure information based on environmental images and IMU data.

[0217] In some examples, the spatial structure information includes only 3D point cloud and planar information.

[0218] The processing module 202 is specifically used to construct optimization equations based on 3D point cloud and planar information; optimize the planar according to the optimization equations to obtain an optimized planar; determine a reference plane based on the optimized planar corresponding to the ground; project the optimized planar representing the boundary onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thus obtaining a reference horizontal plane with a boundary corresponding to the ground; project the optimized planar representing obstacles onto the reference horizontal plane in the direction of gravity to determine the obstacle area of ​​the reference horizontal plane; and generate an electronic fence based on the reference horizontal plane and the obstacle area.

[0219] Figure 16 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application, such as... Figure 16As shown, the processing module 202 also includes a 3D point cloud processing unit 2021, an electronic fence generation unit 2022, a plane detection unit 2023, a pose estimation unit 2024, a depth estimation unit 2025, a mesh recognition unit 2026, and a 3D object recognition unit 2027.

[0220] The pose estimation unit 2024 is used to obtain pose information based on the environmental image and the IMU;

[0221] The 3D point cloud processing unit 2021 is also used to obtain 3D point clouds based on environmental images and IMU;

[0222] The plane detection unit 2023 is also used to obtain a plane from the pose information and 3D point cloud based on plane detection;

[0223] The depth estimation unit 2024 is used to perform depth detection based on the environmental image to obtain depth data;

[0224] Mesh recognition unit 2025 is used to process mesh recognition data based on depth data;

[0225] The 3D object recognition unit 2026 is used to process 3D object recognition information based on depth data.

[0226] The electronic fence generation unit 2022 is also used to generate electronic fences based on 3D point clouds, planar and depth-processed Mesh recognition data and 3D object recognition information.

[0227] In some examples, the automatic electronic fence generation device 200 can identify the reference plane corresponding to the ground through the 3D point cloud processing unit 2021 or the Mesh recognition unit 2025, and then the 3D object recognition unit 2026 determines the reference plane boundary and obstacle area by determining the projection of 3D objects that are more than a first threshold away from the ground based on the 3D object recognition information. Finally, the electronic fence generation unit is also used to generate an electronic fence based on the reference plane, the reference plane boundary and the obstacle area.

[0228] In some examples, the depth estimation module is a TOF sensor, and the depth data is time-of-flight TOF data.

[0229] In some examples, the electronic fence generation unit 2022 is also used to generate an electronic fence centered on a planar point of the user's location in the scene and aligned with the coordinate system of the environment image.

[0230] Figure 17 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application, such as... Figure 17 As shown, the device 200 also includes: a storage module 203

[0231] Storage module 203 is used to save the electronic fence as a historical electronic fence.

[0232] Figure 18 This is a structural block diagram of another automatic electronic fence generation device according to an embodiment of this application, such as... Figure 18 As shown, the processing module 202 also includes a historical electronic fence retrieval unit 2028, which is used to retrieve environmental images from the historical electronic fences already saved in the storage module 203. If a historical electronic fence with similarity is obtained, the weighted score of the historical electronic fence is calculated, and the historical electronic fence with the highest weighted score is determined as the target electronic fence.

[0233] If the pose estimation unit 2024 successfully calculates the pose of the target electronic fence, the electronic fence generation unit 2022 sets the target electronic fence according to the pose difference so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

[0234] Figure 19 This is a structural block diagram of a real-time detection electronic fence device according to an embodiment of this application, such as... Figure 19 As shown, device 300 includes an input module 301 and an electronic fence update module 302. This device 300 can be integrated with any of the devices 200 described above, but is not limited to the modules included in this embodiment.

[0235] The input module 301 is also used to periodically collect real-time environmental images of the scene where the user is located.

[0236] The electronic fence update module 302 is used to update the electronic fence based on the spatial structure information generated from the real-time environmental image if the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold.

[0237] The device provided in this invention realizes the automatic detection and generation of electronic fences, solves the limitation of VR device electronic fences on the site, and also reduces the learning cost of users manually planning electronic fences, which can significantly improve the user experience of VR devices.

[0238] After the electronic fence is generated, this method will also compare the differences and changes of objects in the space in real time. Through difference detection, fusion optimization and other methods, the electronic fence will be automatically updated. The real-time updated electronic fence has high timeliness and can significantly improve the security during use.

[0239] The virtual device provided in this embodiment of the invention can be integrated into the electronic device in the foregoing embodiment, and the method of the above embodiment can be used to generate an electronic fence.

[0240] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0241] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

Claims

1. A method for automatically generating electronic fences, characterized in that, include: Acquire an environmental image, which is a photograph taken of the scene in which the user is located; Spatial structure information is generated based on the environmental image. The spatial structure information includes 3D point cloud and information about at least one plane, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane. An electronic fence is generated based on the spatial structure information; The generation of an electronic fence based on the spatial structure information includes: An optimization equation is constructed based on the information from the 3D point cloud and the plane. The optimized plane is obtained by optimizing the plane according to the optimization equation. Determine the reference plane based on the optimized plane corresponding to the ground; The optimized plane representing the boundary is projected onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thus obtaining a reference horizontal plane with a boundary corresponding to the ground. The obstacle region on the reference horizontal plane is determined by projecting the optimized plane representing the obstacle onto the reference horizontal plane in the direction of gravity. The electronic fence is generated based on the reference horizontal plane and the obstacle area; Alternatively, when the spatial structure information further includes pose information, depth data, mesh recognition data, and 3D object recognition information, the method further includes the following steps before generating the spatial structure information based on the environmental image: Acquire measurement unit (IMU) data, which is obtained by IMU calculation of objects in the user's scene; The step of generating spatial structure information based on the environmental image includes: The spatial structure information is generated based on the environmental image and the IMU data; The step of generating spatial structure information based on the environmental image and the IMU data, and generating an electronic fence based on the spatial structure information, includes: The pose information and the 3D point cloud are obtained based on the environmental image and the IMU data; The plane is obtained by performing plane detection on the pose information and the 3D point cloud; The depth data is obtained by performing depth detection based on the environmental image; The mesh recognition data and the 3D object recognition information are processed based on the depth data; The reference plane and obstacle region are obtained based on the 3D point cloud; Based on the information of the plane, the mesh recognition data after depth data processing, and the 3D object recognition information, the obstacle region is optimized to obtain an optimized obstacle region; An electronic fence is generated based on the reference plane and the optimized obstacle area.

2. The method according to claim 1, characterized in that, The at least one plane includes at least one horizontal plane and other planes, the other planes including horizontal planes or vertical planes.

3. The method according to claim 1, characterized in that, The environmental images are obtained by capturing the user's surroundings using virtual reality (VR) glasses or smart devices with cameras.

4. The method according to claim 1, characterized in that, The depth data is Time-of-Flight (TOF) data.

5. The method according to any one of claims 1 to 4, characterized in that, The generation of the electronic fence includes: The electronic fence is generated by centering on a planar point representing the user's location in the scene and aligning it with the coordinate system of the environmental image.

6. The method according to any one of claims 1 to 4, characterized in that, After generating the electronic fence, the process also includes: Save the electronic fence as a historical electronic fence.

7. The method according to claim 6, characterized in that, After acquiring the environmental image, the process further includes: The environmental image is retrieved from the saved historical electronic fences. If a historical electronic fence with similarity is obtained, the weighted score of the historical electronic fence is calculated, and the historical electronic fence with the highest weighted score is determined as the target electronic fence. The pose of the target electronic fence is solved. If the solution is successful, the target electronic fence is set according to the difference in pose so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

8. A method for automatically generating electronic fences, characterized in that, include: Acquire an environmental image, which is a photograph taken of the scene in which the user is located; Spatial structure information is generated based on the environmental image, and the spatial structure information includes 3D point cloud and 3D object recognition information. Identify the reference plane corresponding to the ground based on the 3D point cloud; Based on the 3D object recognition information, the projection of 3D objects that are more than a first threshold above the ground onto the reference plane is determined, and the boundary of the reference plane and the obstacle area are determined. An electronic fence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area.

9. A method for automatically generating electronic fences, characterized in that, include: Acquire an environmental image, which is a photograph taken of the scene in which the user is located; Spatial structure information is generated based on the environmental image, and the spatial structure information includes mesh recognition data and 3D object recognition information. The reference plane corresponding to the ground is identified based on the Mesh; Based on the 3D object recognition information, the projection of 3D objects that are more than a first threshold above the ground onto the reference plane is determined, and the boundary of the reference plane and the obstacle area are determined. The electronic fence is generated based on the reference plane, the boundary of the reference plane, and the obstacle area.

10. A method for real-time detection of electronic fences, characterized in that, include: Periodically collect real-time environmental images of the user's location; If the difference between the spatial structure information corresponding to the real-time environment image and the existing spatial structure information exceeds a second threshold, the electronic fence is updated based on the spatial structure information generated from the real-time environment image. The electronic fence is generated using the method described in any one of claims 1 to 9.

11. An automatic electronic fence generation device, characterized in that, include: The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located. A processing module is configured to generate spatial structure information based on the environmental image, the spatial structure information including 3D point cloud and information of at least one plane, wherein the plane is determined by the distribution of points of the 3D point cloud on the plane; The processing module is specifically used to construct an optimization equation based on the information of the 3D point cloud and the plane. The plane is optimized according to the optimization equation to obtain an optimized plane; a reference plane is determined based on the optimized plane corresponding to the ground. The optimized plane representing the boundary is projected onto the reference plane in the direction of gravity to determine the boundary of the reference plane, thus obtaining a reference horizontal plane with a boundary corresponding to the ground. The obstacle region on the reference horizontal plane is determined by projecting the optimized plane representing the obstacle onto the reference horizontal plane in the direction of gravity. The electronic fence is generated based on the reference horizontal plane and the obstacle area; Alternatively, when the spatial structure information further includes pose information, depth data, mesh recognition data, and 3D object recognition information, the processing module includes a pose estimation unit, a 3D point cloud processing unit, a plane detection unit, a depth estimation unit, a mesh recognition unit, an electronic fence generation unit, and a 3D object recognition unit. The input module is also used to acquire measurement unit (IMU) data, which is obtained by IMU calculation of objects in the user's scene. The processing module is further configured to generate the spatial structure information based on the environmental image and the IMU data; The pose estimation unit is used to obtain the pose information based on the environmental image and the IMU data; The 3D point cloud processing unit is used to obtain the 3D point cloud based on the environmental image and the IMU data; The plane detection unit is used to perform plane detection on the pose information and the 3D point cloud to obtain the plane; The depth estimation unit is used to obtain the depth data by performing depth detection based on the environmental image; The Mesh recognition unit is used to process the Mesh recognition data based on the depth data; The 3D object recognition unit is used to process the 3D object recognition information based on the depth data; The electronic fence generation unit is used to obtain a reference plane and obstacle area based on the 3D point cloud; Based on the information of the plane, the mesh recognition data after depth data processing, and the 3D object recognition information, the obstacle region is optimized to obtain an optimized obstacle region; An electronic fence is generated based on the reference plane and the optimized obstacle area.

12. The apparatus according to claim 11, characterized in that, The at least one plane includes at least one horizontal plane and other planes, the other planes including horizontal planes or vertical planes.

13. The apparatus according to claim 11, characterized in that, The input module is also used to obtain environmental images by taking pictures of the scene where the user is located.

14. The apparatus according to claim 11, characterized in that, The depth data is Time-of-Flight (TOF) data.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The electronic fence generation unit is also used to generate the electronic fence by centering on a plane point at the user's location in the scene and aligning it with the coordinate system of the environment image.

16. The apparatus according to any one of claims 11 to 14, characterized in that, The device further includes: A storage module is used to save the electronic fence as a historical electronic fence.

17. The apparatus according to claim 16, characterized in that, The processing module also includes a historical electronic fence retrieval unit: The historical electronic fence retrieval unit is used to retrieve the environmental image from the historical electronic fences already saved in the storage module. If a historical electronic fence with similarity is obtained, the weighted score of the historical electronic fence is calculated, and the historical electronic fence with the highest weighted score is determined as the target electronic fence. If the pose estimation unit successfully calculates the pose of the target electronic fence, the electronic fence generation unit sets the target electronic fence according to the difference in pose, so that the coordinate system of the target electronic fence is aligned with the coordinate system of the environmental image.

18. An automatic electronic fence generation device, characterized in that, include: The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located. The processing module is used to generate spatial structure information based on the environmental image, the spatial structure information including 3D point cloud and 3D object recognition information; identify a reference plane corresponding to the ground based on the 3D point cloud; and determine the reference plane boundary and obstacle area based on the projection of 3D objects at a distance exceeding a first threshold from the ground on the reference plane according to the 3D object recognition information. The processing module is also used to generate an electronic fence based on the reference plane, the boundary of the reference plane, and the obstacle area.

19. An automatic electronic fence generation device, characterized in that, include: The input module is used to acquire environmental images, which are captured by photographing the scene where the user is located. The processing module is used to generate spatial structure information based on the environmental image, the spatial structure information including mesh recognition data and 3D object recognition information; identify a reference plane corresponding to the ground based on the mesh; and determine the reference plane boundary and obstacle area based on the projection of 3D objects at a distance exceeding a first threshold from the ground on the reference plane according to the 3D object recognition information. The processing module is also used to generate an electronic fence based on the reference plane, the boundary of the reference plane, and the obstacle area.

20. A real-time detection device for electronic fences, characterized in that, include: The input module is also used to periodically collect real-time environmental images of the user's scene; An electronic fence update module is used to update the electronic fence based on the spatial structure information generated from the real-time environmental image if the difference between the spatial structure information corresponding to the real-time environmental image and the existing spatial structure information exceeds a second threshold. The electronic fence is generated using the method described in any one of claims 1 to 9.

21. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 9.

23. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, performs the method according to any one of claims 1 to 9.

Images