UWB-based AR glasses safety detection method, device and equipment and medium

Abstract

The embodiment of the invention relates to a UWB-based AR glasses safety detection method and device, equipment and a medium, and the method is applied to AR glasses and comprises the steps that communication information sent by target equipment within a first distance is received, and when the distance between the AR glasses and the target equipment is a second distance, the AR glasses obtain a plurality of images including the target equipment based on a plurality of shooting angles; communication is carried out based on the UWB tag of the AR glasses and at least one UWB base station of the target equipment to determine a first distance and a second distance, the second distance is smaller than the first distance, a three-dimensional model corresponding to the target equipment is generated based on the multiple images, relative coordinates of a living object are obtained based on communication information, and the position of the living object is identified in the three-dimensional model based on the relative coordinates of the living object. And displaying the three-dimensional model including the position of the living object on AR glasses. Based on AR glasses, through UWB positioning and distance measurement and image data acquisition, graph construction of a three-dimensional model is realized, so that a user can visually see the three-dimensional model and obtain the position information of a living object, and the use safety of target equipment is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a UWB-based AR glasses safety detection method, device, equipment and medium. Background Art

[0002] Usually, when using the equipment, you need to pay attention to safety issues. For example, before starting the vehicle, you need to detect whether there are animals or other living things around the vehicle.

[0003] In the related art, the user can use the device to ensure that the device environment is safe by observing with his eyes. However, this method is prone to omissions and is difficult to determine the location of living things. It has technical problems such as low efficiency and insufficient accuracy. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a UWB (Ultra Wide Band)-based AR (Augmented Reality) glasses safety detection method, device, equipment and medium.

[0005] The present disclosure provides a UWB-based AR glasses safety detection method, which is applied to AR glasses and includes:

[0006] When receiving communication information sent by a target device within a first distance and being at a second distance from the target device, the AR glasses acquire multiple images including the target device based on multiple shooting angles; wherein the first distance and the second distance are determined based on communication between a UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is less than the first distance;

[0007] Generate a three-dimensional model corresponding to the target device based on the multiple images, obtain relative coordinates of the living thing based on the communication information, and identify the position of the living thing in the three-dimensional model based on the relative coordinates of the living thing;

[0008] A three-dimensional model including the position of the living thing is displayed on the AR glasses.

[0009] Optionally, generating a three-dimensional model corresponding to the target device based on the multiple images includes:

[0010] Based on the matching of the multiple images in a preset large database, the device identification of the target device is obtained, and the original device information corresponding to the device identification is obtained;

[0011] Based on the original device information, obtaining a three-dimensional wireframe diagram of the target device;

[0012] Identify key feature points based on the multiple images to obtain multiple key feature points, and convert the multiple key feature points into point cloud data to generate initial point cloud data;

[0013] Compensating the initial point cloud data based on a preset artificial intelligence algorithm to obtain target point cloud data;

[0014] The target point cloud data is mapped to the three-dimensional wireframe diagram and point cloud stitching is performed to obtain the three-dimensional model.

[0015] Optionally, acquiring relative coordinates of the living thing based on the communication information, and identifying a position of the living thing in the three-dimensional model based on the relative coordinates of the living thing includes:

[0016] Acquire target device absolute position information and living thing position information of the target device based on the communication information;

[0017] Determine the relative position coordinates between the target device and the living thing as the living thing relative coordinates based on the absolute position information of the target device and the position information of the living thing;

[0018] The position of the living thing in the three-dimensional model is determined based on the relative coordinates of the living thing and marked as the position of the living thing.

[0019] Optionally, the UWB-based AR glasses safety detection method further includes:

[0020] Prompt information is generated based on the living thing detection information in the communication information, and the prompt information is played based on the AR glasses.

[0021] The embodiment of the present disclosure provides another UWB-based AR glasses security detection method, which is applied to a target device and includes:

[0022] Communication information is sent to an AR glasses within a first distance so that the AR glasses receive the communication information, obtain relative coordinates of the living thing based on the communication information, identify the position of the living thing in a three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

[0023] Optionally, the UWB-based AR glasses safety detection method further includes:

[0024] transmitting a UWB pulse signal based on at least one UWB base station of the target device and receiving a UWB pulse response signal;

[0025] Based on the presence of a pulse peak at the target position of the UWB pulse response signal, determining that there is a living thing at the target position, and recording living thing detection information and living thing position information;

[0026] The communication information including the living thing detection information and the living thing position information is generated.

[0027] The present disclosure also provides an AR glasses safety detection device based on UWB, which is applied to AR glasses, including:

[0028] A receiving module, used to receive communication information sent by a target device within a first distance;

[0029] a shooting module, configured to, when the AR glasses are at a second distance from the target device, acquire a plurality of images including the target device based on a plurality of shooting angles; wherein the first distance and the second distance are determined based on communication between a UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is less than the first distance;

[0030] A generating module, configured to generate a three-dimensional model corresponding to the target device based on the multiple images;

[0031] an identification module, used for acquiring relative coordinates of the living thing based on the communication information, and identifying a position of the living thing in the three-dimensional model based on the relative coordinates of the living thing;

[0032] A display module is used to display a three-dimensional model including the position of the living thing on the AR glasses.

[0033] The embodiment of the present disclosure also provides another UWB-based AR glasses safety detection device, which is applied to a target device, including:

[0034] A sending module is used to send communication information to the AR glasses within a first distance, so that the AR glasses receive the communication information, obtain the relative coordinates of the living thing based on the communication information, and identify the position of the living thing in the three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

[0035] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the UWB-based AR glasses safety detection method provided in the embodiment of the present disclosure.

[0036] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the UWB-based AR glasses safety detection method provided in the embodiments of the present disclosure.

[0037] The present disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the UWB-based AR glasses safety detection method described in the aforementioned aspect.

[0038] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: the UWB-based AR glasses safety detection solution provided by the embodiment of the present disclosure is applied to AR glasses, including: receiving communication information sent by a target device within a first distance, and when the target device is at a second distance, the AR glasses obtain multiple images including the target device based on multiple shooting angles; the first distance and the second distance are determined by communicating with the UWB tag of the AR glasses and at least one UWB base station of the target device, the second distance is less than the first distance, a three-dimensional model corresponding to the target device is generated based on multiple images, and the relative coordinates of the living creature are obtained based on the communication information, the position of the living creature is identified in the three-dimensional model based on the relative coordinates of the living creature, and the three-dimensional model including the position of the living creature is displayed on the AR glasses. Based on the AR glasses using UWB positioning and ranging, as well as image data acquisition, the graphic construction of the three-dimensional model is realized, so that the user can intuitively see the three-dimensional model and obtain the position information of the living creature, thereby improving the safety of the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A schematic diagram of a UWB-based AR glasses safety detection method provided in an embodiment of the present disclosure;

[0041] Figure 2 Another example diagram of UWB-based AR glasses safety detection provided in an embodiment of the present disclosure;

[0042] Figure 3 Another example diagram of UWB-based AR glasses safety detection provided in an embodiment of the present disclosure;

[0043] Figure 4 An example diagram of a three-dimensional model display provided in an embodiment of the present disclosure;

[0044] Figure 5 A schematic diagram of the structure of a UWB-based AR glasses safety detection device provided in an embodiment of the present disclosure;

[0045] Figure 6 A schematic diagram of the structure of another UWB-based AR glasses safety detection device provided in an embodiment of the present disclosure;

[0046] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0048] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0049] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0050] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0051] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0052] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0053] In order to solve the technical problems that the existing method of using the device to ensure the safety of the device environment by observing with the user's eyes is prone to omissions and it is difficult to determine the location of living things, and there is low efficiency and insufficient accuracy, the present invention proposes to use AR glasses to use UWB positioning and ranging, as well as image data collection, to achieve graphic construction of three-dimensional models, so that users can intuitively see the three-dimensional model and obtain the location information of living things, thereby improving the safety of use of the target device.

[0054] In the disclosed embodiments, AR augmented reality technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0055] In the disclosed embodiments, UWB ultra-wideband technology is a new communication technology that is very different from traditional communication technology. It does not need to use the carrier in the traditional communication system, but transmits data by sending and receiving extremely narrow pulses with nanoseconds or less, so it has a bandwidth of the order of GHz. Therefore, it has the advantages of strong penetration, low power consumption, good anti-multipath effect, high security, low system complexity, and can provide precise positioning accuracy. Therefore, ultra-wideband technology can be applied to the positioning, tracking and navigation of indoor and outdoor stationary or moving objects and people, and can provide very precise positioning accuracy.

[0056] Figure 1 The present invention provides a flowchart of a UWB-based AR glasses safety detection method, which can be performed by a UWB-based AR glasses safety detection device, wherein the device can be implemented by software and / or hardware, and can generally be integrated into an electronic device. Figure 1 As shown, the method is applied to AR glasses, including:

[0057] Step 101: receiving communication information sent by a target device within a first distance, and when the distance from the target device is a second distance, the AR glasses acquire multiple images including the target device based on multiple shooting angles; communicating between a UWB tag of the AR glasses and at least one UWB base station of the target device to determine the first distance and the second distance, and the second distance is smaller than the first distance.

[0058] In the disclosed embodiment, the AR glasses have their own camera module, utilize integrated high-performance chips and advanced imaging technology to capture target devices (such as vehicles, medical equipment, etc.), identify them through AI (Artificial Intelligence) technology, and transform the target devices into digital 3D (3Dimensions) models.

[0059] In the disclosed embodiment, the AR glasses include an Eyebox, which is a screen for displaying the images of the AR glasses. When wearing the AR glasses, the human eye can see a small area of ​​the virtual image. The 3D modeling technology of the AR glasses can be understood as the use of AI vision technology and three-dimensional reconstruction algorithms to achieve accurate scanning and modeling of target devices, and support modeling of humans and animals.

[0060] In the embodiments of the present disclosure, the target device is usually detected in real time through at least one UWB base station, and the ranging information between the target device and the AR glasses, the living thing detection information (including the existence of living things and the location information of living things, etc.), and the absolute location information of the target device obtained through the positioning device of the target device such as GPS (Global Positioning System) can be integrated into communication information and sent to the AR glasses within the first distance.

[0061] In the embodiment of the present disclosure, the first distance and the second distance can be selected and set according to the specific application scenario, and usually the second distance is smaller than the first distance.

[0062] In the disclosed embodiment, at least one UWB base station may be set on the target device. For example, when the target device is a vehicle, a UWB base station may be set on each of the four wheels of the vehicle, the middle of the bottom of the vehicle, and / or the middle of the top of the vehicle.

[0063] In the disclosed embodiment, the UWB tag of the AR glasses communicates with at least one UWB base station of the target device to determine the first distance and the second distance, that is, the UWB tag provided on the AR glasses can measure the distance with the target device with the UWB base station within a certain distance. Specifically, the UWB ranging is a TOF (Time-of-Flight) ranging method, which is a two-way ranging technology. The distance between nodes is measured by using the flight time of the signal between two transceivers. In a line-of-sight environment, the UWB ranging is linearly related to the distance based on the TOF ranging method, and the measurement result is usually more accurate.

[0064] In an embodiment of the present disclosure, the AR glasses have a camera module, which is activated when the distance from the target device is a second distance, and obtains multiple images including the target device based on multiple shooting angles. For example, when the user configures the AR glasses to be at a second distance from the target device (such as a vehicle), the camera module is turned on to take continuous photos to obtain photos of the target device in front at multiple different angles; wherein the multiple angles can be understood as multiple different relative angles between the AR glasses and the target device during the user's movement.

[0065] Step 102: Generate a three-dimensional model corresponding to the target device based on the multiple images, obtain the relative coordinates of the living object based on the communication information, and identify the position of the living object in the three-dimensional model based on the relative coordinates of the living object.

[0066] Step 103: Display a three-dimensional model including the position of the living thing on the AR glasses.

[0067] In the disclosed embodiment, a plurality of images are captured by high-speed continuous shooting through the photographic module of the AR glasses, and the target model and the same target device information are obtained through graphic scanning and matching and a large database connected to the network, and the target device is converted into a three-dimensional wireframe diagram. The plurality of images are recognized to identify key feature points, and then the key feature points are merged on the corresponding three-dimensional wireframe diagram to form a grid outline, and finally simply rendered, and textures are added to create colors with a texture similar to the photo to generate a simple three-dimensional model.

[0068] In an embodiment of the present disclosure, after generating a three-dimensional model, the relative coordinates of the living thing are obtained based on the communication information, and the position of the living thing in the three-dimensional model is identified based on the relative coordinates of the living thing. That is to say, the relative position between the target device and the living thing, that is, the relative coordinates, can be obtained based on the communication information, and then based on the relative coordinates, the position of the living thing in the three-dimensional model can be determined, and the position of the living thing in the three-dimensional model can be identified, and finally the three-dimensional model including the position of the living thing is displayed on the AR glasses.

[0069] The UWB-based AR glasses safety detection solution provided by the disclosed embodiment is applied to AR glasses, including: receiving communication information sent by a target device within a first distance, and when the AR glasses are at a second distance from the target device, the AR glasses obtain multiple images including the target device based on multiple shooting angles; communicating with the UWB tag of the AR glasses and at least one UWB base station of the target device to determine the first distance and the second distance, the second distance is less than the first distance, generating a three-dimensional model corresponding to the target device based on multiple images, and obtaining the relative coordinates of the living creature based on the communication information, identifying the position of the living creature in the three-dimensional model based on the relative coordinates of the living creature, and displaying the three-dimensional model including the position of the living creature on the AR glasses. Based on the AR glasses using UWB positioning and ranging, as well as image data collection, the three-dimensional model is graphically constructed, allowing users to intuitively see the three-dimensional model and obtain the position information of the living creature, thereby improving the safety of the target device.

[0070] Figure 2 This is a flow chart of another UWB-based AR glasses safety detection method provided by the embodiment of the present disclosure. Based on the above embodiment, this embodiment further optimizes the above UWB-based AR glasses safety detection method. Figure 2 As shown, the method is applied to AR glasses, including:

[0071] Step 201: receiving communication information sent by a target device within a first distance, and when the AR glasses are at a second distance from the target device, acquiring multiple images including the target device based on multiple shooting angles.

[0072] It should be noted that step 201 is the same as step 101 , and please refer to the detailed description of step 101 for details.

[0073] Step 202: Based on the multiple images, matching is performed in a preset large database to obtain the device identification of the target device, and the original device information corresponding to the device identification is obtained. Based on the original device information, a three-dimensional wireframe diagram of the target device is obtained.

[0074] In the disclosed embodiment, a large database is pre-set to store image data and other information of many target devices. The device identification can be obtained by matching the image in the large database. The device identification can uniquely identify a target device, such as a vehicle signal, so that the original device information of the target device, such as the target device appearance information, size information, etc., can be obtained based on the device identification, thereby constructing a three-dimensional wireframe diagram of the target device based on the original device information.

[0075] Step 203: perform key feature point recognition based on multiple images to obtain multiple key feature points, and convert the multiple key feature points into point cloud data to generate initial point cloud data.

[0076] Step 204: Compensate the initial point cloud data based on a preset artificial intelligence algorithm to obtain target point cloud data, map the target point cloud data to a three-dimensional wireframe diagram, and perform point cloud stitching to obtain a three-dimensional model.

[0077] Specifically, multiple images are subjected to image recognition to identify the key anchor points of the target device outline, that is, the key feature points of the target device are identified, and the point cloud data layers of multiple images are merged and processed, and then the point clouds are stitched, and the point clouds of all other images are mapped to the three-dimensional wireframe position of the vehicle to perform point cloud stitching.

[0078] More specifically, multiple images, i.e., multiple RGB images, are used to identify key feature points, the two-dimensional key feature points are matched with the corresponding depth map point cloud, the external parameters are obtained based on the internal parameters, the point cloud of one image is converted to the point cloud of another image, and then the point clouds are stitched. The point clouds of all images are mapped to the camera coordinate system of the first image, and the point clouds are stitched to obtain a three-dimensional model.

[0079] It is understandable that there is no guarantee that the image can capture all angles of view of the target device, so the captured point cloud position is limited. It is necessary to use AI algorithms to compensate the point cloud according to the target device. In the large model, the graphics similar to the target device are matched, and point cloud compensation is performed to generate a complete point cloud, that is, the target point cloud data, to further improve the accuracy of obtaining the three-dimensional model.

[0080] In this way, three-dimensional surfaces are constructed, structured point cloud data is connected, and points, lines, and surfaces are generated. AR glasses are based on AR access algorithms and large models, and are equipped with independent computing units, so that their computing power can complete complex calculations and further improve processing efficiency.

[0081] Step 205: Obtain target device absolute position information and living thing position information of the target device based on the communication information; determine relative position coordinates between the target device and the living thing as living thing relative coordinates based on the target device absolute position information and the living thing position information; determine the position of the living thing in the three-dimensional model based on the living thing relative coordinates and mark it as the living thing position.

[0082] In the embodiments of the present disclosure, the communication information may include ranging information between the target device and the AR glasses, living thing detection information (including the presence of living things and the location information of living things, etc.), and absolute location information of the target device obtained through the positioning device of the target device such as GPS, etc.

[0083] Therefore, the target device absolute position information and the living thing position information of the target device are acquired based on the communication information, and then the relative position coordinates between the target device and the living thing are determined as the living thing relative coordinates based on the target device absolute position information and the living thing position information, and the position of the living thing in the three-dimensional model is determined based on the living thing relative coordinates and marked as the living thing position.

[0084] Step 206: Display a three-dimensional model including the location of the living thing on the AR glasses, generate prompt information based on the living thing detection information in the communication information, and play the prompt information based on the AR glasses.

[0085] In the disclosed embodiments, the AR glasses can obtain the distance between the AR glasses and the target device through TOF time-of-flight ranging between the UWB of the AR glasses and the UWB of the target device, and perform radar detection of living things such as animals or humans through the UWB of the target device.

[0086] Specifically, the UWB radar will emit a UWB pulse signal and receive the echo reflected by obstacles (such as kittens, puppies and other animals). By analyzing the echo disturbance, it can determine whether there are objects or people around. When there are people around the UWB radar, the channel pulse response will generate a pulse peak at the corresponding position, which can be used to detect information such as the breathing and heartbeat of living organisms, thereby realizing non-contact, long-distance living detection.

[0087] Specifically, the positioning function of UWB is very similar to the principle of satellites. Through the deployment of known coordinate positioning base stations, such as the UWB installed in the vehicle, the parking position is with coordinate information. Users wearing UWB AR glasses will emit pulses at a certain frequency and continuously measure distances with several base stations. Through the precise coordinates and flight time of multiple points, the position of the tag (AR glasses) is located. All the last parking points of the target device will have accurate GPS coordinate information, that is, the absolute position of the target device. The absolute position of the target device can be used to calculate the position coordinates of other devices (AR glasses, living things, etc.).

[0088] Specifically, the AR glasses acquire multiple images including the target device through multiple shooting angles, perform graphic scanning based on the multiple images, and access the large database connected to the network to acquire the target device identification such as the vehicle model and original device information such as the same model information, convert the target device such as the vehicle into a 3D wireframe, perform simple graphic monochrome rendering, and after processing, present it in the eye-tracking display area of ​​the AR glasses.

[0089] Specifically, the relative position information between the living thing and the target device is calculated and then marked on the rendering (three-dimensional model), thereby realizing the visualization of AR glasses safety detection and conveying visual and voice information to users.

[0090] In the disclosed embodiment, the three-dimensional model including the position of living things displayed on AR glasses can be displayed using the principle of diffraction light waveguide, that is, the diffraction characteristics of the grating are used to design the "light path" to allow light to propagate along the designed path and guide the light emitted by the micro-projection system into the human eye.

[0091] That is to say, after the optical machine completes the imaging process, the waveguide couples the light into its own glass substrate, transmits the light to the front of the eye through the principle of "total reflection" and then releases it. The visualized picture display of the embodiment of the present disclosure is displayed using the diffraction optical waveguide principle; more specifically, the diffraction tube optical waveguide-grating refraction principle controls the propagation path of light by utilizing the wave properties of light, that is, utilizing a periodic grating structure so that light of different wavelengths produces regular changes in propagation direction when passing through, thereby realizing directional propagation of light. The embodiment of the present disclosure uses the grating area for visualization to display the position information of 3D models and target objects such as living things.

[0092] It is understandable that photo modeling technology is based on technologies in fields such as computer vision and deep learning. By training deep neural networks, computers can automatically identify three-dimensional structures in photos and generate realistic 3D models based on these structural information. At the same time, the accuracy of the three-dimensional model can be improved by optimizing algorithms and improving model structures.

[0093] That is to say, the embodiments of the present disclosure use deep learning and other technical means to extract three-dimensional structure information from images and generate high-quality 3D models through AR glasses. At present, most AR glasses with independent algorithms and operations have limited computing power, ROM (Read-Only Memory) is not very large, can only support small-scale calculations, and RAM (Random Access Memory) is not very large. They need to rely on the computing power of mobile phones. When encountering more complex calculations and animations, they need powerful chips, and the processing time is relatively long, and there will be a certain time delay. Taking the AR glasses of the embodiments of the present disclosure, such as AR diffraction optical waveguide smart glasses, AR glasses support the integration of various high-computing chips, have the ability to independently perform complex calculations, hardware supports photography modules, supports BLE (Bluetooth Low Energy), and can also carry UWB inside; AR glasses can also support navigation, music and other multimedia applications, support AI voice interaction and speakers, and can act as TWS (True Wireless Stereo, true wireless headphones), and 3D graphic modeling visualization of AR glasses can use existing camera modules for data collection; UWB's positioning accuracy can reach centimeter level, and recognition is very accurate, whether it is recognition and data transmission.

[0094] Therefore, by taking advantage of the chip computing power of AR glasses, merging graphic layers and converting them into 3D models, combining the characteristics and advantages of AR glasses with the technology of vehicle-side UWB, high-precision positioning and flight ranging technology, high-frequency safety accidents that are prone to occur in life can be solved, further improving the safety of the use of target equipment.

[0095] In order to more comprehensively describe the UWB-based AR glasses security detection method disclosed in the present invention, the following description is based on the target device side. Figure 3 A flowchart of another UWB-based AR glasses safety detection method provided in an embodiment of the present disclosure is shown as follows: Figure 3 As shown, the method is applied to a target device, comprising:

[0096] Step 301: send communication information to the AR glasses within a first distance so that the AR glasses receive the communication information, obtain the relative coordinates of the living thing based on the communication information, identify the position of the living thing in the three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

[0097] In an embodiment of the present disclosure, when the AR glasses are at a second distance from the target device, the AR glasses acquire multiple images including the target device based on multiple shooting angles; wherein the first distance and the second distance are determined based on communication between the UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is smaller than the first distance.

[0098] In an embodiment of the present disclosure, the AR glasses generate a three-dimensional model corresponding to the target device based on multiple images, obtain the relative coordinates of the living thing based on the communication information, identify the position of the living thing in the three-dimensional model based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

[0099] In some embodiments, based on at least one UWB base station transmitting a UWB pulse signal and receiving a UWB pulse response signal, based on the presence of a pulse peak at a target position of the UWB pulse response signal, it is determined that there is a living thing at the target position, and the living thing detection information and the living thing location information are recorded to generate communication information including the living thing detection information and the living thing location information.

[0100] Based on the description of the foregoing embodiments, the AR diffraction optical waveguide AI glasses of the embodiments of the present disclosure support a high-computing-power MCU (Microcontroller Unit), have an independent computing unit, have a camera module, UWB, Bluetooth in hardware, support 4G network communication, and software supports photo taking, voice interaction, gesture interaction, and can quickly respond to user commands; the UWB base station of the embodiments of the present disclosure is installed on a target device with a UWB anchor point, and AR glasses wearing UWB.

[0101] As an example of a scenario, the target device is a vehicle, and there are stray cats resting near the vehicle, crawling on the vehicle engine, or sleeping on the vehicle tires. When a user wearing AR glasses approaches within 20 meters of the vehicle, the AR glasses receive the vehicle's Bluetooth broadcast signal, identify and connect. The user can see the real situation of the vehicle in front through the grating interface of the AR glasses, see the animal under the vehicle, and drive it away in time, avoiding a traffic accident involving an animal.

[0102] Among them, the camera of AR glasses obtains the picture information of the real scene, and after processing, it synthesizes virtual information such as text, images, videos, 3D models into the picture of the real scene, and finally displays it to the user through the display screen. This method relies on video synthesis technology to integrate virtual information with real scenes to provide a rich visual experience.

[0103] Among them, the AR diffraction waveguide grating area screen display includes taking the screen diagonal as the center coordinate to display a visualized screen, such as Figure 4 The horizontal viewing field and the vertical viewing field shown can determine the diagonal viewing field, and the three-dimensional model is displayed with the diagonal viewing field as the central coordinate.

[0104] The disclosed embodiment solves the safety accidents caused by blind spots by taking pictures and scanning with AR glasses, building a three-dimensional model, and AR graphics visualization.

[0105] As an example, collect pictures, AR glasses start continuous shooting and collect multi-angle images of the target device; image processing, use image processing algorithms to process the collected images; point cloud generation, use image processing algorithms to convert processed pictures into point cloud data, point cloud data is the basis of 3D modeling (generate point cloud after identifying key anchor points on the surface of the target device); point cloud processing, filter, merge, segment and other processing on point cloud data to improve the quality and accuracy of point cloud data; surface reconstruction: use surface reconstruction algorithm to convert point cloud data into triangular mesh model, and perform subsequent operations with edges; optimization and editing, optimize and edit the generated 3D model to achieve better effect and completeness, including model stitching, reducing the number of faces, and smoothing; export model, export the completed 3D model for confirming the target position; position calibration, mark the coordinate information and volume data of the target object (i.e. living thing) in the 3D model; visual feedback, display the model information of the vehicle, whether there are living things around the vehicle, and the orientation of the living things in the screen of AR glasses.

[0106] As an example of a usage scenario, step 1: When the user wears AR glasses to go to work and is about to drive, when the user is 20 meters away from the vehicle, the Bluetooth of the AR glasses is connected to the vehicle's Bluetooth, the vehicle's UWB starts real-time ranging, detects living information, and starts reporting (the vehicle has been in a detection state since it was turned off, and the user receives data immediately when he approaches the vehicle, and UWB ranging is performed between the AR glasses and the vehicle); step 2: There is a kitten sleeping on the tire of the vehicle, and the UWB radar detects the occlusion of living things through the fluctuation parameters; step 3: This information (UWB pulse information, ranging information, living thing detection information, coordinate information) is transmitted to the vehicle. information, etc.), which is transmitted to the approaching car owner's AR glasses through the Bluetooth channel; Step 4: When the AR glasses are about 10m away from the vehicle, the camera module of the AR glasses is activated, and continuous photography is started. The photos of the vehicle in front are merged and reorganized to create a grid outline and draw a rough 3D map; Step 5: Similarly, the coordinate information and volume of the living thing can also be calculated, and its position can be displayed in the 3D map; Step 6: The user sees the 3D graphic information and the position information of the living thing in the Eyebox screen of the AR glasses. At the same time, the AR glasses prompt the user through voice, "An object is detected in front, please check before starting the vehicle."

[0107] Therefore, by using the AI ​​algorithm of AR glasses and the support of high-performance flagship chips, UWB is used for positioning and ranging, the camera module is used for image data collection, and the graphic construction of the three-dimensional model is realized, so that users can intuitively see the three-dimensional model and grasp the information of the target object. In addition to being used in the field of vehicle safety detection visualization, it can also be used in the medical field, blind spot detection before driving and other scenarios. For example, in medicine, some complex equipment or machines that realize blind spots can see the full picture, and operation and observation will be more convenient; for blind spot detection, UWB is combined with three-dimensional model construction to prepare to know the people or pets near the vehicle, avoid safety accidents, and greatly improve safety.

[0108] Figure 5 This is a schematic diagram of the structure of a UWB-based AR glasses safety detection device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 5 As shown, the device is applied to AR glasses, including:

[0109] The receiving module 501 is used to receive communication information sent by a target device within a first distance;

[0110] A shooting module 502 is configured to, when the AR glasses are at a second distance from the target device, obtain a plurality of images including the target device based on a plurality of shooting angles; wherein the first distance and the second distance are determined based on communication between a UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is less than the first distance;

[0111] A generating module 503, configured to generate a three-dimensional model corresponding to the target device based on the multiple images;

[0112] An identification module 504, configured to obtain relative coordinates of the living thing based on the communication information, and identify a position of the living thing in the three-dimensional model based on the relative coordinates of the living thing;

[0113] The display module 505 is used to display a three-dimensional model including the position of the living thing on the AR glasses.

[0114] Optionally, the generating module 503 is specifically used for:

[0115] Based on the matching of the multiple images in a preset large database, the device identification of the target device is obtained, and the original device information corresponding to the device identification is obtained;

[0116] Based on the original device information, obtaining a three-dimensional wireframe diagram of the target device;

[0117] Identify key feature points based on the multiple images to obtain multiple key feature points, and convert the multiple key feature points into point cloud data to generate initial point cloud data;

[0118] Compensating the initial point cloud data based on a preset artificial intelligence algorithm to obtain target point cloud data;

[0119] The target point cloud data is mapped to the three-dimensional wireframe diagram and point cloud stitching is performed to obtain the three-dimensional model.

[0120] Optionally, the identification module 504 is specifically used to:

[0121] Acquire target device absolute position information and living thing position information of the target device based on the communication information;

[0122] Determine the relative position coordinates between the target device and the living thing as the living thing relative coordinates based on the absolute position information of the target device and the position information of the living thing;

[0123] The position of the living thing in the three-dimensional model is determined based on the relative coordinates of the living thing and marked as the position of the living thing.

[0124] Optionally, the UWB-based AR glasses safety detection device further includes:

[0125] A prompt generation module is used to generate prompt information based on the living thing detection information in the communication information, and play the prompt information based on the AR glasses.

[0126] Figure 6 This is a schematic diagram of the structure of another UWB-based AR glasses safety detection device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 6 As shown, the device is applied to a target device, including:

[0127] The sending module 601 is used to send communication information to the AR glasses within the first distance, so that the AR glasses receive the communication information, obtain the relative coordinates of the living thing based on the communication information, and identify the position of the living thing in the three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

[0128] Optionally, the UWB-based AR glasses safety detection device further includes: a processing module, which is used to:

[0129] transmitting a UWB pulse signal based on at least one UWB base station of the target device and receiving a UWB pulse response signal;

[0130] Based on the presence of a pulse peak at the target position of the UWB pulse response signal, determining that there is a living thing at the target position, and recording living thing detection information and living thing position information;

[0131] The communication information including the living thing detection information and the living thing position information is generated.

[0132] The UWB-based AR glasses safety detection device provided in the embodiments of the present disclosure can execute the UWB-based AR glasses safety detection method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0133] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the UWB-based AR glasses safety detection method provided by any embodiment of the present disclosure.

[0134] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The electronic device 700 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0136] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0137] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the UWB-based AR glasses safety detection method of the embodiment of the present disclosure are executed.

[0138] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0139] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0141] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: receives communication information sent by a target device within a first distance, and when it is at a second distance from the target device, the AR glasses obtain multiple images including the target device based on multiple shooting angles; determines the first distance and the second distance based on communication between the UWB tag of the AR glasses and at least one UWB base station of the target device, the second distance is smaller than the first distance, generates a three-dimensional model corresponding to the target device based on the multiple images, obtains the relative coordinates of the living thing based on the communication information, identifies the position of the living thing in the three-dimensional model based on the relative coordinates of the living thing, and displays the three-dimensional model including the position of the living thing on the AR glasses.

[0142] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0148] processor;

[0149] a memory for storing instructions executable by the processor;

[0150] The processor is used to read the executable instructions from the memory and execute the instructions to implement any UWB-based AR glasses safety detection method provided in the present disclosure.

[0151] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any UWB-based AR glasses safety detection method as provided in the present disclosure.

[0152] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0153] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0154] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A UWB-based AR glasses safety detection method, characterized in that: Applied to AR glasses, including: When receiving communication information sent by a target device within a first distance and being at a second distance from the target device, the AR glasses acquire multiple images including the target device based on multiple shooting angles; wherein the first distance and the second distance are determined based on communication between a UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is less than the first distance; Generate a three-dimensional model corresponding to the target device based on the multiple images, obtain relative coordinates of the living thing based on the communication information, and identify the position of the living thing in the three-dimensional model based on the relative coordinates of the living thing; A three-dimensional model including the position of the living thing is displayed on the AR glasses.

2. The UWB-based AR glasses safety detection method according to claim 1, characterized in that: The generating a three-dimensional model corresponding to the target device based on the multiple images includes: Based on the matching of the multiple images in a preset large database, the device identification of the target device is obtained, and the original device information corresponding to the device identification is obtained; Based on the original device information, obtaining a three-dimensional wireframe diagram of the target device; Identify key feature points based on the multiple images to obtain multiple key feature points, and convert the multiple key feature points into point cloud data to generate initial point cloud data; Compensating the initial point cloud data based on a preset artificial intelligence algorithm to obtain target point cloud data; The target point cloud data is mapped to the three-dimensional wireframe diagram and point cloud splicing is performed to obtain the three-dimensional model.

3. The UWB-based AR glasses safety detection method according to claim 1, characterized in that: The acquiring the relative coordinates of the living thing based on the communication information, and identifying the position of the living thing in the three-dimensional model based on the relative coordinates of the living thing, includes: Acquire target device absolute position information and living thing position information of the target device based on the communication information; Determine the relative position coordinates between the target device and the living thing as the living thing relative coordinates based on the absolute position information of the target device and the position information of the living thing; The position of the living thing in the three-dimensional model is determined based on the relative coordinates of the living thing and marked as the position of the living thing.

4. The UWB-based AR glasses safety detection method according to claim 1, characterized in that: Also includes: Prompt information is generated based on the living thing detection information in the communication information, and the prompt information is played based on the AR glasses.

5. A UWB-based AR glasses safety detection method, characterized in that: Applied to target devices, including: Communication information is sent to an AR glasses within a first distance so that the AR glasses receive the communication information, obtain relative coordinates of the living thing based on the communication information, identify the position of the living thing in a three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

6. The UWB-based AR glasses safety detection method according to claim 5, characterized in that: Also includes: transmitting a UWB pulse signal based on at least one UWB base station of the target device and receiving a UWB pulse response signal; Based on the presence of a pulse peak at the target position of the UWB pulse response signal, determining that there is a living thing at the target position, and recording living thing detection information and living thing position information; The communication information including the living thing detection information and the living thing position information is generated.

7. A UWB-based AR glasses safety detection device, characterized in that: Applied to AR glasses, including: A receiving module, used to receive communication information sent by a target device within a first distance; a shooting module, configured to, when the AR glasses are at a second distance from the target device, acquire a plurality of images including the target device based on a plurality of shooting angles; wherein the first distance and the second distance are determined based on communication between a UWB tag of the AR glasses and at least one UWB base station of the target device, and the second distance is less than the first distance; A generating module, configured to generate a three-dimensional model corresponding to the target device based on the multiple images; an identification module, used for acquiring relative coordinates of the living thing based on the communication information, and identifying a position of the living thing in the three-dimensional model based on the relative coordinates of the living thing; A display module is used to display a three-dimensional model including the position of the living thing on the AR glasses.

8. A UWB-based AR glasses safety detection device, characterized in that: Applied to target devices, including: A sending module is used to send communication information to the AR glasses within a first distance, so that the AR glasses receive the communication information, obtain the relative coordinates of the living thing based on the communication information, and identify the position of the living thing in the three-dimensional model corresponding to the target device based on the relative coordinates of the living thing, and display the three-dimensional model including the position of the living thing on the AR glasses.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the UWB-based AR glasses safety detection method described in any one of claims 1 to 4 above, or the UWB-based AR glasses safety detection method described in claim 5 or 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the UWB-based AR glasses safety detection method described in any one of claims 1 to 4 above, or the UWB-based AR glasses safety detection method described in claim 5 or 6.

Images