Smart home equipment installation and debugging guidance system and method based on AR technology

Abstract

The invention discloses the technical field of intelligent home equipment installation and debugging, and relates to the technical field of intelligent home equipment installation and debugging. Comprising high-definition cameras, an infrared measurement unit, a scanning recognition module, an image recognition and tracking module, a laser radar, a debugging module, an AR content management module, an AR rendering system, a user gesture voice interaction module, a real-time video stream processing module, a voice system, an equipment joint debugging module and a recording module which are installed at a plurality of key positions in the AR glasses. And the AR content management module is used for collecting spatial data of installation environments of a plurality of indoor key positions in real time, generating a three-dimensional environment model, collecting real scene data, superposing virtual guidance content, and labeling and reminding special places. Static characters and pictures of a traditional paper specification are replaced, manual judgment is replaced by intelligent environment perception, tracking and error correction are operated in real time, and the installation precision and the installation efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of smart home device installation and debugging technology, and in particular to a smart home device installation and debugging guidance system and method based on AR technology. Background Technology

[0002] When installing and debugging smart home devices, users need to read the paper or electronic instruction manual that comes with the device in detail to understand the installation steps, required tools, precautions, etc., which may be difficult for non-professional users to understand. Furthermore, during the installation process, users need to observe the installation location with their naked eyes or other auxiliary tools, such as whether the wall is flat and whether the socket position is compatible. They also need to rely on personal experience to judge whether the equipment is suitable for installation, such as whether the weight exceeds the wall's load-bearing capacity. Misjudgment may lead to rework later. At the same time, during the installation process, users need to repeatedly check the steps against the instruction manual. It is easy to make mistakes due to omissions or misreadings. Inexperienced operation or negligence may also lead to low installation accuracy, resulting in low installation efficiency. At the same time, when users encounter problems during installation, they need to troubleshoot and try to solve them on their own, and may not be able to solve the problems effectively due to a lack of professional knowledge.

[0003] With the increasing demands for smart home device installation and the development of intelligent technologies, developing a smart home device installation and debugging guidance system and method based on AR technology is of great practical significance. Summary of the Invention

[0004] The purpose of this invention is to provide a smart home device installation and debugging guidance system and method based on AR technology, in order to solve the problems that existing paper and electronic manuals are difficult for non-professional users to understand when installing and debugging smart home devices, and that relying on personal experience to evaluate the installation leads to misjudgment of the installation environment and rework, and that the installation accuracy is not high due to personal experience. In addition, problems need to be solved by the user during the installation process.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A smart home device installation and debugging guidance system based on AR technology includes: The environmental perception module includes high-definition cameras, infrared measurement units, scanning and recognition modules, image recognition and tracking modules, LiDAR, debugging modules, AR content management modules, AR rendering systems, user gesture and voice interaction modules, real-time video stream processing modules, voice systems, equipment integration and debugging modules, and recording modules installed in multiple key locations within the AR glasses. It is used to collect spatial data of the installation environment in multiple key locations indoors in real time, generate three-dimensional environmental models, collect real scene data and overlay virtual guidance content, and mark and remind special locations. The AR content management module stores and manages AR guidance content, including a database containing a furniture and equipment manufacturer database and the AR itself database. The database contains equipment installation steps, debugging guides, troubleshooting, etc. The image recognition and tracking module uses computer vision technology to identify the equipment and its components, track the installation progress, and provide real-time feedback to ensure that users install the equipment according to the correct steps. The information collection module uses QR code scanning, RFID identification, and image recognition modules to scan the logos of smart home devices via AR terminals, automatically obtaining information such as device model, parameters, and installation specifications. It then connects to the database within the device's AR content management module to synchronize the latest installation manuals and debugging standards in real time.

[0006] As a further improvement to this technical solution: the environmental perception module includes a camera, a lidar, and an infrared measurement unit to collect spatial data of the installation environment, such as wall dimensions, water and electricity points, and furniture layout, generate a three-dimensional environmental model and upload it to the database, and provide user feedback by marking special locations such as load-bearing walls and pipeline locations.

[0007] As a further improvement to this technical solution: the environment perception module generates the optimal installation path through the rule engine and the content in the machine learning database, including: installation location recommendation, analysis of the environment based on the environment perception module, conversion of the content in the database into a three-dimensional virtual model animation, superimposed on the real scene, and step-by-step operation guidance combined with device functional requirements and environmental compatibility, breaking down the operation steps.

[0008] As a further improvement to this technical solution: the user gesture voice interaction module provides interactive guidance through the voice system user and interaction module, using various forms such as voice prompts, text descriptions, and animation demonstrations. It supports users interacting with AR content through gestures, voice, or touch to obtain more detailed guidance information. The user gesture voice interaction module also includes a remote assistance module, enabling remote communication assistance and providing remote expert assistance. Users can conduct real-time video communication with experts through AR devices for remote guidance. As a further improvement to this technical solution: the device integration module automatically detects the connection status between the device and the home gateway, such as Wi-Fi, Bluetooth, network, etc., and displays the connection progress through an AR interface.

[0009] As a further improvement to this technical solution: the recording module is used to record key data during the installation and debugging process, such as installation time, step completion status, and problems encountered, and provides data analysis functions to help optimize the installation process and improve debugging efficiency. After analysis, the records are uploaded to the database in the AR content management module.

[0010] As a further improvement to this technical solution: a spatial 3D model is generated by the image recognition and tracking module and the environmental perception module to identify key locations such as water and electricity interfaces and load-bearing structures. The system evaluates the installation feasibility using the following formula:

[0011] Where: (F is the feasibility score, S is the spatial matching degree, C is the water and electricity compatibility, and D is the ease of operation; , , (These are weighting coefficients, obtained through training with historical installation data). like( If the threshold is ≥, the AR interface will display the recommended installation area; if ( If the threshold is less than 1, suggestions for environmental adjustments will be provided.

[0012] As a further improvement to this technical solution: the user performs the operation according to the AR prompts, and the AR terminal compares the actual operation with the standard steps in real time through the image recognition and tracking module. If a deviation occurs, the system calculates the deviation value using the following formula:

[0013] in( For positional deviation, (for angular deviation), when ( When the threshold is reached, the AR interface flashes an alarm and displays a correction plan.

[0014] This invention also proposes a method for guiding the installation and debugging of smart home devices based on AR technology, including the following steps: Step 1: Device Information Acquisition and Initialization; Step 2: Installation environment scanning and analysis; Step 3: AR installation path planning and visual guidance; Step 4: Step-by-step operation guidance and real-time error correction; Step 5: Guidance on device networking and function debugging; Step Six: Installation Acceptance and Data Archiving.

[0015] As a further improvement to this technical solution: In step five, the AR terminal prompts the user to start the network mode, and the joint debugging module automatically detects the signal strength RSSI between the device and the gateway. If (RSSI < -80dBm), it prompts "Adjust the device position to a strong signal area".

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention is highly visual and intuitive, lowering the operational threshold. It overlays real-world scenes with 3D virtual content and uses AR technology to transform the equipment installation steps into a 3D virtual model, which is directly superimposed on the real installation scene, replacing the abstract descriptions of traditional text and static images. This allows even users with zero experience to quickly understand the operation logic. 2. Improve installation accuracy through environmental perception and intelligent decision-making. Generate a 3D model of the installation environment using LiDAR and high-definition cameras, automatically identify key points such as water and electricity interfaces, load-bearing walls, and pipelines, avoid safety risks caused by human misjudgment, scientifically recommend the optimal location based on data formulas, without relying on the user's subjective judgment, and can track and correct errors in real time, reducing rework costs. 3. This invention automates joint debugging and fault diagnosis, improves debugging efficiency, and enables continuous optimization through remote assistance and data accumulation. Through the remote assistance module, experts can directly mark fault points in the user's AR view, replacing the inefficient communication of traditional telephone description and imagination of faults.

[0017] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the smart home device installation and debugging guidance system based on AR technology proposed in this invention; Figure 2 This is a schematic diagram of the installation and debugging guidance method for smart home devices based on AR technology proposed in this invention. Detailed Implementation

[0019] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention. The invention is described more specifically in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0020] Please see Figures 1-2In this embodiment of the invention, a smart home device installation and debugging guidance system based on AR technology includes an environmental perception module. The environmental perception module includes a high-definition camera, an infrared measurement unit, a scanning and recognition module, an image recognition and tracking module, a lidar, a debugging module, an AR content management module, an AR rendering system, a user gesture and voice interaction module, a real-time video stream processing module, a voice system, a device integration and debugging module, and a recording module installed in multiple key locations inside the AR glasses. It is used to collect spatial data of the installation environment in multiple key locations indoors in real time, generate a three-dimensional environment model, collect real scene data and overlay virtual guidance content, and mark and remind special locations. The AR content management module stores and manages AR guidance content, including a database containing a database of furniture and equipment manufacturers and the AR itself. The database contains equipment installation steps, debugging guides, troubleshooting, etc. The image recognition and tracking module uses computer vision technology to identify the equipment and its components, track the installation progress, and provide real-time feedback to ensure that users install the equipment according to the correct steps. The information collection module uses QR code scanning, RFID identification, and image recognition modules to scan the logo of smart home devices through AR terminals, automatically obtaining information such as device model, parameters, and installation specifications. It then connects to the database within the device's AR content management module to synchronize the latest installation manuals and debugging standards in real time. Following the AR prompts, users slowly move the AR terminal to scan the installation scene, such as the living room wall and kitchen cabinets. The environmental perception module generates a 3D point cloud model through LiDAR, recording data such as wall dimensions, furniture positions, and water and electricity interfaces. The infrared measurement unit detects hidden obstacles such as concealed wires and water pipes in the wall and marks them in the 3D model through the AR rendering system. Users scan the QR code of smart home devices using the camera of the AR terminal. The information collection module automatically identifies the device model and connects to the manufacturer's database of the AR content management module through the API interface to retrieve the device's exclusive 3D model data, installation manual, debugging parameters, etc.

[0021] Specifically, the environment perception module generates the optimal installation path through the rule engine and the content in the machine learning database, including: installation location recommendation; based on the environment perception module, it analyzes the environment, converts the content in the database into a 3D virtual model animation, overlays it onto the real scene, and combines device functional requirements and environmental compatibility to provide step-by-step operation guidance and break down the operation steps.

[0022] Specifically, the user gesture and voice interaction module provides interactive guidance through the voice system user and interaction module, using various forms such as voice prompts, text descriptions, and animation demonstrations. It supports users to interact with AR content through gestures, voice, or touch to obtain more detailed guidance information. The user gesture and voice interaction module has a remote assistance module that can provide remote communication assistance and remote expert assistance. Users can conduct real-time video communication with experts through AR devices to obtain remote guidance. Users perform operations according to AR prompts, and the image recognition and tracking module captures the operation actions in real time through the camera and quantifies the operation accuracy through the deviation calculation formula.

[0023] Specifically, the device integration module automatically detects the connection status between the device and the home gateway, such as Wi-Fi, Bluetooth, and network, and displays the connection progress through an AR interface.

[0024] The AR terminal prompts "Start network debugging". The device debugging module automatically detects the connection protocol between the device and the home gateway. If a weak signal is detected (RSSI < -80dBm), the AR interface marks the "weak signal area" and prompts "Move the device within 3 meters of the router".

[0025] Specifically, the recording module is used to record key data during the installation and debugging process, such as installation time, completion status of steps, and problems encountered. It provides data analysis functions to help optimize the installation process and improve debugging efficiency. After analysis, the records are uploaded to the database in the AR content management module.

[0026] Specifically, the system generates a spatial 3D model through the image recognition and tracking module and the environmental perception module, identifies key locations such as water and electricity interfaces and load-bearing structures, and evaluates the installation feasibility using the following formula: Where: (F is the feasibility score, S is the spatial matching degree, C is the water and electricity compatibility, and D is the ease of operation; , , (These are weighting coefficients, obtained through training with historical installation data). If the threshold is ≥, the AR interface will display the recommended installation area; if ( If the threshold is less than 1, suggestions for environmental adjustments will be provided.

[0027] Specifically, the user follows the AR prompts to perform the operation. The AR terminal uses an image recognition and tracking module to compare the actual operation with the standard steps in real time. If a deviation occurs, the system calculates the deviation value using the following formula: ,in( For positional deviation, (for angular deviation), when ( When the threshold is reached, the AR interface flashes an alarm and displays a correction plan.

[0028] The method for installing and debugging smart home devices based on AR technology includes the following steps: Step 1: Device Information Acquisition and Initialization; Step 2: Installation environment scanning and analysis; Step 3: AR installation path planning and visual guidance; Step 4: Step-by-step operation guidance and real-time error correction; Step 5: Guidance on device networking and function debugging; Step Six: Installation Acceptance and Data Archiving.

[0029] Specifically, in step five, the AR terminal prompts the user to activate network mode. The debugging module automatically detects the signal strength RSSI between the device and the gateway. If (RSSI < -80dBm), it prompts "Adjust the device position to an area with strong signal". The working principle and process of this invention are as follows: Users scan the QR code of smart home devices with the camera of the AR terminal. The information collection module automatically identifies the device model and connects to the manufacturer's database of the AR content management module through the API interface to retrieve the device's exclusive 3D model data, installation manual, debugging parameters, etc. The AR terminal initializes the interface, automatically loads the device's 3D model preview, and prompts the user to "prepare to scan the installation environment". Following AR prompts, the user slowly moves the AR terminal to scan the installation scene, such as the living room wall and kitchen cabinets. The environmental perception module generates a 3D point cloud model using LiDAR, recording data such as wall dimensions, furniture positions, and water and electricity interfaces. The infrared measurement unit detects hidden obstacles such as concealed wires and water pipes within the wall and marks them in the 3D model using the AR rendering system. The intelligent guidance core module, based on environmental data, uses a feasibility scoring formula (…). Calculate installation feasibility. If (F≥threshold), the AR interface displays a green recommended area; if (F<threshold), it prompts adjustment suggestions. The intelligent guidance core module retrieves the device installation steps from the AR content management module and optimizes the sequence by combining the environmental model. The AR content generation unit converts the steps into a 3D virtual animation. The AR terminal overlays the virtual animation onto the real scene, and users can directly see the corresponding positions of virtual screws and physical walls through AR glasses. The user performs actions according to AR prompts, and the image recognition and tracking module captures the actions in real time using a camera, calculating the deviation using a formula. Quantify operational precision; After installation, the AR terminal prompts "Start network debugging". The device debugging module automatically detects the connection protocol between the device and the home gateway. If a weak signal is detected (RSSI < -80dBm), the AR interface marks the "weak signal area" and prompts "Move the device within 3 meters of the router". The AR terminal generates an acceptance checklist. After the user confirms and clicks "Complete Installation," the recording module automatically stores key data. The data is then uploaded to the dynamic database of the AR content management module, where machine learning is used to optimize guidance strategies. For example, for "frequent errors in wiring steps," the 3D animation details of that step are enhanced.

[0030] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Those skilled in the art can readily implement the present invention based on the accompanying drawings and the above description. However, any modifications, alterations, or variations made by those skilled in the art without departing from the scope of the present invention, utilizing the disclosed technical content, are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, or variations made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the present invention.

Claims

1. A smart home device installation and commissioning guidance system based on AR technology, characterized in that, Comprise: The environmental perception module includes high-definition cameras, infrared measurement units, scanning identification modules, image recognition and tracking modules, laser radars, debugging modules, AR content management modules, AR rendering systems, user gesture and voice interaction modules, real-time video stream processing modules, voice systems, device commissioning modules, and recording modules installed in multiple key positions within the AR glasses, for real-time collection of spatial data of the installation environment of multiple key positions in the room, generation of a three-dimensional environmental model, collection of real scene data and superposition of virtual guidance content, and labeling of special locations for reminders; The AR content management module stores and manages AR guidance content, including a database containing furniture and equipment manufacturer databases and AR self-databases, which contain device installation steps, debugging guidelines, troubleshooting, etc. The image recognition and tracking module uses computer vision technology to identify devices and their components, track installation progress, and provide real-time feedback to ensure that users follow the correct steps for installation. The information collection module scans the identification of smart home devices through the AR terminal, automatically obtains device model, parameters, installation specifications, and other information through two-dimensional code scanning, RFID identification, and image recognition modules, and interfaces with the database in the device AR content management module to synchronize the latest installation manual and debugging standards in real time. 2.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, The environmental perception module includes cameras and laser radars, as well as infrared measurement units, which collect spatial data of the installation environment, such as wall dimensions, water and electricity point locations, and furniture layout, generate a three-dimensional environmental model, and upload it to the database, and provide prompts to users through labeling for special locations such as load-bearing walls and pipeline locations. 3.The AR technology-based smart home device installation and commissioning guidance system of claim 1, characterized in that, The environmental perception module generates the optimal installation path based on the contents of the rule engine and machine learning database, including installation location recommendations, analysis of the environment based on the environmental perception module, conversion of the contents in the database into three-dimensional virtual model animations, superposition on the real scene, combination of device function requirements and environmental compatibility, step-by-step operation guidance, and splitting of operation steps. 4.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, The user gesture and voice interaction module provides interactive guidance through voice prompts, text explanations, and animation demonstrations, supports user interaction with AR content through gestures, voice, or touch, and provides more detailed guidance information. The user gesture and voice interaction module has a remote assistance module that can remotely communicate and assist, providing remote expert assistance functions. Users can communicate with experts in real-time video through AR devices to provide remote guidance. 5.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, The device commissioning module automatically detects the connection status of devices and home gateways, such as Wi-Fi, Bluetooth, and network, and displays the connection progress through the AR interface. 6.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, The recording module records key data during installation and debugging, such as installation time, step completion status, and problems encountered, provides data analysis functions, helps optimize installation processes and improve debugging efficiency, and uploads the records to the database in the AR content management module after analysis. 7.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, Through the image recognition and tracking module and the environment perception module, a spatial three-dimensional model is generated, key points such as water and electricity interfaces and load-bearing structures are recognized, and the system evaluates the installation feasibility through the following formula: ， Wherein: (F is the feasibility score, S is the spatial matching degree, C is the water and electricity compatibility, D is the operation convenience; , , is a weight coefficient, which is obtained by training historical installation data); like( If the threshold is ≥, the AR interface will display the recommended installation area; if ( If the threshold is less than 1, suggestions for environmental adjustments will be provided. 8.The AR technology-based smart home device installation and commissioning guidance system of claim 1, wherein, The user performs the operation according to the AR prompt, the AR terminal compares the actual operation with the standard steps in real time through the image recognition and tracking module, and if there is a deviation, the system calculates the deviation value through the following formula: ， wherein (A is a positional deviation, is an angular deviation), when ( > a threshold value), the AR interface flashes an alert and displays a correction scheme. 9.The method of installing and debugging a smart home device based on AR technology according to any one of claims 1-8, characterized in that, The method comprises the following steps: Step one: device information acquisition and initialization; Step two: installation environment scanning and analysis; Step three: AR installation path planning and visual guidance; Step four: step-by-step operation guidance and real-time error correction; Step five: device networking and function debugging guidance; Step six: installation acceptance and data archiving. 10.The method of AR technology-based smart home device installation and commissioning guidance according to claim 9, wherein, In step five, the AR terminal prompts the user to start the networking mode, and the commissioning module automatically detects the signal strength RSSI of the device and the gateway. If (RSSI<-80dBm), it prompts "Adjust the device position to the signal strong area".

Images