AR glasses-based product accessory detection, positioning, and guiding assembly method and system

Abstract

The application discloses an AR glasses-based product accessory detection, positioning, guiding and assembling method and system, which comprises the following steps: obtaining position data of a mounting base; acquiring a depth image data stream and a video data stream of the mounting base; converting the depth image data stream into point cloud data; transmitting the video data stream to a desktop server; receiving identification data of an accessory matched with the mounting base sent by the desktop server; comparing the identification data of the accessory with the point cloud data, and obtaining three-dimensional coordinate data of the accessory; identifying a two-dimensional code on the accessory, and obtaining a serial number of the accessory; and establishing a simulation guide line between a simulation model of the accessory and a simulation model of the mounting base. The application realizes target identification by using three-dimensional point cloud, and combines two-dimensional code identification assistance, so that the three-dimensional space position of a target accessory can be quickly positioned, three-dimensional coordinates are obtained, and virtual model guiding between objects can be performed, so that quick installation is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection technology, and in particular to a product accessory detection, positioning, guiding and assembling method and system based on AR glasses. BACKGROUND

[0002] With the development of social life, the complexity of products is getting higher and higher. In the assembly process of complex products, there are a large number of accessories of various types and sizes. How to ensure that these accessories can be correctly installed in the appropriate position is crucial. However, there are often various errors in the installation process of accessories, and the quality of installation varies from person to person, which brings challenges to the quality consistency guarantee. These errors not only damage the performance of the product and affect the commercial use, but also the serious appearance quality may even cause safety hazards in the later use.

[0003] At present, the installation and detection technology of complex product accessories mainly relies on manual visual inspection and hand touch judgment method, which is labor-intensive, easy to miss defects, causes great safety hazards, and needs multiple detections, which is extremely inconvenient. SUMMARY

[0004] The embodiment of the present application provides a product accessory detection, positioning, guiding and assembling method based on AR glasses, which solves the technical problem that in the prior art, the installation position of accessories is determined by manual visual inspection and hand touch judgment, which requires multiple judgments and is easy to miss defects, realizes target recognition using three-dimensional point cloud, and combines two-dimensional code recognition to quickly locate the three-dimensional space position of the target accessory and obtain relatively accurate three-dimensional coordinates. Virtual model guidance between objects can be performed to facilitate quick installation.

[0005] The embodiment of the present application provides a product accessory detection, positioning, guiding and assembling method based on AR glasses, which includes the following steps: recognizing and positioning a two-dimensional code on an installation base to obtain position data of the installation base; obtaining depth image data stream and video data stream of the installation base; converting the depth image data stream into point cloud data; transmitting the video data stream to a desktop server; receiving identification data of an accessory matched with the installation base sent by the desktop server, the identification data of the accessory being obtained by target recognition and matching of the installation base based on the video data stream by the desktop server; comparing the identification data of the accessory with the point cloud data and obtaining three-dimensional coordinate data of the accessory; recognizing a two-dimensional code on the accessory and obtaining a serial number of the accessory; and establishing a simulation guide line between a simulation model of the accessory and a simulation model of the installation base according to the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the installation base of the accessory.

[0006] Further, after recognizing and positioning the two-dimensional code on the mounting base, a virtual model of the accessory mounting base is placed and coincides with the accessory mounting base.

[0007] Further, the comparison of the identification data of the accessory with the point cloud data and obtaining the three-dimensional coordinate data of the accessory includes the following steps: coordinate conversion of the two-dimensional coordinate data of the accessory to obtain accessory conversion data; comparison of the accessory conversion data with the point cloud data; if the accessory conversion data has data in the corresponding point cloud region, it is determined that the accessory three-dimensional coordinate data is obtained.

[0008] Further, the comparison of the identification data of the accessory with the point cloud data and obtaining the three-dimensional coordinate data of the accessory further includes the following steps: preprocessing of the point cloud data in the corresponding point cloud region of the accessory conversion data; the preprocessing includes the following steps: taking the mean value of the point cloud data in the corresponding point cloud region of the accessory conversion data; filtering out abnormal points with excessive depth values of the junction points of the accessory edges.

[0009] Further, after the preprocessing, the vector coordinate value of the preprocessed point cloud data is taken as an array, and the average point of the array is taken as a spatial anchor point of the display accessory virtual model.

[0010] The application also provides an AR glasses-based product accessory detection and positioning guiding assembly system, which comprises: an identification and positioning unit configured to recognize and position a two-dimensional code on a mounting base to obtain position data of the mounting base; an acquisition unit configured to acquire depth image data stream and video data stream of the mounting base; a virtual model matching unit configured to place a virtual model of the accessory mounting base and coincide with the accessory mounting base; a conversion unit configured to convert the depth image data stream into point cloud data; a transmission unit configured to transmit the video data stream to a desktop server; a receiving unit configured to receive identification data of an accessory matching the mounting base sent by the desktop server; a comparison unit configured to compare the identification data of the accessory with the point cloud data and obtain three-dimensional coordinate data of the accessory; an accessory identification unit configured to identify a two-dimensional code on the accessory and obtain a serial number of the accessory; and a simulation guiding unit configured to establish a simulation guide line between a simulation model of the accessory and a simulation model of the mounting base according to the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the accessory mounting base.

[0011] Further, the comparing unit comprises a conversion subunit configured to perform coordinate conversion on the two-dimensional coordinate data of the accessory to obtain accessory conversion data; a comparing subunit configured to compare the accessory conversion data with the point cloud data; and a determining subunit configured to determine that the accessory three-dimensional coordinate data is obtained if the accessory conversion data has data in the corresponding point cloud region.

[0012] Further, the comparing unit further comprises a preprocessing subunit configured to preprocess the point cloud data in the corresponding point cloud region of the accessory conversion data, and the preprocessing subunit comprises a mean value taking subunit configured to take a mean value of the point cloud data in the corresponding point cloud region of the accessory conversion data, and a filtering subunit configured to filter out abnormal points with excessively large depth values of the accessory edge intersection points.

[0013] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0014] 1. The target recognition using three-dimensional point clouds, combined with two-dimensional code recognition assistance, can quickly locate the three-dimensional space position of the target accessory, obtain relatively accurate three-dimensional coordinates, and guide the virtual model between objects to facilitate quick installation. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A flowchart of a product accessory detection, positioning, guiding and assembly method based on AR glasses in an embodiment of the present application;

[0016] Figure 2 A flowchart of a comparing step of accessory recognition data and point cloud data in an embodiment of the present application;

[0017] Figure 3 A flowchart of a preprocessing step in an embodiment of the present application;

[0018] Figure 4 A structural schematic diagram of a product accessory detection, positioning, guiding and assembly system based on AR glasses in an embodiment of the present application;

[0019] Figure 5 A structural schematic diagram of a comparing unit in an embodiment of the present application;

[0020] Figure 6 A structural schematic diagram of a preprocessing subunit in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiment of the application discloses a product accessory detection positioning guiding assembly method and system based on AR glasses, and solves the technical problem that in the prior art, accessory installation positions are determined by manual visual inspection and manual touch, which leads to the need for multiple determinations and the possibility of missing defects.

[0022] To solve the above technical problems, the general idea of the technical scheme provided by the application is as follows:

[0023] The embodiment of the application provides a product accessory detection positioning guiding assembly method based on AR glasses, which comprises the following steps: recognizing and positioning an installation base; acquiring a depth image data stream and a video data stream; converting the depth image data stream into point cloud data; transmitting the video data stream to a desktop server; receiving recognition data of an accessory matched with the installation base; comparing the recognition data of the accessory with the point cloud data; recognizing a two-dimensional code on the accessory; and establishing a simulation guiding line between a simulation model of the accessory and a simulation model of the installation base according to three-dimensional coordinate data of the accessory, the recognition data of the accessory, a serial number of the accessory, and position data of the installation base of the accessory.

[0024] To make the above basic method of the embodiment of the application more obvious and easy to understand, the specific embodiments of the application will be described in detail below with reference to the drawings.

[0025] Example One

[0026] Figure 1 The embodiment of the application is a product accessory detection positioning guiding assembly method based on AR glasses, which will be described in detail below through specific steps.

[0027] S11, recognizing and positioning a two-dimensional code on an installation base to obtain position data of the installation base.

[0028] In specific implementation, the installation base can be a socket, a device base or the like, each installation base has an accessory matched therewith, and a two-dimensional code can be pasted on the installation base, which can contain basic information such as a model category of the installation base.

[0029] In specific implementation, a gray-scale camera on the HoloLens eye end can be used to recognize the two-dimensional code on the installation base, the installation base is recognized and positioned, and position data of the installation base is obtained.

[0030] In specific implementation, after the installation base is recognized and positioned, a virtual model of the installation base stored in the eye end can be called, and the virtual model of the installation base is made to coincide with the installation base.

[0031] S12, acquiring a depth image data stream and a video data stream of the installation base.

[0032] In a specific implementation, a depth image data stream and a video data stream of the mounting base can be acquired at the eye end. The depth image data stream is obtained by a depth sensor, and the data thereof mainly includes a frame timestamp, a frame resolution, exposure, gain, etc. The depth sensor uses an AHAT mode, and a corresponding depth image can be obtained by processing a sensor frame. The video data stream is obtained by a picture video sensor, and also includes a frame timestamp, a frame resolution, exposure, gain, etc., and mainly provides video stream data.

[0033] S13, converting the depth image data stream into point cloud data;

[0034] In a specific implementation, the depth image data stream can be converted into point cloud data at the eye end through a coordinate conversion principle.

[0035] In a specific implementation, the calculation process of converting the depth image data into point cloud data is mainly knowledge of multi-view geometry, and the principle thereof is obtained according to an internal and external parameter matrix transformation formula:

[0036] In the setting of the external parameter matrix: since the world coordinate origin and the camera coordinate origin are coincident, i.e., there is no rotation and translation, so

[0037] Since the coordinate origins of the camera coordinate system and the world coordinate system are coincident, the same object in the camera coordinate and the world coordinate has the same depth, i.e.,

[0038] From the above transformation matrix formula, the image point

[0039] Therefore, real-time point cloud information can be obtained through continuous depth images.

[0040] S14, transmitting the video data stream to a desktop server.

[0041] In a specific implementation, the video data stream can be transmitted to the desktop server through a socket module.

[0042] S15, receiving identification data of a matching accessory of the mounting base sent by the desktop server.

[0043] In a specific implementation, the identification data of the accessory is obtained by the desktop server based on target identification matching of the mounting base on the video data stream, and the identification data can include two-dimensional coordinate position, type, confidence, etc. of the accessory.

[0044] S16, compare the identification data of the accessory with the point cloud data, and obtain the three-dimensional coordinate data of the accessory.

[0045] In a specific implementation, as shown in Figure 2 The comparison of the identification data of the accessory with the point cloud data can be as follows:

[0046] S161, perform coordinate conversion on the two-dimensional coordinate data of the accessory to obtain accessory conversion data.

[0047] S162, compare the accessory conversion data with the point cloud data;

[0048] S163, if the accessory conversion data has data in the corresponding point cloud region, it is determined that the accessory has three-dimensional coordinate data.

[0049] S164, pre-process the point cloud data in the point cloud region corresponding to the accessory conversion data.

[0050] In a specific implementation, as shown in Figure 3 The pre-processing step can be as follows:

[0051] S1641, take the average of the point cloud data in the point cloud region corresponding to the accessory conversion data;

[0052] S1642, filter out abnormal points with excessive depth values of the junction points of the accessory edge.

[0053] Pre-processing the point cloud data in the point cloud region corresponding to the accessory conversion data can reduce errors and filter out abnormal points. In addition, after pre-processing, the vector coordinate values of the pre-processed point cloud data can be taken as an array, and the average point of the array can be taken as a spatial anchor point for displaying the virtual model of the accessory.

[0054] S17, identify the two-dimensional code on the accessory, and obtain the serial number of the accessory.

[0055] In a specific implementation, after obtaining the three-dimensional coordinate data of the accessory, the two-dimensional code on the accessory can be obtained through the grayscale camera at the eye end, and the serial number of the accessory can be obtained to distinguish different individuals of the same accessory to meet the installation needs of multiple similar accessories.

[0056] S18, establish a simulation guide line between the simulation model of the accessory and the simulation model of the mounting base.

[0057] In a specific implementation, a simulation guide line is established between the simulation model of the accessory and the simulation model of the mounting base based on the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the accessory mounting base. The established simulation guide line can facilitate the operator to guide the connection between the accessory and the mounting base, and when the accessory is close enough to the mounting base, the guide line is closed and waits for the next identification.

[0058] In summary, the ability to read accessory type and location information in mixed virtual reality is achieved. First, the location of the mounting base is identified. Then, using the image information obtained through the HoloLens glasses, the image is transmitted to the desktop computer, where a pre-trained model is used to identify and locate the product accessories, completing the accessory installation guidance. By combining mixed virtual reality with deep learning object detection and QR code recognition, it can effectively help humans identify the location and type of product accessories and assist in installation. This can achieve high accuracy and stability in actual engineering applications, meeting the needs of actual industrial production.

[0059] In order to enable those skilled in the art to better understand and implement the embodiments of the present application, the following reference is made to Figure 4 This paper introduces a product accessories detection, positioning and guidance assembly system based on AR glasses.

[0060] Example Two

[0061] Reference Figure 4 As shown, an embodiment of the present application provides a product accessory detection, positioning, and assembly guidance system based on AR glasses, the system comprising:

[0062] an identification and positioning unit configured to identify and locate the QR code on the mounting base to obtain position data of the mounting base;

[0063] an acquisition unit configured to acquire a depth image data stream and a video data stream of the mounting base;

[0064] a virtual model matching unit, configured to place the virtual model of the accessory mounting base and overlap the virtual model with the accessory mounting base;

[0065] a conversion unit configured to convert the depth image data stream into point cloud data;

[0066] a transmission unit, configured to transmit the video data stream to a desktop server;

[0067] a receiving unit configured to receive identification data of an accessory matching the mounting base sent by the desktop server;

[0068] The comparison unit is configured to compare the identification data of the accessory with the point cloud data, and obtain three-dimensional coordinate data of the accessory.

[0069] The accessory identification unit is configured to identify a two-dimensional code on the accessory, and obtain a serial number of the accessory.

[0070] The simulation guidance unit is configured to establish a simulation guidance line between a simulation model of the accessory and a simulation model of the installation base according to the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the installation base of the accessory.

[0071] In a specific implementation, as shown in Figure 5 The comparison unit includes:

[0072] The conversion subunit is configured to perform coordinate conversion on the two-dimensional coordinate data of the accessory to obtain accessory conversion data.

[0073] The comparison subunit is configured to compare the accessory conversion data with the point cloud data.

[0074] The identification subunit is configured to identify the three-dimensional coordinate data of the accessory if the accessory conversion data has data in the corresponding point cloud region.

[0075] The preprocessing subunit is configured to pre-process the point cloud data in the corresponding point cloud region of the accessory conversion data.

[0076] In a specific implementation, as shown in Figure 6 The preprocessing subunit includes:

[0077] The mean value taking unit is configured to take a mean value of the point cloud data in the corresponding point cloud region of the accessory conversion data.

[0078] The filtering unit is configured to filter out abnormal points with excessive depth values of the junction points of the accessory edge.

[0079] The various variations and specific implementations of the product accessory detection, positioning, guidance, and assembly method based on AR glasses in the foregoing embodiment one are also applicable to the product accessory detection, positioning, guidance, and assembly system based on AR glasses in the present embodiment. Through the detailed description of the product accessory detection, positioning, guidance, and assembly method based on AR glasses, those skilled in the art can clearly understand the product accessory detection, positioning, guidance, and assembly system based on AR glasses in the present embodiment. Therefore, for the sake of brevity of the specification, the product accessory detection, positioning, guidance, and assembly system based on AR glasses will not be described in detail.

[0080] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the present application can be implemented in software and can be stored on a computer readable medium, which can include random access memory (RAM), read only memory (ROM), magnetic disk or optical disk, or the like. The software implementation can comprise one or more computer program components embodied on one or more computer readable medium(s). The computer readable medium can be resident within the computing device or external to the computing device. The computer program components can also be downloaded into the computing device from an external computer or external storage device.

[0081] The present application is described in reference to the drawings, which are as follows: Figure One Figure One

[0082] Figure One Figure One

[0083] Figure One Figure One

[0084] ​​​​​​​​Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A product accessory detection, positioning and assembly guidance method based on AR glasses, characterized in that: The following steps are involved: Identify and locate the QR code on the mounting base to obtain position data of the mounting base; Acquiring a depth image data stream and a video data stream of the mounting base; Converting the depth image data stream into point cloud data; Transmitting the video data stream to a desktop server; receiving identification data of an accessory that matches the mounting base and is sent by the desktop server, where the identification data of the accessory is obtained by the desktop server performing target recognition and matching on the mounting base based on the video data stream; Comparing the identification data of the accessory with the point cloud data to obtain three-dimensional coordinate data of the accessory; Identify the QR code on the accessory and obtain the serial number of the accessory; Establishing a simulated guide line between a simulated model of the accessory and a simulated model of the mounting base based on the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the mounting base of the accessory; The step of comparing the identification data of the accessory with the point cloud data and obtaining the three-dimensional coordinate data of the accessory comprises the following steps: Performing coordinate transformation on the two-dimensional coordinate data of the accessory to obtain accessory transformation data; Comparing the accessory conversion data with the point cloud data; If the accessory conversion data has data within the corresponding point cloud area, it is considered as the three-dimensional coordinate data of the accessory; The step of comparing the identification data of the accessory with the point cloud data and obtaining the three-dimensional coordinate data of the accessory further includes the following steps: pre-processing the point cloud data within the point cloud area corresponding to the accessory conversion data; The pretreatment comprises the following steps: averaging the point cloud data within the point cloud area corresponding to the accessory conversion data; Abnormal points whose intersection depth values ​​of the accessory edges are too large are filtered out.

2. The method for detecting, positioning, and guiding assembly of product accessories based on AR glasses according to claim 1, characterized in that: After the QR code on the mounting base is identified and located, the virtual model of the accessory mounting base is placed and overlapped with the accessory mounting base.

3. The method for detecting, positioning and guiding assembly of product accessories based on AR glasses according to claim 1, characterized in that: After the preprocessing, the method further includes: taking the vector coordinate values ​​of the preprocessed point cloud data as a set of arrays, and taking the average value point of the array as a spatial anchor point for displaying the virtual model of the accessory.

4. A product accessory detection, positioning and guidance assembly system based on AR glasses, characterized in that: The system comprises: an identification and positioning unit configured to identify and locate the QR code on the mounting base to obtain position data of the mounting base; an acquisition unit configured to acquire a depth image data stream and a video data stream of the mounting base; a virtual model matching unit, configured to place the virtual model of the accessory mounting base and overlap the virtual model with the accessory mounting base; a conversion unit configured to convert the depth image data stream into point cloud data; a transmission unit, configured to transmit the video data stream to a desktop server; a receiving unit configured to receive identification data of an accessory matching the mounting base sent by the desktop server; a comparison unit configured to compare the identification data of the accessory with the point cloud data and obtain three-dimensional coordinate data of the accessory; an accessory identification unit, configured to identify the QR code on the accessory and obtain the serial number of the accessory; a simulation guide unit configured to establish a simulation guide line between a simulation model of the accessory and a simulation model of the mounting base based on the three-dimensional coordinate data of the accessory, the identification data of the accessory, the serial number of the accessory, and the position data of the accessory mounting base; The comparison unit comprises: a conversion subunit configured to perform coordinate conversion on the two-dimensional coordinate data of the accessory to obtain accessory conversion data; a comparison subunit, configured to compare the accessory conversion data with the point cloud data; The identification subunit is configured to identify the accessory conversion data as the three-dimensional coordinate data of the accessory if there is data in the corresponding point cloud area.

5. The product accessory detection, positioning, and assembly guidance system based on AR glasses according to claim 4, characterized in that: The comparison unit further includes a preprocessing subunit, which is configured to preprocess the point cloud data within the point cloud area corresponding to the accessory conversion data. The preprocessing subunit includes: an averaging unit configured to average the point cloud data within a point cloud region corresponding to the accessory conversion data; The filter unit is configured to filter out abnormal points whose depth values ​​of the junction points of the edge of the accessory are too large.

Images