Method and device for measuring distance at specified point in mixed reality

Abstract

The invention relates to a method and a device for measuring distance at a specified point under mixed reality, and belongs to the technical field of space measurement based on MR, and the method comprises the steps: initializing mixed reality equipment, and selecting a measurement mode according to the demands of a user; acquiring the action of the user in the shooting range of the depth camera, performing hand recognition on the scene image at the current moment, and determining the pose of the hand articulation point of the user at the current moment; judging whether the user triggers a target gesture or not according to the pose of the hand joint point at the current moment, and obtaining the pose of a specified joint point in the target gesture; calculating the distance between the specified joint points, judging whether the user triggers a fixed point standard or not according to the distance between the specified joint points, and obtaining the joint pose of the display position of the rendering component; and drawing a virtual distance line segment in real time according to the obtained display position of the distance rendering component, and displaying a distance value. According to the invention, not only is the measurement precision of distance measurement in mixed reality improved, but also the operation becomes simpler and more convenient.

Description

Technical Field

[0001] The present invention relates to a method and device for hand - designated point ranging under mixed reality, belonging to the technical field of spatial measurement based on MR. Background Art

[0002] With the rapid development of fields such as engineering construction and power transmission and transformation, spatial measurement technology has played an increasingly important role in engineering acceptance and on - site detection. Existing measurement technologies usually rely on traditional manual measurement tools (such as tape measures, laser rangefinders, etc.) or high - precision instrument equipment (such as total stations, lidar). However, traditional measurement methods still have some obvious deficiencies. Especially in complex and dynamic construction sites, they cannot provide immediate and accurate measurement data. Traditional manual measurement tools (such as tape measures, laser rangefinders) are widely used in most engineering projects. Although these tools can provide a certain measurement accuracy, their operation is cumbersome and limited by the skill level of the operator. For example, tape measure measurement requires manual alignment and repeated operations. Although the laser rangefinder improves the measurement efficiency, it depends on the operator's visual estimation and hand - held stability, and is prone to measurement errors due to equipment misoperation or environmental interference (such as light changes, wind, etc.). In addition, the accuracy of traditional measurement tools usually reaches the millimeter level, but in large - scale projects and complex environments, the deficiencies in accuracy and efficiency are still relatively prominent.

[0003] High - precision measurement instruments (such as total stations, lidar) can provide relatively high measurement accuracy in large - scale projects. However, these devices generally have the disadvantages of high cost and complex operation. Total stations require professional personnel to operate, and the measurement process is complex, and the measurement results usually need to be manually input and calculated. While lidar can efficiently capture spatial data over a large range, it also has problems such as high equipment cost and high operation threshold. And for application scenarios that require real - time update and interactive feedback (such as construction sites or engineering acceptance sites), the real - time performance and operation convenience of these devices are poor. Therefore, although these high - precision devices can theoretically provide accurate measurement results, they are limited by cost and operation complexity in practical applications.

[0004] Although the measurement applications of existing augmented reality (AR) and mixed reality (MR) technologies have been initially attempted in some scenarios, they still face certain limitations. The core advantage of AR / MR technology lies in its ability to combine virtual information with the real environment, providing users with real-time feedback and interaction capabilities. However, most existing AR / MR measurement technologies rely on the visual perception of devices and the fusion of sensor data. Due to the limitations of device hardware performance and algorithms, measurement accuracy and real-time performance still pose challenges in complex environments. The positioning accuracy of the device, changes in environmental lighting, and occlusion by complex structures can all affect the accuracy of measurement results. In addition, although the gesture recognition and spatial modeling capabilities of existing technologies have developed, in a dynamic construction site, the real-time performance, stability, and interactivity of measurement results are still insufficient and cannot meet the requirements of efficient and accurate measurement. Therefore, the present invention provides a method and device for hand-designated point ranging under mixed reality. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method and device for hand-designated point ranging under mixed reality, which can accurately measure the geometric dimensions of objects in the real world and assist engineers in on-site verification and correction.

[0006] The technical solutions adopted by the present invention to solve its technical problems are as follows: In a first aspect, a method for hand-designated point ranging under mixed reality provided by an embodiment of the present invention includes the following steps: Initialize the mixed reality device and select a measurement method according to the user's needs. The measurement method includes real-time ranging with both index fingers or fixed-point ranging with the thumb and index finger in contact. Obtain the actions of the user within the shooting range of the depth camera, perform hand recognition on the scene image at the current moment, and determine the poses of the user's hand joint points at the current moment. Judge whether the user triggers the target gesture based on the poses of the hand joint points at the current moment, and obtain the poses of the designated joint points in the target gesture. Calculate the distance between the designated joint points, judge whether the user triggers the fixed-point standard according to the distance between the designated joint points, and obtain the joint poses of the display positions of the rendering components. Perform real-time drawing of virtual distance lines and display the distance values according to the obtained distance and the display positions of the rendering components.

[0007] As a possible implementation manner of this embodiment, the real-time ranging with both index fingers is to detect the fingertip positions of the user's both index fingers, and calculate and display the distance between the fingertips of the both index fingers in real time; The fixed-point ranging with the thumb and index finger in contact is that the user determines two measurement points by pinching the thumb and index finger, records these two measurement points and calculates the distance between them.

[0008] As a possible implementation of this embodiment, when obtaining the actions of the user within the shooting range of the depth camera, the far pointer operation of the hand is turned off to prevent the far pointer from being displayed during the positioning process, resulting in misjudgment and accidental touch by the user.

[0009] As a possible implementation of this embodiment, determining whether the user triggers a target gesture based on the pose of the hand joint points at the current moment, and obtaining the pose of the specified joint points in the target gesture includes: In the preset correspondence between gestures and instructions in the measurement method, it is determined whether the user triggers a target gesture according to whether the instruction corresponding to the user's gesture is a fixed-point ranging instruction. Among them, the gesture preset for the real-time ranging measurement of both index fingers is the fingertips of the left index finger and the right index finger; the gesture preset for the fixed-point ranging measurement with the thumb and index finger in contact is the pinching gesture between the thumb and the index finger; If the user triggers a target gesture, obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture.

[0010] As a possible implementation of this embodiment, calculating the distance between the specified joint points, determining whether the user triggers a fixed-point standard according to the distance between the specified joint points, and obtaining the joint pose of the display position of the rendering component includes: According to the Euclidean distance in three-dimensional space, calculate the distance d between joint point A and joint point B: , where, (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) are the spatial coordinates of joint point A and joint point B respectively; Set the determination of whether the user triggers a fixed-point standard. The distance determination standard for the real-time ranging of both index fingers is to trigger the ranging instruction when there are index fingers of both hands within the shooting range of the depth camera; the distance determination standard for the fixed-point ranging with the thumb and index finger in contact is to trigger the ranging fixed-point standard when the distance between the thumb and index finger joint points is less than the filter value; Judge whether to trigger the distance threshold of the fixed-point ranging gesture or detect the positions of both index fingers according to the calculated distance; If so, use the fixed-point ranging gesture for fixed-point ranging; when performing real-time ranging of both index fingers, obtain the joint point A (x 1 , y 1 , z 1 ) and joint point B (x 2 , y 2 , z 2) three-dimensional coordinates; when the thumb and index finger are in contact to measure distance at a fixed point, obtain the joint point A(x 1 ,y 1 ,z 1 ) of the index finger and the joint point B(x 2 ,y 2 ,z 2 ) of the thumb. The position coordinates of the pinch between the three-dimensional coordinates of the two joint points are the three-dimensional coordinates of the positioning point; Otherwise, obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture; judge whether the user triggers the fixed-point distance measurement gesture in the next frame according to the Euclidean distance between the coordinates of joint points A and B.

[0011] As a possible implementation of this embodiment, the real-time drawing of a virtual distance line segment and the display of the distance value according to the display position of the obtained distance rendering component include: After detecting that the user triggers the fixed-point standard, according to the obtained distance data, use the rendering component to draw a virtual distance line segment for real-time distance measurement between the index fingers of both hands or a virtual distance line segment for fixed-point distance measurement with the thumb and index finger in contact in the user's field of view; After the virtual line segment is drawn, calculate and display the measured distance between the two points.

[0012] As a possible implementation of this embodiment, the use of the rendering component to draw a virtual distance line segment for real-time distance measurement between the index fingers of both hands in the user's field of view includes: Initialize the TextMeshPro component and the LineRenderer component; In the frame update function Update in the real-time 3D interactive content creation and operation platform Unity, continuously track the positions of the fingertip joint point A of the left index finger and the fingertip joint point B of the right index finger and obtain the corresponding three-dimensional coordinate data; Assign the position data of joint point A and joint point B obtained in each frame to the points in the initialized LineRenderer component; Connect joint point A and joint point B in the initialized LineRenderer component to obtain a complete distance display line segment; Calculate the distance value of each frame according to the position data of joint point A and joint point B obtained in each frame , and assign the distance value to the initialized TextMeshPro component in real time for distance value display.

[0013] As a possible implementation of this embodiment, the use of the rendering component to draw a virtual distance line segment for fixed-point distance measurement with the thumb and index finger in contact in the user's field of view includes: Initialize the TextMeshPro component and the LineRenderer component; In the frame update function Update in the real-time 3D interactive content creation and operation platform Unity, continuously track the positions of the thumb fingertip joint point A and the index fingertip joint point B and obtain the corresponding three-dimensional coordinate data; Perform filtering value judgment on the position data of joint point A and joint point B obtained in each frame. After passing the judgment, use the pinching position coordinates as a measurement point C, and use the rendering component to render a point position of the measurement point C and display it in the headset device. Subsequently, continue to monitor and trigger the fixed-point filtering standard until there is another measurement point D. Obtain a complete distance display line segment based on the two fixed points of measurement point C and measurement point D, and use the LineRenderer component rendered by measurement point C and measurement point D; calculate the distance between measurement point C and measurement point D, and assign the distance value to the initialized TextMeshPro component for distance value display.

[0014] In a second aspect, a device for measuring distances by hand pointing in mixed reality provided by an embodiment of the present invention includes: An initialization module, configured to initialize the mixed reality device and select a measurement method according to the user's needs, where the measurement method includes real-time distance measurement with both index fingers or fixed-point distance measurement with the thumb and index finger in contact; A hand recognition module, configured to obtain the actions of the user within the shooting range of the depth camera, perform hand recognition on the scene image at the current moment, and determine the poses of the user's hand joints at the current moment; A pose acquisition module, configured to determine whether the user triggers a target gesture according to the poses of the hand joints at the current moment, and acquire the poses of the specified joints in the target gesture; A distance calculation module, configured to calculate the distance between the specified joints, determine whether the user triggers the fixed-point standard according to the distance between the specified joints, and acquire the joint poses at the display positions of the rendering components; A line segment drawing module, configured to perform real-time drawing of a virtual distance line segment and display the distance value according to the acquired display positions of the distance rendering components.

[0015] As a possible implementation manner of this embodiment, the real-time distance measurement with both index fingers is to detect the fingertip positions of the user's both index fingers, and calculate and display the distance between the fingertips of the both index fingers in real time; The fixed-point distance measurement with the thumb and index finger in contact means that the user determines two measurement points by pinching the thumb and index finger, records these two measurement points, and calculates the distance between them.

[0016] The beneficial effects produced by the technical solution of the embodiment of the present invention are as follows: A method for hand-specified point ranging in mixed reality according to an embodiment of the present invention includes the following steps: Initialize the mixed reality device and select a measurement method according to the user's needs. The measurement methods include real-time ranging with both index fingers or fixed-point ranging with the thumb and index finger in contact; Obtain the actions of the user within the shooting range of the depth camera, perform hand recognition on the scene image at the current moment, and determine the poses of the user's hand joint points at the current moment; Determine whether the user triggers a target gesture based on the poses of the hand joint points at the current moment, and obtain the poses of the specified joint points in the target gesture; Calculate the distance between the specified joint points, and determine whether the user triggers a fixed-point standard based on the distance between the specified joint points, and obtain the joint poses of the display positions of the rendering components; After determining that the fixed-point standard is triggered, draw a virtual distance line segment in real time according to the obtained distance and display the distance value at the display position of the rendering component. The present invention combines the depth camera and hand tracking technology to achieve high-precision distance measurement in a mixed reality environment, solves the deficiencies of traditional measurement methods, and provides real-time and accurate measurement results especially in dynamic environments and complex scenes. Through the drawing of virtual distance line segments and instant feedback, the user's interaction experience is enhanced, and the measurement efficiency in project acceptance and construction sites is improved. The present invention not only improves the measurement accuracy of ranging in mixed reality, but also makes the operation simpler and more convenient.

[0017] The present invention provides an efficient and accurate hand-specified point ranging method by combining mixed reality technology and gesture recognition, which can achieve real-time spatial measurement in complex environments. Compared with traditional technologies, the present invention has significant advantages in terms of accuracy, real-time performance, and user experience, and solves a number of technical problems in existing measurement methods.

[0018] The present invention can capture the positions of the user's hand joint points in real time through a depth camera and an inertial sensor, and accurately calculate the distance between specified points through the Euclidean distance formula. Compared with traditional laser rangefinders or manual measurement tools, the system can provide more stable and accurate measurement results, especially in complex scenes, where the accuracy and reliability are effectively improved.

[0019] By combining hand movements and real-time rendering technology, the present invention enables users to quickly perform measurement operations and obtain immediate feedback in a mixed reality environment. Through the real-time rendering function of the device, virtual distance line segments and measurement results can be instantly displayed in the user's field of view, avoiding the problems of cumbersome operation and inability to display results in real time of traditional measurement tools. Users can quickly perform measurements through natural gestures (such as pinching the thumb and index finger) and obtain accurate measurement values in a short time, greatly improving work efficiency and the convenience of on-site operations.

[0020] Compared with traditional measurement tools or high-precision devices, the present invention adopts an intuitive virtual display method. Users can directly see the measurement results through a mixed reality device without complex data entry or calculation. Hand gestures, as an operation means, conform to the natural interaction mode of humans, reducing the learning cost and operation difficulty. In addition, when the user performs a measurement task, the system provides instant prompts and feedback to help the user avoid misoperations and ensure the accuracy of the measurement data. This solution not only improves the operation efficiency but also greatly enhances the interaction experience of on-site staff, making the engineering measurement work more intuitive and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a method for hand-pointing distance measurement in mixed reality according to an exemplary embodiment; Figure 2 is a schematic structural diagram of a device for hand-pointing distance measurement in mixed reality according to an exemplary embodiment; Figure 3 is a flowchart of the specific implementation process of the present invention for hand-pointing distance measurement in mixed reality. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To more clearly illustrate the technical features of the solution of the present invention, the present invention will be elaborated in detail below through specific embodiments and in conjunction with its drawings.

[0023] As Figure 1 shown, a method for hand-pointing distance measurement in mixed reality provided by an embodiment of the present invention includes the following steps: Initialize the mixed reality device and select a measurement method according to the user's needs. The measurement method includes real-time distance measurement with both index fingers or fixed-point distance measurement with the thumb and index finger in contact; Obtain the actions of the user within the shooting range of the depth camera, perform hand recognition on the scene image at the current moment, and determine the pose of the user's hand joint points at the current moment; Judge whether the user triggers a target gesture based on the pose of the hand joint points at the current moment, and obtain the pose of the specified joint points in the target gesture; Calculate the distance between the specified joint points, judge whether the user triggers a fixed-point standard based on the distance between the specified joint points, and obtain the joint pose of the display position of the rendering component; After judging that the fixed-point standard is triggered, perform real-time drawing of a virtual distance line segment and display the distance value according to the obtained distance rendering component display position.

[0024] As a possible implementation manner of this embodiment, the real-time distance measurement with both index fingers is to detect the fingertip positions of the user's both index fingers, and calculate and display the distance between the fingertips of the both index fingers in real time; The method of measuring distance by fitting the thumb and index finger means that the user determines two measurement points by pinching the thumb and index finger, records these two measurement points and calculates the distance between them.

[0025] As a possible implementation of this embodiment, when obtaining the user's actions within the shooting range of the depth camera, the operation of the far pointer of the hand is turned off to prevent the far pointer from being displayed during the process of determining the point, resulting in misjudgment and accidental touch by the user.

[0026] As a possible implementation of this embodiment, determining whether the user triggers the target gesture based on the pose of the hand joint points at the current moment, and obtaining the pose of the specified joint points in the target gesture includes: In the corresponding relationship between the preset gestures and instructions of the measurement method, it is determined whether the user triggers the target gesture according to whether the instruction corresponding to the user's gesture is the fixed-point ranging instruction. Among them, the preset gesture for the real-time ranging measurement of both index fingers is the fingertips of the left index finger and the right index finger; the preset gesture for the fixed-point ranging measurement by fitting the thumb and index finger is the pinching gesture between the thumb and index finger; If the user triggers the target gesture, obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture.

[0027] As a possible implementation of this embodiment, calculating the distance between the specified joint points, determining whether the user triggers the fixed-point standard according to the distance between the specified joint points, and obtaining the joint pose of the display position of the rendering component includes: According to the Euclidean distance in three-dimensional space, calculate the distance d between joint point A and joint point B: , where, (x 1 , y 1 , z 1 ) and (x 2 , y 2 , z 2 ) are the spatial coordinates of joint point A and joint point B respectively; Set the judgment of whether the user triggers the fixed-point standard. The distance judgment standard for the real-time ranging of both index fingers is to trigger the ranging instruction when there are index fingers of both hands within the shooting range of the depth camera; the distance judgment standard for the fixed-point ranging by fitting the thumb and index finger is to trigger the ranging fixed-point standard when the distance between the thumb and index finger joint points is less than the filter value; Judge whether to trigger the distance threshold of the fixed-point ranging gesture or detect the positions of both index fingers according to the calculated distance; If so, use the fixed-point ranging gesture for fixed-point ranging; when performing real-time ranging of both index fingers, obtain the joint point A (x 1 , y 1 , z 1), and the three-dimensional coordinates of the joint point B(x 2 , y 2 , z 2 ); when the thumb and index finger are in contact and the fixed-point distance measurement is performed, the joint point A(x 1 , y 1 , z 1 ) of the index finger and the joint point B(x 2 , y 2 , z 2 ) of the thumb are obtained. The position coordinates of the pinch between the three-dimensional coordinates of the two joint points are the three-dimensional coordinates of the positioning point; Otherwise, obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture; determine whether the user triggers the fixed-point distance measurement gesture in the next frame according to the Euclidean distance between the coordinates of joint points A and B.

[0028] As a possible implementation manner of this embodiment, the real-time drawing of the virtual distance line segment and the display of the distance value according to the obtained position of the rendering component include: After detecting that the user triggers the fixed-point standard, according to the obtained distance data, use the rendering component to draw a virtual distance line segment for real-time distance measurement of the index fingers of both hands or a virtual distance line segment for fixed-point distance measurement with the thumb and index finger in contact in the user's field of view; After the virtual line segment is drawn, calculate and display the measured distance between the two points.

[0029] As a possible implementation manner of this embodiment, the use of the rendering component to draw a virtual distance line segment for real-time distance measurement of the index fingers of both hands in the user's field of view includes: Initialize the TextMeshPro component and the LineRenderer component; In the frame update function Update in the real-time 3D interactive content creation and operation platform Unity, continuously track the positions of the fingertip joint point A of the left index finger and the fingertip joint point B of the right index finger and obtain the corresponding three-dimensional coordinate data; Assign the position data of joint point A and joint point B obtained in each frame to the points in the initialized LineRenderer component; Connect joint point A and joint point B in the initialized LineRenderer component to obtain a complete distance display line segment; Calculate the distance value of each frame according to the position data of joint point A and joint point B obtained in each frame , and assign the distance value to the initialized TextMeshPro component in real time for distance value display.

[0030] As a possible implementation manner of this embodiment, the use of the rendering component to draw a virtual distance line segment for fixed-point distance measurement with the thumb and index finger in contact in the user's field of view includes: Initialize the TextMeshPro component and the LineRenderer component; In the frame update function Update in the real-time 3D interactive content creation and operation platform Unity, continuously track the positions of the thumb fingertip joint point A and the index fingertip joint point B and obtain the corresponding three-dimensional coordinate data; Perform filtering value judgment on the position data of joint point A and joint point B obtained in each frame. After passing the judgment, use the pinching position coordinates as a measurement point C, and use the rendering component to render a point position of the measurement point C and display it in the head-mounted device. Subsequently, continue to listen for the trigger of the fixed-point filtering standard until there is another measurement point D. Obtain a complete distance display line segment based on the two fixed points of measurement point C and measurement point D, and use the LineRenderer component rendered by measurement point C and measurement point D; calculate the distance between measurement point C and measurement point D, and assign the distance value to the initialized TextMeshPro component for distance value display.

[0031] As Figure 2 shown, a hand-specified point ranging device provided by an embodiment of the present invention in mixed reality includes: An initialization module for initializing the mixed reality device and selecting a measurement method according to the user's needs, where the measurement method includes real-time ranging with both index fingers or fixed-point ranging with the thumb and index finger in contact; A hand recognition module for obtaining the actions of the user within the shooting range of the depth camera and performing hand recognition on the scene image at the current moment to determine the pose of the user's hand joints at the current moment; A pose acquisition module for judging whether the user triggers a target gesture based on the pose of the hand joints at the current moment and obtaining the pose of the specified joint in the target gesture; A distance calculation module for calculating the distance between the specified joints, judging whether the user triggers the fixed-point standard based on the distance between the specified joints, and obtaining the joint pose of the rendering component display position; A line segment drawing module for, after judging that the fixed-point standard is triggered, performing real-time drawing of a virtual distance line segment and displaying the distance value according to the obtained distance rendering component display position.

[0032] As Figure 3 shown, the specific implementation process of the present invention for hand-specified point ranging in mixed reality is as follows.

[0033] 1. Initialize the mixed reality device and start real-time ranging with both index fingers or fixed-point ranging with the thumb and index finger in contact.

[0034] First, it is necessary to start the mixed reality device and select the corresponding measurement method according to the user's needs: Real-time distance measurement with both index fingers: By detecting the fingertip positions of the user's both index fingers, calculate and display the distance between the two fingertips in real time; Distance measurement with the thumb and index finger in contact at a fixed point: The user determines two measurement points by pinching the thumb and index finger together, and the system records these two points and calculates the distance between them.

[0035] Initialize the mixed reality device to make it ready, and start the corresponding functions on the device so that the user can start using it. In this case, the user can choose to start the real-time distance measurement with both index fingers or the distance measurement with the thumb and index finger in contact at a fixed point according to actual needs.

[0036] II. Obtain the actions of the user within the shooting range of the depth camera.

[0037] Before the step of obtaining the actions of the user within the shooting range of the depth camera, the hand far pointer operation needs to be turned off to prevent the far pointer from being displayed during the fixed-point process, resulting in misjudgment and accidental touch by the user.

[0038] The depth camera can capture the hand actions of the user, and these actions can be used to determine the positions of the hand joint points of the user.

[0039] III. Perform hand recognition on the scene image at the current moment to determine the poses of the user's hand joint points at the current moment.

[0040] By analyzing the scene image, the hand joint points of the user can be recognized. These joint points can be represented in the image and can be used as the basis for subsequent calculations.

[0041] IV. When it is determined that the user triggers the target gesture based on the poses of the hand joint points at the current moment, obtain the poses of the specified joint points in the target gesture.

[0042] (1) The preset gesture for real-time distance measurement with both index fingers is the fingertip of the left index finger and the fingertip of the right index finger; (2) The preset gesture for distance measurement with the thumb and index finger in contact at a fixed point is the pinching gesture between the thumb and index finger.

[0043] In the corresponding relationship between the preset gestures and instructions, check whether the instruction corresponding to the user's gesture is the fixed-point distance measurement instruction; If the judgment is yes, it is determined that the user triggers the target gesture.

[0044] Then obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture; If not, execute the step: Obtain the actions of the user within the shooting range of the depth camera.

[0045] By comparing the hand joint points of the user with the positions of the expected target gestures, it can be determined whether the user has triggered the target gesture. If the user has triggered the target gesture, the poses of the specified joint points in the target gesture can be obtained for subsequent calculations.

[0046] V. Calculate the distance between the specified joint points; the distance is used to determine whether the user has triggered the fixed-point criterion and obtain the joint poses of the display positions of the rendering components. According to the Euclidean distance in three-dimensional space, for two joint points A(x 1 ,y 1 ,z 1 ) and B(x 2 ,y 2 ,z 2 ), there is the following distance calculation formula: .

[0047] There are different distance judgment criteria for the two modes respectively: (1) Real-time distance measurement between the index fingers of both hands: The distance measurement instruction is triggered when there are index fingers of both hands within the shooting range of the depth camera; (2) Fixed-point distance measurement by fitting the thumb and index finger: The fixed-point distance measurement criterion is triggered when the distance between the thumb and index finger joint points is less than the filter value.

[0048] Judge whether to trigger the distance threshold of the fixed-point distance measurement gesture or detect the positions of the index fingers of both hands according to the calculated distance; If so, the user uses the fixed-point distance measurement gesture for fixed-point distance measurement: (1) Real-time distance measurement between the index fingers of both hands: Obtain the three-dimensional coordinates of two joint points A(x 1 ,y 1 ,z 1 ) and B(x 2 ,y 2 ,z 2 ) of the index fingers of both hands; (2) Fixed-point distance measurement by fitting the thumb and index finger: Obtain the three-dimensional coordinates of two joint points A(x 1 ,y 1 ,z 1 ) of the index finger and B(x 2 ,y 2 ,z 2 ) of the thumb, and the position coordinates of the pinch between them are the three-dimensional coordinates of the positioning point; If not, execute the steps: Obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture; judge whether the user triggers the fixed-point distance measurement gesture in the next frame according to the Euclidean distance between the coordinates of joint points A and B.

[0049] By calculating the distance between specified joint points, it is possible to determine whether the user has triggered the fixed-point criterion. This distance can be used to represent the distance relationship between the user and the target object.

[0050] VI. After determining that the fixed-point criterion has been triggered, a virtual distance line segment is drawn in real time according to the obtained distance to render the component display position. Initialize the TextMeshPro component and the LineRenderer component.

[0051] Drawing a virtual distance line segment during real-time distance measurement with the index fingers of both hands: In the Update function of the frame in Unity, continuously track the positions of the fingertip joint point A of the left index finger and the fingertip joint point B of the right index finger and obtain the corresponding three-dimensional coordinate data; assign the position data of joint point A and joint point B obtained in each frame to the points in the initialized LineRenderer component; connect these two points to obtain a complete distance display line segment; and calculate the distance value for each frame according to the position data of joint point A and joint point B obtained in each frame, using the following formula , and assign the distance value to the initialized TextMeshPro component in real time for distance value display.

[0052] Drawing a virtual distance line segment during fixed-point distance measurement with the thumb and index finger in contact: In the Update function of the frame in Unity, continuously track the positions of the fingertip joint point A of the thumb and the fingertip joint point B of the index finger and obtain the corresponding three-dimensional coordinate data; perform a filtering value judgment on the position data of joint point A and joint point B obtained in each frame. After passing the judgment, use the pinching position coordinates as a measurement point C, and render a point position of the measurement point C using the rendering component and display it in the head-mounted device. Subsequently, continue to monitor and trigger the fixed-point filtering criterion until there are two measurement points C and D. At this time, a complete distance display line segment is obtained based on the two fixed points, through the LineRenderer component rendered by the measurement points C and D; calculate the distance between the measurement points C and D, and assign the distance value to the initialized TextMeshPro component for distance value display.

[0053] By using the distance value as a parameter, the rendering component can draw a virtual distance line segment in real time, making it convenient for the user to intuitively understand and master.

[0054] VII. After rendering is completed, the distance value is displayed.

[0055] When the system detects that the user has triggered the fixed-point criterion, a virtual distance line segment will be drawn in real time in the user's field of view according to the obtained distance data. This line segment will connect the two measured points and be displayed on the device screen together with the virtual marker, helping the user to intuitively view the measurement result. This process is drawn in real time through the rendering component of the device.

[0056] Once the virtual line segment is drawn, the system will immediately calculate and display the measured distance between the two points. This distance value will be presented in the form of virtual text on the device display interface, and the user can directly see the measurement result. After this step is completed, the user can continue with the next measurement or complete the verification.

[0057] Display the rendered distance line segment and its corresponding distance value for the convenience of the user.

[0058] The present invention realizes the method of distance measurement by hand-specified points through mixed reality technology, which not only improves the measurement accuracy, but also makes the operation simpler and more convenient.

[0059] The present invention realizes the high-precision measurement of spatial geometric data based on the MR-based spatial measurement technology, and can be widely applied to fields such as power transmission and transformation project acceptance, building construction, virtual reality, industrial inspection, etc. The present invention utilizes the hardware such as cameras, depth sensors, and inertial sensors in the MR device to achieve precise measurement of the geometric dimensions of objects in the real world, and can provide real-time feedback in complex engineering sites to assist engineers in on-site verification and correction.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for measuring distance at a specified point in mixed reality, characterized in that: The steps include: Initialize the mixed reality device and select a measurement method according to the user's needs, wherein the measurement method includes real-time distance measurement using the index fingers of both hands or fixed-point distance measurement using the thumb and index finger in close contact; Obtain the user's movements within the shooting range of the depth camera, and perform hand recognition on the scene image at the current moment to determine the position and posture of the user's hand joints at the current moment; Determine whether the user has triggered the target gesture based on the current position of the hand joint points, and obtain the position of the specified joint points in the target gesture; Calculate the distance between the specified joint points, determine whether the user triggers the fixed-point standard based on the distance between the specified joint points, and obtain the joint pose of the display position of the rendering component; According to the obtained distance rendering component display position, a virtual distance line segment is drawn in real time and the distance value is displayed.

2. The method for measuring distance from a designated point in mixed reality according to claim 1, characterized in that: The real-time distance measurement of the index fingers of both hands is to detect the fingertip positions of the index fingers of both hands of the user, and calculate and display the distance between the fingertips of the index fingers of both hands in real time; The method of measuring the distance at a fixed point by pinching the thumb and index finger together is that the user determines two measuring points by pinching the thumb and index finger together, records the two measuring points and calculates the distance between them.

3. The method for measuring distance from a designated point in mixed reality according to claim 1, characterized in that: Disables hand far pointer operation when acquiring the user's movements within the depth camera's range.

4. The method for measuring distance from a designated point in mixed reality according to any one of claims 1 to 3, characterized in that: The step of judging whether the user has triggered a target gesture based on the position and posture of the hand joint points at the current moment, and obtaining the position and posture of the specified joint points in the target gesture, includes: In the correspondence between gestures and instructions preset in the measurement mode, whether the user triggers the target gesture is determined based on whether the instruction corresponding to the user gesture is a fixed-point distance measurement instruction. The gestures preset in the real-time distance measurement mode of the index fingers of both hands are the fingertips of the left index finger and the fingertips of the right index finger; the gestures preset in the fixed-point distance measurement mode of the thumb and index finger fitting together are the pinching gestures between the thumb and index finger. If the user triggers a target gesture, obtain the three-dimensional position coordinates of joint point A and joint point B in the target gesture.

5. The method for measuring distance from a designated point in mixed reality according to claim 4, characterized in that: The calculating of the distance between the specified joint points, judging whether the user triggers the fixed point standard according to the distance between the specified joint points, and obtaining the joint pose of the display position of the rendering component includes: According to the Euclidean distance in three-dimensional space, the distance d between joint point A and joint point B is calculated: ， Among them, (x1, y1, z1) and (x2, y2, z2) are the spatial coordinates of joint point A and joint point B respectively; Set the standard for judging whether the user triggers the fixed-point measurement. The distance judgment standard for real-time distance measurement with the index fingers of both hands is that the distance measurement command is triggered when the index fingers of both hands are within the shooting range of the depth camera; the distance judgment standard for fixed-point distance measurement with the thumb and index finger touching is that the distance measurement fixed-point standard is triggered when the distance between the joint points of the thumb and index finger is less than the filter value; The calculated distance is used to determine whether to trigger the distance threshold of the fixed-point distance measurement gesture or detect the position of the index fingers of both hands; If yes, use the fixed-point distance measurement gesture for fixed-point distance measurement; when the index fingers of both hands measure the distance in real time, obtain the three-dimensional coordinates of the joint points A (x1, y1, z1) and B (x2, y2, z2) of the index fingers of both hands; when the thumb and index finger are placed together for fixed-point distance measurement, obtain the three-dimensional coordinates of the two joint points A (x1, y1, z1) of the index finger and B (x2, y2, z2) of the thumb, and the coordinates of the pinched position between the two joint points are the three-dimensional coordinates of the positioning point; Otherwise, obtain the three-dimensional position coordinates of joint points A and B in the target gesture; and determine whether the user triggers a fixed-point ranging gesture in the next frame based on the Euclidean distance between the coordinates of joint points A and B.

6. The method for measuring distance from a designated point in mixed reality according to claim 5, characterized in that: The method of drawing a virtual distance line segment in real time and displaying the distance value according to the acquired distance rendering component display position includes: After detecting that the user has triggered the fixed-point standard, based on the acquired distance data, a rendering component is used to draw in real time in the user's field of view a virtual distance line segment for real-time distance measurement using the index fingers of both hands or a virtual distance line segment for fixed-point distance measurement using the thumb and index finger in close proximity; After the virtual line segment is drawn, the measured distance between the two points is calculated and displayed.

7. The method for measuring distance from a designated point in mixed reality according to claim 6, characterized in that: The method of using a rendering component to draw a virtual distance line segment for real-time distance measurement by the index fingers of both hands in the user's field of view in real time includes: Initialize the TextMeshPro component and LineRenderer component; In the frame update function Update in Unity, a real-time 3D interactive content creation and operation platform, the positions of the left index finger joint point A and the right index finger joint point B are continuously tracked and the corresponding three-dimensional coordinate data are obtained; Assign the position data of joint point A and joint point B obtained in each frame to the points in the initialized LineRenderer component; Connect joint point A and joint point B in the initialized LineRenderer component to obtain a complete distance display line segment; Calculate the distance value of each frame based on the position data of joint point A and joint point B obtained in each frame , assign the distance value to the initialized TextMeshPro component in real time to display the distance value.

8. The method for measuring distance from a designated point in mixed reality according to claim 6, characterized in that: The method of using a rendering component to draw a virtual distance line segment for measuring the distance between the thumb and the index finger in real time in the user's field of view includes: Initialize the TextMeshPro component and LineRenderer component; In the frame update function Update in Unity, a real-time 3D interactive content creation and operation platform, the positions of the thumb joint point A and the index finger joint point B are continuously tracked and the corresponding three-dimensional coordinate data are obtained; The position data of joint point A and joint point B obtained in each frame are filtered and judged. If qualified, the pinched position coordinates are used as a measurement point C, and the measurement point C is rendered as a point position using the rendering component and displayed in the head display device. Then, the fixed-point filtering standard is continuously monitored until there is another measurement point D. A complete distance display line segment based on the two fixed points of measurement point C and measurement point D is obtained, and the LineRenderer component rendered by measurement point C and measurement point D is used; the distance between measurement point C and measurement point D is calculated, and the distance value is assigned to the initialized TextMeshPro component for distance value display.

9. A device for measuring distance to a designated point in mixed reality, characterized in that: include: An initialization module is used to initialize the mixed reality device and select a measurement method according to the user's needs, wherein the measurement method includes real-time distance measurement using the index fingers of both hands or fixed-point distance measurement using the thumb and index finger in close contact; The hand recognition module is used to obtain the user's movements within the shooting range of the depth camera, and perform hand recognition on the scene image at the current moment to determine the position and posture of the user's hand joint points at the current moment; The posture acquisition module is used to determine whether the user triggers the target gesture based on the posture of the hand joint points at the current moment, and obtain the posture of the specified joint points in the target gesture; The distance calculation module is used to calculate the distance between the specified joint points, determine whether the user triggers the fixed point standard according to the distance between the specified joint points, and obtain the joint pose of the display position of the rendering component; The line segment drawing module is used to draw virtual distance line segments in real time and display distance values ​​according to the acquired distance rendering component display position.

10. The device for measuring distance to a designated point in mixed reality according to claim 9, characterized in that: The real-time distance measurement of the index fingers of both hands is to detect the fingertip positions of the index fingers of both hands of the user, and calculate and display the distance between the fingertips of the index fingers of both hands in real time; The method of measuring the distance at a fixed point by pinching the thumb and index finger together is that the user determines two measuring points by pinching the thumb and index finger together, records the two measuring points and calculates the distance between them.

Images