A remotely-guidable lamp mirror system and method

Abstract

The present application relates to the technical field of smart home, in particular to a lamp mirror system and method capable of remote guidance, the lamp mirror system comprises a lamp mirror body, an expert terminal and a cloud server. The purpose of the present application is to provide a lamp mirror system and method capable of remote guidance, computer equipment and storage medium, through integration of video acquisition, three-dimensional posture tracking, augmented reality rendering and cloud data processing, real-time capture of user real state and high-precision virtual marker space anchoring can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, in particular to a lamp mirror system and method capable of remote guidance. BACKGROUND

[0002] At present, in many fields that require high-precision operation and fine guidance, such as remote medical treatment, complex equipment maintenance, skill training, etc., professional guidance personnel need to be able to clearly observe the state of the operator and provide accurate instructions. The traditional remote guidance method mainly relies on video calls, and the guide can only judge the operator's actions and state through two-dimensional video information. This method has the following significant problems:

[0003] 1. Limited information dimension, large understanding deviation: Two-dimensional video cannot fully display the three-dimensional spatial relationship and motion trajectory of the operator's body or face, making it difficult for the guide to accurately understand the operator's real posture and intention, thereby affecting the effectiveness of the guidance.

[0004] 2. Guidance and indication are not intuitive and accurate enough: The guide can only convey instructions through verbal descriptions or vague two-dimensional annotations on the video screen, which are often difficult to accurately correspond to the physical space of the operator's actual operation, easily causing misunderstandings and operation deviations.

[0005] 3. Lack of real-time interaction and immersion: The existing remote guidance lacks deep linkage with the real state of the operator, the user experience is poor, and it is difficult to provide a sense of in-situ guidance.

[0006] In order to overcome these problems, some research has begun to explore the application of augmented reality (AR) technology in remote guidance, trying to superimpose virtual information in the operator's perspective to improve the intuitiveness of guidance. However, existing AR remote guidance systems still have the following deficiencies:

[0007] 1. Insufficient three-dimensional posture perception capability: Most systems cannot capture and analyze user posture data in three-dimensional space in real time and with high precision, resulting in the inability to generate accurate anchor instructions.

[0008] 2. Inaccurate anchoring of AR markers with user's real state: Existing AR markers often cannot achieve precise real-time tracking and anchoring with the user's body or face. Once the user moves or changes posture, the AR marker may deviate from its intended position, reducing the reliability of the guidance.

[0009] 3. Interaction and real-time performance need to be improved: The system's process of receiving and processing expert feedback information and converting it into user-perceptible AR instructions is not efficient enough, affecting the real-time performance and smoothness of the guidance.

[0010] Therefore, the present application aims to solve the problems of limited information dimension, inaccurate guidance, and unstable AR anchoring in the existing remote guidance technology, and provides a novel, more practical and efficient lamp mirror system that can be remotely guided. SUMMARY

[0011] In order to overcome the shortcomings and deficiencies existing in the prior art, the present application aims to provide a lamp mirror system, method, computer device and storage medium that can be remotely guided, which can realize real-time capture of user real state and high-precision spatial anchoring of virtual markers by integrating video acquisition, three-dimensional pose tracking, augmented reality rendering and cloud data processing.

[0012] The present application is realized by the following technical solutions:

[0013] In a first aspect, the present application discloses a lamp mirror system that can be remotely guided, which comprises:

[0014] a lamp mirror body configured to:

[0015] capture a video stream reflecting the real state of a user;

[0016] real-time track three-dimensional pose data of the user;

[0017] receive augmented reality instructions and render augmented reality markers superimposed on the real image of the user on the lamp mirror body;

[0018] an expert terminal configured to:

[0019] display the video stream;

[0020] receive two-dimensional marker input generated by an expert on the video stream;

[0021] and a cloud server configured to:

[0022] receive video stream and three-dimensional pose data from the lamp mirror body;

[0023] receive two-dimensional marker input from the expert terminal;

[0024] determine three-dimensional anchor coordinates of the marker in three-dimensional space according to the two-dimensional marker input and the three-dimensional pose data;

[0025] generate the augmented reality instructions containing the three-dimensional anchor coordinates and send them to the lamp mirror body;

[0026] the augmented reality markers rendered by the lamp mirror body, the position of which on the expert terminal changes synchronously with the movement of the user to achieve the visual effect of being anchored to the body or face.

[0027] In combination with the first aspect, further, the lamp mirror body comprises:

[0028] A controllable lighting system;

[0029] and a color calibration module configured to control the controllable lighting system and capture a calibration image at the beginning of a session to color correct the video stream, thereby providing a video stream to the expert terminal.

[0030] In combination with the first aspect, further, the three-dimensional pose data comprises a three-dimensional mesh model constructed in real time from the user's body or face.

[0031] In combination with the first aspect, further, the cloud server maps the position coordinates of the two-dimensional marker input onto the three-dimensional mesh model to determine the three-dimensional anchor coordinates by a back projection algorithm.

[0032] In combination with the first aspect, further, the two-dimensional marker input comprises instructions generated by at least one of a paint tool, a measurement tool, a geometric shape tool, or a three-dimensional model placement tool.

[0033] In combination with the first aspect, further, the two-dimensional marker input comprises instructions generated by at least one of a paint tool, a measurement tool, a geometric shape tool, or a three-dimensional model placement tool.

[0034] capturing a video stream reflecting the real state of the user by the lamp mirror body and tracking three-dimensional pose data of the user in real time;

[0035] sending the video stream and the three-dimensional pose data to a cloud server by the lamp mirror body;

[0036] displaying the video stream by an expert terminal and receiving a two-dimensional marker input generated by an expert;

[0037] sending the two-dimensional marker input to the cloud server by the expert terminal;

[0038] determining a three-dimensional anchor coordinate of the marker in three-dimensional space by the cloud server according to the two-dimensional marker input and the three-dimensional pose data;

[0039] generating an augmented reality instruction containing the three-dimensional anchor coordinate by the cloud server and sending it to the lamp mirror body;

[0040] receiving and executing the augmented reality instruction by the lamp mirror body, and rendering an augmented reality marker anchored to the three-dimensional anchor coordinate and superimposed on the real image of the user on the expert terminal.

[0041] In combination with the second aspect, further, before the step of capturing the video stream, further comprising:

[0042] automatically calibrating color by the lamp mirror body to ensure color fidelity of the video stream.

[0043] In combination with the second aspect, further comprising the following steps:

[0044] capturing actual operation video of the user by the lamp mirror body;

[0045] comparing and analyzing the actual operation video with the augmented reality mark in real time;

[0046] generating and displaying difference feedback information about user operation accuracy according to the comparison and analysis result.

[0047] In combination with the second aspect, further, the augmented reality instruction is structured data containing the three-dimensional anchor coordinates and rendering attributes such as mark type, color, size, etc.

[0048] In combination with the second aspect, further, the method is applied in scenarios of remote medical diagnosis, remote makeup teaching, remote fitness guidance or remote industrial maintenance guidance.

[0049] In the third aspect, the present application further discloses a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to realize the method of the remote-guidable lamp mirror system as described above.

[0050] In the fourth aspect, the present application further discloses a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the method of the remote-guidable lamp mirror system as described above.

[0051] The present application has the following beneficial effects:

[0052] 1. Provide high-dimensional and accurate remote guidance: the system can capture video stream reflecting the real state of the user and track the three-dimensional pose data of the user in real time. Through the fusion analysis of the two-dimensional mark input by the expert and the three-dimensional pose data of the user by the cloud server, the anchor coordinates of the mark in the three-dimensional space are accurately calculated. This enables the instructor to understand the operation state of the user from the three-dimensional perspective, thereby providing more accurate and more practical guidance.

[0053] 2. Achieving high-precision, real-time augmented reality anchoring: The augmented reality instructions generated by the system contain precise three-dimensional anchor coordinates and can be sent to the lamp body for rendering. The augmented reality markers rendered by the lamp body can change synchronously with the user's movement, anchored to a specific position on the user's body or face, providing visual cues as if they were actually present. This precise anchoring greatly improves the effectiveness of the guidance and the accuracy of the operation, avoiding the problem of ineffective indication due to changes in perspective or posture.

[0054] 3. Enhancing the intuitiveness and ease of understanding of guidance: By superimposing augmented reality markers on the user's real image, the system converts abstract guidance instructions into visual information that the user can directly perceive and understand. For example, it can accurately indicate the bending angle of a user's joint or the movement direction of a certain part of the body, significantly reducing communication costs and understanding biases.

[0055] 4. Improving the interactivity and efficiency of remote guidance: The system can receive two-dimensional marker input generated by the expert terminal in real time and quickly convert it into precise three-dimensional augmented reality instructions. This efficient feedback loop allows experts to provide immediate and accurate feedback and corrections to the user's operation, greatly improving the efficiency and response speed of remote guidance.

[0056] 5. Expanding application fields and improving user experience: The system can be widely used in remote medical surgery guidance, remote technical training, complex equipment operation guidance, sports teaching, and other scenarios, effectively addressing the pain points of remote guidance and improving the overall experience and work efficiency of operators and guides. BRIEF DESCRIPTION OF DRAWINGS

[0057] The invention is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the invention. For ordinary skilled persons in the art, other drawings can be obtained without creative labor based on the following drawings.

[0058] Figure 1 A module schematic diagram of a database system of an embodiment of the invention.

[0059] Figure 2 A flowchart of a method of a remote-guidable lamp system of an embodiment of the invention.

[0060] Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the invention.

[0061] Figure 4 A schematic diagram of a storage medium provided by an embodiment of the invention. DETAILED DESCRIPTION

[0062] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] Example 1

[0064] like Figure 1 As shown, this embodiment discloses a remotely guided lamp-mirror system, which includes:

[0065] The lamp mirror body 101 is configured for:

[0066] Capture video streams that reflect the user's real-time state;

[0067] Real-time tracking of the user's 3D pose data;

[0068] Receive augmented reality instructions and render augmented reality markers superimposed on the user's real image on the lamp mirror body 101;

[0069] Expert terminal 102, configured for:

[0070] Display the video stream;

[0071] Receive two-dimensional marker input generated by an expert on the video stream;

[0072] And cloud server 103, configured for:

[0073] Receive video streams and three-dimensional pose data from the lamp mirror body 101;

[0074] Receive two-dimensional marker input from the expert terminal 102;

[0075] Based on the two-dimensional marker input and the three-dimensional pose data, determine the three-dimensional anchoring coordinates of the marker in three-dimensional space;

[0076] Generate the augmented reality command containing the three-dimensional anchor coordinates and send it to the lamp mirror body 101;

[0077] The augmented reality marker rendered by the lamp mirror body 101 changes position on the expert terminal in sync with the user's movement to achieve a visual effect anchored to the user's body or face.

[0078] The core purpose of designing the lamp-mirror system in this embodiment is to fundamentally solve the two major challenges of information fidelity and command accuracy in remote spatial guidance scenarios.

[0079] Specifically, the design purposes can be divided into the following three points:

[0080] In the background art, whether it is telephone guidance or video call, when it comes to such spatial instructions as here, that corner, a little bit aside, the ambiguity of language and the indirectness of pointing will result in extremely low communication efficiency and extremely high risk of misunderstanding. Especially in the fields of high precision requirements such as medical treatment, maintenance, teaching, etc., such ambiguity is unacceptable. The purpose of the present scheme is to replace the inefficient oral description with a precise, intuitive and unambiguous visual language, and to transmit the expert's intention to the user without loss.

[0081] Traditional remote interaction lacks the sense of presence. Experts cannot stand beside the user and point with their fingers. This distance weakens the immersion of guidance and the trust of users. The purpose of the present scheme is to reconstruct an immersive guidance relationship in the digital space by enabling experts to perform virtual operations on the user's real-world image in real time and directly.

[0082] Many current smart devices (especially smart mirrors) are in the dilemma of not painful and not trivial functions, and unclear value proposition. Users spend a lot of money, but the functions they get are often not a necessity. By transforming the device into a powerful port connected to professional services, its value jumps from information display to professional service carrying, thus fundamentally solving its market positioning problem.

[0083] Based on the above design purposes, the mirror system recorded in the present embodiment can achieve the following effects in actual operation:

[0084] The transformation from description to pointing is realized. The communication mode between the expert and the user changes from me listening to you talking to me looking at you drawing, realizing the seen is the obtained guidance, so that complex spatial instructions can be accurately understood and executed within a few seconds.

[0085] Due to the elimination of the ambiguous zone in language communication, the time of the entire guidance process is greatly shortened, and the operation failure rate caused by misunderstanding also tends to be zero. In the scenes of industrial maintenance and medical diagnosis, this means a significant improvement in productivity and a significant reduction in safety risks.

[0086] The present system makes it possible for professional services that were previously unable or difficult to be carried out online. For example, a dermatologist can remotely and accurately circle the skin lesions that need attention; a makeup master can draw an eyeliner on the user's face; a fitness trainer can mark the student's wrong body posture in real time.

[0087] From the user's perception and rendering, to the expert's observation and operation, to the cloud's core calculation and synchronization, the three work together to form a stable, reliable, and implementable system architecture. It is not just a concept, but a complete solution that defines the core responsibilities and interaction logic of each part and can be implemented.

[0088] Embodiment 2

[0089] In this embodiment, the lamp mirror body 101 includes:

[0090] a controllable lighting system;

[0091] and a color calibration module configured to control the controllable lighting system and capture a calibration image at the beginning of a session to color correct the video stream, thereby providing a video stream to the expert terminal 102.

[0092] After the basic framework of the remote guidance system is constructed as described in the embodiment, the design purpose of the lamp mirror body 101 is to solve a deeper and more important problem in the professional field: the fidelity of visual information. The core purpose is to eliminate the color difference caused by environmental light and hardware differences and ensure that what the expert sees is the true color of the user.

[0093] Specifically, the design purpose can be divided into the following two points:

[0094] Ensure the objective basis for professional judgment: In many professional fields, color is the key information for making correct judgments. For example, a dermatologist needs to judge the nature of a skin lesion based on its true color (fresh red, dark red, or purple); a makeup artist needs to recommend the most suitable makeup product based on the user's true skin color and lip color; an industrial quality inspector needs to determine whether there is a flaw based on the color difference on the surface of the product. An ordinary camera will show a completely different color of the same object under different environmental light (such as daylight during the day, sunset in the evening, incandescent or fluorescent light indoors). The purpose of this embodiment is to remove the interference of environmental light and restore the absolute color of the object through a set of active and standardized calibration process, providing an objective and reliable visual basis for the expert's remote observation, hearing, asking, and cutting.

[0095] Establishing reliable and repeatable evaluation criteria: Professional services require not only the accuracy of a single judgment, but also the consistency between multiple observations. For example, a user needs to track a skin condition over several weeks or evaluate the effect of a skincare product over a period of time. If the color reference for each observation is different due to environmental changes, any color-based longitudinal comparison will lose its meaning. The purpose of this solution is to establish a unified, device-independent color space for each remote session, ensuring that the skin tone seen in the living room today is comparable to the skin tone seen in the bedroom next week after system calibration, thereby enabling reliable and quantifiable effect tracking and evaluation.

[0096] Based on the above design purposes, the embodiment can achieve high visual fidelity. Through active lighting and color calibration, the video stream output by the system is no longer a consumer-level entertainment image, but a professional-level high-fidelity image that meets the requirements of professional diagnosis and evaluation. Experts can make judgments based on the colors they see, greatly improving the reliability and professionalism of remote services.

[0097] In addition, whether the user is at the window in the morning with bright sunshine or in the bedroom at night with dim light, the calibration module can effectively neutralize the influence of environmental light and output standardized colors. This allows remote services to be free from time and location constraints, ensuring the consistency and comparability of evaluation results under any lighting conditions.

[0098] After adding this function, the application range of the system has been greatly expanded. In addition to dermatology, cosmetics, and other fields, it can also be extended to remote art appraisal, textile dye quality inspection, printed matter color difference correction, interior design color matching consultation, and other industries that have extremely high requirements for color fidelity.

[0099] Embodiment 3

[0100] If the three-dimensional pose data is only a set of three-dimensional coordinates of bone joints (e.g., the positions of the wrist, elbow, and shoulder), the AR marker can only be attached to these points. If you want to mark the area between two points, the system can only perform rough interpolation calculations. When the user is active, this interpolation point is difficult to perfectly synchronize with the real skin surface, causing the AR marker to appear to be sliding or floating on the skin, greatly damaging the realism and accuracy.

[0101] In addition, a simple set of bone points cannot support complex visual effects. For example, the system cannot calculate the lighting changes of an AR marker (such as a virtual tattoo) in different orientations of the arm, nor can it allow an AR blush to produce a natural transition in density according to the curvature of the cheek. Bone points also cannot perform area-related analysis.

[0102] Therefore, further, the three-dimensional pose data includes a three-dimensional mesh model constructed in real time according to the user's body or face, and the perception ability of the system is improved from discrete bone points to continuous geometric curves, providing a solid digital foundation for realizing truly realistic and accurate AR interaction.

[0103] Since the system has complete surface topological information, experts can mark any position on the user's body, whether it is the cheek, forehead, or a certain muscle on the back. The AR marker is anchored on a specific triangular face of the Mesh, and when the user moves, turns around, or the muscle contracts, causing the surface to deform, the marker will move completely synchronously, conform to the perspective and morphological changes, realizing a truly skin-conforming visual effect and completely solving the problems of suspension and sliding.

[0104] Further, the 3D Mesh contains normal vector information of each triangular face, which enables the AR rendering engine to calculate lighting, and the 3D Mesh provides a clear and stable target surface. The virtual light rays emitted from the expert's screen can accurately calculate the intersection with this target surface, improving the robustness and accuracy of the calculation and unlocking the ability of geometric morphological analysis.

[0105] Since the expert's input (such as mouse clicks, touch screen swipes) is essentially performed on a two-dimensional plane, while the user's body and the environment in which it is located are three-dimensional. After establishing the 3D Mesh as the digital foundation of the user's geometric morphology, the cloud server 103 maps the position coordinates of the two-dimensional marker input to the three-dimensional mesh model through a back projection algorithm to determine the three-dimensional anchor coordinates, solving the dimension conversion problem.

[0106] Embodiment 4

[0107] The two-dimensional marker input includes instructions generated by at least one of a brush tool, a measurement tool, a geometric shape tool, or a three-dimensional model placement tool.

[0108] This embodiment can provide the most intuitive and most human-friendly qualitative guidance method. By introducing circle drawing, line drawing, arrow pointing, and other methods with quantitative analysis capabilities, experts can quickly define the scope, indicate the direction, emphasize the key points, and conduct free-form visual communication;

[0109] The purpose of this embodiment is to greatly enrich and expand the expression ability and application depth of the system, and to upgrade the system from a simple two-dimensional one-way instruction that can only point to a comprehensive remote professional toolbox to meet the diverse and complex guidance needs of different professional fields.

[0110] Embodiment 5

[0111] As Figure 2As shown, a method of remotely guiding a lamp mirror system, comprising the following steps:

[0112] capturing a video stream reflecting a real state of a user by a lamp mirror body 101, and tracking three-dimensional pose data of the user in real time;

[0113] sending the video stream and the three-dimensional pose data to a cloud server 103 by the lamp mirror body 101;

[0114] displaying the video stream by an expert terminal 102, and receiving a two-dimensional marker input generated by an expert;

[0115] sending the two-dimensional marker input to the cloud server 103 by the expert terminal 102;

[0116] determining a three-dimensional anchor coordinate of the marker in a three-dimensional space according to the two-dimensional marker input and the three-dimensional pose data by the cloud server 103;

[0117] generating an augmented reality instruction containing the three-dimensional anchor coordinate by the cloud server 103, and sending it to the lamp mirror body 101;

[0118] receiving and executing the augmented reality instruction by the lamp mirror body 101, and rendering an augmented reality marker anchored to the three-dimensional anchor coordinate on its expert terminal, which is superimposed on the real image of the user.

[0119] In this embodiment, the complete information flow and processing flow from end to end necessary for realizing remote scene guidance are clearly outlined in time sequence and logical relationship. The defined step sequence and data flow naturally support efficient and low-latency implementation.

[0120] For example, it places time-consuming computing tasks on powerful clouds, and focuses on client tasks on collection and rendering, which can ensure the performance of smooth real-time interaction experience.

[0121] In addition, this embodiment also clearly defines the input and output of the lamp mirror body 101, the cloud and the expert terminal 102, so that the three can be highly decoupled in development.

[0122] Further, the step of automatically performing color calibration by the lamp mirror body 101 to ensure the color fidelity of the video stream.

[0123] Further, it also includes the following steps:

[0124] capturing the actual operation image of the user by the lamp mirror body 101;

[0125] real-time comparison and analysis of the actual operation image and the augmented reality marker;

[0126] According to the comparative analysis result, difference feedback information about user operation accuracy is generated and displayed.

[0127] Further, the augmented reality instruction is structured data containing the three-dimensional anchor coordinates and rendering attributes such as marker type, color, size, etc.

[0128] Further, the method is applied to scenarios of remote medical diagnosis, remote makeup teaching, remote fitness guidance, or remote industrial maintenance guidance.

[0129] Embodiment 6

[0130] In this embodiment, the step of capturing a video stream reflecting the real state of the user by the lamp mirror body 101 and tracking the three-dimensional pose data of the user in real time specifically includes:

[0131] Starting the image sensor and loading the pre-trained pose recognition neural network model;

[0132] At each time point of each frame , a frame of image is captured from the sensor ;

[0133] The image is input into the neural network to obtain a set of original measurement coordinates of three-dimensional joints ;

[0134] The original measurement coordinates are input into the Kalman filter to optimally estimate and predict the real position and velocity of the joint, and a smooth and stable three-dimensional pose data set is output ;

[0135] Steps 2 to 4 are repeated for the next frame.

[0136] In this embodiment, directly using the original joint data output by the neural network will produce high-frequency jitter due to image noise, occlusion, model jitter, etc., affecting the stability of the subsequent AR marker. The Kalman filter is a powerful time domain filtering algorithm that, through the cycle of “prediction-update”, combines the state at the last time and the measurement value at the current time, can effectively filter out noise and provide a better estimate of the target real state.

[0137] The output three-dimensional pose data is extremely smooth and stable, even when the user moves slightly or is partially occluded, the AR marker can still be “tightly fitted” without dramatic jitter, greatly improving the user experience.

[0138]

[0139] In this embodiment, the lamp mirror body 101 sends the video stream and the three-dimensional pose data to the cloud server 103, specifically including:

[0140] The video stream data and the pose data stream are divided into two independent streams;

[0141] The available bandwidth and the delay of the current network are monitored in real time ;

[0142] According to the network state, a rate-distortion optimization algorithm is performed to dynamically adjust the quantization parameter of the video stream and the compression level of the pose data ;

[0143] The optimized and compressed data is packaged into a data packet conforming to the network transmission protocol and is sent.

[0144] The optimization problem of this embodiment can be modeled as an optimization function :

[0145]

[0146] wherein is the total cost function, which is to be minimized; is the video distortion function, which represents the image quality loss when the quantization parameter is ; is the pose data distortion function, which represents the accuracy loss when the compression level is ; is the weight factor, which represents the importance of the pose data relative to the video data; is the Lagrange multiplier, which is used to balance the distortion and the code rate; is the video code rate function, which represents the number of bits generated per second under ; is the pose data code rate function; is the total bandwidth available at the current time.

[0147] Through the above optimization function , network adaptive transmission is realized. When the network is good, the image quality and the pose accuracy are automatically improved; when the network is poor, the low delay and integrity of the pose data are prioritized, and the video image quality is appropriately reduced, thereby ensuring the availability and smoothness of the core AR guidance function.

[0148] Embodiment 8

[0149] In this embodiment, the video stream is displayed by the expert terminal 102, and the two-dimensional marker input generated by the expert is received, specifically including:​​

[0150] A series of discrete original two-dimensional coordinate points drawn by the capture expert on the video picture by mouse or stylus ;

[0151] Preliminary noise filtering is performed on the original coordinate point sequence

[0152] The filtered coordinate points are taken as input, and a B-spline curve fitting algorithm is applied to calculate a set of optimal control points ;

[0153] The final two-dimensional marker input is generated

[0154] Since the trajectory drawn by the expert naturally contains jitter and uneven sampling points, directly transmitting all original points not only has a large data volume, but also the lines are not smooth. A B-spline curve is a piecewise polynomial curve, and only a few control points can define a complex and smooth curve, which is very suitable for representing a hand-drawn trajectory. In addition, it can greatly compress the data volume that needs to be transmitted and reduce network load.

[0155] The mathematical expression of the B-spline curve is:

[0156]

[0157] wherein, is the two-dimensional coordinate point at the parameter of the curve; is the parameter of the curve; is the number of control points; is the two-dimensional coordinate of the th control point; is the th B-spline basis function, which is defined by the De Boor-Cox recursive formula and determines the influence weight of the control point on the curve shape.

[0158] Embodiment 9

[0159] In this embodiment, the step of sending the two-dimensional marker input by the expert terminal 102 to the cloud server 103 specifically includes:

[0160] Analyzing the two-dimensional marker input stream being generated by the expert

[0161] Using an autoregressive model to establish a dynamic model for the motion trajectory of the expert's brush

[0162] While sending the real data at the current time, the AR model is used to predict the possible trajectory in the next short period of time, and this prediction information is packaged and sent together

[0163] The real marker data and the prediction data are sent together.

[0164] In network transmission, delay is inevitable. In order to combat delay, prediction technology can be used, and an autoregressive model assumes that the current state can be linearly combined by the states at several past time points. By modeling the expert hand movement, the server can complete the prediction of the expert action in advance.

[0165] An autoregressive model of order p is expressed as:

[0166]

[0167] wherein, is a two-dimensional coordinate point at time t; is a constant term; is the order of the model, indicating how many past time values are used to predict the current value; is the autoregressive coefficient of the model, indicating the influence weight of the past time value on the current value; is a white noise term, indicating random disturbances that the model cannot predict.

[0168] Embodiment 10

[0169] In this embodiment, the step of determining the three-dimensional anchor coordinates of the marker in three-dimensional space by the cloud server 103 according to the two-dimensional marker input and the three-dimensional pose data specifically includes:

[0170] Ensure that the timestamp of the received two-dimensional marker input is aligned with the timestamp of the three-dimensional pose data of the user terminal;

[0171] For each two-dimensional marker point , use the camera intrinsic matrix and the extrinsic matrix V to construct a three-dimensional ray from the camera or virtual viewpoint passing through the two-dimensional point;

[0172] Calculate the intersection of the three-dimensional ray and the three-dimensional face / body mesh model received from the user terminal;

[0173] Take the first calculated intersection point as the three-dimensional anchor coordinates corresponding to the two-dimensional marker point.

[0174] This step solves how to accurately map the expert's operation on the 2D screen back to the user's 3D body. Its geometric principle is equivalent to picking or ray casting in computer graphics, finding the corresponding object of the 2D screen point in the 3D virtual world.

[0175] The process of finding intersection can be abstracted as a function solving problem :

[0176]

[0177] Where the equation of the ray is:

[0178]

[0179] Where, is the final 3D anchor coordinate; is the 3D ray equation, t is the distance along the ray direction; is the 3D mesh model composed of multiple triangles is the 3D coordinate of the camera or view point; is the 3x3 camera intrinsic matrix; is the 4x4 view matrix; is the homogeneous coordinate representation of ; represents the point that minimizes the objective function.

[0180] Embodiment 11

[0181] In this embodiment, the step of generating augmented reality instructions containing the 3D anchor coordinates by the cloud server 103 and sending them to the lamp mirror body 101 specifically includes:

[0182] Pack the calculated 3D anchor coordinates and the label attributes received from the expert end into a structured data object;

[0183] If the label contains dynamic effects (such as flashing, pulsing), generate a parameterized effect function for it;

[0184] Construct the final AR instructions containing static positions and dynamic effect parameters;

[0185] After serializing the instructions, send them to the designated lamp mirror body 101 through the signaling server.

[0186] In order to make the AR label more expressive and indicative, not only static coordinates need to be delivered, but also a mechanism to describe its dynamic behavior. By parameterizing the dynamic effect as a function, the client can render the animation locally according to this function without the server continuously sending the state of each frame, greatly saving bandwidth.

[0187] An AR instruction containing pulsing effect ​The data structure of the augmented reality marker can be expressed as:

[0188]

[0189] wherein the pulsation effect function defines the size of the marker over time :

[0190]

[0191] wherein is the JSON object of the AR instruction; is the three-dimensional anchor coordinate; is the size or intensity of the marker at the moment; is the base size; is the amplitude; is the damping ratio, which controls the decay speed of the pulse; is the undamped natural frequency; is the damped oscillation frequency; is the phase.

[0192] Embodiment 12

[0193] In this embodiment, the augmented reality instruction received and executed by the lamp mirror body 101, the step of rendering the augmented reality marker anchored to the three-dimensional anchor coordinate on the real image of the user on its expert terminal, specifically includes:

[0194] Receiving and parsing the AR instruction JSON object from the cloud ;

[0195] Obtaining the three-dimensional anchor world coordinate in the instruction , using the current view matrix and the projection matrix , transforming it into screen clipping coordinates ;

[0196] Dividing the component of by its component to obtain the normalized device coordinates;

[0197] Mapping the NDC coordinates to the final screen pixel coordinates ;

[0198] According to the color, size and dynamic effect function in the instruction, the appearance of the AR marker of the current frame is calculated, and the Alpha blending technology is used to smoothly superimpose it on the real image captured by the camera.

[0199] This step is the core process of the standard 3D graphics rendering pipeline, which correctly draws a point existing in the three-dimensional virtual world to the 2D screen. Alpha blending ensures that the virtual AR marker can be naturally blended with the real background image in a semi-transparent manner, rather than being overlaid harshly.

[0200] The core coordinate transformation pipeline can be represented as:

[0201]

[0202] The color calculation formula of alpha blending is:

[0203]

[0204] wherein, is the viewport transformation matrix, used to map NDC to screen resolution; is a 4x4 projection matrix; is a 4x4 view matrix, representing the position and orientation of the camera; is the homogeneous coordinate of the three-dimensional anchor point of the AR marker; is the final pixel color after blending; is the pixel color of the AR marker; is the pixel color of the background, representing the real image of the camera; is the transparency channel value of the AR marker.

[0205] Embodiment 13

[0206] Reference Figure 3 , which shows a structural schematic diagram of a computer device provided by some embodiments of the application. The electronic device 20 includes a processor 200, a memory 201, a bus 202, and a communication interface 203, wherein the processor 200, the communication interface 203, and the memory 201 are connected through the bus 202; the memory 201 stores a computer program that can run on the processor 200, and the processor 200 is configured to operate according to the instructions to perform the steps of the method of any one of embodiments 5-12.

[0207] The memory 201 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The communication between the system network element and at least one other network element is realized through at least one communication interface 203 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0208] The bus 202 can be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. The memory 201 is configured to store programs, and the processor 200 executes the programs after receiving an execution instruction. The method of the remote-guidable headlamp system disclosed in any of the embodiments of the present application can be applied to the processor 200 or implemented by the processor 200.

[0209] The processor 200 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 200. The processor 200 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and combines the hardware to complete the steps of the above method.

[0210] Embodiment 14

[0211] The embodiment also provides a computer readable storage medium corresponding to the method of the remote-guidable headlamp system provided by the foregoing embodiments. Please refer to Figure 4 The computer readable storage medium shown is a CD, and a computer program (i.e., a program product 30) is stored on the CD. When the processor runs the computer program, the method of the remote-guidable headlamp system provided by any of the foregoing embodiments is executed.

[0212] It should be noted that the examples of the computer-readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or other optical, magnetic storage medium, which will not be listed one by one here.

[0213] The computer-readable storage medium provided by the above embodiments of the present application has the same application concept as the method for remotely guiding a lamp mirror system provided by the embodiments of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0214] It should be noted that in the above text, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0215] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0216] In summary, the lamp mirror system and method provided by the present application, computer device and storage medium have the following beneficial effects:

[0217] 1. Provide high-dimensional and accurate remote guidance: The system can capture video streams reflecting the user's real state and track the user's three-dimensional pose data in real time. Through the cloud server 103, the two-dimensional markers input by experts and the user's three-dimensional pose data are analyzed and fused, and the anchor coordinates of the markers in three-dimensional space are accurately calculated. This allows the guide to understand the user's operation state from a three-dimensional perspective, providing more accurate and practical guidance.

[0218] 2. Achieve high-precision and real-time augmented reality anchoring: The augmented reality instructions generated by the system contain accurate three-dimensional anchor coordinates and can be sent to the lamp mirror body 101 for rendering. The augmented reality markers rendered by the lamp mirror body 101 can change synchronously with the user's movement, anchored to a specific position on the user's body or face, providing visual cues as if they were actually present. This precise anchoring greatly improves the effectiveness of guidance and the accuracy of operations, avoiding the problem of indication failure due to changes in perspective or posture.

[0219] 3. Enhance the intuitiveness and ease of understanding of guidance: By superimposing augmented reality markers on the user's real image, the system converts abstract guidance instructions into visual information that users can directly perceive and understand. For example, it can accurately indicate the bending angle of a user's joint or the movement direction of a certain part of the body, significantly reducing communication costs and understanding biases.

[0220] 4. Improve the interactivity and efficiency of remote guidance: The system can receive two-dimensional marker inputs generated by the expert terminal 102 in real time and quickly convert them into accurate three-dimensional augmented reality instructions. This efficient feedback loop allows experts to provide immediate and accurate feedback and corrections to the user's operations, greatly improving the efficiency and response speed of remote guidance.

[0221] 5. Expand application fields and improve user experience: The system can be widely used in remote medical surgery guidance, remote technical training, complex equipment operation guidance, sports teaching, and other scenarios, effectively solving the pain points of remote guidance and improving the overall experience and work efficiency of operators and guides.

[0222] The embodiments of the present application are described above with reference to the drawings, which are only used to illustrate the specific embodiments of the present application, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative, but not restrictive, and those skilled in the art can make many forms without departing from the purpose of the present application and the scope protected by the claims under the inspiration of the present application, which all belong to the protection of the present application. Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not a limitation on the protection scope of the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present application.

Claims

1. A remotely directable lamp mirror system characterized by, The method comprises: a lamp mirror body configured to: capture a video stream reflecting a real state of a user; track three-dimensional pose data of the user in real time; receive augmented reality instructions and render augmented reality markers superimposed on real images of the user on the lamp mirror body; an expert terminal configured to: display the video stream; receive two-dimensional marker input generated by an expert on the video stream; and a cloud server configured to: receive the video stream and the three-dimensional pose data from the lamp mirror body; receive the two-dimensional marker input from the expert terminal; determine three-dimensional anchor coordinates of the two-dimensional marker in three-dimensional space according to the two-dimensional marker input and the three-dimensional pose data; generate the augmented reality instructions containing the three-dimensional anchor coordinates and send them to the lamp mirror body; the augmented reality markers rendered by the lamp mirror body, whose positions on the expert terminal change synchronously with movements of the user to achieve a visual effect anchored to the user's body or face; the three-dimensional pose data comprises a three-dimensional mesh model constructed in real time according to the user's body or face; the cloud server maps position coordinates of the two-dimensional marker input onto the three-dimensional mesh model to determine the three-dimensional anchor coordinates through a back projection algorithm; the step of determining the three-dimensional anchor coordinates of the two-dimensional marker in three-dimensional space according to the two-dimensional marker input and the three-dimensional pose data by the cloud server specifically comprises: aligning a timestamp of the received two-dimensional marker input with a timestamp of the three-dimensional pose data of the user terminal; For each two-dimensional marker point , a three-dimensional ray is constructed from the camera or virtual viewpoint through the two-dimensional marker point using the camera intrinsic matrix and the extrinsic matrix V ; calculating this three-dimensional ray the intersection with the three-dimensional face / body mesh model received from the user end the intersection with the three-dimensional face / body mesh model received from the user end the first intersection point is calculated the three-dimensional anchor coordinates corresponding to the two-dimensional marker point .

2. A remotely directable lamp mirror system according to claim 1, characterized in that the lamp mirror body comprises: a controllable lighting system; and a color calibration module configured to control the controllable lighting system and capture a calibration image at the beginning of a session to perform color correction on the video stream, thereby providing the video stream to the expert terminal.

3. A remotely directable lamp mirror system according to claim 1, characterized in that The two-dimensional marker input comprises instructions generated by at least one of a brush tool, a measurement tool, a geometric shape tool, or a three-dimensional model placement tool.

4. A method of remotely directing a lamp mirror system, characterized by, The method comprises the following steps: capturing, by a lamp mirror body, a video stream reflecting a real state of a user and tracking three-dimensional pose data of the user in real time; sending, by the lamp mirror body, the video stream and the three-dimensional pose data to a cloud server; displaying, by an expert terminal, the video stream and receiving two-dimensional marker input generated by an expert; sending, by the expert terminal, the two-dimensional marker input to the cloud server; determining, by the cloud server, three-dimensional anchor coordinates of the two-dimensional marker in three-dimensional space according to the two-dimensional marker input and the three-dimensional pose data; generating, by the cloud server, augmented reality instructions containing the three-dimensional anchor coordinates and sending them to the lamp mirror body; receiving and executing, by the lamp mirror body, the augmented reality instructions to render augmented reality markers superimposed on real images of the user on its expert terminal, the augmented reality markers being anchored to the three-dimensional anchor coordinates; the three-dimensional pose data comprises a three-dimensional mesh model constructed in real time according to the user's body or face; the cloud server maps position coordinates of the two-dimensional marker input onto the three-dimensional mesh model to determine the three-dimensional anchor coordinates through a back projection algorithm; The step of determining the three-dimensional anchor coordinates of the two-dimensional marker in the three-dimensional space by the cloud server according to the two-dimensional marker input and the three-dimensional pose data specifically comprises: Ensuring that the timestamp of the received two-dimensional marker input is aligned with the timestamp of the three-dimensional pose data of the user terminal; For each two-dimensional marker point , a three-dimensional ray is constructed from the camera or virtual viewpoint through the two-dimensional marker point using the camera intrinsic matrix and the extrinsic matrix V ; calculating this three-dimensional ray the intersection with the three-dimensional face / body mesh model received from the user terminal the intersection with the three-dimensional face / body mesh model received from the user terminal the first intersection point is calculated the three-dimensional anchor coordinates corresponding to the two-dimensional marker point .

5. A method of remotely directing a lamp system according to claim 4, characterized in that Before the step of capturing the video stream, further comprising: The step of automatically performing color calibration by the lamp mirror body to ensure the color fidelity of the video stream.

6. A method of remotely directing a lamp system according to claim 4 or 5, characterized in that Further comprising the following steps: Capturing the actual operation image of the user by the lamp mirror body; Real-time comparative analysis of the actual operation image and the augmented reality marker; According to the comparative analysis result, generate and display the difference feedback information about the user operation accuracy.

7. A method of remotely directing a lamp system according to claim 6, characterized in that The augmented reality instruction is structured data containing the three-dimensional anchor coordinates and the rendering attributes of the marker type, color and size.

8. A method of remotely directing a lamp system as claimed in claim 6, characterized in that, The method is applied to the scenarios of remote medical diagnosis, remote makeup teaching, remote fitness guidance or remote industrial maintenance guidance.

Images