Digital craftsman operation real-time interaction method and system

Abstract

The invention is suitable for the technical field of digital craftsman operation, and particularly relates to a digital craftsman operation real-time interaction method and system, and the method comprises the steps: obtaining a real-time image of craftsman operation performed by a student, real-time motion information of the hands of the student, and a plurality of real-time operation tasks; obtaining first action information and second action information according to the real-time action information; obtaining a plurality of action confidence degrees of the student according to the first action information and the second action information; obtaining a time sequence operation track of the student according to each real-time operation task and each action confidence coefficient; and obtaining AR prompt information according to the time sequence operation track and the real-time action information. Therefore, the digital craftsman operation real-time interaction method provided by the invention can solve the problem of lack of dynamic adaptability in the process of real-time interaction with students.

Description

Technical Field

[0001] The present application belongs to the field of digital craftsman operation technology, and in particular to a real-time interactive method and system for digital craftsman operation. Background Art

[0002] Digital craftsman operations provide digital guidance and specifications to ensure that craftsmen follow the correct procedures, use appropriate tools and techniques, and complete production tasks efficiently and safely. The purpose is to standardize the various behaviors of craftsmen during the operation process.

[0003] The technical challenges with traditional craftsmanship instruction stem primarily from the often delayed feedback from students after their actions, which prevents timely correction of errors and prevents students from adjusting their methods. Furthermore, assessments of student skill often focus solely on the results (such as the quality of the finished product) while ignoring the process itself. Existing digital craftsmanship instruction often relies on pre-set instructional processes and content, lacking the ability to dynamically adjust to students' actual operational situations. Consequently, current digital craftsmanship instruction suffers from a lack of dynamic adaptability during real-time interaction with students. Summary of the Invention

[0004] The embodiments of the present application provide a method and system for real-time interaction of digital craftsmen, which can solve the problem of lack of dynamic adaptability in the process of real-time interaction with students.

[0005] In a first aspect, an embodiment of the present application provides a real-time interactive method for digital craftsman operations, comprising: Acquire real-time images of students performing craftsman operations, real-time motion information of the students' hands, and multiple real-time operation tasks; wherein the real-time motion information is used to reflect changes in the position, acceleration, and angular velocity of the students' hands to date; Obtaining first action information and second action information according to the real-time action information; wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations; Obtaining a plurality of action confidence levels of the student based on the first action information and the second action information; wherein the action confidence levels are used to reflect the student's operation level; Obtaining a student's sequential operation trajectory based on each of the real-time operation tasks and each of the action confidences; wherein the sequential operation trajectory includes a trajectory and a timestamp of the student's execution of each of the corresponding real-time operation tasks, and the trajectory is composed of a series of virtual or real operation nodes passed by the student during the operation process; AR prompt information is obtained according to the temporal operation trajectory and the real-time action information.

[0006] The above technical solutions in the embodiments of the present application have at least the following technical effects: The real-time interactive method for digital craftsman operation provided by the embodiment of the present application obtains the real-time image of the student performing the craftsman operation, the real-time motion information of the student's hand, and multiple real-time operation tasks; obtains the first motion information and the second motion information based on the real-time motion information; obtains multiple motion confidences of the student based on the first motion information and the second motion information; obtains the student's temporal operation trajectory based on each real-time operation task and each motion confidence; obtains AR prompt information based on the temporal operation trajectory and the real-time motion information. Therefore, the real-time interactive method for digital craftsman operation provided by the embodiment of the present application can fully capture the student's operation process, including motion details and operation environment, through real-time images and hand motion information; by distinguishing the motion information of the craftsman operation and other operations, it can accurately identify the student's operation status and avoid misjudgment; the motion confidence provides a quantitative assessment of the student's operation level, making the teaching guidance more objective and accurate; the temporal operation trajectory uses timestamps and operation nodes as the benchmark to intuitively display the student's operation process; combined with the temporal operation trajectory and real-time motion information, it can provide students with accurate AR prompts to help students correct errors in time and improve their operation skills.

[0007] In a possible implementation of the first aspect, obtaining the first action information and the second action information according to the real-time action information includes: Obtaining time domain characteristics and frequency domain characteristics of acceleration and angular velocity according to the real-time motion information; Obtaining the first action information by filtering according to the time domain features and the frequency domain features; The second action information is obtained according to the real-time action information and the first action information.

[0008] In a possible implementation of the first aspect, obtaining the student's sequential operation trajectory according to each of the real-time operation tasks and each of the action confidences includes: Obtaining corresponding operation positions and priorities according to each of the real-time operation tasks; When it is determined according to the real-time image that the current state of the student is an operation state, obtaining a trajectory of the student performing each of the real-time operation tasks according to each of the priorities and the real-time action information; A timestamp of when the student performs each of the real-time operation tasks is obtained according to the confidence level of each of the actions.

[0009] In a possible implementation of the first aspect, when determining, based on the real-time image, that the current state of the student is an operating state, before obtaining, based on the priorities and the real-time action information, a trajectory of the student performing each of the real-time operating tasks, the method further includes: Dividing the real-time image into multiple sub-images according to the positions of the operating object and the student's hand; The student's current state is determined to be an operating state based on each sub-image of the first image and each sub-image of the current frame image in the real-time image; wherein the first image is an image of a preset number of frames before the current frame image.

[0010] In a possible implementation of the first aspect, determining that the student's current state is an operating state based on each sub-image of the first image and each sub-image of the current frame image in the real-time image includes: Obtaining hand motion features of the student based on each sub-image of the first image and each sub-image of the current frame image; wherein the hand motion features include motion trajectory and posture features of the hand; Matching the hand motion features with the motion pattern library to determine whether the student is in a specific operation mode; determining whether the state of the operator has changed according to the sub-image of the operator in the first image and the sub-image of the operator in the current frame image; In the case where it is determined that the student is in a specific operation mode or the state of the operation object has changed, it is determined that the current state of the student is the operation state.

[0011] In a possible implementation of the first aspect, obtaining the student's sequential operation trajectory according to each of the real-time operation tasks and each of the action confidences further includes: When it is determined based on the real-time image that the student's current state is a non-operating state, a plurality of first operation trajectories are predicted based on the real-time operation tasks and the real-time action information; wherein the first operation trajectory refers to a trajectory of the student from the current position to the next operation node; The trajectory of the student performing each of the real-time operation tasks is obtained according to each of the operation positions, each of the priorities and each of the first operation trajectories.

[0012] In a possible implementation of the first aspect, the predicting and obtaining a plurality of first operation trajectories according to the real-time operation tasks and the real-time action information includes: Obtaining the student's previous operation node according to the real-time action information; Obtaining an operable position group according to the last operation node and each priority; Obtaining a corresponding operating distance according to the real-time action information and the operable position group; A plurality of first operation tracks are obtained according to the operation distances.

[0013] In a possible implementation of the first aspect, obtaining multiple action confidence levels of the student based on the first action information and the second action information includes: Subdividing the first action information and the second action information into multiple sub-actions; Calculating a first confidence level for each of the sub-actions; Obtaining a second confidence level of each craftsman's operation in the first action information according to the first confidence level of each sub-action; Obtaining a first duration for the student to perform other operations before performing each craftsman operation according to the first action information and the second action information; The action confidence of each craftsman's operation is obtained according to each second confidence and each first duration.

[0014] In a possible implementation of the first aspect, obtaining AR prompt information according to the time-series operation trajectory and the real-time action information includes: Obtaining deviation node information according to the time sequence operation trajectory and the real-time action information; The AR prompt information is sent according to the deviation node information.

[0015] In a second aspect, an embodiment of the present application provides a real-time interactive system for digital craftsman operations, including: An acquisition module is used to acquire real-time images of students performing craftsman operations, real-time motion information of the students' hands, and multiple real-time operation tasks; wherein the real-time motion information is used to reflect the changes in the position, acceleration, and angular velocity of the students' hands to date; an action information module, configured to obtain first action information and second action information according to the real-time action information; wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations; an action confidence module, configured to obtain a plurality of action confidences of the student based on the first action information and the second action information; wherein the action confidences are used to reflect the student's operation level; A sequential operation trajectory module is configured to obtain a student's sequential operation trajectory based on each of the real-time operation tasks and each of the action confidences; wherein the sequential operation trajectory includes a trajectory and a timestamp of the student's execution of each of the corresponding real-time operation tasks, and the trajectory is composed of a series of virtual or real operation nodes that the student passes through during the operation process; The AR prompt information module is used to obtain AR prompt information according to the temporal operation trajectory and the real-time action information.

[0016] In a third aspect, an embodiment of the present application provides a real-time interactive device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects above is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a real-time interactive device, enables the real-time interactive device to execute any one of the methods described in the first aspect above.

[0019] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a flowchart of a real-time interactive method for digital craftsmen provided in one embodiment of the present application; Figure 2 This is a schematic diagram of the implementation process of steps S200 and S300 in the real-time interactive method for digital craftsman operation provided by an embodiment of the present application; Figure 3 This is a schematic diagram of the implementation process of steps S400, S420, S422 and S500 in the real-time interactive method for digital craftsman operation provided by an embodiment of the present application; Figure 4 This is a structural diagram of a real-time interactive system for digital craftsman operation provided by an embodiment of the present application; Figure 5 It is a structural diagram of the real-time interactive device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0023] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0024] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0025] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0026] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0028] In related technologies, feedback from students after their actions is often delayed, preventing timely correction of errors. This results in students being unable to adjust their operating methods. Furthermore, assessments of students' performance only consider the results (such as the quality of the finished product) while ignoring the process itself. Existing digital craftsmanship instruction often uses pre-set instructional processes and content, lacking the ability to dynamically adjust based on students' actual operational situations. Consequently, current digital craftsmanship systems lack dynamic adaptability during real-time interaction with students.

[0029] To solve the above problems, the embodiment of the present application provides a method and system for real-time interaction of digital craftsman operation. In this method, by obtaining real-time images of students performing craftsman operations, real-time motion information of students' hands and multiple real-time operation tasks; obtaining first motion information and second motion information based on real-time motion information; obtaining multiple motion confidences of students based on the first motion information and the second motion information; obtaining the students' time-series operation trajectory based on each real-time operation task and each motion confidence; obtaining AR prompt information based on the time-series operation trajectory and real-time motion information. Therefore, the real-time interaction method for digital craftsman operation provided by the embodiment of the present application can fully capture the students' operation process, including motion details and operation environment, through real-time images and hand motion information; by distinguishing the motion information of craftsman operation and other operations, it can accurately identify the students' operation status and avoid misjudgment; motion confidence provides a quantitative assessment of the students' operation level, making teaching guidance more objective and accurate; the time-series operation trajectory uses timestamps and operation nodes as the benchmark to intuitively display the students' operation process; combined with the time-series operation trajectory and real-time motion information, it can provide students with accurate AR prompts to help students correct errors in time and improve their operation skills.

[0030] The real-time interactive method for digital craftsman operation provided in the embodiment of the present application can be applied to a real-time interactive device. In this case, the real-time interactive device is the executor of the real-time interactive method for digital craftsman operation provided in the embodiment of the present application. The embodiment of the present application does not impose any restrictions on the specific type of the real-time interactive device.

[0031] For example, the real-time interactive device can be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a desktop computer, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, etc., but is not limited to these.

[0032] In order to better understand the real-time interactive method for digital craftsmen's operations provided in the embodiment of the present application, the specific implementation process of the real-time interactive method for digital craftsmen's operations provided in the embodiment of the present application is exemplarily introduced below.

[0033] Figure 1 A schematic flow chart of a real-time interactive method for digital craftsman operation provided by an embodiment of the present application is shown. The real-time interactive method for digital craftsman operation includes: S100: Acquire a real-time image of a student performing a craftsman operation, real-time motion information of the student's hand, and multiple real-time operation tasks. The real-time motion information is used to reflect the changes in the position, acceleration, and angular velocity of the student's hand so far.

[0034] For example, the built-in camera of the AR device or a depth camera (such as Kinect or Azure Kinect) can be used directly to capture RGB-D images of the student's operating area at a frame rate of 30FPS, and combined with the YOLOv8 target detection model to identify the position of craftsman tools (such as hammers and files) in real time; an IMU sensor (such as MPU6050) can be worn on the student's wrist to record three-dimensional acceleration (m / s²) and angular velocity (rad / s) at a sampling rate of 100Hz, and OpenCV's hand key point detection (such as MediaPipe) can be used to obtain the spatial coordinates of 21 key points of the hand; the current operation task ID (such as "filing plane") can be read through the PLC system and stored in association with the timestamp.

[0035] S200: Obtain first action information and second action information based on the real-time action information, wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations.

[0036] For example, sliding window statistics (such as mean, standard deviation, frequency domain energy, etc.) can be calculated for real-time action information, and the sliding window statistics are input into an LSTM network (input dimension = 12, hidden layer = 64) to distinguish between craftsman operations (such as hammer swinging) and other actions (such as adjusting tools) to obtain first action information and second action information.

[0037] In one possible implementation, see Figure 2 S200, obtaining first action information and second action information according to real-time action information, including: S210 , obtaining time domain features and frequency domain features of acceleration and angular velocity according to the real-time motion information.

[0038] For example, the acceleration and angular velocity collected by the IMU can be used to calculate the statistics (such as mean, standard deviation, peak value, etc.) within a sliding window (such as 0.5 seconds) based on the real-time motion information, that is, the time domain features, and then the fast Fourier transform (FFT) is applied to obtain the power spectral density (PSD), and the main frequency components (such as peak frequency) and frequency band energy (such as the proportion of low-frequency energy from 0 to 5 Hz), that is, the frequency domain features, are extracted.

[0039] S220 , obtaining first action information by filtering according to the time domain features and the frequency domain features.

[0040] For example, a bandpass filter (e.g., 5-15 Hz) can be designed based on the time-domain and frequency-domain characteristics to retain the typical frequencies of the craftsman's operation (e.g., the main hammering frequency of 8 Hz), filter out low-frequency drift and high-frequency noise, and reconstruct the filtered time-domain signal through inverse FFT. A sliding average filter (window = 5 samples) is applied to smooth the acceleration curve and eliminate spike interference. Valid action segments are marked using threshold detection. Dynamic time warping (DTW) is then performed on the filtered signal and matched against a craftsman's operation template (e.g., the PSD of a standard hammering action). A similarity score is output, and the first action information is obtained based on the similarity score.

[0041] S230: Obtain second motion information according to the real-time motion information and the first motion information.

[0042] For example, the residual of the real-time action information and the first action information can be calculated, and features can be extracted from the residual signal to identify non-craftsman operation features (such as slow changes or high-frequency noise). If the time domain standard deviation of the residual signal is >0.2 and the low-frequency energy in the frequency domain is >60%, it is judged to be a slow action such as "adjusting tools" (second action information). If the high-frequency component (>20Hz) of the residual signal accounts for >40%, it is judged to be "environmental interference" or "non-operational action."

[0043] Through steps S210 to S230, time-domain features reflect the instantaneous intensity and stability of the action (e.g., a large standard deviation may indicate jitter), while frequency-domain features reveal the rhythm of the action (e.g., a high frequency component may indicate rapid swinging). Combining these two features allows for distinguishing between craftsman actions (e.g., regular hammering) and non-target actions (e.g., slow movement). Non-target frequency noise (e.g., ambient vibration) is filtered out, improving the signal-to-noise ratio of the craftsman action signal. Template matching ensures that the first action information only contains actions that match the process characteristics, reducing misclassification. Residual analysis achieves a hard separation between "craftsman actions" and "other actions," avoiding error accumulation within a single classifier. Intermediate results, such as frequency peaks and DTW similarity, are traceable, facilitating debugging and personalized adjustments (e.g., differences in the main frequency of different tool operations). By combining signal processing and pattern recognition, high-precision action classification is achieved, providing reliable input for subsequent confidence calculations.

[0044] S300: Obtain multiple action confidence levels of the student based on the first action information and the second action information, wherein the action confidence levels are used to reflect the student's operational level.

[0045] For example, the DTW distance between the hand trajectory and the standard path (such as the standard sinusoidal trajectory of the filing action) can be calculated based on the first action information and the second action information, and the acceleration mutation (such as whether the peak value when the hammer hits is within a reasonable range) can be checked to obtain the action score, which is weighted according to the classification result of S200 (such as the weight of the craftsman operation is 0.7, and the weight of other operations is 0.3), and the action score is scaled to [0,1] using Min-Max, and the weighted action confidence is obtained.

[0046] In one possible implementation, see Figure 2 S300, obtaining multiple action confidences of the student according to the first action information and the second action information, including: S310: Subdivide the first action information and the second action information to obtain a plurality of sub-actions.

[0047] For example, K-means clustering (K = 3-5) can be performed on the time-domain features (such as acceleration standard deviation and angular velocity peak) and frequency-domain features (dominant frequency and frequency band energy) of the first action information (craftsman operation) and the second action information (other operation) to obtain multiple sub-actions. Alternatively, the first and second action information can be aligned with a predefined craftsman operation template (such as a standard action sequence for hammering or filing) through DTW, and the sub-actions can be divided according to the turning points of the matching path. Alternatively, a sliding window can be used to mark the boundaries of a sub-action when the features within the window (such as the angular velocity integral) exceed a threshold. For example, a hammering action can be divided into lifting the hammer (increased acceleration), striking (peak acceleration), and rebounding (negative acceleration).

[0048] S320: Calculate and obtain a first confidence level of each sub-action.

[0049] For example, the DTW distance (D DTW ), confidence level C1=1- , where D max It is the maximum allowable distance, and checks whether the acceleration / angular velocity is within a reasonable range (for example, the peak acceleration of the hammer should be between 10 and 20 m / s²) and whether the duration of the sub-action meets the process requirements (for example, the hammer lifting phase should be less than 0.8 seconds). If it exceeds the range, points will be deducted to obtain the corresponding confidence levels C2 and C3. The first confidence level is obtained by weighted summation of the confidence levels C1, C2, and C3.

[0050] S330: Obtain a second confidence level of each craftsman's operation in the first action information according to the first confidence level of each sub-action.

[0051] For example, a weight can be assigned to each sub-action according to the importance of the process (such as the execution phase weight of hammering = 0.6, the preparation phase = 0.3, and the finishing phase = 0.1), and the second confidence of the corresponding craftsman operation can be obtained based on the weight of each sub-action and the first confidence.

[0052] S340: Obtain a first duration for the student to perform other operations before performing each craftsman operation based on the first action information and the second action information.

[0053] For example, the first duration of the continuous other operations before each craftsman's operation can be recorded based on the first action information and the second action information.

[0054] S350: Obtain the action confidence of each craftsman's operation according to each second confidence and each first duration.

[0055] For example, the corresponding action confidence can be calculated based on the first duration and the second confidence of other operations. For example, the action confidence C=C op ×α,α= , where C op is the second confidence level, is the attenuation coefficient (such as 0.2), and T is the first duration.

[0056] Through steps S310 to S350, complex operations are broken down into interpretable atomic units, and the completion quality of sub-actions is quantified, providing a fine-grained basis for subsequent operation-level confidence. The impact of key sub-actions on the overall operation is highlighted. Operational fluency is quantified. Excessive operation duration may reflect student hesitation or lack of skill. Excessive pauses are dynamically penalized to encourage operational consistency. AR prompts are generated instantly based on sub-action confidence.

[0057] S400: Obtain the student's sequential operation trajectory based on each real-time operation task and each action confidence. The sequential operation trajectory includes the trajectory and timestamp of each real-time operation task performed by the student. The trajectory is composed of a series of operation nodes passed by the student during the operation process.

[0058] It can be understood that the trajectory includes the path and actions of the student's operation.

[0059] For example, a continuous hand trajectory can be obtained based on each real-time operation task and the confidence level of each action. The continuous hand trajectory can be downsampled into key nodes (such as a stationary point with a speed <0.1m / s), and the operation stage (such as "pick up the tool → file → put it down") is marked in combination with the task switching signal (PLC trigger). The time error of the image, IMU and task signal is controlled to <10ms using a synchronized clock to obtain a sequential operation trajectory.

[0060] In one possible implementation, see Figure 3 ,S400, obtain the student's temporal operation trajectory according to each real-time operation task and each action confidence, including: S410: Obtain corresponding operation positions and priorities according to each real-time operation task.

[0061] For example, the corresponding operation position can be obtained according to each real-time operation task, and the priority can be obtained according to the process dependency between each real-time operation task (that is, the subsequent task depends on the completion of the predecessor task) (for example, "hammering workpiece A" needs to be positioned to the coordinates (2.3m, 1.5m, 0.8m), priority = 2; "adjusting fixture B" needs to be positioned to (1.8m, 1.2m, 0.7m), priority = 3), and the status of each real-time operation task can be detected by sensors (for example, after the fixture pressure sensor confirms "clamping", the priority of the hammering task is adjusted from 2 to 3), and a finite state machine (FSM) is used to manage task switching (for example, when the current task is completed, the next highest priority task is automatically activated).

[0062] S420 , when it is determined according to the real-time image that the current state of the student is an operating state, a trajectory of the student performing each real-time operating task is obtained according to each priority and the real-time action information.

[0063] For example, when the student's hand / tool position is captured by an RGB-D camera and a deep learning model (such as YOLOv8) is used to determine that the student is in an operating state (such as holding a tool, touching a workpiece, etc.), a path is generated based on the real-time position of the student's hand and the priority of each real-time operation task, with the current position of the student's hand as the starting point and the target operation position as the end point, avoiding obstacles (such as other equipment), and obtaining the trajectory of the student performing each real-time operation task.

[0064] Optionally, see Figure 3 S420: When it is determined based on the real-time image that the current state of the student is an operation state, before obtaining the trajectory of the student performing each real-time operation task based on each priority and real-time action information, the method further includes: S421 , obtaining a plurality of sub-images according to the positions of the operating object and the student's hand in the real-time image.

[0065] For example, the YOLOv8 model can be used to identify operating objects (including tools (such as hammers and wrenches) and workpieces (such as metal blocks and clamps)) in real-time images, output bounding box coordinates (such as the hammer box: (x1=200, y1=150, x2=350, y2=400)), and use the MediaPipe Hand model to locate the student's hand joints, generate a hand mask, and expand it to a rectangular area that includes the tool operation range (such as expanding the hand box outward by 30% to obtain the operation area box). The sub-image range is dynamically adjusted according to the position of the operating object and the hand to obtain multiple sub-images for each frame of the real-time image.

[0066] S422: Determine that the student's current state is an operating state based on each sub-image of the first image and each sub-image of the current frame image in the real-time image, wherein the first image is an image of a preset number of frames before the current frame image.

[0067] Exemplarily, the HOG feature difference between the first image (the first N frames, such as N=5) and the sub-image of the current frame image can be calculated. If the HOG feature difference between the sub-image of multiple frame images (such as 3 frames) in the first image and the sub-image of the current frame image exceeds a preset difference threshold, it is determined that the student's current state is an operating state.

[0068] Through steps S421 to S422, sub-image segmentation transforms global detection into local key area analysis, improving the accuracy of operating state recognition. Time-series feature matching reduces misjudgments, and dynamic sub-image merging reduces single-frame processing time, meeting real-time requirements.

[0069] For example, see Figure 3 S422, determining that the student's current state is an operating state based on each sub-image of the first image and each sub-image of the current frame image in the real-time image, includes: S4221: Obtain hand motion features of the student based on each sub-image of the first image and each sub-image of the current frame image, wherein the hand motion features include motion trajectory and posture features of the hand.

[0070] For example, the MediaPipe Hands model can be used to extract 21 hand joints (such as fingertips and joints), record their 2D coordinates in consecutive frames (unit: pixel), apply Kalman filtering to eliminate jitter of the hand joints, obtain the Z-axis coordinates of the joints through a binocular camera or a depth sensor (such as Intel RealSense), generate the hand motion trajectory (unit: meter), calculate the angles between adjacent joints (such as the angle θ between the metacarpophalangeal joint (MCP) and the interphalangeal joint (PIP)), and obtain an angle sequence. The angle sequence is input into the LSTM network and classified into postures such as grasping, reaching, and grasping to obtain the hand posture features.

[0071] S4222. Match the hand movement features with the movement pattern library to determine whether the student is in a specific operation mode.

[0072] Exemplarily, operation modes (such as hammering, sawing, measuring) can be predefined to obtain a movement pattern library. Each pattern in the movement pattern library includes: a trajectory template, a DTW distance threshold for the movement trajectory (such as the hammering mode threshold = 0.8), and pose features (such as the hammering mode includes the three-stage pose of "raising - falling - contacting"). Match the hand movement features with each pattern in the movement pattern library in terms of pose and trajectory to determine whether the student is in a specific operation mode. Calculate the edit distance (such as allowing an error of 1 frame) between the pose features of the hand movement features (including the pose sequences of the first image and the current frame image) and each pattern in the movement pattern library. If the DTW distance < DTW distance threshold and the edit distance of the pose < edit distance threshold, it is determined that the student is in a specific operation mode.

[0073] S4223. Determine whether the state of the workpiece has changed based on the sub-image of the workpiece in the first image and the sub-image of the workpiece in the current frame image.

[0074] Exemplarily, the sub-images containing the workpiece can be obtained from the respective sub-images of the first image and the current frame image, and the Farneback algorithm of the optical flow method can be applied to the workpiece sub-images to calculate the pixel displacement field. For example, if the displacement standard deviation > 5 pixels, it is determined as deformation (such as the workpiece is deformed by being struck); calculate the change rate v of the center point coordinates of the workpiece / , for example, if v > 0.5 m / s, it is determined that movement has occurred. If the workpiece is deformed or moves, it is determined that the state of the workpiece has changed.

[0075] S4224. When it is determined that the student is in a specific operation mode or the state of the workpiece has changed, determine that the current state of the student is the operation state.

[0076] Exemplarily, if any of the following conditions is met, it is determined as the operation state: the hand movement features match a specific operation mode, or the state of the workpiece has changed; otherwise, it is determined as the non-operation state.

[0077] Through the above steps S4221 to S4224, Kalman filtering reduces the trajectory error, uses joint angles for LSTM classification to adapt to complex gestures, and uses DTW distance and edit distance to improve the generalization ability and anti-interference ability of operation mode matching.

[0078] S430. Obtain the timestamps of the student's execution of each real-time operation task based on the confidence of each action.

[0079] For example, the action confidence can be associated with the timeline. When the confidence is > 0.7 for 5 consecutive frames, it is marked as the start time of the effective operation, and when the confidence is < 0.3, it is marked as the end time, and the timestamp of the student's execution of each real-time operation task is obtained.

[0080] Through steps S410 to S430, operational safety is ensured through dual spatial and logical constraints. A dynamic adjustment mechanism adapts to different learning stages. Sequential operation trajectories can be integrated with AR navigation and projected into the student's field of view in real time (e.g., displaying a green guide line via HoloLens). Timestamp data can also drive teaching feedback (e.g., "This hammering operation took 0.7 seconds, which is better than the class average of 1.2 seconds").

[0081] In one possible implementation, see Figure 3 S400 obtains the student's temporal operation trajectory according to each real-time operation task and each action confidence, and further includes: S440: When the student's current state is determined to be a non-operating state based on the real-time image, a plurality of first operation trajectories are predicted based on the real-time operation tasks and the real-time action information. The first operation trajectories are trajectories from the student's current position to the next operation node.

[0082] For example, a real-time operation task (such as "assembling a robotic arm") can be decomposed into multiple operation nodes (e.g., "take a screwdriver → take a screw → align it with the threaded hole → tighten it"). Each node contains: spatial coordinates: obtained through AR markers or SLAM technology (e.g., screwdriver placement point coordinates: (1.2, 0.8, 0.5) m); operation constraints: tool type (screwdriver), operation direction (vertical downward), and force range (2-5N). A directed graph is constructed based on these multiple operation nodes, where vertices are operation nodes and edges are transitions between nodes (e.g., "take a screwdriver" → "take a screw" requires a movement of 0.6 m). Path planning is performed using the A* algorithm based on the directed graph. Combined with a library of action patterns (e.g., a "grab-move-place" sequence), the action combination from the current position to the next operation node is predicted to obtain the first operation trajectory.

[0083] Optionally, see Figure 3 S440 predicts and obtains a plurality of first operation trajectories according to the real-time operation tasks and the real-time action information, including: S441, obtaining the student's previous operation node according to the real-time action information.

[0084] For example, the correspondence between actions and operation nodes may be predefined, and then the student's previous operation node may be determined based on real-time action information.

[0085] S442: Obtain an operable position group according to the last operation node and each priority.

[0086] For example, in a directed graph, starting from the last operation node, adjacent nodes are traversed according to priority (breadth-first search, BFS), and operable positions are screened to obtain an operable position group.

[0087] S443: Obtain a corresponding operation distance according to the real-time action information and the operable position group.

[0088] For example, the corresponding actual traversable distance, i.e., the operational distance, can be calculated based on the real-time position of the student's hand and the operable position group through algorithms such as A* and RRT* (rapidly expanding random tree).

[0089] S444: Obtain a plurality of first operation tracks according to the operation distances.

[0090] For example, the corresponding first operation trajectory can be obtained based on each operation distance and by inserting an action label at the operation node.

[0091] Through the above steps S441 to S444, the accuracy of node identification and trajectory prediction is improved, and complex operation processes are supported. Dynamic adjustment of priorities shortens the response time for emergency tasks.

[0092] S450 , obtaining a trajectory of the student performing each real-time operation task according to each operation position, each priority, and each first operation trajectory.

[0093] For example, the trajectory of the future student performing each real-time operation task can be obtained by dynamic optimization using model predictive control (MPC) according to each operation position, each priority and each first operation trajectory, and the trajectory is recalculated at regular intervals (such as 0.5s).

[0094] Through steps S440 to S450, the spatial path and action sequence are combined to adapt to different students' operating styles, automatically optimize the path, and generate multiple trajectories to avoid blocking a single path. MPC updates the trajectory every 0.5 seconds to adapt to dynamic environments and prevent high-priority tasks from occupying the path for a long time.

[0095] S500: Obtain AR prompt information according to the time-series operation trajectory and real-time action information.

[0096] For example, real-time action information can be compared with the sequential operation trajectory. If the confidence level is less than 0.6, an AR prompt message will be displayed. For example, the deviation from the standard path (such as the red dashed line) will be highlighted in the AR glasses. If the angular velocity is continuously low, a prompt "Please increase the wrist swing range" will be displayed. The next operation can also be predicted based on the sequential operation trajectory, indicating the tool placement location in advance.

[0097] In one possible implementation, see Figure 3, S500, obtain AR prompt information according to the time sequence operation trajectory and real-time action information, including: S510 , obtaining deviation node information according to the time sequence operation trajectory and real-time action information.

[0098] It can be understood that the deviation node information includes spatial deviation information and motion deviation information.

[0099] For example, the sequential operation trajectory can be decomposed into a sequence of discrete path points (one sampling point every 0.2 seconds), and associated with an expected action label (such as "grab the screwdriver → move to the workbench → tighten the screw"), the time axis of the sequential operation trajectory and the real-time action information are aligned, and a preset threshold value of the three-dimensional spatial deviation (such as 0.15m) is determined. When the distance between the real-time position of the student's hand and the path point of the sequential operation trajectory exceeds the preset threshold, the deviation is determined to obtain spatial deviation information, and the expected action label of the sequential operation trajectory is compared with the real-time action information (such as "grab the screwdriver") to obtain action deviation information.

[0100] S520: Send AR prompt information according to the deviation node information.

[0101] For example, AR prompt information can be determined based on the deviation node information (such as progressive prompts: first deviation: text prompt; 5 seconds of uncorrected: add arrow guidance; 15 seconds of uncorrected: trigger voice + vibration), and the AR prompt information can be sent to the AR device.

[0102] Through the above steps S510 to S520, the space + motion joint detection reduces the false alarm rate, improves the deviation detection accuracy and reduces the AR prompt response time, making this application more efficient than the traditional method (teacher's verbal guidance).

[0103] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] Corresponding to the real-time interactive method for digital craftsmen's operation described in the above embodiment, the embodiment of the present application also provides a real-time interactive system for digital craftsmen's operation, and each module of the system can implement each step of the real-time interactive method for digital craftsmen's operation. Figure 4 A structural block diagram of a real-time interactive system for digital craftsman operation provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0105] Reference Figure 4 , the system comprises: An acquisition module is used to acquire real-time images of students performing craftsman operations, real-time motion information of the students' hands, and multiple real-time operation tasks; wherein the real-time motion information is used to reflect the changes in the position, acceleration, and angular velocity of the students' hands to date; an action information module, configured to obtain first action information and second action information according to the real-time action information; wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations; an action confidence module, configured to obtain a plurality of action confidences of the student based on the first action information and the second action information; wherein the action confidences are used to reflect the student's operation level; A sequential operation trajectory module is configured to obtain a student's sequential operation trajectory based on each of the real-time operation tasks and each of the action confidences; wherein the sequential operation trajectory includes a trajectory and a timestamp of the student's execution of each of the corresponding real-time operation tasks, and the trajectory is composed of a series of virtual or real operation nodes that the student passes through during the operation process; The AR prompt information module is used to obtain AR prompt information according to the temporal operation trajectory and the real-time action information.

[0106] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0108] The embodiment of the present application also provides a real-time interactive device, Figure 5 This is a structural diagram of a real-time interactive device provided in one embodiment of the present application. Figure 5As shown, the real-time interactive device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown), at least one memory 51 ( Figure 5 Only one is shown in the figure) and a computer program 52 stored in the at least one memory 51 and executable on the at least one processor 50. When the processor 50 executes the computer program 52, the real-time interactive device 5 implements the steps of any of the above-mentioned embodiments of the real-time interactive method for digital craftsmen operation, or implements the functions of the modules / units in the above-mentioned device embodiments.

[0109] For example, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 52 in the real-time interactive device 5.

[0110] The real-time interactive device 5 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The real-time interactive device can include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that Figure 5 This is merely an example of the real-time interactive device 5 and does not constitute a limitation on the real-time interactive device 5 . The real-time interactive device 5 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, buses, etc.

[0111] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0112] In some embodiments, the memory 51 may be an internal storage unit of the real-time interactive device 5, such as the hard drive or memory of the real-time interactive device 5. In other embodiments, the memory 51 may also be an external storage device of the real-time interactive device 5, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the real-time interactive device 5. Furthermore, the memory 51 may include both the internal storage unit of the real-time interactive device 5 and an external storage device. The memory 51 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been output or is about to be output.

[0113] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0114] An embodiment of the present application provides a computer program product. When the computer program product is run on a real-time interactive device, the real-time interactive device implements the steps of any of the above method embodiments.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a real-time interactive device, a recording medium, computer memory, read-only memory (ROM), random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Examples include a USB flash drive, a removable hard drive, a magnetic disk, or an optical disk.

[0116] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the embodiments provided in this application, it should be understood that the disclosed real-time interactive devices and methods can be implemented in other ways. For example, the real-time interactive device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0120] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A real-time interactive method for digital craftsman operation, characterized in that: Applied to a real-time interactive device, the real-time interactive device is communicatively connected to an AR device; the method includes: Acquire real-time images of students performing craftsman operations, real-time motion information of the students' hands, and multiple real-time operation tasks; wherein the real-time motion information is used to reflect changes in the position, acceleration, and angular velocity of the students' hands to date; Obtaining first action information and second action information according to the real-time action information; wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations; Obtaining a plurality of action confidence levels of the student based on the first action information and the second action information; wherein the action confidence levels are used to reflect the student's operation level; Obtaining a student's sequential operation trajectory based on each of the real-time operation tasks and each of the action confidences; wherein the sequential operation trajectory includes a trajectory and a timestamp of the student's execution of each corresponding real-time operation task, and the trajectory is composed of a series of operation nodes passed by the student during the operation process; AR prompt information is obtained according to the temporal operation trajectory and the real-time action information.

2. The real-time interactive method for digital craftsman operation according to claim 1, characterized in that: The obtaining of the first action information and the second action information according to the real-time action information includes: Obtaining time domain characteristics and frequency domain characteristics of acceleration and angular velocity according to the real-time motion information; Obtaining the first action information by filtering according to the time domain features and the frequency domain features; The second action information is obtained according to the real-time action information and the first action information.

3. The real-time interactive method for digital craftsman operation according to claim 1, characterized in that: The step of obtaining the student's sequential operation trajectory according to each of the real-time operation tasks and each of the action confidences includes: Obtaining corresponding operation positions and priorities according to each of the real-time operation tasks; When it is determined according to the real-time image that the current state of the student is an operation state, obtaining a trajectory of the student performing each of the real-time operation tasks according to each of the priorities and the real-time action information; A timestamp of when the student performs each of the real-time operation tasks is obtained according to the confidence level of each of the actions.

4. The real-time interactive method for digital craftsman operation according to claim 3, characterized in that: When it is determined according to the real-time image that the current state of the student is an operating state, before obtaining a trajectory of the student performing each of the real-time operating tasks according to each of the priorities and the real-time action information, the method further includes: Dividing the real-time image into multiple sub-images according to the positions of the operating object and the student's hand; The student's current state is determined to be an operating state based on each sub-image of the first image and each sub-image of the current frame image in the real-time image; wherein the first image is an image of a preset number of frames before the current frame image.

5. The real-time interactive method for digital craftsman operation according to claim 4, characterized in that: The determining that the current state of the student is an operating state according to each sub-image of the first image and each sub-image of the current frame image in the real-time image includes: Obtaining hand motion features of the student based on each sub-image of the first image and each sub-image of the current frame image; wherein the hand motion features include motion trajectory and posture features of the hand; Matching the hand motion features with the motion pattern library to determine whether the student is in a specific operation mode; determining whether the state of the operator has changed according to the sub-image of the operator in the first image and the sub-image of the operator in the current frame image; In the case where it is determined that the student is in a specific operation mode or the state of the operation object has changed, it is determined that the current state of the student is the operation state.

6. The real-time interactive method for digital craftsman operation according to claim 3, characterized in that: The method of obtaining the student's sequential operation trajectory according to each of the real-time operation tasks and each of the action confidences further includes: When it is determined based on the real-time image that the student's current state is a non-operating state, a plurality of first operation trajectories are predicted based on the real-time operation tasks and the real-time action information; wherein the first operation trajectory refers to a trajectory of the student from the current position to the next operation node; The trajectory of the student performing each of the real-time operation tasks is obtained according to each of the operation positions, each of the priorities and each of the first operation trajectories.

7. The real-time interactive method for digital craftsman operation according to claim 6, characterized in that: The predicting of a plurality of first operation trajectories according to each of the real-time operation tasks and the real-time action information includes: Obtaining the student's previous operation node according to the real-time action information; Obtaining an operable position group according to the last operation node and each priority; Obtaining a corresponding operating distance according to the real-time action information and the operable position group; A plurality of first operation tracks are obtained according to the operation distances.

8. The real-time interactive method for digital craftsman operation according to claim 1, characterized in that: The obtaining of a plurality of action confidence levels of the student according to the first action information and the second action information includes: Subdividing the first action information and the second action information into multiple sub-actions; Calculating a first confidence level for each of the sub-actions; Obtaining a second confidence level of each craftsman's operation in the first action information according to the first confidence level of each sub-action; Obtaining a first duration for the student to perform other operations before performing each craftsman operation according to the first action information and the second action information; The action confidence of each craftsman's operation is obtained according to each second confidence and each first duration.

9. The real-time interactive method for digital craftsman operation according to claim 1, characterized in that: The obtaining of AR prompt information according to the time sequence operation trajectory and the real-time action information includes: Obtaining deviation node information according to the time sequence operation trajectory and the real-time action information; The AR prompt information is sent according to the deviation node information.

10. A digital craftsman operation real-time interactive system, characterized in that: include: An acquisition module is used to acquire real-time images of students performing craftsman operations, real-time motion information of the students' hands, and multiple real-time operation tasks; wherein the real-time motion information is used to reflect the changes in the position, acceleration, and angular velocity of the students' hands to date; an action information module, configured to obtain first action information and second action information according to the real-time action information; wherein the first action information is action information when the student performs a craftsman operation, and the second action information is action information when the student performs other operations; an action confidence module, configured to obtain a plurality of action confidences of the student based on the first action information and the second action information; wherein the action confidences are used to reflect the student's operation level; A sequential operation trajectory module is configured to obtain a student's sequential operation trajectory based on each of the real-time operation tasks and each of the action confidences; wherein the sequential operation trajectory includes a trajectory and a timestamp of the student's execution of each of the corresponding real-time operation tasks, and the trajectory is composed of a series of virtual or real operation nodes that the student passes through during the operation process; The AR prompt information module is used to obtain AR prompt information according to the temporal operation trajectory and the real-time action information.

Images