Kniftsman operation guidance method, device and equipment based on augmented reality

Abstract

The invention is suitable for the technical field of augmented reality, and particularly relates to a craftsman operation guidance method, device and equipment based on augmented reality, and the method comprises the steps: receiving the real-time voice of AR teaching equipment and the real-time image of the AR teaching equipment; determining a solution based on the real-time image and the real-time voice; determining real-time virtual operation information based on the real-time image and the solution; and obtaining AR prompt content according to the real-time virtual operation information, the solution and the real-time image, and sending the AR prompt content to the AR learning equipment. The craftsman operation guidance method based on augmented reality can solve the problem that students lack personalized guidance.

Description

Technical Field

[0001] The present application belongs to the field of augmented reality technology, and in particular, relates to a method, device and equipment for guiding craftsmen's operation based on augmented reality. Background Art

[0002] Craftsman operating instructions are detailed instructions and specifications provided to ensure that craftsmen follow the correct procedures, use appropriate tools and techniques during their work, and complete production tasks efficiently and safely. They aim to regulate the various behaviors of craftsmen during their work.

[0003] In the existing technology, teachers provide craftsman operation guidance by observing the students' operation results. Students cannot understand in real time whether their operation is correct or how to improve it. Every student has different learning styles and abilities, but traditional teaching methods often adopt a one-size-fits-all approach and cannot provide personalized guidance for each student. Therefore, the current augmented reality-based craftsman operation guidance method lacks personalized guidance. Summary of the invention

[0004] The embodiments of the present application provide a method, device and equipment for guiding craftsmen's operation based on augmented reality, which can solve the problem of lack of personalized guidance.

[0005] In a first aspect, an embodiment of the present application provides a method for guiding craftsmen to perform operations based on augmented reality, including: Receiving real-time voice of the AR teaching device and real-time image of the AR learning device; Determine a solution based on the real-time image and the real-time voice; wherein the solution includes images and text for explaining the correct operation of each operation step; Determine real-time virtual operation information based on the real-time image and the solution; wherein the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, wherein the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation; AR prompt content is obtained according to the real-time virtual operation information, the solution and the real-time image, and sent to the AR learning device; wherein the AR prompt content is an image or text generated using AR technology.

[0006] The above technical solutions in the embodiments of the present application have at least the following technical effects: The augmented reality-based craftsman operation guidance method provided in the embodiment of the present application receives the real-time voice of the AR teaching device and the real-time image of the AR learning device; determines the solution based on the real-time image and the real-time voice; determines the real-time virtual operation information based on the real-time image and the solution; obtains the AR prompt content according to the real-time virtual operation information, the solution and the real-time image, and sends it to the AR learning device. Therefore, the augmented reality-based craftsman operation guidance method provided in the embodiment of the present application can provide timely feedback and guidance by obtaining the real-time voice in the teaching process and the real-time images in the operation process of each student, so as to help students correct mistakes immediately and improve learning efficiency; and can provide personalized guidance to students based on the solution and the real-time virtual operation information and real-time images of each student.

[0007] In a possible implementation manner of the first aspect, determining a solution based on the real-time image and the real-time voice includes: Identify the image position of the operating object according to the real-time image; Obtaining corresponding object image features according to the image position of each of the operating objects; Obtaining a first task according to the object image feature and the real-time voice; wherein the first task includes an operation object name and operation requirements; According to the first task, the solution is retrieved from a database.

[0008] In a possible implementation manner of the first aspect, determining the real-time virtual operation information based on the real-time image and the solution includes: Determine the difficulty and risk level of each operation step based on the solution; Determine a first time period based on the difficulty of the operation and the degree of danger; wherein the first time period is the time required for each operation step of the student; Determining whether the dangerous operation step exists according to the degree of danger; If the dangerous operation step exists, creating the virtual operation environment using AR technology according to the real-time image and the first time period; The real-time virtual operation information is obtained.

[0009] In a possible implementation manner of the first aspect, determining the first time period based on the operation difficulty and the danger level includes: Determine an initial time period based on the difficulty of the operation and the degree of danger; wherein the initial time period is the time required for each operation step of the student determined initially; Receiving a first adjustment request from the AR learning device; wherein the first adjustment request is used to adjust the initial time period; Sending the first adjustment request and the initial time period to the AR teaching device; Receiving adjustment confirmation information sent by the AR teaching device; wherein the adjustment confirmation information includes whether the adjustment is agreed, the adjustment operation steps and the adjusted time period; The initial time period is adjusted according to the adjustment confirmation information to obtain the first time period.

[0010] In a possible implementation manner of the first aspect, obtaining AR prompt content according to the real-time virtual operation information, the solution, and the real-time image, and sending the AR prompt content to the AR learning device includes: In the first time period, first real-time operation information is obtained based on the real-time image; wherein the first real-time operation information includes images and timestamps of each operation step before each student receives guidance; Performing image comparison between the first real-time operation information and the image in the solution to obtain first difference information; wherein the first difference information includes a position difference and a size difference between an operation object in the image of the first real-time operation information and a corresponding operation object in the image in the solution; determining whether the first real-time operation information is incorrect according to the first difference information; If the first real-time operation information of the student is incorrect, generating corresponding AR prompt content according to the solution and the first real-time operation information; Performing image comparison between the real-time virtual operation information and the image in the solution to obtain second difference information; wherein the second difference information includes a position difference and a size difference between an operation object in the image of the real-time virtual operation information and a corresponding operation object in the image in the solution; judging whether the real-time virtual operation information is incorrect according to the second difference information; If the real-time virtual operation information of the student is incorrect, generating corresponding AR prompt content according to the solution and the real-time virtual operation information; If it is determined according to the real-time image and the solution that a student has not completed the corresponding previous operation step at the first timestamp, generating corresponding AR prompt content according to the first real-time operation information and the solution; wherein the first timestamp refers to the time point when the student starts to perform an operation step other than the initial operation step within the first time period; The AR prompt content is sent to the AR learning device.

[0011] In a possible implementation manner of the first aspect, after determining whether the real-time virtual operation information is incorrect according to the second difference information, the method further includes: If the real-time virtual operation information of the student is incorrect, generating corresponding real-time warning information according to the real-time virtual operation information and the solution, and sending the real-time warning information to the AR teaching device; In a possible implementation manner of the first aspect, after determining whether the first real-time operation information is incorrect according to the first difference information, the method further includes: If the first real-time operation information of the student is incorrect and the corresponding operation step is the dangerous operation step, the corresponding real-time warning information is generated according to the first real-time operation information and the solution, and the real-time warning information is sent to the AR teaching device and the AR learning device.

[0012] In a possible implementation manner of the first aspect, after sending the AR prompt content to the AR learning device, the method further includes: Determine key areas during the student's operation based on the real-time image and the solution; Determine a presentation position of the AR prompt content according to the key area; calculating the angle between the student's sight line and the presentation position; Determining whether the angle exceeds an angle range; If the angle does not exceed the angle range, the presentation position of the corresponding AR prompt content is adjusted according to the angle, and the presentation position is sent to the AR learning device.

[0013] In a possible implementation manner of the first aspect, determining whether the angle exceeds an angle range includes: If the angle exceeds the angle range, a prompt message is sent to the AR learning device; wherein the prompt message is sound or visual prompt message.

[0014] In a second aspect, an embodiment of the present application provides a craftsman operation guidance device based on augmented reality, comprising: An acquisition module, used for receiving the real-time voice of the AR teaching device and the real-time image of the AR learning device; A solution module, used to determine a solution based on the real-time image and the real-time voice; wherein the solution includes images and texts for explaining the correct operation of each operation step; A virtual operation module, for determining real-time virtual operation information based on the solution; the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, wherein the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation; A guidance module is used to obtain AR prompt content based on the real-time virtual operation information, the solution and the real-time image, and send it to the AR learning device; wherein the AR prompt content is an image or text generated using AR technology.

[0015] In a third aspect, an embodiment of the present application provides an augmented reality-based craftsman operation guidance device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the above-mentioned first aspects when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects above is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an augmented reality-based craftsman operation guidance device, the augmented reality-based craftsman operation guidance device executes any of the methods described in the first aspect above.

[0018] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a flowchart of a craftsman operation guidance method based on augmented reality provided in one embodiment of the present application; Figure 2 It is a schematic diagram of the implementation process of steps S200, S300 and S320 in the craftsman operation guidance method based on augmented reality provided in one embodiment of the present application; Figure 3It is a schematic diagram of the implementation process of steps S400, S430 and S460 in the craftsman operation guidance method based on augmented reality provided in one embodiment of the present application; Figure 4 It is a schematic diagram of the implementation process of steps S490 and S4904 in the craftsman operation guidance method based on augmented reality provided in one embodiment of the present application; Figure 5 is a structural schematic diagram of a craftsman operation guidance device based on augmented reality provided in an embodiment of the present application; Figure 6 It is a structural diagram of the craftsman operation guidance device based on augmented reality provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0023] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0025] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] In related technologies, teachers provide craftsman operation guidance by observing students' operation results. Students cannot understand in real time whether their operations are correct or how to improve them. Each student has different learning styles and abilities, but traditional teaching methods often adopt a one-size-fits-all approach and cannot provide personalized guidance for each student. Therefore, the current augmented reality-based craftsman operation guidance method lacks personalized guidance.

[0028] To solve the above problems, the embodiments of the present application provide a method, device and equipment for craftsman operation guidance based on augmented reality. In this method, by receiving the real-time voice of the AR teaching device and the real-time image of the AR learning device; determining the solution based on the real-time image and the real-time voice; determining the real-time virtual operation information based on the real-time image and the solution; obtaining the AR prompt content according to the real-time virtual operation information, the solution and the real-time image, and sending it to the AR learning device. Therefore, the method for craftsman operation guidance based on augmented reality provided by the embodiments of the present application can provide timely feedback and guidance by obtaining the real-time voice in the teaching process and the real-time images in the operation process of each student, so as to help students correct mistakes immediately and thus improve learning efficiency; it can provide personalized guidance to students based on the solution and the real-time virtual operation information and real-time images of each student.

[0029] The augmented reality-based craftsman operation guidance method provided in the embodiment of the present application can be applied to an augmented reality-based craftsman operation guidance device. At this time, the augmented reality-based craftsman operation guidance device is the executor of the augmented reality-based craftsman operation guidance method provided in the embodiment of the present application. The embodiment of the present application does not impose any restrictions on the specific type of the augmented reality-based craftsman operation guidance device.

[0030] For example, the augmented reality-based craftsman operation guidance device can be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a mobile phone, a tablet computer, a wearable device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a desktop computer, a smart screen, a smart TV, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle network terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, etc., but not limited to these.

[0031] In order to better understand the augmented reality-based craftsman operation guidance method provided in the embodiment of the present application, the specific implementation process of the augmented reality-based craftsman operation guidance method provided in the embodiment of the present application is exemplarily introduced below.

[0032] Figure 1 A schematic flow chart of a method for guiding a craftsman based on augmented reality provided in an embodiment of the present application is shown. The method for guiding a craftsman based on augmented reality includes: S100, receiving real-time voice of the AR teaching device and real-time image of the AR learning device.

[0033] Exemplarily, real-time voice can be received from AR teaching devices (such as head-mounted AR displays, fixed AR projection systems, etc., the AR teaching devices are equipped with microphones) and real-time images of each student from AR learning devices (such as head-mounted AR displays, fixed AR projection systems, etc., the AR learning devices are equipped with cameras).

[0034] S200, determining a solution based on the real-time image and the real-time voice, wherein the solution includes images and texts for explaining the correct operation of each operation step.

[0035] Exemplarily, a solution may be retrieved from a database based on real-time images and real-time voice.

[0036] In one possible implementation, see Figure 2 , S200, based on real-time image and real-time voice determination solutions, including: S210, recognizing the image position of the operating object according to the real-time image.

[0037] It can be understood that the operating objects refer to objects that need to be identified, operated or processed in the craftsman's operating environment, such as resistors, capacitors, diodes, power lines and signal lines.

[0038] Exemplarily, a target detection model in deep learning (such as Faster R-CNN, YOLO, SingleShot MultiBox Detector, etc.) can be used to output the bounding box of each operator, that is, the image position of the operator. Among them, the pre-processed historical image data containing the operator and the historical annotated image data marking the image position of the operator can be input into the YOLO model, and forward propagation and back propagation can be performed. The gradient is calculated according to the loss function (such as the cross entropy loss function, the bounding box regression loss function, etc.), the parameters of the YOLO model are updated, the training process is repeated, and the performance of the YOLO model is evaluated using the validation set until the performance of the YOLO model (such as indicators such as accuracy and recall rate) reaches the set standard or the number of training times reaches the upper limit, and the target detection model is obtained.

[0039] S220, obtaining corresponding object image features according to the image position of each operation object.

[0040] For example, a traditional feature extraction algorithm (such as Scale-Invariant Feature Transform, Speeded-Up Robust Features, Histogram of Oriented Gradients, etc.) or a feature extraction network in deep learning (such as the feature layer of VGG and ResNet) can be used to automatically extract the feature vector of each operation object, that is, the object image feature, from the image.

[0041] S230, obtaining a first task according to the object image feature and the real-time voice, wherein the first task includes an operation object name and an operation requirement.

[0042] For example, the teacher's voice instructions can be converted into text using speech recognition technology (such as Google Speech-to-Text, Mozilla DeepSpeech, etc.) to obtain speech recognition results, and the object image features and speech recognition results can be input into the multimodal fusion model to obtain the first task. Among them, a model architecture suitable for multimodal input can be selected, such as the attention mechanism or fusion network (such as LSTM+CNN, Transformer, etc.) in deep learning, by inputting the training set into the LSTM+CNN model (wherein the training set includes object image features and speech recognition results), forward propagation and back propagation are performed, the gradient is calculated according to the loss function (such as the cross entropy loss function, the bounding box regression loss function, etc.), the parameters of the LSTM+CNN model are updated, the training process is repeated, and the performance of the LSTM+CNN model is evaluated using the validation set until the performance of the LSTM+CNN model (such as indicators such as accuracy and recall rate) reaches the set standard or the number of training times reaches the upper limit, and a multimodal fusion model is obtained. For example, the first task is shown in Table 1 below.

[0043] Table 1 S240, according to the first task, retrieve a solution from the database.

[0044] Exemplarily, a relational database management system such as MySQL or PostgreSQL may be used to store and retrieve solutions. During retrieval, a corresponding query statement is generated according to the first task, and the query statement is executed in the database to obtain a solution.

[0045] Through the above steps S210 to S240, targeted solutions can be provided in real time based on the teacher's real-time voice and the real-time images during the student's operation, which is conducive to improving teaching efficiency and enabling students to obtain correct operation guidance in a timely manner.

[0046] S300, determining real-time virtual operation information based on the real-time image and the solution, wherein the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, and the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation.

[0047] Exemplarily, dangerous operation steps can be determined according to the solution, a virtual operation environment can be created based on real-time images and the solution, and real-time virtual operation information can be obtained from the AR learning device.

[0048] In one possible implementation, see Figure 2 , S300, determines real-time virtual operation information based on real-time images and solutions, including: S310, determining the operation difficulty and risk level of each operation step based on the solution.

[0049] Exemplarily, the semantic information of each operation step can be extracted from the solution, and the historical database matching can be performed based on the semantic information to determine the operation difficulty and risk level, as shown in Table 2 below.

[0050] Table 2 S320, determining a first time period based on the difficulty of the operation and the degree of danger, wherein the first time period is the time required for each operation step of the student.

[0051] Exemplarily, for each operation step, a corresponding time period, ie, an initial time period, may be preliminarily determined according to the operation difficulty and risk level, and then the initial time period may be adjusted to obtain the first time period.

[0052] Optionally, see Figure 2 , S320, determining a first time period based on the difficulty and danger level of the operation, including: S321, determining an initial time period based on the difficulty and risk level of the operation, wherein the initial time period is the time required for each operation step of the student determined initially.

[0053] Exemplarily, the initial time period can be calculated based on the difficulty of operation and the degree of danger, for example, initial time period = difficulty coefficient × danger coefficient × basic time, wherein the difficulty coefficient and danger coefficient can be quantified based on the difficulty of operation and the degree of danger of the step (for example, low = 1, medium = 1.5, high = 4), the basic time can be a fixed time unit (for example, 5 minutes), the difficulty of operation and the degree of danger of each operation step are shown in Table 2 above, and the initial time period is shown in Table 3 below.

[0054] Table 3 S322: Receive a first adjustment request from the AR learning device, wherein the first adjustment request is used to adjust an initial time period.

[0055] Exemplarily, the craftsman operation guidance device may receive a first adjustment request from the AR learning device. For example, the operation steps are as shown in Table 3 above, and the first adjustment request is: [welding resistance, delay 5 minutes].

[0056] S323: Send the first adjustment request and the initial time period to the AR teaching device.

[0057] Exemplarily, the craftsman operation guidance device may send the first adjustment request and the initial time period to the AR teaching device.

[0058] S324, receiving adjustment confirmation information sent by the AR teaching device, wherein the adjustment confirmation information includes whether the adjustment is agreed, the adjustment operation steps and the adjusted time period.

[0059] Exemplarily, the craftsman operation guidance device may receive the adjustment confirmation information sent by the AR teaching device. For example, the initial time period is as shown in Table 3 above, the first adjustment request is: [welding resistance, +5 minutes], and the adjustment confirmation information is: [Agree, welding resistance, 20 minutes].

[0060] S325: Adjust the initial time period according to the adjustment confirmation information to obtain a first time period.

[0061] Exemplarily, the craftsman operation guidance device may adjust the initial time period according to the adjustment confirmation information to obtain the first time period.

[0062] Through the above steps S321 to S325, students and teachers are allowed to flexibly adjust the time period of the operation steps according to actual conditions, which is conducive to improving the flexibility and adaptability of the teaching process. Through the confirmation and adjustment of the teacher, it is conducive to improving the rationality and accuracy of the time period setting.

[0063] S330, judging whether there is any dangerous operation step according to the degree of danger.

[0064] Exemplarily, whether there is a dangerous operation step can be determined according to the degree of danger. For example, the operation difficulty and danger level of each operation step are shown in Table 2 above, and welding resistors and welding diodes are dangerous operation steps.

[0065] S340: If there are any dangerous operation steps, a virtual operation environment is created using AR technology based on the real-time image and the first time period.

[0066] Exemplarily, 3D modeling software (such as Blender, Maya, etc.) can be used to create operating objects in a virtual operating environment based on the operating objects in the dangerous operating steps and the operating results of each student in the real-time image before the dangerous operating steps, and an AR technology platform (such as Unity, Unreal Engine, etc.) is selected to use AR markers or position tracking technology to locate and display the operating objects in the dangerous operating steps, and a virtual operating environment is presented to students through AR glasses or display screens within half of the first time period corresponding to the dangerous operating steps (i.e., the virtual operation time, as shown in Table 4 below).

[0067] Table 4 S350, obtaining real-time virtual operation information.

[0068] For example, the craftsman operation guidance device may receive real-time virtual operation information from the AR learning device.

[0069] Through the above steps S310 to S350, the teaching guidance strategy can be dynamically adjusted according to the actual operation of the students, the difficulty of the operation and the degree of danger of the steps, which is conducive to improving the teaching efficiency and quality. By creating a virtual operation environment, students can perform simulated operations in a safe environment, which is conducive to reducing the risks and costs in actual operations. Real-time capture and recording of students' operation behaviors provides data support for teaching analysis and feedback.

[0070] S400: Obtain AR prompt content based on the real-time virtual operation information, solution and real-time image, and send it to the AR learning device. The AR prompt content is an image or text generated using AR technology.

[0071] Exemplarily, the real-time virtual operation information and the real-time image can be compared with the images in the solution respectively to determine whether AR prompt content is needed, and the AR prompt content can be sent to the AR learning device.

[0072] In one possible implementation, see Figure 3 , S400, obtains AR prompt content according to real-time virtual operation information, solutions and real-time images, and sends it to the AR learning device, including: S410, obtaining first real-time operation information based on real-time images within a first time period, wherein the first real-time operation information includes images and timestamps of each operation step before each student receives guidance.

[0073] For example, in the first time period, the student's operation behavior can be captured in real time by a camera to obtain a real-time image, and the image and timestamp of each operation step can be recorded based on the real-time image.

[0074] S420: Compare the first real-time operation information with the image in the solution to obtain first difference information, wherein the first difference information includes the position difference and size difference between the operation object in the image of the first real-time operation information and the corresponding operation object in the image in the solution.

[0075] Exemplarily, a target detection algorithm (such as YOLO, Single Shot MultiBox Detector, Faster R-CNN, etc.) can be used to detect the bounding box of the operator in the image, and a classification model (such as convolutional neural network, support vector machine, random forest, gradient boosting tree, etc.) can be used to identify the type of each operator in the first real-time operation information and the type of each operator in the corresponding solution, and the position difference and size difference can be calculated by the bounding box of the same type of operator. Among them, the pre-processed image data containing the operator can be input into the convolutional neural network model, and the weights and biases of the convolutional neural network model can be updated by the back propagation algorithm, and the training process can be repeated. The performance of the convolutional neural network model is evaluated using the validation set until the performance of the convolutional neural network model (such as indicators such as accuracy and recall rate) reaches the set standard or the number of training times reaches the upper limit, and the classification model is obtained. For example, the bounding box of the operator A in the first real-time operation information detected by the YOLO model is (x1, y1, w1, h1), and the bounding box of the operator A in the image in the corresponding solution is (x2, y2, w2, h2). The position difference is: (x1-x2, y1-y2), and the size difference is: (w1 / w2, h1 / h2), where A is the type of the operator, (x1, y1) and (x2, y2) are the coordinates of the upper left corner of the bounding box, and (w1, h1) and (w2, h2) are the width and height of the bounding box.

[0076] S430: Determine whether the first real-time operation information is incorrect according to the first difference information.

[0077] Exemplarily, whether the first real-time operation information is wrong can be determined based on whether the position difference and size difference in the first difference information exceed a preset tolerance range. For example, the preset tolerance range is: .

[0078] Optionally, see Figure 3 S430, after determining whether the first real-time operation information is incorrect according to the first difference information, the method further includes: S4301: If the first real-time operation information of the student is incorrect and the corresponding operation step is a dangerous operation step, a corresponding real-time warning information is generated according to the first real-time operation information and the solution, and the real-time warning information is sent to the AR teaching device and the AR learning device.

[0079] Exemplarily, when a student's first real-time operation information is incorrect and the corresponding operation step is a dangerous operation step, a real-time warning message should be immediately generated based on the first real-time operation information and the solution, and sent to the teacher's AR teaching device. The real-time warning message should include student information, incorrect operation steps, possible risks, and corresponding correct operations. For example, the real-time warning message is: "Warning! Student [student name] made an error when performing [dangerous operation step]. This operation may cause [specific risks, such as electric shock, mechanical injury, etc.]. Please pay attention to and refer to the following operations: [correct operation]".

[0080] Through the above step S4301, when a student makes a mistake in a dangerous operation step, the teacher and the student can be reminded in time so that the teacher can quickly take intervention measures to prevent the occurrence of safety accidents, which is conducive to improving the operation safety of the students and reducing the risks in the teaching process.

[0081] S440: If the first real-time operation information of the student is incorrect, corresponding AR prompt content is generated according to the solution and the first real-time operation information.

[0082] For example, one or more AR prompts can be generated based on the student's first real-time operation information and the text in the solution used to explain the correct operation of the corresponding operation steps. For example, the AR prompt is: "Attention! Your operation is wrong. The correct operation method is: [correct operation]. Please follow this method."

[0083] S450, performing image comparison between the real-time virtual operation information and the image in the solution to obtain second difference information, wherein the second difference information includes the position difference and size difference between the operation object in the image of the real-time virtual operation information and the corresponding operation object in the image in the solution.

[0084] Exemplarily, a target detection algorithm (such as YOLO, Single Shot MultiBox Detector, Faster R-CNN, etc.) can be used to detect the bounding box of each operator in the image, and a classification model (such as convolutional neural network, support vector machine, random forest, gradient boosting tree, etc.) can be used to identify the type of each operator in the real-time virtual operation information and the solution, and the position difference and size difference are calculated by the bounding box of the same type of operator. For example, the bounding box of operator B in the real-time virtual operation information detected by the YOLO model is (x3, y3, w3, h3), and the bounding box of operator B in the image in the corresponding solution is (x4, y4, w4, h4), then the position difference is: (x3-x4, y3-y4), and the size difference is: (w3 / w4, h3 / h4), where B is the type of operator, (x3, y3) and (x4, y4) are the coordinates of the upper left corner of the bounding box, and (w3, h3) and (w4, h4) are the width and height of the bounding box.

[0085] S460: Determine whether the real-time virtual operation information is incorrect according to the second difference information.

[0086] Exemplarily, whether the real-time virtual operation information is wrong can be determined based on whether the position difference and size difference in the second difference information exceed a preset tolerance range. For example, the preset tolerance range is: .

[0087] Optionally, see Figure 3 S460, after determining whether the real-time virtual operation information is incorrect according to the second difference information, the method further includes: S4601: If there is any error in the student's real-time virtual operation information, corresponding real-time warning information is generated according to the real-time virtual operation information and the solution, and the real-time warning information is sent to the AR teaching device.

[0088] For example, when a student's real-time virtual operation information is incorrect, a real-time warning message should be immediately generated based on the real-time virtual operation information and the solution, and sent to the teacher's AR teaching device. The real-time warning message should include the student information and the operation steps of the incorrect operation. For example, the real-time warning message is: "Student [student name] made an error when performing [dangerous operation step (virtual)]".

[0089] Through the above step S4601, teachers can be reminded in time to pay attention to students' operating errors in the virtual operating environment so that they can give timely guidance and corrections, which is conducive to improving students' learning effects in the virtual operating environment, improving the safety of students' actual operations, and reducing risks in the teaching process.

[0090] S470: If the student's real-time virtual operation information is incorrect, corresponding AR prompt content is generated according to the solution and the real-time virtual operation information.

[0091] For example, one or more AR prompts can be generated based on the student's real-time virtual operation information and the text in the solution that describes the correct operation of the corresponding operation steps. For example, the AR prompt is: "Attention! Your operation is wrong. The correct operation method is: [correct operation]. Please follow this method."

[0092] S480: If it is determined based on the real-time image and the solution that a student has not completed the corresponding previous operation step at the first timestamp, a corresponding AR prompt content is generated based on the first real-time operation information and the solution. The first timestamp refers to the time point when the student starts to perform an operation step other than the initial operation step within the first time period.

[0093] For example, it can be determined based on the real-time image and the solution that a student has not completed the previous operation step at the first timestamp (i.e., when starting the next operation step), and then an AR prompt content is generated based on the first real-time operation information and the solution. For example, the AR prompt content is: "Attention! Your previous operation step is still not completed. The correct operation method of the previous operation step is: [Correct operation]".

[0094] S490: Send the AR prompt content to the AR learning device.

[0095] Exemplarily, the craftsman operation guidance device can send AR prompt content to the AR learning device of the corresponding student.

[0096] Through the above steps S410 to S490, the student's operation behavior is captured and compared in real time, errors are discovered and corrected in time, and the accuracy and effectiveness of teaching are improved. AR prompt content provides students with intuitive guidance and feedback to help them quickly understand and master operation skills. The presentation position of the AR prompt content is dynamically adjusted according to the student's line of sight and position, so that students can clearly see the prompt content, which is conducive to improving teaching effectiveness and learning experience.

[0097] Optionally, see Figure 4 , S490, after sending the AR prompt content to the AR learning device, the method further includes: S4901, Identify critical areas during student operations based on real-time images and solutions.

[0098] Exemplarily, edge detection algorithms (such as Canny, Prewitt, Sobel, etc.) can be used to obtain edge information of the operating object in the real-time image and the image of the solution (wherein the edge information includes the shape, size and position information of the operating object), and based on the edge information of the operating object in the real-time image and the image of the solution, image segmentation technology (such as GrabCut algorithm, region growing algorithm, etc.) is used to extract the key area in the real-time image.

[0099] S4902: Determine a presentation position of the AR prompt content according to the key area.

[0100] It can be understood that the presentation position is the three-dimensional position of the AR prompt content.

[0101] Exemplarily, a camera that can directly measure depth (such as a stereo camera, a ToF camera, or a structured light camera) can be used to obtain the depth value of each pixel, and each point in the key area can be converted into a point in three-dimensional space, that is, the two-dimensional coordinates (x, y) are combined with the depth value z to form three-dimensional coordinates (x, y, z). In the three-dimensional space, the average position of all points in the key area is calculated to obtain the center point (Cx, Cy, Cz). Based on historical data and industry standards, an offset (Dx, Dy, Dz) of the AR prompt content relative to the center point of the key area can be preset, and the offset is added to the center point to obtain the presentation position of the AR prompt content (Px, Py, Pz) = (Cx + Dx, Cy + Dy, Cz + Dz).

[0102] S4903, calculating the angle between the student's sight line and the presentation position.

[0103] For example, the head posture of the student can be tracked by the camera, sensor or inertial measurement unit (IMU) of the VR learning device, and the craftsman operates the guidance device to obtain the rotation matrix R (3x3 matrix) representing the head posture. , where V is the sight direction vector, R T is the transpose of the rotation matrix R. For example, if the presentation position of the AR prompt content is (Px, Py, Pz), then the AR prompt content presentation position vector P=(Px−Ox,Py−Oy,Pz−Oz) (where O=(Ox,Oy,Oz) is the position of the student's head, which can be set as the origin). , where θ is the angle between the student’s line of sight and the presentation location.

[0104] S4904, determining whether the angle exceeds the angle range.

[0105] For example, a reasonable angle range may be determined based on historical data and industry standards, for example, the angle range is [45°, 90°].

[0106] For example, see Figure 4 , S4904, determining whether the angle exceeds the angle range, including: S49041: If the angle exceeds the angle range, a prompt message is sent to the AR learning device, wherein the prompt message is a sound or visual prompt message.

[0107] Exemplarily, when the included angle (such as 30°) exceeds the angle range (such as [45°, 90°]), a prompt message (such as a voice prompt message: "Please adjust your sight to view the prompt content" or a visual prompt message: displaying an arrow in the AR interface pointing to the correct sight direction) can be sent to the AR learning device.

[0108] Through the above step S49041, by sending prompt information, students can pay attention to the AR prompt content in time and improve learning efficiency.

[0109] S4905: If the angle does not exceed the angle range, adjust the presentation position of the corresponding AR prompt content according to the angle, and send the presentation position to the AR learning device.

[0110] For example, the presentation position of the AR prompt content can be rotated and adjusted according to the size of the angle, so that the AR prompt content is perpendicular to the student's line of sight, and the presentation position is sent to the AR learning device.

[0111] Through the above steps S4901 to S4905, the presentation position of the AR prompt content during the student operation process is both accurate and easy to observe, which helps to reduce misunderstandings or omissions caused by line of sight deviation, improve students' attention and understanding, and thus improve teaching effectiveness and learning efficiency.

[0112] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0113] Corresponding to the augmented reality-based craftsman operation guidance method described in the above embodiment, the embodiment of the present application also provides an augmented reality-based craftsman operation guidance device, and each module of the device can implement each step of the augmented reality-based craftsman operation guidance method. Figure 5 A structural block diagram of an augmented reality-based craftsman operation guidance device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0114] Reference Figure 5 , the device comprises: An acquisition module, used for receiving the real-time voice of the AR teaching device and the real-time image of the AR learning device; A solution module, used to determine a solution based on the real-time image and the real-time voice; wherein the solution includes images and texts for explaining the correct operation of each operation step; A virtual operation module, for determining real-time virtual operation information based on the solution; the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, wherein the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation; A guidance module is used to obtain AR prompt content based on the real-time virtual operation information, the solution and the real-time image, and send it to the AR learning device; wherein the AR prompt content is an image or text generated using AR technology.

[0115] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0116] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0117] The present application also provides a craftsman operation guidance device based on augmented reality. Figure 6 This is a schematic diagram of the structure of an augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment provided in one embodiment of the present application. Figure 6 As shown, the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6 of this embodiment includes: at least one processor 60 ( Figure 6Only one is shown), at least one memory 61 ( Figure 6 Only one is shown in the figure) and a computer program 62 stored in the at least one memory 61 and executable on the at least one processor 60. When the processor 60 executes the computer program 62, the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6 implements the steps in any of the above-mentioned augmented reality-based craftsman operation guidance method embodiments, or enables the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6 to implement the functions of each module / unit in the above-mentioned device embodiments.

[0118] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the augmented reality-based craftsman operation guidance device 6.

[0119] The augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6 can be a computing device such as a desktop computer, a notebook, a handheld computer, and a cloud server. The augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment can include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art can understand that Figure 6 It is merely an example of an augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6 and does not constitute a limitation of the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, buses, etc.

[0120] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0121] In some embodiments, the memory 61 may be an internal storage unit of the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device 6, such as a hard disk or memory of the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device 6. In other embodiments, the memory 61 may also be an external storage device of the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device 6. Further, the memory 61 may also include both the internal storage unit of the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device 6 and an external storage device. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0122] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0123] An embodiment of the present application provides a computer program product. When the computer program product runs on an augmented reality-based craftsman operation guidance device, the augmented reality-based craftsman operation guidance device enables the device to implement the steps in any of the above-mentioned method embodiments.

[0124] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance equipment and method can be implemented in other ways. For example, the above-described augmented reality-based craftsman operation guidance device / augmented reality-based craftsman operation guidance device embodiment is merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for guiding craftsmen's operation based on augmented reality, characterized in that: Applied to a craftsman operation guidance device, the craftsman operation guidance device is respectively connected to an AR teaching device at a teacher end and an AR learning device at a student end, and the method includes: Receiving real-time voice of the AR teaching device and real-time image of the AR learning device; Determine a solution based on the real-time image and the real-time voice; wherein the solution includes images and text for explaining the correct operation of each operation step; Determine real-time virtual operation information based on the real-time image and the solution; wherein the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, wherein the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation; AR prompt content is obtained according to the real-time virtual operation information, the solution and the real-time image, and sent to the AR learning device; wherein the AR prompt content is an image or text generated using AR technology.

2. The method for guiding craftsmen to operate based on augmented reality according to claim 1, characterized in that: The determining of a solution based on the real-time image and the real-time voice includes: Identify the image position of the operating object according to the real-time image; Obtaining corresponding object image features according to the image position of each of the operating objects; Obtaining a first task according to the object image feature and the real-time voice; wherein the first task includes an operation object name and operation requirements; According to the first task, the solution is retrieved from a database.

3. The method for guiding craftsmen to operate based on augmented reality according to claim 1, characterized in that: The determining of real-time virtual operation information based on the real-time image and the solution includes: Determine the difficulty and risk level of each operation step based on the solution; Determine a first time period based on the difficulty of the operation and the degree of danger; wherein the first time period is the time required for each operation step of the student; Determining whether the dangerous operation step exists according to the degree of danger; If the dangerous operation step exists, creating the virtual operation environment using AR technology according to the real-time image and the first time period; The real-time virtual operation information is obtained.

4. The method for guiding craftsmen to operate based on augmented reality according to claim 3, characterized in that: The determining the first time period based on the operation difficulty and the danger level includes: Determine an initial time period based on the difficulty of the operation and the degree of danger; wherein the initial time period is the time required for each operation step of the student determined initially; Receiving a first adjustment request from the AR learning device; wherein the first adjustment request is used to adjust the initial time period; Sending the first adjustment request and the initial time period to the AR teaching device; Receiving adjustment confirmation information sent by the AR teaching device; wherein the adjustment confirmation information includes whether the adjustment is agreed, the adjustment operation steps and the adjusted time period; The initial time period is adjusted according to the adjustment confirmation information to obtain the first time period.

5. The method for guiding craftsmen to operate based on augmented reality according to claim 4, characterized in that: The obtaining of AR prompt content according to the real-time virtual operation information, the solution and the real-time image, and sending the content to the AR learning device includes: In the first time period, first real-time operation information is obtained based on the real-time image; wherein the first real-time operation information includes images and timestamps of each operation step before each student receives guidance; Performing image comparison between the first real-time operation information and the image in the solution to obtain first difference information; wherein the first difference information includes a position difference and a size difference between an operation object in the image of the first real-time operation information and a corresponding operation object in the image in the solution; determining whether the first real-time operation information is incorrect according to the first difference information; If the first real-time operation information of the student is incorrect, generating corresponding AR prompt content according to the solution and the first real-time operation information; Performing image comparison between the real-time virtual operation information and the image in the solution to obtain second difference information; wherein the second difference information includes a position difference and a size difference between an operation object in the image of the real-time virtual operation information and a corresponding operation object in the image in the solution; judging whether the real-time virtual operation information is incorrect according to the second difference information; If the real-time virtual operation information of the student is incorrect, generating corresponding AR prompt content according to the solution and the real-time virtual operation information; If it is determined according to the real-time image and the solution that a student has not completed the corresponding previous operation step at the first timestamp, generating corresponding AR prompt content according to the first real-time operation information and the solution; wherein the first timestamp refers to the time point when the student starts to perform an operation step other than the initial operation step within the first time period; The AR prompt content is sent to the AR learning device.

6. The method for guiding craftsmen to operate based on augmented reality according to claim 5, characterized in that: After determining whether the real-time virtual operation information is incorrect according to the second difference information, the method further includes: If the real-time virtual operation information of the student is incorrect, generating corresponding real-time warning information according to the real-time virtual operation information and the solution, and sending the real-time warning information to the AR teaching device; And / or, after determining whether the first real-time operation information is incorrect according to the first difference information, the method further includes: If the first real-time operation information of the student is incorrect and the corresponding operation step is the dangerous operation step, the corresponding real-time warning information is generated according to the first real-time operation information and the solution, and the real-time warning information is sent to the AR teaching device and the AR learning device.

7. The method for guiding craftsmen to operate based on augmented reality according to claim 5, characterized in that: After sending the AR prompt content to the AR learning device, the method further includes: Determine key areas during the student's operation based on the real-time image and the solution; Determine a presentation position of the AR prompt content according to the key area; calculating the angle between the student's sight line and the presentation position; Determining whether the angle exceeds an angle range; If the angle does not exceed the angle range, the presentation position of the corresponding AR prompt content is adjusted according to the angle, and the presentation position is sent to the AR learning device.

8. The method for guiding craftsmen to operate based on augmented reality according to claim 7, characterized in that: The determining whether the angle exceeds an angle range includes: If the angle exceeds the angle range, a prompt message is sent to the AR learning device; wherein the prompt message is sound or visual prompt message.

9. A craftsman operation guidance device based on augmented reality, characterized in that: Applied to a craftsman operation guidance device, the craftsman operation guidance device is respectively connected to the AR teaching device on the teacher side and the AR learning device on the student side, including: An acquisition module, used for receiving the real-time voice of the AR teaching device and the real-time image of the AR learning device; A solution module, used to determine a solution based on the real-time image and the real-time voice; wherein the solution includes images and texts for explaining the correct operation of each operation step; A virtual operation module, for determining real-time virtual operation information based on the solution; the real-time virtual operation information includes images and timestamps of dangerous operation steps performed by each student in the virtual operation environment, wherein the dangerous operation steps refer to steps that may cause physical harm to the student and / or damage to the equipment during the operation; A guidance module is used to obtain AR prompt content based on the real-time virtual operation information, the solution and the real-time image, and send it to the AR learning device; wherein the AR prompt content is an image or text generated using AR technology.

10. A craftsman operation guidance device based on augmented reality, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images