A method for three-dimensional guided operation based on industrial scenes

Abstract

The present invention discloses a method for three-dimensional guided operation based on industrial scenes, involving the fields of AR and AI visual technology, including making a device model for identifying target instruments in a real environment, using yolov5 to train a model for identifying target equipment; displaying a virtual prompt panel in space, using the ARCore framework to enhance the understanding of the environment, motion tracking, and illumination estimation; using the hands detection module of the MediaPipe open source framework to detect the human hand joint nodes through the trained model, and return the three-dimensional position in the image to perform gesture tracking; according to the coordinate position of the gesture joint obtained in the gesture tracking, processing is performed, converted into data for judgment, and judgment is performed. This method can greatly reduce the operating threshold and the possibility of errors of operators, thereby improving the safety management and control level of smart factories.

Description

Technical Field

[0001] The present invention relates to the field of AR and AI visual technology, and in particular to a method for three-dimensional guided operations based on industrial scenarios. Background Art

[0002] With the advancement of smart factory reform, factory management continues to develop and innovate. It is very common to use instruments in factories. How to further improve the accuracy and safety of operators in operating instruments, ensure the normal operation of the factory, and promote safe factory operations has become a focus of many factories. Therefore, guiding the use of instruments has become an indispensable and important part of safe production management.

[0003] Although factories have paid enough attention to the use of instruments over the years, in the past, it was mostly through paper manuals or through the teaching of experience between teachers and apprentices. However, from the actual effect, these methods are all based on experience and memory, and are too dependent on the learning and memory ability of operators. Therefore, the effect is not satisfactory, and instrument usage errors frequently occur. Especially in the context of the rapid advancement of smart factories, the need for safe use of instruments needs to be addressed urgently. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a three-dimensional guided operation method based on industrial scenarios, which assists users in using instruments normally and avoids erroneous operations through virtual panel indications and simultaneous recognition of gestures, thereby significantly reducing the irregular behaviors of operators during instrument operation and avoiding the occurrence of safety accidents.

[0006] In order to solve the above technical problems, the present invention provides the following technical solution, a method for three-dimensional guided operation based on industrial scenes, comprising:

[0007] Create a device model for identifying target instruments in the real environment, and use yolov5 to train a model for identifying target devices; display a virtual prompt panel in the space, and use the ARCore framework to enhance the understanding of the environment, including motion tracking and lighting estimation; use the hands detection module of the MediaPipe open source framework to detect the hand joint nodes through the trained model, and return the three-dimensional position in the image for gesture tracking; process the coordinate positions of the gesture joints obtained in gesture tracking, convert them into data that can be judged, and make judgments.

[0008] As a preferred solution of the method for three-dimensional guided operations based on industrial scenarios described in the present invention, wherein: the enhanced understanding of the environment means that ARCore will detect corner points, edges and texture feature points in the scene, and track the movement of the feature points to obtain the position and direction in the device; the motion tracking means that ARCore uses the IMU of the device to measure the movement and rotation of the device; the lighting estimation means that ARCore analyzes camera images to estimate the light direction and intensity of the environment.

[0009] As a preferred solution of the method for three-dimensional guided operation based on industrial scenarios described in the present invention, the detection of human hand joint nodes includes a gesture recognition algorithm model based on convolutional neural networks and recurrent neural networks, combined with the hands detection module of the MediaPipe open source framework, which detects 21 joint nodes on the human hand through a trained model, and returns the three-dimensional position in the image, annotates the nodes in the image, connects them with lines, obtains complete gestures, and further performs gesture recognition through a deep learning model.

[0010] As a preferred solution of the method for three-dimensional guided operation based on industrial scene described in the present invention, the judgment includes processing the coordinate positions of the gesture joints obtained in gesture tracking and converting them into data for judgment, and the specific calculation is as follows:

[0011] Euclidean distance calculation, for each pair of joint points before and after, calculate the Euclidean distance:

[0012] ,

[0013] in, , , is the joint point coordinate at the current moment, , , is the coordinate of the joint point at the previous moment;

[0014] Average moving distance calculation, calculate the average moving distance of all related nodes:

[0015] ,

[0016] in, is the number of joint points, is the index of the joint point, It is The distance to move the joint.

[0017] As a preferred solution of the method for three-dimensional guided operation based on industrial scene of the present invention, the determination further includes angle calculation, which calculates the angle change between each pair of joint points at the previous and next moments:

[0018] ,

[0019] in, It is The angle between the joint points; , is the current moment The x and y coordinates of each joint point; , It is the previous moment The x and y coordinates of the joint points;

[0020] Average angle calculation, calculate the average angle change of all related nodes:

[0021] ,

[0022] in, is the number of joint points, and angle is the average change angle of the joint.

[0023] As a preferred solution of the method for three-dimensional guided operation based on industrial scene described in the present invention, wherein: the gesture tracking includes extracting the spatial features of each image frame using a convolutional neural network CNN, assuming that the image is , the convolution kernel is , then the convolutional layer output is calculated as follows:

[0024] ,

[0025] in is the output of the convolutional layer, is the activation function, Indicates that the convolution kernel is in a The value of the mth row and nth column in the matrix, M represents the number of rows of the convolution kernel, and N represents the number of columns of the convolution kernel. represents the index of the row in the output feature map of the convolutional layer, is the pixel value of the input image, Represents the index of the column in the convolutional layer output feature map, is the bias term;

[0026] The pooling layer reduces the dimension by taking the maximum value or average value of the local area. Suppose the size of the pooling area is , then the pooling layer output is calculated as follows:

[0027] ,

[0028] Based on the extracted spatial features, the long short-term memory network in the recurrent neural network RNN ​​is used to capture the time series features of the gesture. Assuming the time step is t, the input of the LSTM unit is and , the output is and , the calculation process is as follows:

[0029] ,

[0030] in, Represents the activation value of the forget gate; The sigmoid activation function compresses the output value to between (0, 1); Represents the weight matrix of the forget gate; Represents the hidden state at the previous moment and the current input connection; Represents the bias vector of the forget gate;

[0031] ,

[0032] in, represents the activation value of the input gate; represents the weight matrix of the input gate; represents the bias vector of the input gate;

[0033] ,

[0034] in, Represents candidate memory, through the current input and the hidden state at the previous moment Calculated and used to generate new cell states; It is a hyperbolic tangent activation function, which compresses the output value to between (-1, 1); represents the weight matrix used to generate candidate memories; represents the bias vector used to generate candidate memories;

[0035] ,

[0036] in, Represents the cell state at the current moment, combining the cell state at the previous moment and the current candidate memory; The part that represents the cell state at the previous moment controlled by the forget gate; represents the portion of the current candidate memory controlled by the input gate;

[0037] ,

[0038] in, represents the activation value of the output gate; Represents the weight matrix of the output gate; represents the bias vector of the output gate;

[0039] ,

[0040] in, The hidden state at the current moment, combined with the current cell state and the activation value of the output gate ; Indicates cell status go through The value after the activation function is used to generate the output hidden state;

[0041] Combine the distance and angle features extracted by MediaPipe with the features extracted by CNN and LSTM to form a fused feature vector :

[0042] ,

[0043] Among them, angle is the average change angle of the joint;

[0044] Use a fully connected layer to fuse the feature vector Input to the Softmax classifier to get the probability distribution of gesture categories :

[0045] ,

[0046] in, is the probability distribution vector calculated by the Softmax function, indicating the probability that the input belongs to each gesture category; and are weight matrix and bias vector respectively; the specific calculation formula of Softmax function is:

[0047] ,

[0048] in, is the probability distribution vector A component in the , indicating that the input belongs to a specific gesture category The probability of represents the output vector, ; R represents the total number of categories; e represents the base of the natural logarithm, Represents the corresponding category of the model in the output vector z of the fully connected layer The component of The unnormalized fraction of It represents the first Quantity The exponential transformation of Represents the kth component in the model output vector z where k is an index variable ranging from 1 to R.

[0049] Train the fusion model, assuming that the training data set is ,in is the index of the sample, It is The input features of samples, It is The label of each sample represents the true category or label of the sample; the loss function is the cross entropy loss, the optimization algorithm is the Adam optimizer, and the cross entropy loss function is defined as follows:

[0050] ,

[0051] in, Represented as the total number of training samples, Represented as the model prediction The output prediction probability of samples;

[0052] The gesture recognition model optimization algorithm uses the Adam optimizer, and the update rules are as follows:

[0053] gradient calculate:

[0054] ,

[0055] in, represents the learnable parameters in the gesture recognition model; Indicated in The parameters calculated at the iteration The gradient of It means that the loss function L has parameters The gradient at Represents the loss function, which is the cross entropy loss function, with parameters The gradient represents the direction and rate of change of the loss function under the current parameter value;

[0056] First moment estimate and the second moment estimate Update:

[0057] ,

[0058] ,

[0059] in, Indicates that at time step The first moment estimate of ; Indicates that at time step The first moment estimate of ; is the decay rate of the first-order moment estimate, which is =0.9; Indicates that at time step The second moment estimate of ; Indicates that at time step The second moment estimate of ; is the decay rate of the second-order moment estimate, which is =0.999;

[0060] Bias correction:

[0061] ,

[0062] ,

[0063] in, Indicated in At the iteration, the bias-corrected first-order moment estimate; Indicated in At the iteration, the bias-corrected second-order moment estimate; represents the decay rate of the first-order moment estimate of power, as time steps increase, reduce; represents the decay rate of the second-order moment estimate of power, as time steps increase, reduce;

[0064] After bias correction, the parameters are updated using the following formula:

[0065] ,

[0066] in, is the learning rate, which controls the step size of each parameter update; To prevent division by zero, the constant is taken as θ t Indicates The parameter value after iterations; θ t-1 Indicates The parameter value before iteration.

[0067] As a preferred solution of the method for three-dimensional guidance operation based on industrial scene according to the present invention, wherein: according to the output gesture category probability , select the category with the highest probability as the recognized gesture category , gesture category Determined by the following formula:

[0068] ,

[0069] in, Represents the final prediction result of the gesture category, that is, the gesture category predicted by the model; It is used to find the parameter that maximizes the function value, that is, to find the The largest value; The index representing the gesture category; ∈{1,2,3,4} corresponds to four different gesture categories; Indicates category The predicted probability is obtained by calculating the Softmax function, indicating that the output image belongs to the category possibility.

[0070] As a preferred solution of the method for three-dimensional guided operation based on industrial scenarios described in the present invention, the judgment also includes determining whether the operation is accurate, comparing the returned gesture with the actual operation manual, and providing feedback through a virtual panel; the comparison and judgment means that the actual operation manual asks the operator to click switch A, and the actual system identifies whether the gesture performed by the operator on switch A is a click gesture.

[0071] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for three-dimensional guided operation based on an industrial scene when executing the computer program.

[0072] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for three-dimensional guided operation based on an industrial scenario.

[0073] Beneficial effects of the present invention: The present invention proposes to provide a method for three-dimensional guided operations based on industrial scenarios, which uses AR technology to add a virtual instruction panel to prompt operators to operate the correct steps and actions of the instrument, recognize user gestures, and promptly stop operators' wrong actions, thereby meeting the urgent needs of factories for safe use of instruments. Compared with the prior art, the present invention has significant substantive characteristics and obvious progress. Specifically, the present invention identifies the actual instruments and equipment to be operated, combines AR capabilities, displays the operation prompt panel in real space, and prompts operators in real time whether the operation is accurate, as well as the next specific operation, etc., which greatly reduces the operating threshold and the possibility of errors for operators, thereby improving the safety management and control level of smart factories. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0075] Figure 1 A schematic flow chart of a method for three-dimensional guided operation based on an industrial scenario provided for one embodiment of the present invention.

[0076] Figure 2 A schematic diagram of human joint nodes of a method for three-dimensional guided operation based on an industrial scenario provided by an embodiment of the present invention.

[0077] Figure 3 A schematic flow chart of a method for three-dimensional guided operation based on an industrial scenario provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0079] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0080] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0081] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0082] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0083] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0084] Example 1, reference Figure 1-Figure 3 , which is the first embodiment of the present invention, provides a method for three-dimensional guided operation based on industrial scenes, including:

[0085] Combining a variety of advanced technologies, including instrument recognition, AR 3D, gesture recognition and action confirmation, a more natural and intuitive interactive method is adopted, which significantly reduces the threshold of operation and improves the level of factory safety production. The following is the data of relevant instrument guidance methods on the market. By comparison, it can be proved that the method of the present invention has more advantages in safety and efficiency.

[0086] Table 1 Comparison of existing instrument guidance methods

[0087]

[0088] From the above data, it can be seen that: Operation error rate: The operation error rate of the present invention reaches 2.2%, which is significantly lower than the existing instrument guidance method. Safety accident rate: The safety accident rate of the present invention is only 1.3%, and the operator injury rate is only 0.5%, which is significantly lower than the existing method. Equipment damage rate: The equipment damage rate of the present invention is only 2.1%, which is much lower than the existing method.

[0089] In actual experiments, we adjusted the instrument guidance method for a large manufacturing plant from the previous traditional method: relying on paper manuals and mentor-apprentice guidance, to the present invention.

[0090] like Figure 1 As shown in , the specific steps of a method for three-dimensional guided operation based on industrial scenes are as follows:

[0091] Create a device model for identifying target instruments in the real environment, and use yolov5 to train a model for identifying target devices; display a virtual prompt panel in the space, and use the ARCore framework to enhance the understanding of the environment, motion tracking, and lighting estimation; use the hands detection module of the MediaPipe open source framework to detect the human hand joint nodes through the trained model, and return the three-dimensional position in the image for gesture tracking; process the coordinate positions of the gesture joints obtained in gesture tracking, convert them into data that can be judged, and make judgments.

[0092] The enhanced understanding of the environment means that ARCore will detect corners, edges, and texture feature points in the scene, track the movement of the feature points, and obtain the position and direction in the device; the motion tracking means that ARCore uses the device's IMU to measure the movement and rotation of the device; the light estimation means that ARCore analyzes camera images and estimates the light direction and intensity of the environment.

[0093] The detection of human hand joint nodes includes a gesture recognition algorithm model based on convolutional neural network and recurrent neural network, combined with the hands detection module of MediaPipe open source framework, to detect 21 joint nodes on the human hand through a trained model (refer to Figure 2 ), and returns the 3D position in the image, marks the nodes in the image, connects them with lines, obtains the complete gesture, and further performs gesture recognition through a deep learning model.

[0094] in, Figure 2 The middle joint nodes are wrist joint 0, thumb carpometacarpal joint 1, thumb metacarpophalangeal joint 2, thumb interphalangeal joint 3, thumb fingertip 4, index finger metacarpophalangeal joint 5, index finger interphalangeal joint 6, index finger distal interphalangeal joint 7, index fingertip 8, middle finger metacarpophalangeal joint 9, middle finger interphalangeal joint 10, middle finger distal interphalangeal joint 11, middle finger tip 12, ring finger metacarpophalangeal joint 13, ring finger interphalangeal joint 14, ring finger distal interphalangeal joint 15, ring finger tip 16, little finger metacarpophalangeal joint 17, little finger interphalangeal joint 18, little finger distal interphalangeal joint 19, little finger tip 20.

[0095] The determination includes processing the coordinate positions of the gesture joints obtained in the gesture tracking and converting them into data for determination. The specific calculation is as follows:

[0096] Euclidean distance calculation, for each pair of joint points before and after, calculate the Euclidean distance:

[0097] ,

[0098] in, , , is the joint point coordinate at the current moment, , , is the coordinate of the joint point at the previous moment;

[0099] Average moving distance calculation, calculate the average moving distance of all related nodes:

[0100] ,

[0101] in, is the number of joint points, is the index of the joint point, It is The distance to move the joint.

[0102] The determination also includes angle calculation, which calculates the angle change between each pair of joint points at the previous and next moments:

[0103] ,

[0104] in, It is The angle between the joint points; , is the current moment The x and y coordinates of each joint point; , It is the previous moment The x and y coordinates of the joint points;

[0105] Average angle calculation, calculate the average angle change of all related nodes:

[0106] ,

[0107] in, is the number of joint points, and angle is the average change angle of the joint.

[0108] The gesture tracking includes extracting the spatial features of each image frame using a convolutional neural network (CNN). , the convolution kernel is , then the convolutional layer output is calculated as follows:

[0109] ,

[0110] in is the output of the convolutional layer, is the activation function, Indicates that the convolution kernel is in a The value of the mth row and nth column in the matrix, m represents the index of the convolution kernel in the height (number of rows) direction, M represents the number of rows of the convolution kernel, n represents the index of the convolution kernel in the width (number of columns), and N represents the number of columns of the convolution kernel. represents the index of the row in the output feature map of the convolutional layer, is the pixel value of the input image, Represents the index of the column in the convolutional layer output feature map, is the bias term;

[0111] The pooling layer reduces the dimension by taking the maximum value or average value of the local area. Suppose the size of the pooling area is , then the pooling layer output is calculated as follows:

[0112] ,

[0113] Based on the extracted spatial features, the long short-term memory network in the recurrent neural network RNN ​​is used to capture the time series features of the gesture. Assuming the time step is t, the input of the LSTM unit is and , the output is and , the calculation process is as follows:

[0114] ,

[0115] in, Represents the activation value of the forget gate; The sigmoid activation function compresses the output value to between (0, 1); Represents the weight matrix of the forget gate; Represents the hidden state at the previous moment and the current input connection; Represents the bias vector of the forget gate;

[0116] ,

[0117] in, represents the activation value of the input gate; represents the weight matrix of the input gate; represents the bias vector of the input gate;

[0118] ,

[0119] in, Represents candidate memory, through the current input and the hidden state at the previous moment Calculated and used to generate new cell states; It is a hyperbolic tangent activation function, which compresses the output value to between (-1, 1); represents the weight matrix used to generate candidate memories; represents the bias vector used to generate candidate memories;

[0120] ,

[0121] in, Represents the cell state at the current moment, combining the cell state at the previous moment and the current candidate memory; The part that represents the cell state at the previous moment controlled by the forget gate; represents the portion of the current candidate memory controlled by the input gate;

[0122] ,

[0123] in, represents the activation value of the output gate; Represents the weight matrix of the output gate; represents the bias vector of the output gate;

[0124] ,

[0125] in, The hidden state at the current moment, combined with the current cell state and the activation value of the output gate ; Indicates cell status go through The value after the activation function is used to generate the output hidden state;

[0126] Combine the distance and angle features extracted by MediaPipe with the features extracted by CNN and LSTM to form a fused feature vector :

[0127] ,

[0128] Among them, angle is the average change angle of the joint;

[0129] Use a fully connected layer to fuse the feature vector Input to the Softmax classifier to get the probability distribution of gesture categories :

[0130] ,

[0131] in, is the probability distribution vector calculated by the Softmax function, indicating the probability that the input belongs to each gesture category; and are weight matrix and bias vector respectively; the specific calculation formula of Softmax function is:

[0132] ,

[0133] in, is the probability distribution vector A component in the , indicating that the input belongs to a specific gesture category The probability of represents the output vector, ; R represents the total number of categories; e represents the base of the natural logarithm, Represents the corresponding category of the model in the output vector z of the fully connected layer The component of The unnormalized fraction of It represents the first Quantity The exponential transformation of Represents the kth component in the model output vector z where k is an index variable ranging from 1 to R.

[0134] Train the fusion model, assuming that the training data set is ,in is the index of the sample, It is The input features of samples, It is The label of each sample represents the true category or label of the sample; the loss function is the cross entropy loss, the optimization algorithm is the Adam optimizer, and the cross entropy loss function is defined as follows:

[0135] ,

[0136] in, Represented as the total number of training samples, Represented as the model prediction The output prediction probability of samples;

[0137] The gesture recognition model optimization algorithm uses the Adam optimizer, and the update rules are as follows:

[0138] gradient calculate:

[0139] ,

[0140] in, represents the learnable parameters in the gesture recognition model; Indicated in The parameters calculated at the iteration The gradient of It means that the loss function L has parameters The gradient at Represents the loss function, which is the cross entropy loss function, with parameters The gradient represents the direction and rate of change of the loss function under the current parameter value;

[0141] First moment estimate and the second moment estimate Update:

[0142] ,

[0143] ,

[0144] in, Indicates that at time step The first moment estimate of ; Indicates that at time step The first moment estimate of ; is the decay rate of the first-order moment estimate, which is =0.9; Indicates that at time step The second moment estimate of ; Indicates that at time step The second moment estimate of ; is the decay rate of the second-order moment estimate, which is =0.999;

[0145] Bias correction:

[0146] ,

[0147] ,

[0148] in, Indicated in At the iteration, the bias-corrected first-order moment estimate; Indicated in At the iteration, the bias-corrected second-order moment estimate; represents the decay rate of the first-order moment estimate of power, as time steps increase, reduce; represents the decay rate of the second-order moment estimate of power, as time steps increase, reduce;

[0149] After bias correction, the parameters are updated using the following formula:

[0150] ,

[0151] in, is the learning rate, which controls the step size of each parameter update; To prevent division by zero, the constant is taken as θ t Indicates The parameter value after iterations; θ t-1 Indicates The parameter value before iteration.

[0152] According to the output gesture category probability , select the category with the highest probability as the recognized gesture category , gesture category Determined by the following formula:

[0153] ,

[0154] in, Represents the final prediction result of the gesture category, that is, the gesture category predicted by the model; It is used to find the parameter that maximizes the function value, that is, to find the The largest value; The index representing the gesture category; ∈{1,2,3,4} corresponds to four different gesture categories; Indicates category The predicted probability is obtained by calculating the Softmax function, indicating that the output image belongs to the category possibility.

[0155] The judgment also includes determining whether the operation is accurate, comparing the returned gesture with the actual operation manual, and providing feedback through the virtual panel; the comparison and judgment means that the actual operation manual requires the operator to click switch A, and the actual system recognizes whether the gesture performed by the operator on switch A is a click gesture.

[0156] After the system recognizes that the operator's gesture is a click operation, it prompts on the demand panel: "Correct operation."

[0157] The above is a guide method for the use of instruments in a large manufacturing plant. Based on this invention, we have obtained the following comparison results of the operating error rate and the safety accident rate:

[0158] Table 2 Comparison of guidance methods for instrument use in a large manufacturing plant

[0159]

[0160] Referring to the results in Table 1 and Table 2, we can see whether our invention can effectively reduce the operating error rate and safety accident rate of factory instrument use, thereby enhancing the factory's safety production management capabilities. It can be seen that this invention has broad application prospects in industrial scenarios.

[0161] Embodiment 2, the second embodiment of the present invention, is different from the previous embodiment in that:

[0162] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0167] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for three-dimensional guided operation based on industrial scenes, characterized in that: include, Create a device model for identifying target instruments in real environments, and use yolov5 to train the model for identifying target devices; Display a virtual prompt panel in the space, using the ARCore framework to enhance environmental understanding, motion tracking, and lighting estimation; The enhanced understanding of the environment means that ARCore detects corners, edges, and texture feature points in the scene, tracks the movement of feature points, and obtains the position and direction of the device; the motion tracking means that ARCore uses the device's IMU to measure the movement and rotation of the device; the light estimation means that ARCore analyzes camera images and estimates the direction and intensity of the light in the environment; Use the hands detection module of the MediaPipe open source framework to detect the hand joint nodes through the trained model and return the 3D position in the image for gesture tracking; According to the coordinate positions of the gesture joints obtained in gesture tracking, they are processed, converted into data that can be judged, and then judged.

2. The method for three-dimensional guided operation based on industrial scene according to claim 1, characterized in that: The detection of hand joint nodes includes a gesture recognition algorithm model based on convolutional neural networks and recurrent neural networks, combined with the hands detection module of the MediaPipe open source framework, which detects 21 joint nodes on the hand through a trained model, returns the three-dimensional position in the image, marks the nodes in the image, connects them with lines, obtains a complete gesture, and further performs gesture recognition through a deep learning model.

3. The method for three-dimensional guidance of industrial scenes according to claim 2, characterized in that: The determination includes processing the coordinate positions of the gesture joints obtained in the gesture tracking and converting them into data for determination. The specific calculation is as follows: Euclidean distance calculation, for each pair of joint points before and after, calculate the Euclidean distance: in, is the joint point coordinate at the current moment, is the coordinate of the joint point at the previous moment; Average moving distance calculation, calculate the average moving distance of all related nodes: Where λ is the number of joint points, i is the index of the joint point, and d i is the moving distance of the i-th pair of joints.

4. The method for three-dimensional guidance of industrial scenes according to claim 3, characterized in that: The determination also includes angle calculation, which calculates the angle change between each pair of joint points at the previous and next moments: Among them, θ i is the angle between the i-th pair of joint points; are the x and y coordinates of the i-th joint point at the current moment; are the x and y coordinates of the i-th joint point at the previous moment; Average angle calculation, calculate the average angle change of all related nodes: Among them, λ is the number of joint points, and angle is the average change angle of the joint.

5. The method for three-dimensional guidance of industrial scenes according to claim 4, characterized in that: The gesture tracking includes extracting the spatial features of each image frame using a convolutional neural network (CNN). Let the image be X and the convolution kernel be K. The convolution layer output is calculated as follows: Where O is the output of the convolutional layer, f() is the activation function, and K mn represents the value of the mth row and nth column of the convolution kernel in an M×N matrix, M represents the number of rows of the convolution kernel, N represents the number of columns of the convolution kernel, φ represents the index of the row in the output feature map of the convolution layer, X (φ+m-1)(ψ+n-1) is the pixel value of the input image, ψ represents the index of the column in the output feature map of the convolutional layer, and b is the bias term; The pooling layer reduces the dimension by taking the maximum or average value of the local area. Assuming the size of the pooling area is p×p, the output of the pooling layer is calculated as follows: Based on the extracted spatial features, the long short-term memory network in the recurrent neural network RNN ​​is used to capture the time series features of the gesture. Assume that the time step is t and the input of the LSTM unit is h t-1 and c t-1 , the output is h t and c t , the calculation process is as follows: f t =σ(W f ·[h t-1 ,x t ]+b f ) Among them, f t represents the activation value of the forget gate; σ is the sigmoid activation function, which compresses the output value to between (0,1); W f represents the weight matrix of the forget gate; [h t-1 , x t ] represents the hidden state h at the previous moment t-1 and the current input x t connection; b f Represents the bias vector of the forget gate; i t =σ(W i ·[h t-1 ,x t ]+b i ) Among them, i t represents the activation value of the input gate; W i represents the weight matrix of the input gate; b i represents the bias vector of the input gate; in, Represents candidate memory, through the current input x t and the hidden state h at the previous moment t-1 Calculated and used to generate new cell states; tanh is the hyperbolic tangent activation function, which compresses the output value to between (-1,1); W c represents the weight matrix used to generate candidate memories; b c represents the bias vector used to generate candidate memories; Among them, c t Represents the cell state at the current moment, combining the cell state at the previous moment and the current candidate memory; f t ·c t-1 The part that represents the cell state at the previous moment controlled by the forget gate; represents the portion of the current candidate memory controlled by the input gate; the t =σ(W o ·[h t-1 ,x t ]+b o ) Among them, O t Represents the activation value of the output gate; W o represents the weight matrix of the output gate; b o represents the bias vector of the output gate; h t =o t ·tanh(c t ) Among them, h t The hidden state at the current moment, combined with the current cell state c t and the activation value o of the output gate t ; tanh(c t ) represents the cell state c t The value after the tanh activation function is used to generate the output hidden state; Combine the distance and angle features extracted by MediaPipe with the features extracted by CNN and LSTM to form a fused feature vector g: g=[h t ,movement,angle] Among them, angle is the average change angle of the joint; Use the fully connected layer to input the fused feature vector g into the Softmax classifier to obtain the gesture category probability distribution y: y=softmax(W·g+b) Among them, y is the probability distribution vector calculated by the Softmax function, which indicates the probability that the input belongs to each gesture category; W and b are the weight matrix and bias vector respectively; the specific calculation formula of the Softmax function is: Among them, y r is a component in the probability distribution vector y, representing the probability that the input belongs to a specific gesture category r; z represents the output vector, z = W g + b; R represents the total number of categories; e represents the base of the natural logarithm, z r Represents the component of the model corresponding to category r in the output vector z of the fully connected layer, representing the unnormalized score of category r, It represents the rth component z in the model output vector z r The exponential transformation of Represents the kth component z in the model output vector z k exponential transformation, where k is an index variable ranging from 1 to R; Train the fusion model, assuming that the training data set is (x (α) ,y (α) ), where α is the index of the sample, x (α) is the input feature of the αth sample, y (α) is the label of the αth sample, indicating the true category or label of the sample; the loss function is the cross entropy loss, the optimization algorithm is the Adam optimizer, and the cross entropy loss function is defined as follows: Where T represents the total number of training samples. Represents the output prediction probability of the αth sample predicted by the model; The gesture recognition model optimization algorithm uses the Adam optimizer, and the update rules are as follows: Gradient g t calculate: Where θ represents the learnable parameters in the gesture recognition model; g t represents the gradient of the parameter θ calculated at the tth iteration; Represents the loss function L in parameter θ t-1 The gradient at L(θ t-1 ) represents the loss function, which is the cross entropy loss function, and the parameter θ t-1 The gradient represents the direction and rate of change of the loss function under the current parameter value; First-order moment estimate m t and the second-order moment estimate v t Update: m t =β1m t-1 +(1-β1)g t Among them, m t represents the first-order moment estimate at time step t; m t-1 represents the first-order moment estimate at time step t-1; β1 is the decay rate of the first-order moment estimate, which is β1 = 0.9; v t represents the second-order moment estimate at time step t; v t-1 represents the second-order moment estimate at time step t-1; β2 is the decay rate of the second-order moment estimate, and its value is β2=0.999; Bias correction: in, represents the bias-corrected first-order moment estimate at the tth iteration; represents the bias-corrected second-order moment estimate at the tth iteration; Represents the decay rate β1 of the first-order moment estimate to the power of t. As the time step t increases, reduce; represents the decay rate β2 of the second-order moment estimate to the power of t. As the time step t increases, reduce; After bias correction, the parameters are updated using the following formula: Among them, η is the learning rate, which controls the step size of each parameter update; ∈ is a constant to prevent division by zero, and its value is 10 -8 θ t represents the parameter value after the tth iteration; θ t-1 Represents the parameter value before the tth iteration.

6. A method for three-dimensional guidance of industrial scenes according to claim 5, characterized in that: According to the output gesture category probability y, the category with the largest probability is selected as the recognized gesture category C. The gesture category C is determined by the following formula: Among them, C represents the final prediction result of the gesture category, that is, the gesture category predicted by the model; It is used to find the parameter that maximizes the function value, that is, to find the parameter that makes y τ The largest τ value; τ represents the index of the gesture category; τ∈{1,2,3,4} corresponds to four different gesture categories; y τ Represents the predicted probability of category τ, which is calculated by the Softmax function and indicates the possibility that the output image belongs to category τ.

7. A method for three-dimensional guidance of operations based on industrial scenarios as claimed in claim 6, characterized in that: The judgment also includes determining whether the operation is accurate, comparing the returned gesture with the actual operation manual, and providing feedback through the virtual panel; the comparison and judgment means that the actual operation manual requires the operator to click switch A, and the actual system recognizes whether the gesture performed by the operator on switch A is a click gesture.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images