Generative MR industrial inspection method based on large model and natural man-machine interaction

Abstract

The invention discloses a generative MR industrial inspection method based on a large model and natural man-machine interaction, and belongs to the technical field of intelligent inspection. In order to solve the technical problems of long development period and high cost of MR inspection, the method comprises the following steps: acquiring a scene image sequence to be inspected by using MR head display equipment, constructing an inspection scene image data set, and training a multi-modal large model by using the inspection scene image data set to obtain category sequence information data to be inspected; acquiring corresponding spatial positioning information and interaction information, constructing an inspection module data set based on the acquired spatial positioning information and interaction information, and then training an inspection module by using the inspection module data set to obtain to-be-detected information of an inspection sequence of the inspection module; fitting the category sequence information to be inspected and to-be-detected information of the inspection sequence of each inspection module to obtain an inspection execution model; and executing a generative MR industrial inspection process based on the large model and natural man-machine interaction by using the inspection execution model.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent inspection, and in particular relates to a generative MR industrial inspection method based on a large model and natural human-computer interaction. Background Art

[0002] Inspection work is crucial to ensure the normal operation of industrial systems (including production lines, large-scale production equipment, and important pipelines). This work can efficiently maintain and repair industrial equipment and promptly eliminate problems that may exist in long-term operation, thereby fundamentally improving the safety, efficiency, and stability of industrial equipment. In recent years, with the gradual promotion of mixed reality technology in industrial applications, the smart inspection system in which inspectors wear mixed reality headsets or glasses has gradually been applied in industrial sites.

[0003] Mixed reality MR intelligent inspection is that the inspector wears an MR headset to enter the industrial site. The MR headset runs a software customized for the industrial site, and performs spatial positioning based on the MR headset's own capabilities (such as real-time photography and comparison with background data to determine which workstation / workstation / inspection point the inspector is at), or determines the inspector's position in the factory based on external sensors (cameras or iBeacons, etc.). Then, according to the location information, the corresponding inspection task information is displayed on the MR headset, such as work orders, SOPs, or digital twin models of corresponding work sections, to supervise, guide, or assist inspection operations. However, the problems with the existing technology are: the industrial system is very large, and different industrial categories have different requirements for inspection; even in the same industrial category, different enterprises and factories have different requirements for inspection; even in the same factory, the same production line, and the same equipment, when producing different products, the requirements for inspection are different.

[0004] The current application of MR inspection needs to be customized according to different production tasks. The customization process is cumbersome and requires not only the participation of professional program developers, but also engineers with rich production experience and rich inspection experience to participate in the planning, including writing scripts, setting standards, and managing processes. In addition, it is also necessary to go to the industrial site for testing and joint debugging. The cost of application development is very high. This is why, although MR intelligent inspection has set a benchmark in some industrial industries, it has not been widely promoted. Summary of the invention

[0005] The problem to be solved by the present invention is the technical difficulties of long development cycle and high cost of MR inspection, and a generative MR industrial inspection method based on large model and natural human-computer interaction is proposed.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A generative MR industrial inspection method based on large models and natural human-computer interaction includes the following steps:

[0008] S1. Use the MR head display device to collect the scene image sequence to be inspected, build the scene image dataset for inspection, use the scene image dataset for inspection to train the multimodal large model, and obtain the category sequence information data to be inspected;

[0009] S2. Based on the MR head display device, corresponding spatial positioning information and interaction information are collected, wherein the interaction information includes eye tracking information, body movement information, voice information, and gesture information, and an inspection module data set is constructed based on the collected spatial positioning information and interaction information. Then, the inspection module data set is used to train the inspection module to obtain the intended detection information of the inspection sequence of the inspection module;

[0010] S3. Fitting the category sequence information to be inspected obtained in step S1 and the inspection sequence information of each inspection module obtained in step S2 to obtain an inspection execution model;

[0011] S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

[0012] Furthermore, the specific implementation method of step S1 includes the following steps:

[0013] S1.1. Collect image sequences of the scene to be inspected using the MR head-mounted display device;

[0014] S1.2. Annotate the elements in the scene image sequence to be inspected collected in step S1.1, wherein the elements include target objects, actions, and states, and then perform image preprocessing, including normalization, denoising, and enhancement, to obtain a scene image dataset for inspection;

[0015] S1.3. Construct loss functions for multimodal large model training, including mean square error loss function and cross entropy loss function;

[0016] The mean square error loss function MSE is used for regression problems and calculates the average square difference between the predicted value and the true value. For numerical prediction tasks in image sequences, the formula is:

[0017]

[0018] Where n is the number of samples, y i is the true value, is the predicted value;

[0019] The cross entropy loss function H(p,q) is used for classification tasks in image sequences, and the formula is:

[0020]

[0021] Among them, p(x) is the true probability distribution, q(x) is the predicted probability distribution;

[0022] S1.4. Adjust the loss function of the multimodal large model training constructed in step S1.3:

[0023] Add temporal coherence constraints to the loss function, calculate the difference in prediction values ​​between adjacent frames, and add the difference in prediction values ​​between adjacent frames as a loss term to the total loss function;

[0024] Adding optical flow loss to image sequences with motion information constrains the model's prediction of object motion. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames.

[0025] S1.5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the inspection scene image dataset, and the optimization algorithm adopts the stochastic gradient descent method;

[0026] The training step is to input the inspection scene image dataset into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, use the optimization algorithm to update the model's parameters to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times.

[0027] Furthermore, during the training process of step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers, and the optimal hyperparameter combination is found through the cross-validation method.

[0028] Furthermore, the specific implementation method of step S2 includes the following steps:

[0029] S2.1. Collect corresponding spatial positioning information and interaction information based on the MR head display device, including using eye tracking equipment to collect the target's eye information in different scenarios, including eye movement trajectory, gaze point, and scanning speed, and using GPS or indoor positioning system to obtain the target's spatial position information. During the inspection process, record the target's interaction with the inspection module, including the time and order of the target viewing a specific device or inspecting a certain area;

[0030] S2.2. Inspection module parameter collection: Collect the parameters of each inspection module, including the characteristics of the inspection object, the indicator range under normal conditions, and common fault types;

[0031] S2.3. Perform data preprocessing and feature extraction on the data collected in step S2.1 and step S2.2 to construct an inspection module data set;

[0032] The data preprocessing method includes data cleaning and data screening;

[0033] The feature extraction, for eyeball information, extracts the distribution of gaze points, gaze time, and scanning path features; for spatial positioning information, extracts the position coordinates, moving speed, and direction features of the target; for inspection module parameters, extracts the values ​​and change trend features of key indicators;

[0034] S2.4. Divide the inspection module data set into a training set and a test set, select a neural network or a support vector machine or a decision tree as the machine learning algorithm, build a machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module, using batch gradient descent or stochastic gradient descent method to accelerate the training process;

[0035] Use the test set to evaluate the trained machine learning model of the inspection module. The evaluation indicators include accuracy, recall rate, and F1 value. According to the evaluation results, optimize the machine learning model of the inspection module, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method.

[0036] S2.5. Based on the machine learning model of the optimized inspection module, the parameters of the inspection module including the inspector's position, gaze point, and operation execution parameters are obtained; the spatial positioning information is used to determine the location scene of the target and find the corresponding inspection module; the interaction information is used to map the interaction information and the parameters of the corresponding inspection module, and the intended detection information under the inspection module is obtained through training.

[0037] Furthermore, the specific implementation method of step S2.5 includes the following steps:

[0038] S2.5.1. Use eye tracking equipment to collect the target's eye information in different scenes and collect the eye direction vector Where E x is the x-axis direction vector of the eyeball, E y is the y-axis direction vector of the eyeball, E z is the z-axis direction vector of the eyeball; set the eyeball viewing angle range to the horizontal viewing angle range α and the vertical viewing angle range β;

[0039] S2.5.2. Use GPS or indoor positioning system to obtain the spatial position information of the target and determine the position coordinates of the target in three-dimensional space, set as Where P x is the x-axis coordinate of the target in three-dimensional space, P y is the y-axis coordinate of the target in three-dimensional space, P z is the z-axis coordinate of the target in three-dimensional space;

[0040] S2.5.3. Establish a reference coordinate system to determine the relative position of the target and the eyeball;

[0041] S2.5.4. Calculate the sight vector. The vector from the eyeball position to the target is obtained by subtracting the eyeball direction vector from the target's position coordinates. The formula is:

[0042]

[0043] S2.5.5. Calculate the angle between the line of sight and the axis of the reference coordinate system. Use the dot product of the vectors and the modulus to calculate the angle between the line of sight vector and the axis of the reference coordinate system. Calculate using the following formula:

[0044]

[0045] Among them, Cos(θ x ) is the cosine value of the angle between the line of sight and the x-axis of the reference coordinate system, is the unit vector of the x-axis in the reference coordinate system;

[0046] S2.5.6. Determine whether the target is within the field of view based on the visual angle of the eyeball and the angle between the line of sight and the coordinate axis. If the angle between the line of sight and the x-axis is within the horizontal visual angle range α, and the angle between the line of sight and the y-axis is within the vertical visual angle range β, then the target is within the field of view.

[0047] Furthermore, the specific implementation method of step S4 includes the following steps:

[0048] S4.1. After the inspection starts, the spatial position information of the MR head display device is obtained, the spatial position information is input into the inspection execution model, the inspection module and the information to be detected that should appear at this time are calculated and determined, and transmitted to the MR head display device to generate the display information to be inspected;

[0049] S4.2. Obtain the interactive information collected by the MR head display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship in the inspection execution model;

[0050] S4.3. When the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head display device for display;

[0051] S4.4. After all inspection modules have been completed, the data collected by the MR headset device is again sent to the inspection execution model for training to obtain an optimized inspection execution model.

[0052] Beneficial effects of the present invention:

[0053] The present invention describes a generative MR industrial inspection method based on a large model and natural human-computer interaction. The back end adopts a multimodal large model for industrial subdivisions, and the front end adopts natural human-computer interaction inputs including voice, gaze point, gesture, spatial position, body movement, etc. On the one hand, it collects multimodal large model training data, and on the other hand, it automatically generates an input interface for the application. Finally, the MR head-mounted display device is used as the terminal hardware carrier to collect the user's voice, gaze point, gesture, spatial position and other information, which is transmitted to the multimodal large model at the back end through the network to generate corresponding inspection tasks. After the MR head-mounted display device receives it, it is displayed at the corresponding time and space.

[0054] The generative MR industrial inspection method based on a large model and natural human-computer interaction described in the present invention uses the training corpus collected by MR to train the large model, which can quickly accumulate a large amount of actual training corpus and improve the training efficiency of the large model in the subdivided field; based on the trained multimodal large model, the inspection task is generated in real time according to the comprehensive feedback data of the corresponding MR head display, and the data in the real-time inspection process can also optimize the multimodal data of the inspection personnel for training, thereby improving the accuracy of the inspection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a generative MR industrial inspection method based on a large model and natural human-computer interaction described in the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0057] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0058] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 The detailed instructions are as follows:

[0059] Embodiment 1:

[0060] A generative MR industrial inspection method based on large models and natural human-computer interaction includes the following steps:

[0061] S1. Use the MR head display device to collect the scene image sequence to be inspected, build the scene image dataset for inspection, use the scene image dataset for inspection to train the multimodal large model, and obtain the category sequence information data to be inspected;

[0062] Furthermore,

[0063] The specific implementation method of step S1 includes the following steps:

[0064] S1.1. Collect image sequences of the scene to be inspected using the MR head-mounted display device;

[0065] S1.2. Annotate the elements in the scene image sequence to be inspected collected in step S1.1, wherein the elements include target objects, actions, and states, and then perform image preprocessing, including normalization, denoising, and enhancement, to obtain a scene image dataset for inspection;

[0066] S1.3. Construct loss functions for multimodal large model training, including mean square error loss function and cross entropy loss function;

[0067] The mean square error loss function MSE is used for regression problems and calculates the average square difference between the predicted value and the true value. For numerical prediction tasks in image sequences, the formula is:

[0068]

[0069] Where n is the number of samples, y i is the true value, is the predicted value;

[0070] The cross entropy loss function H(p,q) is used for classification tasks in image sequences, and the formula is:

[0071]

[0072] Among them, p(x) is the true probability distribution, q(x) is the predicted probability distribution;

[0073] S1.4. Adjust the loss function of the multimodal large model training constructed in step S1.3:

[0074] Add temporal coherence constraints to the loss function, calculate the difference in prediction values ​​between adjacent frames, and add the difference in prediction values ​​between adjacent frames as a loss term to the total loss function;

[0075] Adding optical flow loss to image sequences with motion information constrains the model's prediction of object motion. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames.

[0076] S1.5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the inspection scene image dataset, and the optimization algorithm adopts the stochastic gradient descent method;

[0077] The training step is to input the inspection scene image dataset into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, use the optimization algorithm to update the model's parameters to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times.

[0078] Furthermore, during the training process of step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers, and the optimal hyperparameter combination is found through the cross-validation method.

[0079] S2. Based on the MR head display device, corresponding spatial positioning information and interaction information are collected, wherein the interaction information includes eye tracking information, body movement information, voice information, and gesture information, and an inspection module data set is constructed based on the collected spatial positioning information and interaction information. Then, the inspection module data set is used to train the inspection module to obtain the intended detection information of the inspection sequence of the inspection module;

[0080] Furthermore, the specific implementation method of step S2 includes the following steps:

[0081] S2.1. Collect corresponding spatial positioning information and interaction information based on the MR head display device, including using eye tracking equipment to collect the target's eye information in different scenarios, including eye movement trajectory, gaze point, and scanning speed, and using GPS or indoor positioning system to obtain the target's spatial position information. During the inspection process, record the target's interaction with the inspection module, including the time and order of the target viewing a specific device or inspecting a certain area;

[0082] S2.2. Inspection module parameter collection: Collect the parameters of each inspection module, including the characteristics of the inspection object, the indicator range under normal conditions, and common fault types;

[0083] S2.3. Perform data preprocessing and feature extraction on the data collected in step S2.1 and step S2.2 to construct an inspection module data set;

[0084] The data preprocessing method includes data cleaning and data screening;

[0085] The feature extraction, for eyeball information, extracts the distribution of gaze points, gaze time, and scanning path features; for spatial positioning information, extracts the position coordinates, moving speed, and direction features of the target; for inspection module parameters, extracts the values ​​and change trend features of key indicators;

[0086] S2.4. Divide the inspection module data set into a training set and a test set, select a neural network or a support vector machine or a decision tree as the machine learning algorithm, build a machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module, using batch gradient descent or stochastic gradient descent method to accelerate the training process;

[0087] Use the test set to evaluate the trained machine learning model of the inspection module. The evaluation indicators include accuracy, recall rate, and F1 value. According to the evaluation results, optimize the machine learning model of the inspection module, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method.

[0088] S2.5. Based on the machine learning model of the optimized inspection module, the parameters of the inspection module including the inspector's position, gaze point, and operation execution parameters are obtained; the spatial positioning information is used to determine the location scene of the target and find the corresponding inspection module; the interaction information is used to map the interaction information and the parameters of the corresponding inspection module, and the intended detection information under the inspection module is obtained through training.

[0089] Furthermore, the specific implementation method of step S2.5 includes the following steps:

[0090] S2.5.1. Use eye tracking equipment to collect the target's eye information in different scenes and collect the eye direction vector Where E x is the x-axis direction vector of the eyeball, E y is the y-axis direction vector of the eyeball, E z is the z-axis direction vector of the eyeball; set the eyeball viewing angle range to the horizontal viewing angle range α and the vertical viewing angle range β;

[0091] S2.5.2. Use GPS or indoor positioning system to obtain the spatial position information of the target and determine the position coordinates of the target in three-dimensional space, set as Where P x is the x-axis coordinate of the target in three-dimensional space, P y is the y-axis coordinate of the target in three-dimensional space, P z is the z-axis coordinate of the target in three-dimensional space;

[0092] S2.5.3. Establish a reference coordinate system to determine the relative position of the target and the eyeball;

[0093] S2.5.4. Calculate the sight vector. The vector from the eyeball position to the target is obtained by subtracting the eyeball direction vector from the target's position coordinates. The formula is:

[0094]

[0095] S2.5.5. Calculate the angle between the line of sight and the axis of the reference coordinate system. Use the dot product of the vectors and the modulus to calculate the angle between the line of sight vector and the axis of the reference coordinate system. Calculate using the following formula:

[0096]

[0097] Among them, Cos(θ x ) is the cosine value of the angle between the line of sight and the x-axis of the reference coordinate system, is the unit vector of the x-axis in the reference coordinate system;

[0098] S2.5.6. Determine whether the target is within the field of view based on the visual angle of the eyeball and the angle between the line of sight and the coordinate axis. If the angle between the line of sight and the x-axis is within the horizontal visual angle range α, and the angle between the line of sight and the y-axis is within the vertical visual angle range β, then the target is within the field of view.

[0099] S3. Fitting the category sequence information to be inspected obtained in step S1 and the inspection sequence information of each inspection module obtained in step S2 to obtain an inspection execution model;

[0100] The following are some examples:

[0101] Suppose an inspection engineer wears an MR headset, stands in front of an equipment cabinet (position and head orientation), looks at a meter on the cabinet, points his finger at the meter, and says (voice): "The pressure gauge reads 0.5 MPa, which is normal. You can go to the next inspection point." He also says: "There is an abnormality. If the pressure gauge reading exceeds 0.7 MPa, you need to go to equipment A immediately to check the flow meter reading." The corpus information collected here is shown in the table:

[0102] Table 1

[0103]

[0104] Furthermore, by three-dimensionally rendering the factory digital twin in the MR headset, inspectors wearing the MR headset can patrol the virtual digital factory, avoiding the need to collect corpus at the industrial site. Without interrupting normal industrial production, training corpus can be obtained quickly and at low cost.

[0105] Furthermore, compared with traditional MR inspection applications, this inspection training scheme can adaptively train the inspection execution model suitable for the scenario by training the parameters obtained from the first operation of the MR headset;

[0106] Furthermore, this training process involves: adaptation of input and output data dimensions; alignment of multiple input data, including alignment in time, space and data scale; setting of training objective function (loss function) and its association with inspection application.

[0107] S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

[0108] Furthermore, the specific implementation method of step S4 includes the following steps:

[0109] S4.1. After the inspection starts, the spatial position information of the MR head display device is obtained, the spatial position information is input into the inspection execution model, the inspection module and the information to be detected that should appear at this time are calculated and determined, and transmitted to the MR head display device to generate the display information to be inspected;

[0110] S4.2. Obtain the interactive information collected by the MR head display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship in the inspection execution model;

[0111] S4.3. When the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head display device for display;

[0112] S4.4. After all inspection modules have been completed, the data collected by the MR headset device is again sent to the inspection execution model for training to obtain an optimized inspection execution model.

[0113] Furthermore, when the inspector's operation information is identified as inaccurate or out of error range by the large model during the inspection process, the large model will generate comparison information between the two, send it to the MR head display for display, and generate a selection confirmation message to let the inspector select one of the operation information. If the selected operation information is the information previously trained and collected by the large model, the operation information for re-executing the inspection module will be generated, allowing the inspector to perform an operation check on the inspection module again; if the selected operation information is the current operation information, it will be fed back to the large model to train the inspection module again, and generate a prompt message that the inspection module is to be re-tested, allowing the inspector to re-test the inspection module after a certain period of time.

[0114] Furthermore, the actual sensing data of the MR terminal at the scene is used to generate inspection task information suitable for the corresponding inspection module based on the previously generated large model and MR spatial positioning information, and instruct the inspection engineer to complete the inspection task. And when a difference is found between the operation and the data of the large model, comparison information can be generated to let the inspector confirm whether the inspection module needs to be retrained.

[0115] An example is as follows: an inspection engineer wears an MR headset to enter an industrial site. At this time, the sensor functions of the MR headset are all turned on, and prompts are transmitted in real time to the backend large model server. The response of the large model server is displayed in the MR headset in the form of voice broadcast or text or other visual annotations (arrows, annotation boxes, PDF documents, pictures, etc.), as shown in Table 2:

[0116] Table 2

[0117]

[0118]

[0119] An example of a model response is as follows:

[0120] 1 Voice or text: "Turn your head and look at device B";

[0121] 2 Visual annotation: "The arrow points to device B";

[0122] 3 Voice or text: "Go to device B now";

[0123] 4 Visual annotation: "The annotation box highlights the indicator light L and switch S on the device";

[0124] 5 Voice or text: "Check indicator light L: if it shows green, it is normal. If it shows yellow, please turn on switch S. If it shows red, please contact the factory command center immediately."

[0125] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0126] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A generative MR industrial inspection method based on large models and natural human-computer interaction, characterized in that: The steps include: S1. Use the MR head display device to collect the scene image sequence to be inspected, build the scene image dataset for inspection, use the scene image dataset for inspection to train the multimodal large model, and obtain the category sequence information data to be inspected; S2. Based on the MR head display device, corresponding spatial positioning information and interaction information are collected, wherein the interaction information includes eye tracking information, body movement information, voice information, and gesture information, and an inspection module data set is constructed based on the collected spatial positioning information and interaction information. Then, the inspection module data set is used to train the inspection module to obtain the intended detection information of the inspection sequence of the inspection module; S3. Fitting the category sequence information to be inspected obtained in step S1 and the inspection sequence information of each inspection module obtained in step S2 to obtain an inspection execution model; S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

2. A generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1.

1. Collect image sequences of the scene to be inspected using the MR head-mounted display device; S1.

2. Annotate the elements in the scene image sequence to be inspected collected in step S1.1, wherein the elements include target objects, actions, and states, and then perform image preprocessing, including normalization, denoising, and enhancement, to obtain a scene image dataset for inspection; S1.

3. Construct loss functions for multimodal large model training, including mean square error loss function and cross entropy loss function; The mean square error loss function MSE is used for regression problems and calculates the average square difference between the predicted value and the true value. For numerical prediction tasks in image sequences, the formula is: Where n is the number of samples, y i is the true value, is the predicted value; The cross entropy loss function H(p,q) is used for classification tasks in image sequences, and the formula is: Among them, p(x) is the true probability distribution, q(x) is the predicted probability distribution; S1.

4. Adjust the loss function of the multimodal large model training constructed in step S1.3: Add temporal coherence constraints to the loss function, calculate the difference in prediction values ​​between adjacent frames, and add the difference in prediction values ​​between adjacent frames as a loss term to the total loss function; Adding optical flow loss to image sequences with motion information constrains the model's prediction of object motion. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames. S1.

5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the inspection scene image dataset, and the optimization algorithm adopts the stochastic gradient descent method; The training step is to input the inspection scene image dataset into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, use the optimization algorithm to update the model's parameters to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times.

3. A generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 2, characterized in that: During the training process of step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers, and the optimal hyperparameter combination is found through the cross-validation method.

4. A generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 3, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Collect corresponding spatial positioning information and interaction information based on the MR head display device, including using eye tracking equipment to collect the target's eye information in different scenarios, including eye movement trajectory, gaze point, and scanning speed, and using GPS or indoor positioning system to obtain the target's spatial position information. During the inspection process, record the target's interaction with the inspection module, including the time and order of the target viewing a specific device or inspecting a certain area; S2.

2. Inspection module parameter collection: Collect the parameters of each inspection module, including the characteristics of the inspection object, the indicator range under normal conditions, and common fault types; S2.

3. Perform data preprocessing and feature extraction on the data collected in step S2.1 and step S2.2 to construct an inspection module data set; The data preprocessing method includes data cleaning and data screening; The feature extraction, for eyeball information, extracts the distribution of gaze points, gaze time, and scanning path features; for spatial positioning information, extracts the position coordinates, movement speed, and direction features of the target; For inspection module parameters, extract the values ​​and change trend characteristics of key indicators; S2.

4. Divide the inspection module data set into a training set and a test set, select a neural network or a support vector machine or a decision tree as the machine learning algorithm, build a machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module, using batch gradient descent or stochastic gradient descent method to accelerate the training process; Use the test set to evaluate the trained machine learning model of the inspection module. The evaluation indicators include accuracy, recall rate, and F1 value. According to the evaluation results, optimize the machine learning model of the inspection module, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method. S2.

5. Based on the optimized machine learning model of the inspection module, the parameters of the inspection module including the inspector's position, gaze point, and operation execution parameters are obtained; the location scene of the target is determined by using the spatial positioning information, and the inspection module adapted to the corresponding one is found; By utilizing the interactive information, the interactive information and the parameters of the corresponding inspection module are mapped, and the intended detection information under the inspection module is obtained through training.

5. A generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 4, characterized in that: The specific implementation method of step S2.5 includes the following steps: S2.5.

1. Use eye tracking equipment to collect the target's eye information in different scenes and collect the eye direction vector Where E x is the x-axis direction vector of the eyeball, E y is the y-axis direction vector of the eyeball, E z is the z-axis direction vector of the eyeball; set the eyeball viewing angle range to the horizontal viewing angle range α and the vertical viewing angle range β; S2.5.

2. Use GPS or indoor positioning system to obtain the spatial location information of the target and determine the position coordinates of the target in three-dimensional space, set as Where P x is the x-axis coordinate of the target in three-dimensional space, P y is the y-axis coordinate of the target in three-dimensional space, P z is the z-axis coordinate of the target in three-dimensional space; S2.5.

3. Establish a reference coordinate system to determine the relative position of the target and the eyeball; S2.5.

4. Calculate the sight vector. The vector from the eyeball position to the target is obtained by subtracting the eyeball direction vector from the target's position coordinates. The formula is: S2.5.

5. Calculate the angle between the line of sight and the axis of the reference coordinate system. Use the dot product of the vectors and the modulus to calculate the angle between the line of sight vector and the axis of the reference coordinate system. Calculate using the following formula: Among them, Cos(θ x ) is the cosine value of the angle between the line of sight and the x-axis of the reference coordinate system, is the unit vector of the x-axis in the reference coordinate system; S2.5.

6. Determine whether the target is within the field of view based on the visual angle of the eyeball and the angle between the line of sight and the coordinate axis. If the angle between the line of sight and the x-axis is within the horizontal visual angle range α, and the angle between the line of sight and the y-axis is within the vertical visual angle range β, then the target is within the field of view.

6. A generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 5, characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. After the inspection starts, the spatial position information of the MR head display device is obtained, the spatial position information is input into the inspection execution model, the inspection module and the information to be detected that should appear at this time are calculated and determined, and transmitted to the MR head display device to generate the display information to be inspected; S4.

2. Obtain the interactive information collected by the MR head display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship in the inspection execution model; S4.

3. When the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head display device for display; S4.

4. After all inspection modules have been completed, the data collected by the MR headset device is again sent to the inspection execution model for training to obtain an optimized inspection execution model.

Images