A Generative MR Industrial Inspection Method Based on Large Models and Natural Human-Computer Interaction

Abstract

This invention discloses a generative MR industrial inspection method based on a large model and natural human-computer interaction, belonging to the field of intelligent inspection technology. To address the technical challenges of long development cycles and high costs in MR inspection, this invention includes: acquiring scene image sequences to be inspected using an MR head-mounted display device to construct an inspection scene image dataset; training a multimodal large model using the inspection scene image dataset to obtain category sequence information data to be inspected; acquiring corresponding spatial positioning and interaction information; constructing an inspection module dataset based on the acquired spatial positioning and interaction information; training inspection modules using the inspection module dataset to obtain the proposed detection information of the inspection sequence of the inspection module; fitting the category sequence information to be inspected and the proposed detection information of the inspection sequences of each inspection module to obtain an inspection execution model; and using the inspection execution model to execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

Description

Technical Field

[0001] This invention belongs to the field of intelligent inspection technology, specifically relating to a generative MR industrial inspection method based on large models and natural human-computer interaction. Background Technology

[0002] Inspection work is crucial for ensuring the normal operation of industrial systems, including production lines, large production equipment, and critical pipelines. This work efficiently maintains and repairs industrial equipment, promptly eliminating potential problems that may have arisen during long-term operation, thereby fundamentally improving the safety, efficiency, and stability of industrial equipment. In recent years, with the gradual promotion of Mixed Reality (MRP) technology in industrial applications, intelligent inspection systems where inspectors wear MRP headsets or glasses are increasingly being used in industrial settings.

[0003] Mixed Reality (MR) intelligent inspection involves inspectors wearing MR headsets to enter industrial sites. The headsets run software customized for that specific industrial environment, using their own spatial positioning capabilities (such as real-time photography and comparison with backend data to determine the workstation / station / inspection point) or externally deployed sensors (cameras or iBeacon, etc.) to determine the inspector's location within the factory. Based on this location information, the MR headset displays corresponding inspection task information, such as work orders, standard operating procedures (SOPs), or digital twin models of the corresponding work section, to supervise, guide, or assist inspection operations. However, existing technologies have limitations: industrial systems are vast, and different industrial sectors have varying inspection requirements; even within the same industrial sector, different companies and factories have different requirements; and even within the same factory, on the same production line, using the same equipment, the inspection requirements can differ depending on the product being produced.

[0004] Current MR inspection applications need to be customized according to different production tasks. The customization process is complicated and requires not only professional programmers but also engineers with rich production and inspection experience to participate in the planning, including writing scripts, setting standards, and streamlining processes. Moreover, on-site testing and debugging are required in the industrial field. The cost of application development is very high. This is why, although MR intelligent inspection has set a benchmark in some industrial sectors, it has not been widely promoted. Summary of the Invention

[0005] The problem this invention aims to solve is the technical challenge of long development cycles and high costs in MR inspection. It proposes a generative MR industrial inspection method based on large models and natural human-computer interaction.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A generative MR industrial inspection method based on large models and natural human-computer interaction includes the following steps:

[0008] S1. Use an MR head-mounted display device to collect scene image sequences to be inspected, construct a scene image dataset for inspection, and use the scene image dataset for inspection to train a multimodal large model to obtain category sequence information data to be inspected.

[0009] S2. Collect corresponding spatial positioning information and interaction information based on the MR head-mounted display device. The interaction information includes eye tracking information, body movement information, voice information and gesture information. Construct an inspection module dataset based on the collected spatial positioning information and interaction information. Then, use the inspection module dataset to train the inspection module to obtain the proposed detection information of the inspection sequence of the inspection module.

[0010] S3. Fit the category sequence information to be inspected obtained in step S1 and the proposed detection information of the inspection sequence of each inspection module obtained in step S2 to obtain the inspection execution model.

[0011] S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

[0012] Furthermore, the specific implementation method of step S1 includes the following steps:

[0013] S1.1. Use an MR head-mounted display device to acquire a sequence of scene images to be inspected;

[0014] S1.2. The elements in the scene image sequence to be inspected collected in step S1.1 are labeled. The elements include target objects, actions, and states. Then, image preprocessing is performed, including normalization, denoising, and enhancement, to obtain the scene image dataset to be inspected.

[0015] S1.3. Construct the loss function for training a multimodal large model, including the mean squared error loss function and the cross-entropy loss function;

[0016] The mean squared error loss function (MSE) is used for regression problems to calculate the mean squared difference between the predicted and true values. For numerical prediction tasks in image sequences, the formula is:

[0017]

[0018] Where n is the number of samples, y i It is the actual value. It is a predicted value;

[0019] The cross-entropy loss function H(p,q) is used for classification tasks in image sequences, and its formula is:

[0020]

[0021] Where p(x) is the true probability distribution and q(x) is the predicted probability distribution;

[0022] S1.4. Adjust the loss function for training the multimodal large model constructed in step S1.3:

[0023] Add a temporal coherence constraint to the loss function, calculate the difference between the predicted values ​​of adjacent frames, and add the difference between the predicted values ​​of adjacent frames as a loss term to the total loss function;

[0024] An optical flow loss constraint model is added to an image sequence with motion information to predict the motion of objects. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames.

[0025] S1.5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the scene image dataset from the inspection, and the optimization algorithm adopts the stochastic gradient descent method.

[0026] The training steps are as follows: input the scene image dataset of the inspection into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, update the model's parameters using the optimization algorithm to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times.

[0027] Furthermore, during the training process in step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers, and the optimal combination of hyperparameters is found through cross-validation.

[0028] Furthermore, the specific implementation method of step S2 includes the following steps:

[0029] S2.1. Collect corresponding spatial positioning information and interaction information based on MR head-mounted display devices, including using eye-tracking devices to collect eye information of the target in different scenarios, including eye movement trajectory, gaze point, and saccade speed; using GPS or indoor positioning systems to obtain the target's spatial location information; and recording the interaction behavior between the target and the inspection module during the inspection process, including the time and order in which the target views specific devices and inspects a certain area.

[0030] S2.2. Inspection Module Parameter Collection: Collect parameters for each inspection module, including the characteristics of the inspection object, the index range under normal conditions, and common fault types;

[0031] S2.3. Perform data preprocessing and feature extraction on the data collected in steps S2.1 and S2.2 to construct the inspection module dataset;

[0032] The data preprocessing method includes data cleaning and data filtering;

[0033] The feature extraction includes, for eye information, extracting the distribution of gaze points, gaze time, and saccade path features; for spatial positioning information, extracting the target's position coordinates, movement speed, and direction features; and for inspection module parameters, extracting the values ​​and trend features of key indicators.

[0034] S2.4. Divide the inspection module dataset into a training set and a test set, select a neural network, support vector machine, or decision tree as the machine learning algorithm, construct the machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module. Use batch gradient descent or stochastic gradient descent methods to accelerate the training process.

[0035] The machine learning model of the inspection module was evaluated using a test set, with evaluation metrics including accuracy, recall, and F1 score. Based on the evaluation results, the machine learning model of the inspection module was optimized, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method.

[0036] S2.5. Based on the optimized inspection module's machine learning model, the parameters of the inspection module are obtained, including the inspector's position, gaze point, and operation execution parameters; using spatial positioning information, the location scene of the target is determined, and the corresponding inspection module is found; using interaction information, the interaction information and the parameters of the corresponding inspection module are mapped, and the proposed detection information under the inspection module is trained.

[0037] Furthermore, the specific implementation method of step S2.5 includes the following steps:

[0038] S2.5.1. Use an eye-tracking device to collect eye information of the target in different scenarios, and acquire the eye direction vector. Where E x E is the x-axis vector of the eyeball. y E is the y-axis direction vector of the eyeball. z Let be the z-axis direction vector of the eyeball; set the eyeball's visual field range as the horizontal visual field range α and the vertical visual field range β;

[0039] S2.5.2. Obtain the spatial location information of the target using GPS or an indoor positioning system, determine the target's position coordinates in three-dimensional space, and set them as follows: Where P x Let P be the x-axis coordinate of the target in three-dimensional space. y Let P be the y-axis coordinate of the target in three-dimensional space. z Let z be the z-axis coordinate of the target in three-dimensional space;

[0040] S2.5.3. Establish a reference coordinate system to determine the relative position of the target and the eyeball;

[0041] S2.5.4. Calculate the gaze vector. The gaze vector is obtained by subtracting the eye direction vector from the target's position coordinates. The formula is:

[0042]

[0043] S2.5.5. Calculate the angle between the line of sight and the coordinate axes of the reference coordinate system. Use the dot product and magnitude of the vectors to calculate the angle between the line of sight vector and the coordinate axes of the reference coordinate system, using the following formula:

[0044]

[0045] Wherein, Cos(θ) x The value of ) represents the cosine of the angle between the line of sight and the x-axis of the reference coordinate system. The x-axis is the unit vector in the reference coordinate system;

[0046] S2.5.6. Determine whether the target is within the field of vision. Based on the eye's field of view and the angle between the line of sight and the coordinate axis, determine whether the target is within the field of vision. If the angle between the line of sight and the x-axis is within the horizontal field of vision α, and the angle between the line of sight and the y-axis is within the vertical field of vision β, then the target is within the field of vision.

[0047] Furthermore, the specific implementation method of step S4 includes the following steps:

[0048] S4.1. After the inspection starts, the spatial position information of the MR head-mounted display device is obtained. The spatial position information is input into the inspection execution model to calculate and determine the inspection module that should appear at this time and the information to be detected. The information is then transmitted to the MR head-mounted display device to generate the display information to be inspected.

[0049] S4.2. Obtain the interactive information collected by the MR head-mounted display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship within the inspection execution model;

[0050] S4.3. Once the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head-mounted display device for display.

[0051] S4.4. After all inspection modules have been completed, the data collected by the MR headset device is fed back into the inspection execution model for training to obtain an optimized inspection execution model.

[0052] The beneficial effects of this invention are:

[0053] The present invention discloses a generative MR industrial inspection method based on a large model and natural human-computer interaction. The backend adopts a multimodal large model for specific industrial fields, while the frontend adopts natural human-computer interaction input including voice, gaze points, gestures, spatial position, and body movements. On the one hand, it collects training data for the multimodal large model, and on the other hand, it automatically generates the input interface for the application. Finally, an MR head-mounted display device is used as the terminal hardware carrier to collect the user's voice, gaze points, gestures, spatial position and other information, and transmits it to the backend multimodal large model through the network to generate corresponding inspection tasks. After receiving the data, the MR head-mounted display device displays the data in the corresponding time and space.

[0054] The present invention discloses a generative MR industrial inspection method based on a large model and natural human-computer interaction. It uses training data collected by MR to train a large model, which can quickly accumulate a large amount of actual training data and improve the training efficiency of the large model in subdivided fields. Based on the trained multimodal large model, inspection tasks are generated in real time according to the comprehensive feedback data of the corresponding MR headset. The data in the real-time inspection process can also optimize the multimodal data of the inspection personnel and improve the accuracy of the inspection tasks. Attached Figure Description

[0055] Figure 1 This is a flowchart of a generative MR industrial inspection method based on large models and natural human-computer interaction, as described in this invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0057] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0058] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 Detailed explanation is as follows:

[0059] Example 1:

[0060] A generative MR industrial inspection method based on large models and natural human-computer interaction includes the following steps:

[0061] S1. Use an MR head-mounted display device to collect scene image sequences to be inspected, construct a scene image dataset for inspection, and use the scene image dataset for inspection to train a multimodal large model to obtain category sequence information data to be inspected.

[0062] Furthermore,

[0063] The specific implementation method of step S1 includes the following steps:

[0064] S1.1. Use an MR head-mounted display device to acquire a sequence of scene images to be inspected;

[0065] S1.2. The elements in the scene image sequence to be inspected collected in step S1.1 are labeled. The elements include target objects, actions, and states. Then, image preprocessing is performed, including normalization, denoising, and enhancement, to obtain the scene image dataset to be inspected.

[0066] S1.3. Construct the loss function for training a multimodal large model, including the mean squared error loss function and the cross-entropy loss function;

[0067] The mean squared error loss function (MSE) is used for regression problems to calculate the mean squared difference between the predicted and true values. For numerical prediction tasks in image sequences, the formula is:

[0068]

[0069] Where n is the number of samples, y i It is the actual value. It is a predicted value;

[0070] The cross-entropy loss function H(p,q) is used for classification tasks in image sequences, and its formula is:

[0071]

[0072] Where p(x) is the true probability distribution and q(x) is the predicted probability distribution;

[0073] S1.4. Adjust the loss function for training the multimodal large model constructed in step S1.3:

[0074] Add a temporal coherence constraint to the loss function, calculate the difference between the predicted values ​​of adjacent frames, and add the difference between the predicted values ​​of adjacent frames as a loss term to the total loss function;

[0075] An optical flow loss constraint model is added to an image sequence with motion information to predict the motion of objects. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames.

[0076] S1.5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the scene image dataset from the inspection, and the optimization algorithm adopts the stochastic gradient descent method.

[0077] The training steps are as follows: input the scene image dataset of the inspection into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, update the model's parameters using the optimization algorithm to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times.

[0078] Furthermore, during the training process in step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers, and the optimal combination of hyperparameters is found through cross-validation.

[0079] S2. Collect corresponding spatial positioning information and interaction information based on the MR head-mounted display device. The interaction information includes eye tracking information, body movement information, voice information and gesture information. Construct an inspection module dataset based on the collected spatial positioning information and interaction information. Then, use the inspection module dataset to train the inspection module to obtain the proposed detection information of the inspection sequence of the inspection module.

[0080] Furthermore, the specific implementation method of step S2 includes the following steps:

[0081] S2.1. Collect corresponding spatial positioning information and interaction information based on MR head-mounted display devices, including using eye-tracking devices to collect eye information of the target in different scenarios, including eye movement trajectory, gaze point, and saccade speed; using GPS or indoor positioning systems to obtain the target's spatial location information; and recording the interaction behavior between the target and the inspection module during the inspection process, including the time and order in which the target views specific devices and inspects a certain area.

[0082] S2.2. Inspection Module Parameter Collection: Collect parameters for each inspection module, including the characteristics of the inspection object, the index range under normal conditions, and common fault types;

[0083] S2.3. Perform data preprocessing and feature extraction on the data collected in steps S2.1 and S2.2 to construct the inspection module dataset;

[0084] The data preprocessing method includes data cleaning and data filtering;

[0085] The feature extraction includes, for eye information, extracting the distribution of gaze points, gaze time, and saccade path features; for spatial positioning information, extracting the target's position coordinates, movement speed, and direction features; and for inspection module parameters, extracting the values ​​and trend features of key indicators.

[0086] S2.4. Divide the inspection module dataset into a training set and a test set, select a neural network, support vector machine, or decision tree as the machine learning algorithm, construct the machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module. Use batch gradient descent or stochastic gradient descent methods to accelerate the training process.

[0087] The machine learning model of the inspection module was evaluated using a test set, with evaluation metrics including accuracy, recall, and F1 score. Based on the evaluation results, the machine learning model of the inspection module was optimized, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method.

[0088] S2.5. Based on the optimized inspection module's machine learning model, the parameters of the inspection module are obtained, including the inspector's position, gaze point, and operation execution parameters; using spatial positioning information, the location scene of the target is determined, and the corresponding inspection module is found; using interaction information, the interaction information and the parameters of the corresponding inspection module are mapped, and the proposed detection information under the inspection module is trained.

[0089] Furthermore, the specific implementation method of step S2.5 includes the following steps:

[0090] S2.5.1. Use an eye-tracking device to collect eye information of the target in different scenarios, and acquire the eye direction vector. Where E x E is the x-axis vector of the eyeball. y E is the y-axis direction vector of the eyeball. z Let be the z-axis direction vector of the eyeball; set the eyeball's visual field range as the horizontal visual field range α and the vertical visual field range β;

[0091] S2.5.2. Obtain the spatial location information of the target using GPS or an indoor positioning system, determine the target's position coordinates in three-dimensional space, and set them as follows: Where P x Let P be the x-axis coordinate of the target in three-dimensional space. y Let P be the y-axis coordinate of the target in three-dimensional space. z Let z be the z-axis coordinate of the target in three-dimensional space;

[0092] S2.5.3. Establish a reference coordinate system to determine the relative position of the target and the eyeball;

[0093] S2.5.4. Calculate the gaze vector. The gaze vector is obtained by subtracting the eye direction vector from the target's position coordinates. The formula is:

[0094]

[0095] S2.5.5. Calculate the angle between the line of sight and the coordinate axes of the reference coordinate system. Use the dot product and magnitude of the vectors to calculate the angle between the line of sight vector and the coordinate axes of the reference coordinate system, using the following formula:

[0096]

[0097] Wherein, Cos(θ) x The value of ) represents the cosine of the angle between the line of sight and the x-axis of the reference coordinate system. The x-axis is the unit vector in the reference coordinate system;

[0098] S2.5.6. Determine whether the target is within the field of vision. Based on the eye's field of view and the angle between the line of sight and the coordinate axis, determine whether the target is within the field of vision. If the angle between the line of sight and the x-axis is within the horizontal field of vision α, and the angle between the line of sight and the y-axis is within the vertical field of vision β, then the target is within the field of vision.

[0099] S3. Fit the category sequence information to be inspected obtained in step S1 and the proposed detection information of the inspection sequence of each inspection module obtained in step S2 to obtain the inspection execution model.

[0100] Examples are given below:

[0101] Suppose an inspection engineer wearing an MR headset stands in front of an equipment cabinet, their gaze directed at an instrument on the cabinet, and they point to that instrument, saying (in voice): "Pressure gauge reading 0.5 MPa, normal reading, proceed to the next inspection point." They then add: "There is an anomaly. If the pressure gauge reading exceeds 0.7 MPa, immediately proceed to equipment A and check the flow rate gauge reading." The collected speech data is shown in the table below.

[0102] Table 1

[0103]

[0104] Furthermore, by creating a digital twin of the factory in 3D rendering within an MR headset, inspectors wearing MR headsets can tour the virtual digital factory, avoiding the need to collect data on-site. This allows for the rapid and low-cost acquisition of training data without disrupting normal industrial production.

[0105] Furthermore, compared to traditional MR inspection applications, this inspection training scheme can adaptively train an inspection execution model suitable for the scenario by training the parameters obtained from the first operation of the MR headset.

[0106] Furthermore, this training process involves: adapting the dimensions of input and output data; aligning various input data, including alignment in terms of time, space, and data scale; and setting the training objective function (loss function) and its association with inspection applications.

[0107] S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

[0108] Furthermore, the specific implementation method of step S4 includes the following steps:

[0109] S4.1. After the inspection starts, the spatial position information of the MR head-mounted display device is obtained. The spatial position information is input into the inspection execution model to calculate and determine the inspection module that should appear at this time and the information to be detected. The information is then transmitted to the MR head-mounted display device to generate the display information to be inspected.

[0110] S4.2. Obtain the interactive information collected by the MR head-mounted display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship within the inspection execution model;

[0111] S4.3. Once the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head-mounted display device for display.

[0112] S4.4. After all inspection modules have been completed, the data collected by the MR headset device is fed back into the inspection execution model for training to obtain an optimized inspection execution model.

[0113] Furthermore, during the inspection process, if the inspector's operational information is identified as inaccurate or outside the error range by the large model, the large model will generate comparison information between the two, send it to the MR headset for display, and generate a selection confirmation message, allowing the inspector to choose one of the operational information. If the selected operational information is information previously collected during the large model's training, it will generate operational information to re-execute the current inspection module, allowing the inspector to check the module again. If the selected operational information is the current operational information, it will provide feedback to the large model to retrain the inspection module and generate a prompt message indicating that the module needs to be inspected again, allowing the inspector to check the module again after a certain period.

[0114] Furthermore, utilizing the actual sensor data from the MR terminal on-site, and based on the previously generated large model and MR spatial positioning information, inspection task information adapted to the corresponding inspection module is generated, instructing inspection engineers to complete the inspection tasks. Moreover, when discrepancies are found between the operational data and the large model data, comparison information is generated, allowing inspectors to confirm whether the inspection module needs retraining.

[0115] For example, when an inspection engineer wears an MR headset to enter the industrial site, all the sensors in the MR headset are activated, transmitting prompts in real time to the backend large model server. The response from the large model server is displayed in the MR headset in the form of voice broadcast, text, or other visual annotations (arrows, annotation boxes, PDF documents, images, etc.), as shown in Table 2.

[0116] Table 2

[0117]

[0118]

[0119] The model's response example is as follows:

[0120] 1. Voice or text: "Turn your head and look at device B";

[0121] 2. Visual annotation: "Arrow points to device B";

[0122] 3. Voice or text: "Go to device B immediately";

[0123] 4. Visual annotation: "Highlight the indicator light L and switch S on the device in the annotation box";

[0124] 5. Voice or text message: "Check indicator L: If it is green, it is normal; if it is yellow, please turn on switch S; if it is red, please contact the factory control center immediately."

[0125] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A generative MR industrial inspection method based on large models and natural human-computer interaction, characterized in that, Includes the following steps: S1. Use an MR head-mounted display device to collect scene image sequences to be inspected, construct a scene image dataset for inspection, and use the scene image dataset for inspection to train a multimodal large model to obtain category sequence information data to be inspected. The specific implementation method of step S1 includes the following steps: S1.

1. Use an MR head-mounted display device to acquire a sequence of scene images to be inspected; S1.

2. The elements in the scene image sequence to be inspected collected in step S1.1 are labeled. The elements include target objects, actions, and states. Then, image preprocessing is performed, including normalization, denoising, and enhancement, to obtain the scene image dataset to be inspected. S1.

3. Construct the loss function for training a multimodal large model, including the mean squared error loss function and the cross-entropy loss function; The mean squared error loss function (MSE) is used for regression problems to calculate the mean squared difference between the predicted and true values. For numerical prediction tasks in image sequences, the formula is: Where n is the number of samples, y i It is the actual value. It is a predicted value; The cross-entropy loss function H(p,q) is used for classification tasks in image sequences, and its formula is: Where p(x) is the true probability distribution and q(x) is the predicted probability distribution; S1.

4. Adjust the loss function for training the multimodal large model constructed in step S1.3: Add a temporal coherence constraint to the loss function, calculate the difference between the predicted values ​​of adjacent frames, and add the difference between the predicted values ​​of adjacent frames as a loss term to the total loss function; An optical flow loss constraint model is added to an image sequence with motion information to predict the motion of objects. The optical flow loss measures the difference between the motion predicted by the model and the actual motion by calculating the displacement of pixels between adjacent frames. S1.

5. Based on the loss function adjusted in step S1.4, the multimodal large model is trained using the scene image dataset from the inspection, and the optimization algorithm adopts the stochastic gradient descent method. The training steps are as follows: input the scene image dataset of the inspection into the multimodal large model, calculate the loss between the model's predicted value and the true value according to the loss function, update the model's parameters using the optimization algorithm to minimize the loss function, and repeat the above steps until the model converges or reaches the predetermined number of training times. S2. Collect corresponding spatial positioning information and interaction information based on the MR head-mounted display device. The interaction information includes eye tracking information, body movement information, voice information and gesture information. Construct an inspection module dataset based on the collected spatial positioning information and interaction information. Then, use the inspection module dataset to train the inspection module to obtain the proposed detection information of the inspection sequence of the inspection module. S3. Fit the category sequence information to be inspected obtained in step S1 and the proposed detection information of the inspection sequence of each inspection module obtained in step S2 to obtain the inspection execution model. S4. Using the inspection execution model obtained in step S3, execute a generative MR industrial inspection process based on a large model and natural human-computer interaction.

2. The generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 1, characterized in that, During the training process in step S1, the hyperparameters of the multimodal large model are adjusted, including the learning rate, batch size, and number of network layers. The optimal combination of hyperparameters is found through cross-validation.

3. The generative MR industrial inspection method based on a large model and natural human-computer interaction according to claim 2, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. Collect corresponding spatial positioning information and interaction information based on MR head-mounted display devices, including using eye-tracking devices to collect eye information of the target in different scenarios, including eye movement trajectory, gaze point, and saccade speed; using GPS or indoor positioning systems to obtain the target's spatial location information; and recording the interaction behavior between the target and the inspection module during the inspection process, including the time and order in which the target views specific devices and inspects a certain area. S2.

2. Inspection Module Parameter Collection: Collect parameters for each inspection module, including the characteristics of the inspection object, the index range under normal conditions, and common fault types; S2.

3. Perform data preprocessing and feature extraction on the data collected in steps S2.1 and S2.2 to construct the inspection module dataset; The data preprocessing method includes data cleaning and data filtering; The feature extraction process includes extracting the distribution of gaze points, gaze duration, and saccade path features for eye information; and extracting the target's position coordinates, movement speed, and direction features for spatial positioning information. For the parameters of the inspection module, extract the values ​​and trend characteristics of key indicators; S2.

4. Divide the inspection module dataset into a training set and a test set, select a neural network, support vector machine, or decision tree as the machine learning algorithm, construct the machine learning model of the inspection module, and then use the training set to train the machine learning model of the inspection module. Use batch gradient descent or stochastic gradient descent methods to accelerate the training process. The machine learning model of the inspection module was evaluated using a test set, with evaluation metrics including accuracy, recall, and F1 score. Based on the evaluation results, the machine learning model of the inspection module was optimized, including increasing the amount of data, adjusting the model structure, and improving the feature extraction method. S2.

5. Based on the optimized inspection module's machine learning model, the parameters of the inspection module are obtained, including the inspector's position, gaze point, and operation execution parameters; using spatial positioning information, the location scene of the target is determined, and the corresponding inspection module is found to be suitable. By utilizing interactive information, the interactive information and the parameters of the corresponding inspection module are mapped to train and obtain the proposed detection information under that inspection module.

4. The generative MR industrial inspection method based on large models and natural human-computer interaction according to claim 3, characterized in that, The specific implementation method of step S2.5 includes the following steps: S2.5.

1. Use an eye-tracking device to collect eye information of the target in different scenarios and acquire the eye direction vector. Where E x E is the x-axis vector of the eyeball. y E is the y-axis direction vector of the eyeball. z Let be the z-axis direction vector of the eyeball; set the eyeball's visual field range as the horizontal visual field range α and the vertical visual field range β; S2.5.

2. Obtain the spatial location information of the target using GPS or an indoor positioning system, determine the target's position coordinates in three-dimensional space, and set them as follows: Where P x Let P be the x-axis coordinate of the target in three-dimensional space. y Let P be the y-axis coordinate of the target in three-dimensional space. z Let z be the z-axis coordinate of the target in three-dimensional space; S2.5.

3. Establish a reference coordinate system to determine the relative position of the target and the eyeball; S2.5.

4. Calculate the gaze vector. The gaze vector is obtained by subtracting the eye direction vector from the target's position coordinates. The formula is: S2.5.

5. Calculate the angle between the line of sight and the coordinate axes of the reference coordinate system. Use the dot product and magnitude of the vectors to calculate the angle between the line of sight vector and the coordinate axes of the reference coordinate system, using the following formula: Wherein, Cos(θ) x The value of ) represents the cosine of the angle between the line of sight and the x-axis of the reference coordinate system. The x-axis is the unit vector in the reference coordinate system; S2.5.

6. Determine whether the target is within the field of vision. Based on the eye's field of view and the angle between the line of sight and the coordinate axis, determine whether the target is within the field of vision. If the angle between the line of sight and the x-axis is within the horizontal field of vision α, and the angle between the line of sight and the y-axis is within the vertical field of vision β, then the target is within the field of vision.

5. The generative MR industrial inspection method based on a large model and natural human-computer interaction according to claim 4, characterized in that, The specific implementation method of step S4 includes the following steps: S4.

1. After the inspection starts, the spatial position information of the MR head-mounted display device is obtained. The spatial position information is input into the inspection execution model to calculate and determine the inspection module that should appear at this time and the information to be detected. The information is then transmitted to the MR head-mounted display device to generate the display information to be inspected. S4.

2. Obtain the interactive information collected by the MR head-mounted display device, input it into the inspection execution model, and determine the inspection result based on the mapping relationship within the inspection execution model; S4.

3. Once the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR head-mounted display device for display. S4.

4. After all inspection modules have been completed, the data collected by the MR headset device is fed back into the inspection execution model for training to obtain an optimized inspection execution model.

Images