Virtual reality intelligent education method and system based on industrial multi-mode large model driving

Abstract

The invention relates to a virtual reality intelligent education method and system based on industrial multi-mode large model driving, and the method comprises the steps: employing a scanning function to scan a target object in a virtual reality scene; obtaining multi-modal information of a target object, and encoding and formatting the multi-modal information; the multi-modal information is input into a multi-modal large model for analysis processing, and a processing result is generated and fed back to a user; questioning information input by a user is obtained, the questioning information is input into the multi-mode large model to be extracted and processed, answer information is generated in real time, and meanwhile the multi-mode large model conducts intelligent error correction according to industrial knowledge involved in the questioning information; and judging whether the user continues to ask questions or not, returning to the previous step when the user continues to ask questions, and otherwise, closing the system dialogue function. According to the invention, personalized learning experience can be provided, a safe, convenient and efficient learning mode is provided for the user, and errors can be corrected and feedback can be provided in real time.

Description

Technical Field

[0001] The present invention relates to the field of intelligent education technology, and in particular to a virtual reality intelligent education method and system driven by an industrial multimodal large model. Background Art

[0002] Most of the existing education systems rely on preset static teaching content and lack intelligent and personalized learning experiences. Virtual reality technology can provide virtual teaching scenes such as simulated laboratories and historical scene reproduction to enhance students' learning interest and comprehension ability.

[0003] However, traditional virtual reality education systems mainly rely on preset teaching content and cannot be dynamically adjusted according to learners' real-time performance. Factors such as learners' operation level and knowledge mastery are difficult to affect the system's feedback and content generation, limiting the possibility of personalized learning. In addition, traditional virtual reality education systems usually provide post-event feedback and cannot provide real-time guidance to learners during operation. When learners encounter problems, they often need to complete the entire step before receiving feedback, which reduces learning efficiency.

[0004] Therefore, it is necessary to provide new virtual reality intelligent education methods and systems driven by industrial multimodal large models. Summary of the invention

[0005] Based on the above-mentioned problems existing in the prior art, the purpose of the embodiments of the present invention is to provide a virtual reality intelligent education method and system driven by an industrial multimodal large model, which can provide a personalized learning experience, provide users with a safe, convenient and efficient learning method, and can correct errors and provide feedback in real time.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a virtual reality intelligent education method driven by an industrial multimodal large model, comprising:

[0007] S1, in the virtual reality scene, the target object is scanned using the scanning function;

[0008] S2, obtaining multimodal information of the target object, and encoding and formatting the multimodal information;

[0009] S3, input the multimodal information into the multimodal large model for analysis and processing, generate processing results and feed them back to the user;

[0010] S4, obtaining the question information input by the user, inputting the question information into the multimodal large model for extraction and processing and generating answer information in real time, and the multimodal large model performs intelligent error correction based on the industrial knowledge involved in the question information;

[0011] S5, judging whether the user continues to ask questions. If the user continues to ask questions, the process returns to step S4, otherwise the system dialogue function is closed.

[0012] Furthermore, in S1, in the virtual reality scene, scanning the target object using a scanning function includes:

[0013] Step S11, the user identifies the target object and calls the scanning function of the system;

[0014] Step S12, searching and locking the target object in the virtual reality scene;

[0015] Step S13, scanning the target object by using the scanning tool built into the system according to the scanning technology adopted.

[0016] Furthermore, the scanning technology includes laser scanning, structured light scanning and photogrammetry. The laser scanning acquires the three-dimensional coordinate information of the surface of the target object by emitting a laser beam and measuring the reflection of the laser on the surface of the object; the structured light scanning projects a specific pattern of light onto the surface of the target object and calculates the three-dimensional shape of the object according to the deformation of the pattern; the photogrammetry takes a large number of photos of the object from different angles and then uses a computer vision algorithm to reconstruct the three-dimensional shape of the object.

[0017] Furthermore, in S2, the step of acquiring multimodal information of the target object and encoding and formatting the multimodal information includes:

[0018] Step S21, obtaining multimodal information of the target object, including three-dimensional model information and image information of the target object;

[0019] Step S22, encoding and formatting the multimodal information, including first classifying and integrating the multimodal information, and then encoding and formatting the integrated multimodal information;

[0020] Among them, obtaining the three-dimensional model information of the target object includes the system accurately obtaining the coordinate information of each point on the surface of the object through scanning technology, and then constructing a three-dimensional model of the object, thereby completely presenting the geometric shape of the object, including whether its overall outline is a regular geometric shape or a complex irregular shape; obtaining the three-dimensional model information of the target object also includes the shape of each component of the target object and the connection relationship between each component; obtaining the image information of the target object includes the system taking image information of the object from different angles through the scanning function.

[0021] Furthermore, in S3, the step of inputting the multimodal information into the multimodal large model for analysis and processing, generating processing results and feeding back to the user includes:

[0022] Step S31, the multimodal large model identifies and extracts features of the object based on the multimodal information;

[0023] Step S32, the multimodal large model analyzes and obtains relevant industrial knowledge of the target object based on the results of recognition and feature extraction;

[0024] In step S33, the relevant industrial knowledge of the target object obtained through analysis is used as a processing result, and the processing result is fed back to the user in the virtual reality scene.

[0025] Furthermore, in S4, the question information input by the user is obtained, and the question information is input into the multimodal large model for extraction and processing and real-time generation of answer information. At the same time, the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information, including:

[0026] Step S41, the user inputs question information through the microphone device equipped by the system, and the system uses speech recognition technology to convert the question information in audio mode into text form and input it into the multimodal large model;

[0027] Step S42, the multimodal large model combines natural language processing technology to analyze the text-based question information and extract key question information;

[0028] Step S43, the multimodal large model makes a comprehensive judgment based on the key question information, combined with the related multimodal information, extracted language features, fused knowledge graph and industrial knowledge, and generates answer information in real time;

[0029] In step S44, the multimodal large model identifies the industrial knowledge elements involved in the question information, establishes an intelligent error correction mechanism based on industrial knowledge, and the multimodal large model corrects the errors in the industrial knowledge involved in the user's question information.

[0030] Furthermore, the establishment of an intelligent error correction mechanism based on industrial knowledge includes error correction related to product knowledge, error correction related to industrial process and technology knowledge, error correction related to industrial equipment operation and maintenance knowledge, and error correction related to industrial standards and specifications knowledge.

[0031] A virtual reality intelligent education system driven by an industrial multimodal large model is applied to the above-mentioned virtual reality intelligent education method driven by an industrial multimodal large model, and the system includes:

[0032] A target scanning module is used to scan a target object using a scanning function in a virtual reality scene;

[0033] An information acquisition module is used to acquire multimodal information of a target object and encode and format the multimodal information;

[0034] An analysis and processing module is used to input multimodal information into a large multimodal model for analysis and processing;

[0035] A result feedback module is used to feed back the generated processing results to the user;

[0036] The question-answering error correction module is used to obtain the question information input by the user, input the question information into the multimodal large model for extraction and processing, and generate answer information in real time. At the same time, the multimodal large model performs intelligent error correction based on the industrial knowledge involved in the question information;

[0037] The judgment return module is used to judge whether the user continues to ask questions. If the user continues to ask questions, it returns, otherwise it closes the system dialogue function.

[0038] The embodiment of the present invention further provides a network side server, including:

[0039] At least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned virtual reality intelligent education method driven by industrial multimodal large models.

[0040] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned virtual reality intelligent education method driven by an industrial multimodal large model.

[0041] The beneficial effects of the present invention are as follows: in a virtual reality scene, a scanning function is used to scan a target object; multimodal information of the target object is obtained, and the multimodal information is encoded and formatted; the multimodal information is input into a multimodal large model for analysis and processing, and a processing result is generated and fed back to a user; question information input by a user is obtained, and the question information is input into the multimodal large model for extraction and processing and answer information is generated in real time, and at the same time, the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information; it is determined whether the user continues to ask questions, and when the user continues to ask questions, the previous step is returned, otherwise the system dialogue function is closed. The virtual reality intelligent education method driven by the industrial multimodal large model of the present invention can provide a personalized learning experience, has strong interactive capabilities and flexibility, provides users with a safe, convenient and efficient learning method, and can correct errors and provide feedback in real time; by scanning objects to obtain multimodal information, it can effectively understand the needs of users, and answer questions in real time through the intelligent dialogue function, which can not only improve learning efficiency, but also enhance the user's sense of participation and immersion; the accuracy and richness of information are ensured through the multimodal large model, so that users can deeply understand the structure, function and operation method of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] In the figure:

[0044] Figure 1 A flowchart of a virtual reality intelligent education method driven by an industrial multimodal large model provided in the first embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a module of a virtual reality intelligent education system driven by an industrial multimodal large model provided in the second embodiment of the present invention;

[0046] Figure 3 It is a structural diagram of a network-side server provided according to a third embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] First embodiment:

[0049] The first embodiment of the present invention provides a virtual reality intelligent education method driven by an industrial multimodal large model, including: in a virtual reality scene, scanning a target object using a scanning function; obtaining multimodal information of the target object, and encoding and formatting the multimodal information; inputting the multimodal information into the multimodal large model for analysis and processing, generating processing results and feeding back to the user; obtaining question information input by the user, inputting the question information into the multimodal large model for extraction and processing and generating answer information in real time, and the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information; judging whether to continue asking questions, when the user continues to ask questions, returning to the previous step, otherwise closing the system dialogue function. The virtual reality intelligent education method driven by an industrial multimodal large model of the present invention can provide a personalized learning experience, has strong interactive capabilities and flexibility, provides users with a safe, convenient and efficient learning method, and can correct errors and provide feedback in real time.

[0050] The following is a detailed description of the implementation details of the virtual reality intelligent education method based on industrial multimodal large model driving in this embodiment. The following content is only for the convenience of understanding the implementation details, which is not necessary for the implementation of this solution. The specific process of this embodiment is as follows: Figure 1 As shown, this embodiment is applied to a virtual reality intelligent education system driven by an industrial multimodal large model.

[0051] Step S1: In a virtual reality scene, a target object is scanned using a scanning function.

[0052] Specifically, the specific steps of using the scanning function to scan the target object in the virtual reality scene are as follows:

[0053] In step S11, the user identifies the target object and calls the scanning function of the system.

[0054] Optionally, the system of this embodiment is equipped with a virtual reality system with a scanning function. Before scanning, the user needs to clearly identify the target object to be analyzed. For example, in education and training, students need to have an in-depth understanding of the structure of a certain teaching model. The user can first take a general look in the virtual reality scene to clearly identify the location or approximate appearance of the target object; then call up the scanning function on the system's operating interface. The system usually sets the scanning function in a more conspicuous position so that the user can easily find it. On the operating interface, the scanning function may appear in the form of an icon, button, or menu option.

[0055] As an example, the scanning function icon in the system is a pattern similar to a camera lens, placed in the toolbar at the bottom of the screen; or it is an option in a virtual menu called up by handle operation, and is set with a text label of the scanned object.

[0056] Step S12: searching and locking the target object in the virtual reality scene.

[0057] Optionally, the user searches for the target object in the virtual reality scene through the interactive method provided by the operating system; after finding the target object, the user needs to lock the target object through specific operations in order to perform accurate scanning. The interactive method provided by the system includes dragging with a mouse or moving the viewing angle through a motion capture handle.

[0058] As an example, in a virtual reality scene, the user needs to move the motion capture handle to rotate the view and observe closely, so as to find the target object among many teaching models and parts. When the target object is found, the cursor or the user's sight is kept on the target object, and the mouse button or the trigger button of the motion capture handle is pressed to confirm, so that the virtual aiming frame accurately covers the target object. When the color of the aiming frame changes to the color of successful locking, it means that the target object has been locked.

[0059] Step S13, scanning the target object by using the scanning tool built into the system according to the scanning technology adopted.

[0060] Optionally, scanning techniques include laser scanning, structured light scanning, and photogrammetry.

[0061] As an example, laser scanning acquires the three-dimensional coordinate information of the surface of a target object by emitting a laser beam and measuring the reflection of the laser on the surface of the object.

[0062] As an example, structured light scanning projects a specific pattern of light onto the surface of a target object and calculates the 3D shape of the object based on the deformation of the pattern.

[0063] As an example, photogrammetry involves taking a large number of photos of an object from different angles and then using computer vision algorithms to reconstruct the object's 3D shape.

[0064] Optionally, during the scanning process, the user needs to keep the scanning tool aimed at the target object and perform corresponding operations according to the system prompts.

[0065] For example, for some larger or complex-shaped objects, it may be necessary to slowly move the scanning tool around the object to ensure that all surfaces of the object can be completely scanned; or, as required by the system, press a button within a specific time interval to obtain scanning data at different angles.

[0066] Step S2, obtaining multimodal information of the target object, and encoding and formatting the multimodal information.

[0067] Specifically, the specific steps of obtaining the multimodal information of the target object and encoding and formatting the multimodal information are as follows:

[0068] Step S21, obtaining multimodal information of the target object, including three-dimensional model information and picture information of the target object.

[0069] Optionally, obtaining the three-dimensional model information of the target object includes the system accurately obtaining the coordinate information of each point on the surface of the object through scanning technology, and then constructing a three-dimensional model of the object, thereby fully presenting the geometric shape of the object, including whether its overall outline is a regular geometric shape or a complex irregular shape.

[0070] Optionally, acquiring the three-dimensional model information of the target object also includes the shapes of various components of the target object and the connection relationship between the various components.

[0071] As an example, for a mechanical target object, the specific shapes of various gears, shafts, housings and other components can be obtained, as well as how they fit together and are connected together.

[0072] Optionally, the 3D model information of the target object may also include the material characteristics of the target object. When a metal object is scanned by laser scanning, the glossiness, roughness and other material characteristics of the metal can be initially reflected on the model based on factors such as the reflection of the laser on the surface of the object; for objects of different materials, the model will present their material characteristics in different ways. For example, scanning a wooden object may show the texture of the wood, while scanning a plastic object may show the smoothness or transparency of the plastic and other related characteristics.

[0073] Optionally, obtaining the image information of the target object includes the system taking image information of the object from different angles through a scanning function. The image can fully display the appearance of the object, presenting the color, surface details, light and shadow effects and other features of the object from the front, side, back and various tilt angles. By obtaining the image information of the target object, richer detailed information can be provided than the three-dimensional model information.

[0074] Furthermore, the multimodal information of the target object is obtained, which may also include audio information and text information of the target object.

[0075] Optionally, obtaining audio information of the target object includes setting up a function of synchronously collecting audio when scanning the object. Then, for some objects with sound-emitting parts or that make sounds during operation, the operation sound can be used as a supplementary understanding of the characteristics of the object. In the subsequent analysis or display, it can allow users to feel the operating status of the object more realistically from an auditory level. Obtaining text information of the target object includes that the system may obtain the name, model, brand and other identification information of the object through association with an existing database or manual labeling. The identification information is very important for clarifying the identity and classification of the object. In the subsequent analysis and processing, it is convenient for users to quickly understand the basic situation of the object. In addition, through database association or manual labeling, the system can also obtain text information such as the function description and operation guide of the object.

[0076] Step S22, encoding and formatting the multimodal information, including first classifying and integrating the multimodal information, and then encoding and formatting the integrated multimodal information.

[0077] Optionally, classifying the multimodal information includes classifying it according to the type and characteristics of the information. For example, the three-dimensional model information of the object is classified into one category, and the image information taken at different angles is classified separately according to factors such as shooting angle and resolution.

[0078] Optionally, the system integrates data of different categories to form a set of organically combined multimodal information. During the integration process, it is necessary not only to ensure that the model and the picture can correspond to each other in space, such as making a certain face of the model match the corresponding face shown in a specific picture in spatial position through key point matching, but also to establish semantic associations between them, such as associating a certain component on the model with the area in the picture that shows the appearance of the component and the related text description.

[0079] Optionally, the integrated multimodal information is encoded and formatted, including encoding the 3D model information in common formats such as FBX, OBJ, etc., and formatting and adjusting its coordinate system, size, rotation angle, etc.; encoding the image information in formats such as JPEG, PNG, etc., and formatting the resolution, color balance, etc.; if there is text information, using UTF-8 encoding format, and setting its font, size, color and other appearance attributes as well as presentation method.

[0080] Step S3, input the multimodal information into the multimodal large model for analysis and processing, generate processing results and feed them back to the user.

[0081] Specifically, the specific steps of inputting the multimodal information into the multimodal large model for analysis and processing, generating processing results and feeding back to the user are as follows:

[0082] Step S31: the multimodal large model identifies and extracts features of the object based on the multimodal information.

[0083] Optionally, the multimodal large model uses computer vision technology to identify the three-dimensional model information and image information in the input multimodal information. By analyzing the shape, color, texture and other visual features of the object, combined with the existing object model database for comparison, the type, name and other information of the object can be determined.

[0084] As an example, given a 3D model and related pictures of an industrial part, the model can identify whether it is a certain type of bolt, a specific type of gear, etc.

[0085] Furthermore, if the input multimodal information includes text information, the object name, description, etc. mentioned in the input text information are analyzed and compared with a known industrial product knowledge base or name database.

[0086] Optionally, the modal model extracts features of the object based on multimodal information, including in-depth extraction of various visual features from the image information and 3D model information of the object. In addition to basic shape, color, and texture features, it also includes key part features of the object, such as key connection parts of mechanical parts, operation panel parts of equipment, etc., posture features of the object, such as the tilt angle of the object in space, rotation state, etc., and the motion features of the object.

[0087] Step S32: The multimodal large model analyzes and obtains relevant industrial knowledge of the target object based on the results of recognition and feature extraction.

[0088] Optionally, the relevant industrial knowledge of the target object includes the overall function and usage of the target object. The multimodal large model infers the role of each component in the realization of the overall function of the target object by analyzing the relationship between the component features of the target object and comparing them with the component function library of known industrial objects, and then infers the overall function of the target object by combining the functional inferences of each component. Based on the component layout displayed in the three-dimensional model information, the multimodal large model extracts the size, interface type, material and other features of the components in the three-dimensional model of the target object, compares them with the corresponding component features of other known industrial objects, determines the component compatibility between them, and then infers the usage of the target object.

[0089] In step S33, the relevant industrial knowledge of the target object obtained through analysis is used as a processing result, and the processing result is fed back to the user in the virtual reality scene.

[0090] Optionally, in the virtual reality scene, a special information display area or mechanism is set up to display the processing results to the user in real time and intuitively. The display area can be a fixed floating window located at a suitable position in the user's field of view, or a prompt box that appears dynamically according to the interaction between the user and the target object.

[0091] As an example, when a user focuses on a target object in a virtual reality scene by gazing at it or pointing it with a motion capture handle, a display window or prompt box of the processing result will automatically pop up, displaying relevant text information.

[0092] Furthermore, interactive functions are added to the displayed processing results in the virtual reality scene. For example, the user can expand or collapse different parts of the processing results, such as the purpose and functional characteristics of the object, by operating the motion capture handle, so as to view a certain content in more detail or quickly browse the overall information; or when the user clicks on a specific component name mentioned in the text, the virtual reality scene will highlight the component on the target object accordingly, so that the user can more intuitively see the actual position and appearance of the described component on the object.

[0093] Step S4, obtain the question information input by the user, input the question information into the multimodal large model for extraction and processing, and generate answer information in real time. At the same time, the multimodal large model performs intelligent error correction based on the industrial knowledge involved in the question information.

[0094] Specifically, the specific steps of obtaining the user's question information, inputting the question information into the multimodal large model for extraction and processing, and generating answer information in real time are as follows:

[0095] In step S41, the user inputs question information through the microphone device equipped by the system. The system uses speech recognition technology to convert the question information in audio mode into text form and inputs it into the multimodal large model.

[0096] In step S42, the multimodal large model combines natural language processing technology to analyze the text-based question information and extract key question information.

[0097] As an example, it is clear whether the user is asking about the function of a certain component of the object, the overall manufacturing process, or other aspects.

[0098] In step S43, the multimodal large model makes a comprehensive judgment based on the key question information, combined with related multimodal information, extracted language features, integrated knowledge graphs and industrial knowledge, and generates answer information in real time.

[0099] Since a knowledge graph is usually built inside a large multimodal model, which stores a wealth of industrial domain knowledge and common information about various objects, when processing user questions, the extracted language features and semantic information will be integrated with the relevant knowledge in the knowledge graph.

[0100] As an example, when a user asks about the main functions of a target object, the multimodal large model analyzes the components of the equipment and its possible functional implementation methods from the three-dimensional model information, refers to details such as the equipment appearance and operating interface shown in the picture, and combines the functional description of this type of equipment in the knowledge graph and the general understanding of the functions of this type of equipment in the industry to generate a detailed and accurate answer.

[0101] Specifically, the specific steps of the multimodal large model performing intelligent error correction based on the industrial knowledge involved in the question information include:

[0102] In step S44, the multimodal large model identifies the industrial knowledge elements involved in the question information, establishes an intelligent error correction mechanism based on industrial knowledge, and the multimodal large model corrects the errors in the industrial knowledge involved in the user's question information.

[0103] Specifically, an intelligent error correction mechanism based on industrial knowledge should be established, including error correction related to product knowledge, error correction related to industrial process and technology knowledge, error correction related to industrial equipment operation and maintenance knowledge, and error correction related to industrial standards and specifications knowledge.

[0104] As an example, error correction related to product knowledge includes when the user's description of the characteristics of an industrial product in a question is inconsistent with the knowledge mastered by the model, and when the user mistakenly classifies an industrial product into another product category, the model will point it out and correct it, and the multimodal large model will make corrections.

[0105] As an example, error correction related to industrial processes and technological knowledge includes when users make step errors when describing industrial production processes and when the industrial process parameters mentioned by users are unreasonable, the model will give correction suggestions.

[0106] As an example, the multimodal big model will provide correction suggestions for industrial equipment operation and maintenance knowledge related errors, including when users make mistakes in describing the operation steps of industrial equipment and when users make unreasonable references to the maintenance cycle of industrial equipment.

[0107] As an example, error correction related to knowledge of industrial standards and specifications includes when a user makes misunderstandings when referring to industrial standards or specifications and when a user makes errors when applying industrial standards or specifications, the multimodal large model will provide correction suggestions.

[0108] Furthermore, by establishing an intelligent error correction mechanism based on industrial knowledge, the multimodal large model can effectively correct errors in industrial knowledge involved in user question information, accurately understand the user's intentions, and give accurate answers that are more in line with actual industrial conditions, helping users to better understand relevant knowledge in the industrial field.

[0109] Step S5, determining whether the user continues to ask questions. If the user continues to ask questions, the process returns to step S4, otherwise the system dialogue function is closed.

[0110] The first embodiment of the present invention provides a virtual reality intelligent education method based on industrial multimodal large model driving, including in a virtual reality scene, using a scanning function to scan a target object; obtaining multimodal information of the target object, and encoding and formatting the multimodal information; inputting the multimodal information into the multimodal large model for analysis and processing, generating processing results and feeding back to the user; obtaining the question information input by the user, inputting the question information into the multimodal large model for extraction and processing and generating answer information in real time, and the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information; judging whether to continue the user to ask questions, when the user continues to ask questions, returning to the previous step, otherwise closing the system dialogue function. The virtual reality intelligent education method based on industrial multimodal large model driving of the present invention has strong interactive ability and flexibility, and can provide users with personalized learning experience; by scanning objects to obtain multimodal information, it can effectively understand the needs of users, and answer questions in real time through the intelligent dialogue function, which can not only improve learning efficiency, but also enhance the user's sense of participation and immersion; the accuracy and richness of information are ensured through the multimodal large model, so that users can deeply understand the structure, function and operation method of the object.

[0111] The virtual reality intelligent education method based on industrial multimodal large model driving of the present invention can significantly reduce the cost in training. Compared with traditional physical training, the virtual reality system reduces the dependence on actual equipment and venues, which not only saves expensive equipment maintenance costs, but also reduces the cost of venue rental and tool purchase; through multimodal large models combined with virtual reality technology to simulate industrial operation scenes, learners can be trained anytime and anywhere, without being restricted by time and space. At the same time, through dynamic content generation, the system can automatically adjust the difficulty of training to ensure that learners learn at the most suitable rhythm, greatly improving the efficiency of training; training in a virtual environment avoids the safety risks that may be caused by direct operation of real equipment; through virtual reality scenes to simulate complex industrial equipment operations, users can master the operating steps and safety specifications, reducing equipment damage and accidents caused by misoperation.

[0112] Second implementation method:

[0113] like Figure 2 As shown, the second embodiment of the present invention provides a virtual reality intelligent education system driven by an industrial multimodal large model, and the system includes: a target scanning module 201, an information acquisition module 202, an analysis and processing module 203, a result feedback module 204, a question and answer error correction module 205, and a judgment return module 206.

[0114] Specifically, the target scanning module 201 is used to scan the target object using the scanning function in the virtual reality scene; the information acquisition module 202 is used to obtain the multimodal information of the target object and encode and format the multimodal information; the analysis and processing module 203 is used to input the multimodal information into the multimodal large model for analysis and processing; the result feedback module 204 is used to feed back the generated processing results to the user; the question and answer error correction module 205 is used to obtain the question information input by the user, input the question information into the multimodal large model for extraction and processing and generate answer information in real time, and the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information; the judgment return module 206 is used to judge whether the user continues to ask questions. When the user continues to ask questions, it returns, otherwise the system dialogue function is closed.

[0115] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0116] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0117] A third embodiment of the present invention relates to a network side server, such as Figure 3 As shown, it includes at least one processor 302; and a memory 301 that is communicatively connected to the at least one processor 302; wherein the memory 301 stores instructions that can be executed by the at least one processor 302, and the instructions are executed by the at least one processor 302 so that the at least one processor 302 can execute the above-mentioned data processing method.

[0118] The memory 301 and the processor 302 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 302 and the memory 301 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 302 is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 302.

[0119] The processor 302 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 301 can be used to store data used by the processor 302 when performing operations.

[0120] The fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the virtual reality intelligent education method based on industrial multimodal large model driving in the first embodiment.

[0121] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0122] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can know all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual reality intelligent education method based on industrial multimodal large model drive, characterized in that: include: S1, in the virtual reality scene, the target object is scanned using the scanning function; S2, obtaining multimodal information of the target object, and encoding and formatting the multimodal information; S3, input the multimodal information into the multimodal large model for analysis and processing, generate processing results and feed them back to the user; S4, obtaining the question information input by the user, inputting the question information into the multimodal large model for extraction and processing and generating answer information in real time, and the multimodal large model performs intelligent error correction based on the industrial knowledge involved in the question information; S5, judging whether the user continues to ask questions. If the user continues to ask questions, the process returns to step S4, otherwise the system dialogue function is closed.

2. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 1 is characterized in that: In S1, in the virtual reality scene, scanning the target object using a scanning function includes: Step S11, the user identifies the target object and calls the scanning function of the system; Step S12, searching and locking the target object in the virtual reality scene; Step S13, scanning the target object by using the scanning tool built into the system according to the scanning technology adopted.

3. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 2 is characterized in that: The scanning technology includes laser scanning, structured light scanning and photogrammetry. The laser scanning acquires the three-dimensional coordinate information of the surface of the target object by emitting a laser beam and measuring the reflection of the laser on the surface of the object. The structured light scanning calculates the three-dimensional shape of the object according to the deformation of the pattern by projecting a specific pattern of light onto the surface of the target object. Photogrammetry involves taking a large number of photos of an object from different angles and then using computer vision algorithms to reconstruct the three-dimensional shape of the object.

4. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 3 is characterized in that: In S2, the step of acquiring multimodal information of the target object and encoding and formatting the multimodal information includes: Step S21, obtaining multimodal information of the target object, including three-dimensional model information and image information of the target object; Step S22, encoding and formatting the multimodal information, including first classifying and integrating the multimodal information, and then encoding and formatting the integrated multimodal information; Among them, obtaining the three-dimensional model information of the target object includes the system accurately obtaining the coordinate information of each point on the surface of the object through scanning technology, and then constructing a three-dimensional model of the object, thereby completely presenting the geometric shape of the object, including whether its overall outline is a regular geometric shape or a complex irregular shape; obtaining the three-dimensional model information of the target object also includes the shape of each component of the target object and the connection relationship between each component; obtaining the image information of the target object includes the system taking image information of the object from different angles through the scanning function.

5. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 1 is characterized in that: In S3, the multimodal information is input into the multimodal large model for analysis and processing, and the processing results are generated and fed back to the user, including: Step S31, the multimodal large model identifies and extracts features of the object based on the multimodal information; Step S32, the multimodal large model analyzes and obtains relevant industrial knowledge of the target object based on the results of recognition and feature extraction; In step S33, the relevant industrial knowledge of the target object obtained through analysis is used as a processing result, and the processing result is fed back to the user in the virtual reality scene.

6. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 1 is characterized in that: In S4, the question information input by the user is obtained, and the question information is input into the multimodal large model for extraction and processing and real-time generation of answer information. At the same time, the multimodal large model performs intelligent error correction according to the industrial knowledge involved in the question information, including: Step S41, the user inputs question information through the microphone device equipped by the system, and the system uses speech recognition technology to convert the question information in audio mode into text form and input it into the multimodal large model; Step S42, the multimodal large model combines natural language processing technology to analyze the text-based question information and extract key question information; Step S43, the multimodal large model makes a comprehensive judgment based on the key question information, combined with the related multimodal information, extracted language features, fused knowledge graph and industrial knowledge, and generates answer information in real time; In step S44, the multimodal large model identifies the industrial knowledge elements involved in the question information, establishes an intelligent error correction mechanism based on industrial knowledge, and the multimodal large model corrects the errors in the industrial knowledge involved in the user's question information.

7. The virtual reality intelligent education method based on industrial multimodal large model drive according to claim 6 is characterized in that: The establishment of an intelligent error correction mechanism based on industrial knowledge includes error correction related to product knowledge, error correction related to industrial process and technology knowledge, error correction related to industrial equipment operation and maintenance knowledge, and error correction related to industrial standards and specifications knowledge.

8. A virtual reality intelligent education system driven by an industrial multimodal large model, characterized by: Applied to the virtual reality intelligent education method based on industrial multimodal large model drive described in claim 1, the system comprises: A target scanning module is used to scan a target object using a scanning function in a virtual reality scene; An information acquisition module is used to acquire multimodal information of a target object and encode and format the multimodal information; An analysis and processing module is used to input multimodal information into a large multimodal model for analysis and processing; A result feedback module is used to feed back the generated processing results to the user; The question-answering error correction module is used to obtain the question information input by the user, input the question information into the multimodal large model for extraction and processing, and generate answer information in real time. At the same time, the multimodal large model performs intelligent error correction based on the industrial knowledge involved in the question information; The judgment return module is used to judge whether the user continues to ask questions. If the user continues to ask questions, it returns, otherwise it closes the system dialogue function.

9. A network side server, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual reality intelligent education method based on industrial multimodal large model driving as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the virtual reality intelligent education method based on industrial multimodal large model driving described in any one of claims 1 to 7 is implemented.

Images