Multi-person collaborative virtual interaction method based on MR recording and broadcasting technology

Abstract

The application relates to an artificial intelligence technology and discloses a multi-person cooperative virtual interaction method based on MR recording and broadcasting technology, which comprises the following steps: constructing a teaching auxiliary courseware according to relevant course data of any course, expanding the teaching auxiliary courseware to obtain an assembly interaction system, performing orthogonal transformation on a three-dimensional virtual model constructed based on the assembly interaction system to obtain a transformed model, mapping the transformed model to a world coordinate system and constructing a virtual simulation platform corresponding to the transformed model in the world coordinate system, creating a stream socket corresponding to a user terminal and establishing a connection between the user terminal and the virtual simulation system based on the stream socket, and enabling a user corresponding to the user terminal to perform a virtual interaction operation in the virtual simulation system when a content interaction request is received. The application further discloses a multi-person cooperative virtual interaction device based on the MR recording and broadcasting technology. The application can solve the problem of low efficiency of multi-person cooperative virtual interaction based on the MR recording and broadcasting technology.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-person collaborative virtual interaction method based on MR recording and broadcasting technology. Background Technology

[0002] With the rapid development of computer technology, virtual reality (VR) technology in modern industry can simulate and optimize assembly processes, improve the efficiency of conceptual design, simplify design, and more effectively carry out industrial production, achieving the integration of VR and intelligent manufacturing and promoting the development and upgrading of modern industry. In teaching scenarios, VR technology is often used to provide students with convenient and quick practical experiences. However, VR technology typically allows only one person to operate it simultaneously, leading to low interactivity, lack of interaction, and wasted resources. Therefore, there is an urgent need to propose a more efficient multi-person collaborative virtual interaction method. Summary of the Invention

[0003] This invention provides a multi-person collaborative virtual interaction method based on MR recording and broadcasting technology, the main purpose of which is to solve the problem of low efficiency in multi-person collaborative virtual interaction methods.

[0004] To achieve the above objectives, the present invention provides a multi-person collaborative virtual interaction method based on MR recording and broadcasting technology, comprising:

[0005] Obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into teaching aids based on a hypertext structure.

[0006] The teaching aids are expanded using a pre-acquired knowledge base and an assembly model interaction module to obtain an assembly interaction system.

[0007] Based on the assembly interaction system, a corresponding 3D virtual model is constructed. The 3D virtual model is then orthogonally transformed according to a pre-constructed orthogonal projection transformation matrix to obtain the transformed model. The transformed model is then mapped to a preset world coordinate system. Finally, a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system. The pre-constructed orthogonal projection transformation matrix is:

[0008]

[0009] Among them, T o This is the orthogonal projection transformation matrix, where right, left, top, bottom, far, and near are all formal parameters, and t x t y and t zThese are fixed parameters on the x, y, and z axes.

[0010] Obtain user terminals corresponding to multiple different users, call a preset socket function to create a stream socket corresponding to the user terminal, and establish a connection between the user terminal and the virtual simulation system based on the stream socket;

[0011] When a content interaction request is received, the user corresponding to the user terminal is instructed to perform a virtual interaction operation in the virtual simulation system.

[0012] Optionally, dividing the relevant course materials into different course chapters includes:

[0013] Extract multiple different course titles from the relevant course materials, and perform type identification on the multiple course titles to obtain the type of the course title;

[0014] The course titles of the aforementioned types are used as dividing nodes to divide the relevant course materials into different course chapters.

[0015] Optionally, the combination of the different course chapters into teaching aids based on a hypertext structure includes:

[0016] Using the course names of the relevant course materials as the structural starting point and multiple course chapters as structural nodes, a hypertext structure is constructed to obtain the courseware chapter division structure.

[0017] Based on the connections between various knowledge points, the chapter division structure of the courseware is expanded to obtain a standard division structure, which is then output as teaching aids.

[0018] Optionally, the step of expanding the courseware chapter division structure based on the connections between various knowledge points to obtain a standard division structure includes:

[0019] Select any one of the course chapters in the aforementioned course chapter division structure as the target chapter;

[0020] The knowledge point extraction model is used to extract multiple different knowledge points from the target chapter.

[0021] Based on the connections between the knowledge points, multiple knowledge points are associated to obtain a standard partitioning structure.

[0022] Optionally, the step of using a pre-acquired knowledge base and assembly model interaction module to extend the teaching aids to obtain an assembly interaction system includes:

[0023] An expert system module consisting of a help information section and a record section is constructed, and the expert system module is embedded into the assembly model interaction module to obtain a standard assembly interaction module;

[0024] The knowledge base, the standard assembly interaction module, and the detection module are connected according to the preset connection relationship to obtain the assembly interaction system.

[0025] Optionally, the construction of the corresponding three-dimensional virtual model based on the assembly interaction system includes:

[0026] The data simulation subsystem is used to construct the data simulation model and related characteristic simulation model of the assembly interaction system.

[0027] Receive external control data and parse the external control data to obtain control commands;

[0028] The control command is transmitted to the 3D model creation subsystem, and the 3D model creation subsystem is used to load the data simulation model and the related characteristic simulation model to obtain a 3D virtual model.

[0029] Optionally, before mapping the transformed model to a preset world coordinate system, the method further includes:

[0030] The transformed model is then subjected to translation, rotation, and scaling transformations.

[0031] Optionally, the step of calling a preset socket function to create a stream socket corresponding to the user terminal includes:

[0032] A terminal socket is created according to the preset socket function, and the terminal socket is bound to the specified port;

[0033] The connection function is invoked to listen for connection requests. If the virtual simulation system accepts the connection request, the terminal socket is returned as a stream socket.

[0034] Optionally, the step of performing an orthogonal transformation on the 3D virtual model according to a pre-constructed orthogonal projection transformation matrix to obtain the transformed model includes:

[0035] Extract the three-dimensional data from the three-dimensional virtual model, and multiply the orthogonal projection transformation matrix with the three-dimensional data in the three-dimensional virtual model;

[0036] The corresponding model is constructed based on the data obtained from the multiplication calculation, and the transformed model is obtained.

[0037] In this embodiment of the invention, teaching aids are constructed using relevant course materials from any course, and these teaching aids are then expanded to enrich the generated assembly interaction system. Based on this assembly interaction system, a three-dimensional virtual model is constructed, and a series of model processing and platform construction steps are performed to generate a platform suitable for virtual reality simulation. Stream sockets are used to enable interactive control of the same simulation content, thereby achieving virtual interactive operation. Therefore, the multi-person collaborative virtual interaction method, device, electronic device, and computer-readable storage medium based on MR recording and broadcasting technology proposed in this invention can solve the problem of low efficiency in multi-person collaborative virtual interaction based on MR recording and broadcasting technology. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a multi-person collaborative virtual interaction method based on MR recording and broadcasting technology provided in an embodiment of the present invention;

[0039] Figure 2 A functional block diagram of a multi-person collaborative virtual interaction device based on MR recording and broadcasting technology provided in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the multi-person collaborative virtual interaction method based on MR recording and broadcasting technology, according to an embodiment of the present invention.

[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] This invention provides a multi-user collaborative virtual interaction method based on MR recording and broadcasting technology. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this invention: a server, a terminal, etc. In other words, the multi-user collaborative virtual interaction method based on MR recording and broadcasting technology can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0044] Reference Figure 1 The diagram shown is a flowchart illustrating a multi-person collaborative virtual interaction method based on MR recording technology according to an embodiment of the present invention. In this embodiment, the multi-person collaborative virtual interaction method based on MR recording technology includes:

[0045] S1. Obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into a teaching aid courseware in the form of a hypertext structure.

[0046] In the embodiments of the present invention, the any course may refer to a variety of courses related to the field in different fields, including but not limited to courses related to trade and finance in the financial field, or courses in the aerospace-related field, or the automotive field. At the same time, the relevant materials may be refined materials for any part specified in the course. For example, the relevant course materials for the any course may refer to materials about an engine.

[0047] Specifically, the dividing the relevant course materials into different course chapters includes:

[0048] Extract multiple different course titles from the relevant course materials, and perform type recognition on the multiple course titles to obtain the types of the course titles.

[0049] Divide the relevant course materials into different course chapters with the course titles of the preset type as the division nodes.

[0050] Specifically, the relevant course materials contain multiple paragraphs and the corresponding course titles. Among them, the course titles include different forms and corresponding meanings. For example, the course title "One" is a first-level title, the course title "1" is a second-level title, and the course title "(1)" is a third-level title. Performing type recognition on the multiple course titles means identifying that the course titles are first-level titles, second-level titles, third-level titles or other undefined titles. If the preset type is a first-level title, then divide the relevant course materials into different course nodes with the course titles of the first-level title as the division nodes.

[0051] For example, in the relevant course materials of engine disassembly, the content corresponding to different first-level titles is "Introduction", "Crankshaft Connecting Rod Mechanism", "Valve Train", "Fuel Supply System", "Ignition System", "Cooling System", "Lubrication System" and "Starting System". Therefore, divide the relevant course materials into different course chapters with different first-level titles as the division nodes, where the different course chapters are "Introduction", "Crankshaft Connecting Rod Mechanism", "Valve Train", "Fuel Supply System", "Ignition System", "Cooling System", "Lubrication System" and "Starting System".

[0052] Further, the combining the different course chapters into a teaching aid courseware in the form of a hypertext structure includes:

[0053] Using the course names of the relevant course materials as the structural starting point and multiple course chapters as structural nodes, a hypertext structure is constructed to obtain the courseware chapter division structure.

[0054] Based on the connections between various knowledge points, the chapter division structure of the courseware is expanded to obtain a standard division structure, which is then output as teaching aids.

[0055] In detail, the structure of information organization in course chapters is mainly divided into two types. One is to arrange information in a specific order to facilitate the organization of materials; this is called a linear structure. The other is to use a non-linear network structure, which mainly divides information according to its inherent relevance and independence; this structure is called a hypertext structure. In this scheme, because there is a lot of interactive information among the components in the engine structure and the linear characteristics are not obvious, a hypertext structure is suitable. Utilizing the interconnected communication characteristics of network structure nodes allows for a better presentation of teaching content and the teaching process.

[0056] For example, if the course title of the relevant course materials is "Engine", then "Engine" is used as the starting point of the structure, and multiple course chapters such as "Introduction", "Crankshaft and Connecting Rod Mechanism", "Valve Train Mechanism", "Supply System", "Ignition System", "Cooling System", "Hot System" and "Starting System" are used as structural nodes to construct the hypertext structure, resulting in the courseware chapter division structure.

[0057] Specifically, the process of expanding the courseware chapter division structure based on the connections between various knowledge points to obtain a standard division structure includes:

[0058] Select any one of the course chapters in the aforementioned course chapter division structure as the target chapter;

[0059] The knowledge point extraction model is used to extract multiple different knowledge points from the target chapter.

[0060] Based on the connections between the knowledge points, multiple knowledge points are associated to obtain a standard partitioning structure.

[0061] Specifically, any course chapter in the course chapter division structure is selected as the target chapter. The target chapter can be "crankshaft and connecting rod mechanism". Multiple different knowledge points in the target chapter are extracted using a knowledge point extraction model, which can be a convolutional neural network model or a bidirectional long short-term memory network, etc. The extracted knowledge points are "overview", "body assembly", "piston and connecting rod assembly", "crankshaft and flywheel", "piston", "piston ring", "piston pin", and "connecting rod". Based on the relationships between these knowledge points, a standard division structure is obtained. The knowledge point "piston and connecting rod assembly" has a total-to-part relationship with "piston", "piston ring", "piston pin", and "connecting rod". That is, "piston and connecting rod assembly" is the total knowledge point, and "piston", "piston ring", "piston pin", and "connecting rod" are the sub-knowledge points of the total knowledge point.

[0062] S2. Using the pre-acquired knowledge base and assembly model interaction module, the teaching aid courseware is expanded to obtain the assembly interaction system.

[0063] In this embodiment of the invention, since the teaching aids are often based on theoretical knowledge, but the structure of the engine is very complex, traditional text, pictures, videos or Flash teaching resources have limitations in structural display, which will increase the cognitive load of learners and thus cause the embarrassing situation of poor teaching effect. Therefore, it is necessary to expand the teaching aids to obtain the assembly interaction system.

[0064] Specifically, the step of using a pre-acquired knowledge base and an assembly model interaction module to expand the teaching aids to obtain an assembly interaction system includes:

[0065] An expert system module consisting of a help information section and a record section is constructed, and the expert system module is embedded into the assembly model interaction module to obtain a standard assembly interaction module;

[0066] The knowledge base, the standard assembly interaction module, and the detection module are connected according to the preset connection relationship to obtain the assembly interaction system.

[0067] In detail, the help information section can display or hide help information. When learners assemble parts, it provides correct installation prompts, error prompts, and corresponding text and color changes. The recording section allows learners to replay the assembly process for self-assessment. The knowledge base includes a 3D model library, a parts information library, and other basic information. Learners can select different types of parts to view relevant descriptions, such as the part's name, assembly location, and functional description. The assembly model interaction module features animated demonstrations of the assembly process and the ability to self-control and adjust the assembly process. Learners can use this module to independently select the assembly order of parts, retrieve the relevant parts, and perform real-time assembly operations.

[0068] S3. Based on the assembly interaction system, construct a corresponding three-dimensional virtual model, perform orthogonal transformation on the three-dimensional virtual model according to the pre-constructed orthogonal projection transformation matrix to obtain the transformed model, map the transformed model to a preset world coordinate system, and select a target simulation tool to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system.

[0069] In this embodiment of the invention, the corresponding three-dimensional virtual model is constructed through the assembly interaction system, which enables the engine to perform functions such as dragging, rotating, and changing the viewpoint in a three-dimensional environment. Combined with virtual reality technology, it can achieve collaborative virtual interaction.

[0070] Specifically, the construction of the corresponding three-dimensional virtual model based on the assembly interaction system includes:

[0071] The interactive data in the assembly interaction system is input into the pre-acquired 3D modeling software to obtain the 3D virtual model corresponding to the assembly interaction system.

[0072] Specifically, the 3D modeling software is Pro / Engine, which is an integrated CAD / CAM / CAE 3D software. It is known for its parametric nature and is the earliest user of parametric technology, holding an important position in the field of 3D modeling software.

[0073] In another embodiment of the present invention, the step of constructing a corresponding three-dimensional virtual model based on the assembly interaction system includes:

[0074] The data simulation subsystem is used to construct the data simulation model and related characteristic simulation model of the assembly interaction system.

[0075] Receive external control data and parse the external control data to obtain control commands;

[0076] The control command is transmitted to the 3D model creation subsystem, and the 3D model creation subsystem is used to load the data simulation model and the related characteristic simulation model to obtain a 3D virtual model.

[0077] In detail, the relevant characteristic simulation models include fuel data simulation models, fuel control simulation models, air data simulation models, air source simulation models, exhaust simulation models, ignition system data models, and continuous motion data models. The data simulation subsystem includes engine simulation modules, fuel simulation modules, etc. The 3D model creation subsystem loads the model based on the received external control data. Model loading refers to displaying the real-time animation, motion, and special effects of the 3D model. In this embodiment of the invention, the display methods of the 3D virtual model include overall display, perspective display, partial display, and dynamically split display.

[0078] Further, the step of performing an orthogonal transformation on the 3D virtual model based on a pre-constructed orthogonal projection transformation matrix to obtain the transformed model includes:

[0079] Extract the three-dimensional data from the three-dimensional virtual model, and multiply the orthogonal projection transformation matrix with the three-dimensional data in the three-dimensional virtual model;

[0080] The corresponding model is constructed based on the data obtained from the multiplication calculation, and the transformed model is obtained.

[0081] Specifically, the pre-constructed orthogonal projection transformation matrix is:

[0082]

[0083] Among them, T o This is the orthogonal projection transformation matrix, where right, left, top, bottom, far, and near are all formal parameters, and t x t y and t z These are fixed parameters on the x, y, and z axes.

[0084] Furthermore, before mapping the transformed model to a preset world coordinate system, the method further includes:

[0085] The transformed model is then subjected to translation, rotation, and scaling transformations.

[0086] In detail, the transformed model is placed in a unified world coordinate system after a series of 3D transformations such as translation, rotation and scaling, which facilitates subsequent observation and operations.

[0087] Specifically, a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system. The target simulation tool refers to the Vega system, a visually driven software platform widely used in the development of simulation and virtual reality systems.

[0088] The virtual simulation platform is a computer simulation system that can create and experience virtual worlds. It can use computers to generate three-dimensional virtual environments to simulate real-world scenes. Users can interact with the virtual world through various sensing devices, participate in understanding and influencing events within it. It has the ability to allow people to personally experience and immerse themselves in this virtual environment and interact with it. Virtual reality simulation can provide a transferable experience, and learners can repeatedly try in training until they reach the optimal state or the desired result.

[0089] S4. Obtain user terminals corresponding to multiple different users, call a preset socket function to create a stream socket corresponding to the user terminal, and establish a connection between the user terminal and the virtual simulation system based on the stream socket.

[0090] In this embodiment of the invention, multiple user terminals corresponding to different users are obtained. These multiple different users can refer to user A, user B, or user C, and the terminal corresponding to each user refers to the interaction device between the site and the user. These interaction devices can be desktop computers, laptops, handheld computers, or other smart terminal devices, such as tablets. In this solution, user A can use a desktop computer, user B can use a laptop, and user C can use a tablet.

[0091] Specifically, the step of calling a preset socket function to create a stream socket corresponding to the user terminal includes:

[0092] A terminal socket is created according to the preset socket function, and the terminal socket is bound to the specified port;

[0093] The connection function is invoked to listen for connection requests. If the virtual simulation system accepts the connection request, the terminal socket is returned as a stream socket.

[0094] In detail, the preset socket function is Socket. Socket is used to describe IP address and port, and is a handle of a communication link. A complete Socket communication program generally includes several steps such as creating Socket, opening input / output streams connected to Socket, performing read or write operations on Socket according to a certain protocol, and closing Socket.

[0095] Furthermore, a connection is established between the user terminal and the virtual simulation system based on the stream socket. This connection is more convenient and facilitates easier access to and development of various network applications. In this solution, Socket network communication technology can be used to achieve interactive control of the same simulation content. For example, in a virtual simulation system courseware for engine disassembly, multiple users such as User A, User B, User C, etc., can learn together online. Multiple users can simultaneously perform course content such as disassembling the same engine model and observe each other's operating status.

[0096] S5. When a content interaction request is received, the user corresponding to the user terminal is instructed to perform a virtual interaction operation in the virtual simulation system.

[0097] In this embodiment of the invention, when a content interaction request is received, the user corresponding to the user terminal can perform virtual interactive operations in the virtual simulation system. That is, user A, user B, and user C can learn together online. Multiple users can simultaneously perform course content such as disassembling the same engine model, and can observe each other's operation status. Meanwhile, the virtual simulation system contains rich and vivid courseware on disassembling engine models, thus helping users achieve efficient and convenient learning, and solving the problem of collaborative operation between multiple simulation applications in the same space.

[0098] In this embodiment of the invention, teaching aids are constructed using relevant course materials from any course, and these aids are then expanded to enrich the resulting assembly interaction system. A three-dimensional virtual model is built based on this system, and a series of model processing and platform construction steps are performed to generate a platform suitable for virtual reality simulation. Stream sockets are used to enable interactive control of the same simulation content, thereby achieving virtual interactive operation. Therefore, the multi-person collaborative virtual interaction method based on MR recording technology proposed in this invention can solve the problem of low efficiency in multi-person collaborative virtual interaction based on MR recording technology.

[0099] like Figure 2 The diagram shown is a functional block diagram of a multi-person collaborative virtual interaction device based on MR recording and broadcasting technology provided in an embodiment of the present invention.

[0100] The multi-user collaborative virtual interactive device 100 based on MR recording and broadcasting technology described in this invention can be installed in an electronic device. Depending on the functions implemented, the multi-user collaborative virtual interactive device 100 based on MR recording and broadcasting technology may include a courseware generation module 101, a platform construction module 102, a system connection module 103, and a virtual interaction module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0101] In this embodiment, the functions of each module / unit are as follows:

[0102] The courseware generation module 101 is used to obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into teaching aids courseware based on a hypertext structure.

[0103] The platform construction module 102 is used to expand the teaching aid courseware using a pre-acquired knowledge base and an assembly model interaction module to obtain an assembly interaction system. Based on the assembly interaction system, a corresponding three-dimensional virtual model is constructed. The three-dimensional virtual model is orthogonally transformed according to a pre-constructed orthogonal projection transformation matrix to obtain a transformed model. The transformed model is mapped to a preset world coordinate system, and a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system.

[0104] The system connection module 103 is used to obtain user terminals corresponding to multiple different users, call a preset socket function to create a stream socket corresponding to the user terminal, and establish a connection between the user terminal and the virtual simulation system based on the stream socket.

[0105] The virtual interaction module 104 is used to, when receiving a content interaction request, instruct the user corresponding to the user terminal to perform a virtual interaction operation in the virtual simulation system.

[0106] In detail, the specific implementation methods of each module of the multi-person collaborative virtual interaction device 100 based on MR recording and broadcasting technology are as follows:

[0107] Step 1: Obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into teaching aids based on a hypertext structure.

[0108] In the embodiments of the present invention, the arbitrary course may refer to a variety of courses related to different fields, including but not limited to courses related to trade and finance in the financial field, or courses in the aerospace-related field, which may also be in the automotive field. At the same time, the relevant materials may be refined materials for any part specified in the course. For example, the relevant course materials for the arbitrary course may refer to materials of an engine.

[0109] Specifically, the dividing the relevant course materials into different course chapters includes:

[0110] Extracting a plurality of different course titles from the relevant course materials and performing type recognition on the plurality of course titles to obtain the types of the course titles;

[0111] Using the course titles of the preset type as division nodes to divide the relevant course materials into different course chapters.

[0112] Specifically, the relevant course materials contain a plurality of paragraphs and corresponding course titles. Among them, the course titles include different forms and corresponding meanings. For example, the course title "One" is a first-level title, the course title "1" is a second-level title, and the course title "(1)" is a third-level title. Performing type recognition on the plurality of course titles means recognizing that the course titles are first-level titles, second-level titles, third-level titles or other titles not defined. If the preset type is a first-level title, then using the course titles of the first-level title as division nodes to divide the relevant course materials into different course nodes.

[0113] For example, in the relevant course materials on engine disassembly, the content corresponding to different first-level titles is "Introduction", "Crankshaft Connecting Rod Mechanism", "Valve Train", "Fuel Supply System", "Ignition System", "Cooling System", "Lubrication System" and "Starting System". Therefore, using different first-level titles as division nodes to divide the relevant course materials into different course chapters, where the different course chapters are "Introduction", "Crankshaft Connecting Rod Mechanism", "Valve Train", "Fuel Supply System", "Ignition System", "Cooling System", "Lubrication System" and "Starting System".

[0114] Furthermore, the combining the different course chapters into a teaching auxiliary courseware in the form of a hypertext structure includes:

[0115] Using the course name of the relevant course materials as the structural starting point and using the plurality of course chapters as structural nodes to construct a hypertext structure to obtain a courseware chapter division structure;

[0116] Based on the connections between various knowledge points, the chapter division structure of the courseware is expanded to obtain a standard division structure, which is then output as teaching aids.

[0117] In detail, the structure of information organization in course chapters is mainly divided into two types. One is to arrange information in a specific order to facilitate the organization of materials; this is called a linear structure. The other is to use a non-linear network structure, which mainly divides information according to its inherent relevance and independence; this structure is called a hypertext structure. In this scheme, because there is a lot of interactive information among the components in the engine structure and the linear characteristics are not obvious, a hypertext structure is suitable. Utilizing the interconnected communication characteristics of network structure nodes allows for a better presentation of teaching content and the teaching process.

[0118] For example, if the course title of the relevant course materials is "Engine", then "Engine" is used as the starting point of the structure, and multiple course chapters such as "Introduction", "Crankshaft and Connecting Rod Mechanism", "Valve Train Mechanism", "Supply System", "Ignition System", "Cooling System", "Hot System" and "Starting System" are used as structural nodes to construct the hypertext structure, resulting in the courseware chapter division structure.

[0119] Specifically, the process of expanding the courseware chapter division structure based on the connections between various knowledge points to obtain a standard division structure includes:

[0120] Select any one of the course chapters in the aforementioned course chapter division structure as the target chapter;

[0121] The knowledge point extraction model is used to extract multiple different knowledge points from the target chapter.

[0122] Based on the connections between the knowledge points, multiple knowledge points are associated to obtain a standard partitioning structure.

[0123] Specifically, any course chapter in the course chapter division structure is selected as the target chapter. The target chapter can be "crankshaft and connecting rod mechanism". Multiple different knowledge points in the target chapter are extracted using a knowledge point extraction model, which can be a convolutional neural network model or a bidirectional long short-term memory network, etc. The extracted knowledge points are "overview", "body assembly", "piston and connecting rod assembly", "crankshaft and flywheel", "piston", "piston ring", "piston pin", and "connecting rod". Based on the relationships between these knowledge points, a standard division structure is obtained. The knowledge point "piston and connecting rod assembly" has a total-to-part relationship with "piston", "piston ring", "piston pin", and "connecting rod". That is, "piston and connecting rod assembly" is the total knowledge point, and "piston", "piston ring", "piston pin", and "connecting rod" are the sub-knowledge points of the total knowledge point.

[0124] Step 2: Using the pre-acquired knowledge base and assembly model interaction module, the teaching aid courseware is expanded to obtain the assembly interaction system.

[0125] In this embodiment of the invention, since the teaching aids are often based on theoretical knowledge, but the structure of the engine is very complex, traditional text, pictures, videos or Flash teaching resources have limitations in structural display, which will increase the cognitive load of learners and thus cause the embarrassing situation of poor teaching effect. Therefore, it is necessary to expand the teaching aids to obtain the assembly interaction system.

[0126] Specifically, the step of using a pre-acquired knowledge base and an assembly model interaction module to expand the teaching aids to obtain an assembly interaction system includes:

[0127] An expert system module consisting of a help information section and a record section is constructed, and the expert system module is embedded into the assembly model interaction module to obtain a standard assembly interaction module;

[0128] The knowledge base, the standard assembly interaction module, and the detection module are connected according to the preset connection relationship to obtain the assembly interaction system.

[0129] In detail, the help information section can display or hide help information. When learners assemble parts, it provides correct installation prompts, error prompts, and corresponding text and color changes. The recording section allows learners to replay the assembly process for self-assessment. The knowledge base includes a 3D model library, a parts information library, and other basic information. Learners can select different types of parts to view relevant descriptions, such as the part's name, assembly location, and functional description. The assembly model interaction module features animated demonstrations of the assembly process and the ability to self-control and adjust the assembly process. Learners can use this module to independently select the assembly order of parts, retrieve the relevant parts, and perform real-time assembly operations.

[0130] Step 3: Construct a corresponding 3D virtual model based on the assembly interaction system, perform orthogonal transformation on the 3D virtual model according to the pre-constructed orthogonal projection transformation matrix to obtain the transformed model, map the transformed model to a preset world coordinate system, and select a target simulation tool to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system.

[0131] In this embodiment of the invention, the corresponding three-dimensional virtual model is constructed through the assembly interaction system, which enables the engine to perform functions such as dragging, rotating, and changing the viewpoint in a three-dimensional environment. Combined with virtual reality technology, it can achieve collaborative virtual interaction.

[0132] Specifically, the construction of the corresponding three-dimensional virtual model based on the assembly interaction system includes:

[0133] The interactive data in the assembly interaction system is input into the pre-acquired 3D modeling software to obtain the 3D virtual model corresponding to the assembly interaction system.

[0134] Specifically, the 3D modeling software is Pro / Engine, which is an integrated CAD / CAM / CAE 3D software. It is known for its parametric nature and is the earliest user of parametric technology, holding an important position in the field of 3D modeling software.

[0135] In another embodiment of the present invention, the step of constructing a corresponding three-dimensional virtual model based on the assembly interaction system includes:

[0136] The data simulation subsystem is used to construct the data simulation model and related characteristic simulation model of the assembly interaction system.

[0137] Receive external control data and parse the external control data to obtain control commands;

[0138] The control command is transmitted to the 3D model creation subsystem, and the 3D model creation subsystem is used to load the data simulation model and the related characteristic simulation model to obtain a 3D virtual model.

[0139] In detail, the relevant characteristic simulation models include fuel data simulation models, fuel control simulation models, air data simulation models, air source simulation models, exhaust simulation models, ignition system data models, and continuous motion data models. The data simulation subsystem includes engine simulation modules, fuel simulation modules, etc. The 3D model creation subsystem loads the model based on the received external control data. Model loading refers to displaying the real-time animation, motion, and special effects of the 3D model. In this embodiment of the invention, the display methods of the 3D virtual model include overall display, perspective display, partial display, and dynamically split display.

[0140] Further, the step of performing an orthogonal transformation on the 3D virtual model based on a pre-constructed orthogonal projection transformation matrix to obtain the transformed model includes:

[0141] Extract the three-dimensional data from the three-dimensional virtual model, and multiply the orthogonal projection transformation matrix with the three-dimensional data in the three-dimensional virtual model;

[0142] The corresponding model is constructed based on the data obtained from the multiplication calculation, and the transformed model is obtained.

[0143] Specifically, the pre-constructed orthogonal projection transformation matrix is:

[0144]

[0145] Among them, T o This is the orthogonal projection transformation matrix, where right, left, top, bottom, far, and near are all formal parameters, and t x t y and t z These are fixed parameters on the x, y, and z axes.

[0146] Furthermore, before mapping the transformed model to a preset world coordinate system, the method further includes:

[0147] The transformed model is then subjected to translation, rotation, and scaling transformations.

[0148] In detail, the transformed model is placed in a unified world coordinate system after a series of 3D transformations such as translation, rotation and scaling, which facilitates subsequent observation and operations.

[0149] Specifically, a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system. The target simulation tool refers to the Vega system, a visually driven software platform widely used in the development of simulation and virtual reality systems.

[0150] The virtual simulation platform is a computer simulation system that can create and experience virtual worlds. It can use computers to generate three-dimensional virtual environments to simulate real-world scenes. Users can interact with the virtual world through various sensing devices, participate in understanding and influencing events within it. It has the ability to allow people to personally experience and immerse themselves in this virtual environment and interact with it. Virtual reality simulation can provide a transferable experience, and learners can repeatedly try in training until they reach the optimal state or the desired result.

[0151] Step 4: Obtain user terminals corresponding to multiple different users, call the preset socket function to create the stream socket corresponding to the user terminal, and establish the connection between the user terminal and the virtual simulation system based on the stream socket.

[0152] In this embodiment of the invention, multiple user terminals corresponding to different users are obtained. These multiple different users can refer to user A, user B, or user C, and the terminal corresponding to each user refers to the interaction device between the site and the user. These interaction devices can be desktop computers, laptops, handheld computers, or other smart terminal devices, such as tablets. In this solution, user A can use a desktop computer, user B can use a laptop, and user C can use a tablet.

[0153] Specifically, the step of calling a preset socket function to create a stream socket corresponding to the user terminal includes:

[0154] A terminal socket is created according to the preset socket function, and the terminal socket is bound to the specified port;

[0155] The connection function is invoked to listen for connection requests. If the virtual simulation system accepts the connection request, the terminal socket is returned as a stream socket.

[0156] In detail, the preset socket function is Socket. Socket is used to describe IP address and port, and is a handle of a communication link. A complete Socket communication program generally includes several steps such as creating Socket, opening input / output streams connected to Socket, performing read or write operations on Socket according to a certain protocol, and closing Socket.

[0157] Furthermore, a connection is established between the user terminal and the virtual simulation system based on the stream socket. This connection is more convenient and facilitates easier access to and development of various network applications. In this solution, Socket network communication technology can be used to achieve interactive control of the same simulation content. For example, in a virtual simulation system courseware for engine disassembly, multiple users such as User A, User B, User C, etc., can learn together online. Multiple users can simultaneously perform course content such as disassembling the same engine model and observe each other's operating status.

[0158] Step 5: When a content interaction request is received, instruct the user corresponding to the user terminal to perform a virtual interaction operation in the virtual simulation system.

[0159] In this embodiment of the invention, when a content interaction request is received, the user corresponding to the user terminal can perform virtual interactive operations in the virtual simulation system. That is, user A, user B, and user C can learn together online. Multiple users can simultaneously perform course content such as disassembling the same engine model, and can observe each other's operation status. Meanwhile, the virtual simulation system contains rich and vivid courseware on disassembling engine models, thus helping users achieve efficient and convenient learning, and solving the problem of collaborative operation between multiple simulation applications in the same space.

[0160] In this embodiment of the invention, teaching aids are constructed using relevant course materials from any course, and these aids are then expanded to enrich the resulting assembly interaction system. A three-dimensional virtual model is built based on this system, and a series of model processing and platform construction steps are performed to generate a platform suitable for virtual reality simulation. Stream sockets are used to enable interactive control of the same simulation content, thereby achieving virtual interactive operation. Therefore, the multi-person collaborative virtual interaction device based on MR recording technology proposed in this invention can solve the problem of low efficiency in multi-person collaborative virtual interaction based on MR recording technology.

[0161] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a multi-person collaborative virtual interaction method based on MR recording and broadcasting technology, according to an embodiment of the present invention.

[0162] The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a multi-person collaborative virtual interaction program based on MR recording and broadcasting technology.

[0163] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a multi-user collaborative virtual interactive program based on MR recording technology, but also to temporarily store data that has been output or will be output.

[0164] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., multi-user collaborative virtual interactive programs based on MR recording technology) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0165] The communication interface 12 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0166] The bus 13 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 13 can be divided into an address bus, a data bus, a control bus, etc. The bus 13 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0167] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0168] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0169] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0170] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0171] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0172] The multi-user collaborative virtual interaction program based on MR recording and broadcasting technology stored in the memory 11 of the electronic device is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0173] Obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into teaching aids based on a hypertext structure.

[0174] The teaching aids are expanded using a pre-acquired knowledge base and an assembly model interaction module to obtain an assembly interaction system.

[0175] Based on the assembly interaction system, a corresponding three-dimensional virtual model is constructed. The three-dimensional virtual model is orthogonally transformed according to the pre-constructed orthogonal projection transformation matrix to obtain the transformed model. The transformed model is mapped to a preset world coordinate system, and a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system.

[0176] Obtain user terminals corresponding to multiple different users, call a preset socket function to create a stream socket corresponding to the user terminal, and establish a connection between the user terminal and the virtual simulation system based on the stream socket;

[0177] When a content interaction request is received, the user corresponding to the user terminal is instructed to perform a virtual interaction operation in the virtual simulation system.

[0178] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0179] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0180] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0181] Obtain relevant course materials for any course, divide the relevant course materials into different course chapters, and combine the different course chapters into teaching aids based on a hypertext structure.

[0182] The teaching aids are expanded using a pre-acquired knowledge base and an assembly model interaction module to obtain an assembly interaction system.

[0183] Based on the assembly interaction system, a corresponding three-dimensional virtual model is constructed. The three-dimensional virtual model is orthogonally transformed according to the pre-constructed orthogonal projection transformation matrix to obtain the transformed model. The transformed model is mapped to a preset world coordinate system, and a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system.

[0184] Obtain user terminals corresponding to multiple different users, call a preset socket function to create a stream socket corresponding to the user terminal, and establish a connection between the user terminal and the virtual simulation system based on the stream socket;

[0185] When a content interaction request is received, the user corresponding to the user terminal is instructed to perform a virtual interaction operation in the virtual simulation system.

[0186] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0187] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0189] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0190] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0191] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0192] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-person collaborative virtual interaction method based on MR recording and broadcasting technology, characterized in that, The method includes: acquiring relevant course materials for any course, dividing the relevant course materials into different course chapters, and combining the different course chapters into teaching aids based on a hypertext structure; using a pre-acquired knowledge base and an assembly model interaction module to perform courseware expansion processing on the teaching aids to obtain an assembly interaction system. The step of using a pre-acquired knowledge base and assembly model interaction module to expand the teaching aids to obtain an assembly interaction system includes: constructing an expert system module consisting of a help information section and a record section, and embedding the expert system module into the assembly model interaction module to obtain a standard assembly interaction module; and connecting the knowledge base, the standard assembly interaction module, and the detection module according to a preset connection relationship to obtain the assembly interaction system. Based on the assembly interaction system, a corresponding 3D virtual model is constructed. The 3D virtual model is then orthogonally transformed according to a pre-constructed orthogonal projection transformation matrix to obtain the transformed model. The transformed model is then mapped to a preset world coordinate system. Finally, a target simulation tool is selected to construct a virtual simulation platform corresponding to the transformed model in the preset world coordinate system. The pre-constructed orthogonal projection transformation matrix is: Where To is the orthogonal projection transformation matrix, right, left, top, bottom, far, and near are all formal parameters, and tx, ty, and ty are fixed parameters on the x-axis, y-axis, and z-axis, respectively. Multiple user terminals corresponding to different users are obtained, and a preset socket function is called to create a stream socket corresponding to the user terminal. A connection between the user terminal and the virtual simulation system is established based on the stream socket. When a content interaction request is received, the user corresponding to the user terminal is instructed to perform a virtual interaction operation in the virtual simulation system.

2. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, The step of dividing the relevant course materials into different course chapters includes: extracting multiple different course titles from the relevant course materials, identifying the types of the multiple course titles to obtain the types of the course titles; and dividing the relevant course materials into different course chapters using the course titles of the preset types as dividing nodes.

3. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, The method of combining different course chapters into teaching aids based on a hypertext structure includes: using the course name of the relevant course materials as the structural starting point and multiple course chapters as structural nodes to construct a hypertext structure to obtain a course chapter division structure; expanding the course chapter division structure based on the connections between various knowledge points to obtain a standard division structure, and outputting the standard division structure as teaching aids.

4. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 3, characterized in that, The step of expanding the course chapter division structure based on the connections between knowledge points to obtain a standard division structure includes: selecting any course chapter in the course chapter division structure as the target chapter; extracting multiple different knowledge points in the target chapter using a knowledge point extraction model; and performing association processing on the multiple knowledge points based on the connections between them to obtain the standard division structure.

5. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, The construction of the corresponding three-dimensional virtual model based on the assembly interaction system includes: using a data simulation subsystem to construct a data simulation model and a related characteristic simulation model corresponding to the assembly interaction system; receiving external control data and parsing the external control data to obtain control commands; transmitting the control commands to a three-dimensional model creation subsystem and using the three-dimensional model creation subsystem to load the data simulation model and the related characteristic simulation model to obtain a three-dimensional virtual model.

6. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, Before mapping the transformed model to a preset world coordinate system, the method further includes performing translation, rotation, and scaling transformations on the transformed model respectively.

7. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, The step of calling a preset socket function to create a stream socket corresponding to the user terminal includes: creating a terminal socket according to the preset socket function and binding the terminal socket to a specified port; calling a connection function to listen for connection requests, and if the virtual simulation system accepts the connection request, returning the terminal socket as a stream socket.

8. The multi-person collaborative virtual interaction method based on MR recording and broadcasting technology as described in claim 1, characterized in that, The step of performing orthogonal transformation on the three-dimensional virtual model according to the pre-constructed orthogonal projection transformation matrix to obtain the transformed model includes: extracting the three-dimensional data from the three-dimensional virtual model, multiplying the orthogonal projection transformation matrix with the three-dimensional data from the three-dimensional virtual model; and constructing a corresponding model based on the data obtained from the multiplication calculation to obtain the transformed model.

Images