Electronic circuit experiment teaching method and system based on mixed reality

Abstract

The invention discloses an electronic circuit experiment teaching method and system based on mixed reality, and relates to the technical field of virtual-real interaction, and the method specifically comprises the steps: obtaining an electronic component, splitting the electronic component, and constructing an electronic component model database; identifying and tracking the physical object, and acquiring identification information and real-time position data of the physical object; obtaining an external input instruction, and analyzing the external input instruction to obtain an operation instruction; calling related data in the electronic component model and / or the theory knowledge base according to the operation instruction and the identification information of the physical object; and displaying the related data, and performing corresponding interactive operation on the related data according to the received gesture data or eye movement data or real-time position data and a preset rule. According to the invention, a real-time interactive learning environment is provided, so that students can understand abstract concepts more intuitively and perform operation practice in a virtual environment, the learning efficiency is improved, and the sense of participation and interest of the students are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of virtual-reality interaction technology, and more specifically to an electronic circuit experiment teaching method and system based on mixed reality. Background Art

[0002] At present, electronic circuit experimental teaching occupies an important position in the education of electronic information majors, aiming to strengthen students' theoretical knowledge and cultivate their practical innovation ability. However, traditional teaching methods are often limited by the time and space constraints of physical laboratories, insufficient experimental teaching staff, and the large gap between theory and practice. When students transition from theoretical learning to practical operation, they need to master a lot of operational skills, which poses a challenge within the limited teaching time. At the same time, it is difficult for teachers to provide sufficient personalized guidance to each student, which affects the course progress and teaching effect.

[0003] However, existing technical solutions have some limitations. On the one hand, they only implement a basic web-based operating platform, lack AR / MR support, and fail to provide a real-time interactive learning experience; on the other hand, these solutions are mostly focused on specific course content, such as "Basics of Computer Software Technology" and "Computer Practice", and do not achieve cross-platform compatibility, resulting in a limited scope of application and low popularity.

[0004] Therefore, developing an electronic circuit experiment teaching system and method that can provide an interactive and personalized learning environment to improve the teaching quality and students' learning efficiency is an urgent problem that technicians in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides an electronic circuit experiment teaching method and system based on mixed reality, which overcomes the above-mentioned defects.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An electronic circuit experiment teaching method based on mixed reality, the specific steps are:

[0008] Acquire electronic components and split the electronic components, generate electronic component models, and build an electronic component model database;

[0009] Identify and track physical objects, and obtain identification information and real-time location data of the physical objects;

[0010] Obtaining external input instructions, and parsing the external input instructions to obtain operation instructions;

[0011] Calling the electronic component model and / or related data in the theoretical knowledge base according to the operation instruction and the identification information of the physical object;

[0012] The relevant data is displayed, and corresponding interactive operations are performed on the relevant data according to preset rules based on the received gesture data or eye movement data or the real-time position data.

[0013] Furthermore, the steps of constructing the electronic component model database are:

[0014] Acquire the electronic components and classify the electronic components;

[0015] Separate any type of electronic components according to body, pins and parameters;

[0016] Modeling the body, pins and parameters of any type of electronic component in proportion to form a model of any type of electronic component;

[0017] The models of various electronic components are summarized to construct the electronic component model database.

[0018] Furthermore, the steps of constructing the theoretical knowledge base are: acquiring relevant knowledge of electronic circuits, and constructing the theoretical knowledge base based on the relevant knowledge.

[0019] Furthermore, the relevant data includes electronic component models, electronic component parameters and schematic diagrams.

[0020] Furthermore, triggering, clicking, dragging, rotating, scaling and marking operations are performed according to the gesture data.

[0021] Furthermore, a labeling operation is performed according to the eye movement data.

[0022] Furthermore, a drag operation is performed according to the real-time position data.

[0023] An electronic circuit experiment teaching system based on mixed reality, comprising:

[0024] A database construction module is used to split electronic components, generate electronic component models, and construct an electronic component model database;

[0025] A physical identification and tracking module, used to identify and track physical objects, and transmit identification information and real-time location data of the physical objects to the processing module and the interactive identification module respectively;

[0026] The instruction parsing module is used to obtain external input instructions, parse the external input instructions, and transmit the obtained operation instructions to the processing module;

[0027] The processing module is used to call the relevant data of the electronic component model database and the theoretical knowledge base according to the operation instruction and the identification information of the physical object, and transmit the relevant data to the interactive identification module;

[0028] The interaction recognition module is used to display the relevant data and operate the relevant data according to preset rules based on the received gesture data or eye movement data or the real-time position data.

[0029] Furthermore, the physical identification and tracking module includes an image acquisition unit, an image classification unit, and an image recognition unit;

[0030] The image acquisition unit is used to convert the world coordinates of the electronic component into a camera coordinate system and then capture an image of the electronic component;

[0031] The image classification unit is used to input the captured image into a pre-trained classification model to obtain the category of the electronic component;

[0032] The image recognition unit is used to call a corresponding recognition model according to the category of the electronic component to obtain parameter recognition of the electronic component.

[0033] Furthermore, it also includes a feedback unit for receiving a simulation circuit, judging whether the simulation is correct according to the theoretical knowledge base, and outputting an optimization solution based on the operation result of the simulation circuit.

[0034] It can be seen from the above technical solutions that the present invention discloses an electronic circuit experiment teaching method and system based on mixed reality, which has the following beneficial effects compared with the prior art:

[0035] Enhanced learning experience: The system uses AR / MR functions to provide a real-time interactive learning environment, allowing students to understand abstract concepts more intuitively and practice operations in a virtual environment, which not only improves learning efficiency but also enhances students' sense of participation and interest;

[0036] Personalized guidance: Through the intelligent assistance system, each student can get personalized operation guidance and instant feedback, which makes up for the lack of teachers' inability to provide full one-on-one guidance to all students;

[0037] Cross-platform compatibility: Developed using the Unity engine and OpenXR interface, with the support of multi-platform plug-ins, the software can be quickly ported across different hardware platforms, increasing the flexibility and applicability of the system, and helping to improve the sharing of teaching resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the method provided by the present invention;

[0040] Figure 2 This is a schematic diagram of resistor splitting provided by the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The embodiment of the present invention discloses an electronic circuit experiment teaching method based on mixed reality, such as Figure 1 As shown, the specific steps are:

[0043] Step 1: Obtain electronic components and split them, generate electronic component models, and build an electronic component model database;

[0044] Step 2: Identify and track the physical object, and obtain the identification information and real-time location data of the physical object;

[0045] Step 3: Obtain external input instructions, and parse the external input instructions to obtain operation instructions;

[0046] Step 4: Call the electronic component model and / or related data in the theoretical knowledge base according to the operation instruction and the identification information of the physical object;

[0047] Step 5: Display the relevant data, and perform corresponding interactive operations on the relevant data according to preset rules based on the received gesture data or eye movement data or real-time position data.

[0048] In one embodiment, the steps of constructing the electronic component model database are:

[0049] Obtain electronic components and classify them;

[0050] Split any type of electronic components according to body, pins and parameters;

[0051] Model the body, pins and parameters of any type of electronic components respectively in proportion to form a model of any type of electronic components;

[0052] The models of various electronic components are summarized to build an electronic component model database.

[0053] Furthermore, the construction of the electronic component model is described using a resistor as an example:

[0054] Use the 3D modeling software C4d to make the pins of the model as follows Figure 2 The split design shown in the figure splits the pins into horizontal and vertical segments, so that the breadboard sockets at different distances can be connected. The length adjustment only needs to change the vertical scale value of the horizontal pins, and at the same time, the curved part is translated in the corresponding direction to complete the splicing of the two parts. Through this split logic, the modeling of color ring resistors and wires can be realized.

[0055] The model was built with reference to the proportions in reality, and can restore the impact of the corresponding parameters on the appearance. For example, the resistor color ring is independent of the main body, and the color of the color ring can be changed according to the specific resistance value; the wire can be changed to different colors.

[0056] In one embodiment, the steps of constructing the theoretical knowledge base are: acquiring relevant knowledge about electronic circuits, and constructing the theoretical knowledge base based on the relevant knowledge.

[0057] In one embodiment, the relevant data includes electronic component models, electronic component parameters, and operating principle diagrams.

[0058] In one embodiment, triggering, clicking, dragging, rotating, scaling, and marking operations are performed according to gesture data.

[0059] Furthermore, gestures commonly used in experiments include pinching, clicking, dragging, rotating, and zooming, which meet the user's basic input operations in the mixed reality world and realize functions such as virtual experimental data, recognition triggering, and integrated circuit structure preview.

[0060] Virtual experimental materials: The system will provide a window interface to display experimental materials consisting of pictures and texts. By detecting gestures (click, drag), you can select the display of specific content, or flip the pages up and down or close them.

[0061] IC structure preview: Use two fingers to pinch and drag to select and move the displayed virtual IC package model; click to zoom or associate the display, and drag to decompose.

[0062] At the same time, in order to quickly call the recognition function, based on convenience considerations, the experiment designed a trigger gesture in which the left thumb and index finger pinch the component and pull it in front of the eyes, which is in line with the user's habit of reading parameter information on the component.

[0063] In one embodiment, the labeling operation is performed based on the eye movement data.

[0064] Furthermore, according to the infrared camera used by the MR headset to monitor human eye attention in real time, the area where the user's current attention is focused can be obtained, and the area where the user's current attention is focused can be captured. When the line of sight falls on the corresponding object, the gaze can be enhanced to enhance the relevant information, which can help the user obtain information when it is inconvenient to operate with both hands.

[0065] In one embodiment, the dragging operation is performed based on the real-time position data.

[0066] On the other hand, this embodiment discloses an electronic circuit experiment teaching system based on mixed reality, including:

[0067] A database construction module is used to split electronic components, generate electronic component models, and construct an electronic component model database;

[0068] A physical identification and tracking module, used to identify and track physical objects, and transmit identification information and real-time location data of the physical objects to the processing module and the interactive identification module respectively;

[0069] The instruction parsing module is used to obtain external input instructions, parse the external input instructions, and transmit the obtained operation instructions to the processing module;

[0070] The processing module is used to call the relevant data of the electronic component model database and the theoretical knowledge base according to the operation instruction and the identification information of the physical object, and transmit the relevant data to the interactive identification module;

[0071] The interaction recognition module is used to display the relevant data and operate the relevant data according to preset rules based on the received gesture data or eye movement data or the real-time position data.

[0072] Furthermore, the system selects Unity as the basic engine of the software and OpenXR as the interface for AR\MR devices. Together with plug-ins that support multiple platforms, the software can be quickly ported to various glasses platforms. During the deployment process, the sensor input, spatial mapping, and image rendering tasks that occupy most of the performance are deployed on local devices to ensure that the device can run stably at 60FPS and ensure the most basic immersive experience. Deploying functions involving AI reasoning, such as image recognition, which have relatively low real-time requirements, to cloud servers can achieve lower latency than running on local devices, as well as an overall better user experience, while greatly increasing the battery life of the device.

[0073] This system opens relevant service interfaces to colleges and educational institutions for relevant learners to call, so as to maximize the rational use of AI servers.

[0074] In one embodiment, the physical identification and tracking module includes an image acquisition unit, an image classification unit, and an image recognition unit;

[0075] An image acquisition unit, used for converting the world coordinates of the electronic component into a camera coordinate system and then capturing an image of the electronic component;

[0076] An image classification unit, used for inputting the captured image into a pre-trained classification model to obtain the category of the electronic component;

[0077] The image recognition unit is used to call the corresponding recognition model according to the category of the electronic component to obtain parameter recognition of the electronic component.

[0078] In one embodiment, a feedback unit is further included, which is used to receive the simulation circuit, determine whether the simulation is correct according to the theoretical knowledge base, and output an optimization solution based on the operation result of the simulation circuit.

[0079] Furthermore, the system can provide prompts based on the user's current operation, follow the standard process, prevent wrong operation, and ensure the correctness of circuit construction. For example, by identifying the oscilloscope waveform, it can give suggestions for possible adjustments. If the waveform is detected to be too dense, the horizontal scale can be rotated; or if the waveform is unstable, the trigger level can be adjusted to stabilize the waveform.

[0080] Furthermore, the system can realize the circuit pre-building function, so that students can use mobile phones or MR headsets to place virtual breadboards anytime and anywhere to build and simulate circuits. Among them, relevant experiments can be designed according to course needs. This embodiment designs three experiments. Experiment 1 builds a circuit on a virtual breadboard and tests its correctness; Experiment 2 is the teaching of the use of signal sources and oscilloscopes; Experiment 3 is to build a calculus circuit on a breadboard and adjust the oscilloscope to display the correct waveform.

[0081] Experiment 1

[0082] First, build an electronic component model database;

[0083] Secondly, through Vuforia's object recognition, a virtual breadboard appears after scanning the image. First, number each socket on the breadboard, set the first socket as the starting point in advance, and obtain the distance between the sockets. Starting from the starting point, a new socket is obtained by looping each time a distance is increased, and numbered. Then use a dictionary to save the socket status (empty in the initial state). When clicking to insert a component, the dictionary is accessed. If the socket is empty, it is saved after insertion.

[0084] Again, click the component icon to display the 3D component model. The component and breadboard are bound in the center of the screen. Shake the camera to move the component. At the same time, aim at the breadboard socket in the center of the screen to display the socket information synchronously. Click to put in the pin. The code will fix the pin position. At this time, the fixed pin will not move when the camera is moved. The component pin is inserted into the socket visually. The system provides a component deletion function. Double-click the placed component. The system calls the Destory function to destroy the selected object and update the socket status in the dictionary.

[0085] Finally, a netlist will be generated based on the circuit built by the user, and the netlist file can be passed to the linked SPICE library for circuit simulation. After completion, the breadboard project is saved as an XML file for easy viewing and modification later.

[0086] Net table meaning: component name-parameter-net name 1-net name 2;

[0087] XML save structure: component name-parameter-network name 1-network name 2;

[0088] Experiment 2

[0089] This experiment uses the 555 chip as an example to realize the three-dimensional display function of the chip's internal structure, making abstract knowledge concrete and helping students better understand its circuit structure, packaging process and other experimental principles.

[0090] First, the integrated operational amplifier uses network resources. The model is cut and opened, and the internal circuit is modeled. The chip is first of normal size, and its size can be enlarged to a size that is easy to observe by zooming in with both hands. (When the system detects that both hands collide with the three-dimensional model, and pinches both hands, and the distance between the two hands gradually increases, the model will be proportionally enlarged to the size set by the user) Then drag the upper part of the package shell and align it to open the cover.

[0091] The system will display the chip layout and schematic diagram in front, provide a cross-sectional view of each structure on the layout, and inform the corresponding device on the schematic diagram, so that users can better understand the design, manufacturing, and working principles of integrated circuits.

[0092] When a finger is detected clicking on the corresponding area of ​​the schematic or layout, the layout structure or circuit connection of the structure will be displayed on the corresponding image. At the same time, the cross-sectional structure will also be displayed at the corresponding position of the layout.

[0093] Experiment 3

[0094] This experiment uses experimental circuits as an example to design a full-process circuit experiment guide, which can help students establish a standardized process concept.

[0095] Tracking breadboard: Use the 3D object recognition function in Vuforia, use Vuforia's object tracking app in advance, place the breadboard on a piece of reference paper, and perform a full-scale photo scan to extract feature points, which are used to match feature points during tracking and obtain feature data. When the program recognizes the breadboard, it will display light blue boxes around the object in the four corners.

[0096] Course Guide: After placing the breadboard, the course's graphic materials will be presented in the form of a page directly in front of the user's field of vision, and the complete circuit image will be displayed on the breadboard. The circuit that the user needs to build in this lesson has already been mapped to the jack.

[0097] Gaze function: When the user's gaze stays on an object for about 0.8 seconds using the gaze function of hololens2, the information of the object will pop up (the gaze information comes from an input method provided by MRTK. The device that supports eye movement capture will emit a ray in space to the place where the eyes are looking. When the ray touches the object hung with the "gazeable script", various functions after the gaze can be triggered). The displayed information is as follows:

[0098] For example, when staring at an operational amplifier chip, when contact with the ray is detected, a schematic diagram of the corresponding pin will be superimposed on the chip; or when contact with a resistor is detected, the resistance value will be displayed above the resistor.

[0099] Gaze works on both real and virtual objects because virtual objects can be set to be invisible in the virtual coordinate system while still being effective.

[0100] Furthermore, this embodiment also includes intelligent assistance for experimental operation:

[0101] Component parameter identification:

[0102] We use EasyDL, a one-stop deep learning platform based on PaddlePaddle. Using the image segmentation function, we can segment and identify components such as resistor color rings, chip pins, ceramic capacitors, and electrolytic capacitors. Specifically:

[0103] The trigger recognition method is based on the posture judgment of the gesture. When the user holds an object in his left hand and moves it in front of his eyes, the device will trigger the photo taking based on the current gesture and the distance between the user's fingertips and eyes.

[0104] In order to reduce the interference of the background in the image and better identify the components on the user's hand, the hand recognition component in MRTK can be used to convert the world coordinates of the fingertip components into the pixel coordinate system of the image, so as to directly capture the image of the component and upload it to the server model for recognition, and then return the result. (The recognition method is that the system will obtain the finger joint coordinates of the current hand recognition and transform it to the camera coordinate system; since the camera coordinate system is approximately the human eye coordinate system in reality, it can be transformed into the coordinate system of the HoloLens camera through translation; therefore, the pixel area where the fingertips of the person in the photo are located can be obtained, and it can be captured and sent to the server model)

[0105] First, the image will be sent to the component classification model, which can detect which of the following components it is: resistor, electrolytic capacitor, ceramic capacitor, diode, transistor, chip, etc. (Classification model: first collect the data set, use Baidu Cloud platform for training, online deployment, and then realize network call)

[0106] Call the resistor color ring segmentation model trained on the Baidu cloud platform, specifically: by collecting data sets, uploading data to the cloud platform for area annotation, and selecting instance segmentation models for training. Each color ring can be identified, and by manually annotating the resistor data set for area segmentation, the resistor color ring can be well segmented, the area of ​​each box can be returned, and the corresponding resistance value can be calculated according to the color of the color ring, and the image can be marked on the device using OpenCV. Similarly, the identification of integrated circuit pins can also be realized, and the function of each pin is given. In addition, combined with text recognition, the parameter information of ceramic capacitors and electrolytic capacitors can be obtained.

[0107] Verify circuit correctness:

[0108] Run the Segment Anything image segmentation model on the remote host, send the captured photos to the model, and the image segmentation results will be returned. Generate the corresponding circuit netlist file, and pass it to Spice for circuit verification to determine the correctness of the circuit connection.

[0109] Oscilloscope teaching:

[0110] First, when the oscilloscope is identified, the functions of each button can be displayed synchronously, and then the trained oscilloscope waveform anomaly model is called. The image classification function of the Baidu cloud platform is used for construction, and the data sets of the oscilloscope in different situations are collected and labeled for training. This model uses a large amount of data for training, and the judgment of waveform density and stability is basically accurate. When a waveform anomaly is detected, corresponding adjustment suggestions are given synchronously, such as prompting students to adjust the level trigger level knob or the horizontal scale knob.

[0111] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0112] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An electronic circuit experiment teaching method based on mixed reality, characterized in that: The specific steps are: Acquire electronic components and split the electronic components, generate electronic component models, and build an electronic component model database; Identify and track physical objects, and obtain identification information and real-time location data of the physical objects; Obtaining external input instructions, and parsing the external input instructions to obtain operation instructions; Calling the electronic component model and / or related data in the theoretical knowledge base according to the operation instruction and the identification information of the physical object; The relevant data is displayed, and corresponding interactive operations are performed on the relevant data according to preset rules based on the received gesture data or eye movement data or the real-time position data.

2. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: The steps for constructing the electronic component model database are as follows: Acquire the electronic components and classify the electronic components; Separate any type of electronic components according to body, pins and parameters; Modeling the body, pins and parameters of any type of electronic component in proportion to form a model of any type of electronic component; The models of various electronic components are summarized to construct the electronic component model database.

3. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: The steps of constructing the theoretical knowledge base are: acquiring relevant knowledge of electronic circuits, and constructing the theoretical knowledge base based on the relevant knowledge.

4. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: The relevant data includes electronic component models, electronic component parameters and schematic diagrams.

5. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: Trigger, click, drag, rotate, scale, and mark operations are performed according to the gesture data.

6. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: A labeling operation is performed according to the eye movement data.

7. The electronic circuit experiment teaching method based on mixed reality according to claim 1 is characterized in that: A drag operation is performed according to the real-time position data.

8. An electronic circuit experiment teaching system based on mixed reality, characterized in that: include: A database construction module is used to split electronic components, generate electronic component models, and construct an electronic component model database; A physical identification and tracking module, used to identify and track physical objects, and transmit identification information and real-time location data of the physical objects to the processing module and the interactive identification module respectively; The instruction parsing module is used to obtain external input instructions, parse the external input instructions, and transmit the obtained operation instructions to the processing module; The processing module is used to call the relevant data of the electronic component model database and the theoretical knowledge base according to the operation instruction and the identification information of the physical object, and transmit the relevant data to the interactive identification module; The interaction recognition module is used to display the relevant data and operate the relevant data according to preset rules based on the received gesture data or eye movement data or the real-time position data.

9. The electronic circuit experiment teaching system based on mixed reality according to claim 8, characterized in that: The physical identification and tracking module includes an image acquisition unit, an image classification unit, and an image recognition unit; The image acquisition unit is used to convert the world coordinates of the electronic component into a camera coordinate system and then capture an image of the electronic component; The image classification unit is used to input the captured image into a pre-trained classification model to obtain the category of the electronic component; The image recognition unit is used to call a corresponding recognition model according to the category of the electronic component to obtain parameter recognition of the electronic component.

10. The electronic circuit experiment teaching system based on mixed reality according to claim 9, characterized in that: It also includes a feedback unit for receiving a simulation circuit, determining whether the simulation is correct according to the theoretical knowledge base, and outputting an optimization solution based on the operation result of the simulation circuit.

Images